2024-04-17T19:28:59.228 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-17T19:28:59.228 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --archive /home/teuthworker/archive/pdonnell-2024-04-17_17:06:06-fs-wip-pdonnell-testing-20240417.021458-debug-distro-default-smithi/7660566 --verbose --name pdonnell-2024-04-17_17:06:06-fs-wip-pdonnell-testing-20240417.021458-debug-distro-default-smithi --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2024-04-17_17:06:06-fs-wip-pdonnell-testing-20240417.021458-debug-distro-default-smithi/7660566/orig.config.yaml 2024-04-17T19:28:59.288 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-17T19:28:59.372 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-04-17_17:06:06-fs-wip-pdonnell-testing-20240417.021458-debug-distro-default-smithi/7660566 branch: wip-pdonnell-testing-20240417.021458-debug description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: pdonnell@redhat.com first_in_suite: false job_id: '7660566' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-04-17_17:06:06-fs-wip-pdonnell-testing-20240417.021458-debug-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-pdonnell-testing-20240417.021458-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: true mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - Replacing daemon mds sha1: f3a98d3f110319c567ec86bf70955c141495ccf6 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: f3a98d3f110319c567ec86bf70955c141495ccf6 kclient: syntax: v1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240417.021458-debug sha1: f3a98d3f110319c567ec86bf70955c141495ccf6 owner: scheduled_pdonnell@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 4494 sha1: f3a98d3f110319c567ec86bf70955c141495ccf6 sleep_before_teardown: 0 subset: 496/512 suite: fs suite_branch: wip-pdonnell-testing-20240417.021458-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_f3a98d3f110319c567ec86bf70955c141495ccf6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: f3a98d3f110319c567ec86bf70955c141495ccf6 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDm42Cu+zPyGmJHkLKzD0ySzoi04P1QY9nWBJ94YKWJ7BQFVm0GmklFrtakxdR+DcokzKXLG5p7f3ZJusv/XSSQ= smithi037.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKJdJX3KscOjByZ1RIC3C9USJdBQlNeoTTCYVqNOfcu/LA1THBL7/idc3lx9iF9y2GKJOFUAL4DeUNUORXpiBuk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-17_17:06:06 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-17T19:28:59.373 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_f3a98d3f110319c567ec86bf70955c141495ccf6/qa; will attempt to use it 2024-04-17T19:28:59.373 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_f3a98d3f110319c567ec86bf70955c141495ccf6/qa/tasks 2024-04-17T19:28:59.374 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-17T19:28:59.375 INFO:teuthology.task.internal:Checking packages... 2024-04-17T19:28:59.399 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'f3a98d3f110319c567ec86bf70955c141495ccf6' 2024-04-17T19:28:59.399 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-17T19:28:59.399 INFO:teuthology.packaging:ref: None 2024-04-17T19:28:59.399 INFO:teuthology.packaging:tag: None 2024-04-17T19:28:59.399 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240417.021458-debug 2024-04-17T19:28:59.399 INFO:teuthology.packaging:sha1: f3a98d3f110319c567ec86bf70955c141495ccf6 2024-04-17T19:28:59.399 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-pdonnell-testing-20240417.021458-debug 2024-04-17T19:28:59.741 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3146-gf3a98d3f-1jammy 2024-04-17T19:28:59.742 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-17T19:28:59.751 INFO:teuthology.task.internal:no buildpackages task found 2024-04-17T19:28:59.751 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-17T19:28:59.811 INFO:teuthology.task.internal:Saving configuration 2024-04-17T19:28:59.831 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-17T19:28:59.842 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-17T19:28:59.870 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-17_17:06:06-fs-wip-pdonnell-testing-20240417.021458-debug-distro-default-smithi/7660566', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-17 19:19:34.165317', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII5Y3HQJTZgFBiOzYS40VDrrbO1Bm+v4+AEXqJ9IUSSw'} 2024-04-17T19:28:59.890 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-17_17:06:06-fs-wip-pdonnell-testing-20240417.021458-debug-distro-default-smithi/7660566', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-17 19:19:34.166813', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOfbEhts/xyQj/XYaw66nxb78S+eHnHiUXMMcM0AsME2'} 2024-04-17T19:28:59.891 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-17T19:28:59.902 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-17T19:28:59.902 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-04-17T19:28:59.902 INFO:teuthology.run_tasks:Running task console_log... 2024-04-17T19:29:00.004 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5d735313a0>, signals=[15]) 2024-04-17T19:29:00.004 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-17T19:29:00.015 INFO:teuthology.task.internal:Opening connections... 2024-04-17T19:29:00.015 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-04-17T19:29:00.017 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T19:29:00.108 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi037.front.sepia.ceph.com 2024-04-17T19:29:00.109 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T19:29:00.189 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-17T19:29:00.200 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-04-17T19:29:00.205 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-04-17T19:29:00.206 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-04-17T19:29:00.253 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-17T19:29:00.254 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="Ubuntu" 2024-04-17T19:29:00.254 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="22.04" 2024-04-17T19:29:00.254 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-17T19:29:00.254 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_CODENAME=jammy 2024-04-17T19:29:00.254 INFO:teuthology.orchestra.run.smithi029.stdout:ID=ubuntu 2024-04-17T19:29:00.254 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE=debian 2024-04-17T19:29:00.254 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-17T19:29:00.254 INFO:teuthology.orchestra.run.smithi029.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-17T19:29:00.254 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-17T19:29:00.254 INFO:teuthology.orchestra.run.smithi029.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-17T19:29:00.254 INFO:teuthology.orchestra.run.smithi029.stdout:UBUNTU_CODENAME=jammy 2024-04-17T19:29:00.255 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-04-17T19:29:00.276 DEBUG:teuthology.orchestra.run.smithi037:> uname -m 2024-04-17T19:29:00.281 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2024-04-17T19:29:00.281 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release 2024-04-17T19:29:00.328 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-17T19:29:00.328 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="Ubuntu" 2024-04-17T19:29:00.328 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="22.04" 2024-04-17T19:29:00.328 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-17T19:29:00.328 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_CODENAME=jammy 2024-04-17T19:29:00.328 INFO:teuthology.orchestra.run.smithi037.stdout:ID=ubuntu 2024-04-17T19:29:00.328 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE=debian 2024-04-17T19:29:00.328 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-17T19:29:00.329 INFO:teuthology.orchestra.run.smithi037.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-17T19:29:00.329 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-17T19:29:00.329 INFO:teuthology.orchestra.run.smithi037.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-17T19:29:00.329 INFO:teuthology.orchestra.run.smithi037.stdout:UBUNTU_CODENAME=jammy 2024-04-17T19:29:00.329 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2024-04-17T19:29:00.351 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-17T19:29:00.363 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-17T19:29:00.403 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-17T19:29:00.403 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-04-17T19:29:00.406 DEBUG:teuthology.orchestra.run.smithi037:> test '!' -e /home/ubuntu/cephtest 2024-04-17T19:29:00.411 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-17T19:29:00.425 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-17T19:29:00.425 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-04-17T19:29:00.450 DEBUG:teuthology.orchestra.run.smithi037:> test -z $(ls -A /var/lib/ceph) 2024-04-17T19:29:00.463 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-17T19:29:00.520 INFO:teuthology.run_tasks:Running task kernel... 2024-04-17T19:29:00.535 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-04-17T19:29:00.535 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-17T19:29:00.536 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-17T19:29:00.536 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-17T19:29:00.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T19:29:00.541 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing 2024-04-17T19:29:00.711 DEBUG:teuthology.task.kernel:sha1 for client.0 is 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-17T19:29:00.712 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8"... 2024-04-17T19:29:00.712 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-04-17T19:29:00.717 INFO:teuthology.orchestra.run.smithi029.stdout:5.15.0-56-generic 2024-04-17T19:29:00.718 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-17T19:29:00.718 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-17T19:29:00.718 INFO:teuthology.task.kernel:Installing linux-firmware on client.0... 2024-04-17T19:29:00.718 DEBUG:teuthology.orchestra.run.smithi029:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2024-04-17T19:29:00.777 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2024-04-17T19:29:00.920 INFO:teuthology.orchestra.run.smithi029.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2024-04-17T19:29:00.920 INFO:teuthology.orchestra.run.smithi029.stderr:hint: is subject to change. To configure the initial branch name to use in all 2024-04-17T19:29:00.920 INFO:teuthology.orchestra.run.smithi029.stderr:hint: of your new repositories, which will suppress this warning, call: 2024-04-17T19:29:00.920 INFO:teuthology.orchestra.run.smithi029.stderr:hint: 2024-04-17T19:29:00.920 INFO:teuthology.orchestra.run.smithi029.stderr:hint: git config --global init.defaultBranch 2024-04-17T19:29:00.920 INFO:teuthology.orchestra.run.smithi029.stderr:hint: 2024-04-17T19:29:00.921 INFO:teuthology.orchestra.run.smithi029.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-04-17T19:29:00.921 INFO:teuthology.orchestra.run.smithi029.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2024-04-17T19:29:00.921 INFO:teuthology.orchestra.run.smithi029.stderr:hint: 2024-04-17T19:29:00.921 INFO:teuthology.orchestra.run.smithi029.stderr:hint: git branch -m 2024-04-17T19:29:00.921 INFO:teuthology.orchestra.run.smithi029.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2024-04-17T19:29:00.924 DEBUG:teuthology.orchestra.run.smithi029:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2024-04-17T19:29:00.988 DEBUG:teuthology.orchestra.run.smithi029:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2024-04-17T19:29:01.043 DEBUG:teuthology.orchestra.run.smithi029:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2024-04-17T19:31:15.415 INFO:teuthology.orchestra.run.smithi029.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2024-04-17T19:31:15.416 INFO:teuthology.orchestra.run.smithi029.stderr: * [new branch] main -> origin/main 2024-04-17T19:31:15.454 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20190312 -> 20190312 2024-04-17T19:31:15.454 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20190416 -> 20190416 2024-04-17T19:31:15.454 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20190514 -> 20190514 2024-04-17T19:31:15.454 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20190618 -> 20190618 2024-04-17T19:31:15.454 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20190717 -> 20190717 2024-04-17T19:31:15.454 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20190815 -> 20190815 2024-04-17T19:31:15.454 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20190923 -> 20190923 2024-04-17T19:31:15.454 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20191022 -> 20191022 2024-04-17T19:31:15.455 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20191215 -> 20191215 2024-04-17T19:31:15.455 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20200122 -> 20200122 2024-04-17T19:31:15.455 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20200316 -> 20200316 2024-04-17T19:31:15.455 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20200421 -> 20200421 2024-04-17T19:31:15.455 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20200519 -> 20200519 2024-04-17T19:31:15.455 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20200619 -> 20200619 2024-04-17T19:31:15.455 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20200721 -> 20200721 2024-04-17T19:31:15.455 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20200817 -> 20200817 2024-04-17T19:31:15.455 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20200918 -> 20200918 2024-04-17T19:31:15.455 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20201022 -> 20201022 2024-04-17T19:31:15.455 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20201118 -> 20201118 2024-04-17T19:31:15.455 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20201218 -> 20201218 2024-04-17T19:31:15.456 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20210208 -> 20210208 2024-04-17T19:31:15.456 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20210315 -> 20210315 2024-04-17T19:31:15.456 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20210511 -> 20210511 2024-04-17T19:31:15.456 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20210716 -> 20210716 2024-04-17T19:31:15.456 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20210818 -> 20210818 2024-04-17T19:31:15.463 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20210919 -> 20210919 2024-04-17T19:31:15.463 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20211027 -> 20211027 2024-04-17T19:31:15.465 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20211216 -> 20211216 2024-04-17T19:31:15.465 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20220209 -> 20220209 2024-04-17T19:31:15.465 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20220310 -> 20220310 2024-04-17T19:31:15.465 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20220411 -> 20220411 2024-04-17T19:31:15.465 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20220509 -> 20220509 2024-04-17T19:31:15.465 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20220610 -> 20220610 2024-04-17T19:31:15.466 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20220708 -> 20220708 2024-04-17T19:31:15.466 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20220815 -> 20220815 2024-04-17T19:31:15.466 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20220913 -> 20220913 2024-04-17T19:31:15.466 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20221012 -> 20221012 2024-04-17T19:31:15.466 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20221109 -> 20221109 2024-04-17T19:31:15.466 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20221214 -> 20221214 2024-04-17T19:31:15.466 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20230117 -> 20230117 2024-04-17T19:31:15.466 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20230210 -> 20230210 2024-04-17T19:31:15.466 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20230310 -> 20230310 2024-04-17T19:31:15.467 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20230404 -> 20230404 2024-04-17T19:31:15.467 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20230515 -> 20230515 2024-04-17T19:31:15.467 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20230625 -> 20230625 2024-04-17T19:31:15.467 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20230804 -> 20230804 2024-04-17T19:31:15.467 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20230919 -> 20230919 2024-04-17T19:31:15.467 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20231030 -> 20231030 2024-04-17T19:31:15.468 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20231111 -> 20231111 2024-04-17T19:31:15.468 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20231211 -> 20231211 2024-04-17T19:31:15.468 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20240115 -> 20240115 2024-04-17T19:31:15.469 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20240220 -> 20240220 2024-04-17T19:31:15.469 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20240312 -> 20240312 2024-04-17T19:31:15.469 INFO:teuthology.orchestra.run.smithi029.stderr: * [new tag] 20240410 -> 20240410 2024-04-17T19:31:19.921 INFO:teuthology.orchestra.run.smithi029.stderr:Updating files: 47% (1731/3609) Updating files: 48% (1733/3609) Updating files: 49% (1769/3609) Updating files: 50% (1805/3609) Updating files: 51% (1841/3609) Updating files: 52% (1877/3609) Updating files: 53% (1913/3609) Updating files: 54% (1949/3609) Updating files: 55% (1985/3609) Updating files: 56% (2022/3609) Updating files: 57% (2058/3609) Updating files: 58% (2094/3609) Updating files: 59% (2130/3609) Updating files: 60% (2166/3609) Updating files: 61% (2202/3609) Updating files: 61% (2215/3609) Updating files: 62% (2238/3609) Updating files: 63% (2274/3609) Updating files: 64% (2310/3609) Updating files: 65% (2346/3609) Updating files: 66% (2382/3609) Updating files: 67% (2419/3609) Updating files: 68% (2455/3609) Updating files: 69% (2491/3609) Updating files: 70% (2527/3609) Updating files: 71% (2563/3609) Updating files: 71% (2570/3609) Updating files: 72% (2599/3609) Updating files: 73% (2635/3609) Updating files: 74% (2671/3609) Updating files: 75% (2707/3609) Updating files: 76% (2743/3609) Updating files: 77% (2779/3609) Updating files: 78% (2816/3609) Updating files: 79% (2852/3609) Updating files: 80% (2888/3609) Updating files: 81% (2924/3609) Updating files: 82% (2960/3609) Updating files: 83% (2996/3609) Updating files: 83% (3016/3609) Updating files: 84% (3032/3609) Updating files: 85% (3068/3609) Updating files: 86% (3104/3609) Updating files: 87% (3140/3609) Updating files: 88% (3176/3609) Updating files: 89% (3213/3609) Updating files: 90% (3249/3609) Updating files: 91% (3285/3609) Updating files: 92% (3321/3609) Updating files: 93% (3357/3609) Updating files: 94% (3393/3609) Updating files: 95% (3429/3609) Updating files: 96% (3465/3609) Updating files: 97% (3501/3609) Updating files: 98% (3537/3609) Updating files: 99% (3573/3609) Updating files: 100% (3609/3609) Updating files: 100% (3609/3609), done. 2024-04-17T19:31:19.926 INFO:teuthology.orchestra.run.smithi029.stdout:HEAD is now at e6786998 Merge branch 'mt7922' into 'main' 2024-04-17T19:31:19.951 INFO:teuthology.task.kernel:Downloading kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.0... 2024-04-17T19:31:19.951 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-17T19:31:20.100 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/ubuntu/jammy/flavors/default/pool/main/l/linux-6.9.0-rc3-g3f60ed14f7c1/ 2024-04-17T19:31:20.100 DEBUG:teuthology.orchestra.run.smithi029:> rm -f /tmp/linux-image.deb && echo linux-image-6.9.0-rc3-g3f60ed14f7c1_6.9.0-rc3-g3f60ed14f7c1-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://1.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/ubuntu/jammy/flavors/default/pool/main/l/linux-6.9.0-rc3-g3f60ed14f7c1/ --input-file=- 2024-04-17T19:31:20.103 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi029.front.sepia.ceph.com to complete... 2024-04-17T19:31:23.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17 19:31:23 URL:https://1.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/ubuntu/jammy/flavors/default/pool/main/l/linux-6.9.0-rc3-g3f60ed14f7c1/linux-image-6.9.0-rc3-g3f60ed14f7c1_6.9.0-rc3-g3f60ed14f7c1-1_amd64.deb [44839714/44839714] -> "/tmp/linux-image.deb" [1] 2024-04-17T19:31:23.073 INFO:teuthology.orchestra.run.smithi029.stderr:FINISHED --2024-04-17 19:31:23-- 2024-04-17T19:31:23.073 INFO:teuthology.orchestra.run.smithi029.stderr:Total wall clock time: 2.9s 2024-04-17T19:31:23.073 INFO:teuthology.orchestra.run.smithi029.stderr:Downloaded: 1 files, 43M in 2.7s (15.7 MB/s) 2024-04-17T19:31:23.077 INFO:teuthology.task.kernel:Installing kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.0... 2024-04-17T19:31:23.078 DEBUG:teuthology.orchestra.run.smithi029:> sudo dpkg -i /tmp/linux-image.deb 2024-04-17T19:31:23.381 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package linux-image-6.9.0-rc3-g3f60ed14f7c1. 2024-04-17T19:31:26.323 INFO:teuthology.orchestra.run.smithi029.stdout:(Reading database ... 129542 files and directories currently installed.) 2024-04-17T19:31:26.327 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack /tmp/linux-image.deb ... 2024-04-17T19:31:26.452 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking linux-image-6.9.0-rc3-g3f60ed14f7c1 (6.9.0-rc3-g3f60ed14f7c1-1) ... 2024-04-17T19:31:31.374 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up linux-image-6.9.0-rc3-g3f60ed14f7c1 (6.9.0-rc3-g3f60ed14f7c1-1) ... 2024-04-17T19:31:31.500 INFO:teuthology.orchestra.run.smithi029.stderr:update-initramfs: Generating /boot/initrd.img-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T19:32:03.920 INFO:teuthology.orchestra.run.smithi029.stderr:Sourcing file `/etc/default/grub' 2024-04-17T19:32:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-17T19:32:03.967 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2024-04-17T19:32:04.293 INFO:teuthology.orchestra.run.smithi029.stderr:Found linux image: /boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T19:32:04.316 INFO:teuthology.orchestra.run.smithi029.stderr:Found initrd image: /boot/initrd.img-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T19:32:04.604 INFO:teuthology.orchestra.run.smithi029.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-17T19:32:04.620 INFO:teuthology.orchestra.run.smithi029.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-17T19:32:04.692 INFO:teuthology.orchestra.run.smithi029.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-17T19:32:04.698 INFO:teuthology.orchestra.run.smithi029.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-17T19:32:05.122 INFO:teuthology.orchestra.run.smithi029.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-17T19:32:05.123 INFO:teuthology.orchestra.run.smithi029.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-17T19:32:05.123 INFO:teuthology.orchestra.run.smithi029.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-17T19:32:05.198 INFO:teuthology.orchestra.run.smithi029.stderr:done 2024-04-17T19:32:05.442 DEBUG:teuthology.orchestra.run.smithi029:> dpkg-deb -c /tmp/linux-image.deb 2024-04-17T19:32:05.450 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./ 2024-04-17T19:32:05.450 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./boot/ 2024-04-17T19:32:05.450 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 7002012 2024-04-11 18:50 ./boot/System.map-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T19:32:05.469 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 172073 2024-04-11 18:50 ./boot/config-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T19:32:05.470 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13120512 2024-04-11 18:50 ./boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T19:32:05.493 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./etc/ 2024-04-17T19:32:05.493 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./etc/kernel/ 2024-04-17T19:32:05.494 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./etc/kernel/postinst.d/ 2024-04-17T19:32:05.494 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./etc/kernel/postrm.d/ 2024-04-17T19:32:05.494 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./etc/kernel/preinst.d/ 2024-04-17T19:32:05.494 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./etc/kernel/prerm.d/ 2024-04-17T19:32:05.494 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/ 2024-04-17T19:32:05.494 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/ 2024-04-17T19:32:05.494 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/ 2024-04-17T19:32:05.494 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/ 2024-04-17T19:32:05.494 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/ 2024-04-17T19:32:05.494 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/ 2024-04-17T19:32:05.494 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ 2024-04-17T19:32:05.494 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 507569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/aesni-intel.ko 2024-04-17T19:32:05.497 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-17T19:32:05.497 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-17T19:32:05.497 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-17T19:32:05.497 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 97953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-17T19:32:05.497 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 109905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-17T19:32:05.497 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 155249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-17T19:32:05.497 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-17T19:32:05.499 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-17T19:32:05.499 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 87009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-17T19:32:05.499 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-17T19:32:05.500 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 50305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-17T19:32:05.500 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 50377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-17T19:32:05.500 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 54513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-17T19:32:05.500 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 44657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-17T19:32:05.500 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 48049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-17T19:32:05.500 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 62873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-17T19:32:05.500 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 53569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-17T19:32:05.500 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-17T19:32:05.500 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-17T19:32:05.500 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/ 2024-04-17T19:32:05.501 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/ 2024-04-17T19:32:05.501 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/ 2024-04-17T19:32:05.501 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 49081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-17T19:32:05.501 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpuid.ko 2024-04-17T19:32:05.501 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/msr.ko 2024-04-17T19:32:05.501 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/ 2024-04-17T19:32:05.501 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 444425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-17T19:32:05.502 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1035881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-17T19:32:05.502 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 2849609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm.ko 2024-04-17T19:32:05.715 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ 2024-04-17T19:32:05.716 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 65025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/af_alg.ko 2024-04-17T19:32:05.716 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/algif_hash.ko 2024-04-17T19:32:05.716 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/algif_skcipher.ko 2024-04-17T19:32:05.716 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ansi_cprng.ko 2024-04-17T19:32:05.716 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/anubis.ko 2024-04-17T19:32:05.716 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/arc4.ko 2024-04-17T19:32:05.716 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/asymmetric_keys/ 2024-04-17T19:32:05.716 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2024-04-17T19:32:05.716 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/ 2024-04-17T19:32:05.716 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_memcpy.ko 2024-04-17T19:32:05.716 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_pq.ko 2024-04-17T19:32:05.716 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-17T19:32:05.716 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_tx.ko 2024-04-17T19:32:05.716 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_xor.ko 2024-04-17T19:32:05.717 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/raid6test.ko 2024-04-17T19:32:05.717 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authenc.ko 2024-04-17T19:32:05.717 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authencesn.ko 2024-04-17T19:32:05.717 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blake2b_generic.ko 2024-04-17T19:32:05.717 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_common.ko 2024-04-17T19:32:05.717 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_generic.ko 2024-04-17T19:32:05.717 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 40889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/camellia_generic.ko 2024-04-17T19:32:05.717 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast5_generic.ko 2024-04-17T19:32:05.717 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast6_generic.ko 2024-04-17T19:32:05.717 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast_common.ko 2024-04-17T19:32:05.717 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 43281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ccm.ko 2024-04-17T19:32:05.717 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cmac.ko 2024-04-17T19:32:05.717 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crc32_generic.ko 2024-04-17T19:32:05.717 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 59129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cryptd.ko 2024-04-17T19:32:05.717 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 48113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_engine.ko 2024-04-17T19:32:05.718 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_simd.ko 2024-04-17T19:32:05.718 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_user.ko 2024-04-17T19:32:05.718 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cts.ko 2024-04-17T19:32:05.718 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/des_generic.ko 2024-04-17T19:32:05.718 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/dh_generic.ko 2024-04-17T19:32:05.718 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/echainiv.ko 2024-04-17T19:32:05.718 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/essiv.ko 2024-04-17T19:32:05.718 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/fcrypt.ko 2024-04-17T19:32:05.718 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/khazad.ko 2024-04-17T19:32:05.718 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lrw.ko 2024-04-17T19:32:05.718 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4.ko 2024-04-17T19:32:05.718 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4hc.ko 2024-04-17T19:32:05.718 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/md4.ko 2024-04-17T19:32:05.718 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/michael_mic.ko 2024-04-17T19:32:05.719 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcbc.ko 2024-04-17T19:32:05.719 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcrypt.ko 2024-04-17T19:32:05.719 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/rmd160.ko 2024-04-17T19:32:05.719 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/seed.ko 2024-04-17T19:32:05.719 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/serpent_generic.ko 2024-04-17T19:32:05.719 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 169065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tcrypt.ko 2024-04-17T19:32:05.719 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tea.ko 2024-04-17T19:32:05.719 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_common.ko 2024-04-17T19:32:05.719 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_generic.ko 2024-04-17T19:32:05.719 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/vmac.ko 2024-04-17T19:32:05.719 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 38209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/wp512.ko 2024-04-17T19:32:05.719 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xcbc.ko 2024-04-17T19:32:05.719 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xor.ko 2024-04-17T19:32:05.719 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xts.ko 2024-04-17T19:32:05.719 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xxhash_generic.ko 2024-04-17T19:32:05.720 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ 2024-04-17T19:32:05.720 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ 2024-04-17T19:32:05.720 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_extlog.ko 2024-04-17T19:32:05.720 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 31921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-17T19:32:05.720 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_pad.ko 2024-04-17T19:32:05.720 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/apei/ 2024-04-17T19:32:05.720 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 44705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/apei/einj.ko 2024-04-17T19:32:05.720 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ec_sys.ko 2024-04-17T19:32:05.720 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/platform_profile.ko 2024-04-17T19:32:05.720 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbs.ko 2024-04-17T19:32:05.720 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbshc.ko 2024-04-17T19:32:05.720 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 119033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/video.ko 2024-04-17T19:32:05.720 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ 2024-04-17T19:32:05.721 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/acard-ahci.ko 2024-04-17T19:32:05.721 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 96825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci.ko 2024-04-17T19:32:05.721 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci_platform.ko 2024-04-17T19:32:05.721 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 107737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci.ko 2024-04-17T19:32:05.721 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 42001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci_platform.ko 2024-04-17T19:32:05.721 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_acpi.ko 2024-04-17T19:32:05.721 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ali.ko 2024-04-17T19:32:05.721 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 38329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_amd.ko 2024-04-17T19:32:05.721 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_artop.ko 2024-04-17T19:32:05.721 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atiixp.ko 2024-04-17T19:32:05.721 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atp867x.ko 2024-04-17T19:32:05.721 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd640.ko 2024-04-17T19:32:05.721 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd64x.ko 2024-04-17T19:32:05.721 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cypress.ko 2024-04-17T19:32:05.721 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_efar.ko 2024-04-17T19:32:05.721 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt366.ko 2024-04-17T19:32:05.722 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 38913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt37x.ko 2024-04-17T19:32:05.722 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-17T19:32:05.722 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-17T19:32:05.722 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it8213.ko 2024-04-17T19:32:05.722 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 41289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it821x.ko 2024-04-17T19:32:05.722 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_jmicron.ko 2024-04-17T19:32:05.722 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 60257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_legacy.ko 2024-04-17T19:32:05.722 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_marvell.ko 2024-04-17T19:32:05.722 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_mpiix.ko 2024-04-17T19:32:05.722 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_netcell.ko 2024-04-17T19:32:05.722 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ninja32.ko 2024-04-17T19:32:05.722 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87410.ko 2024-04-17T19:32:05.722 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87415.ko 2024-04-17T19:32:05.723 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_oldpiix.ko 2024-04-17T19:32:05.723 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_opti.ko 2024-04-17T19:32:05.723 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_optidma.ko 2024-04-17T19:32:05.723 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pcmcia.ko 2024-04-17T19:32:05.723 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 43009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-17T19:32:05.723 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-17T19:32:05.723 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_piccolo.ko 2024-04-17T19:32:05.723 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_radisys.ko 2024-04-17T19:32:05.723 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rdc.ko 2024-04-17T19:32:05.723 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rz1000.ko 2024-04-17T19:32:05.723 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sch.ko 2024-04-17T19:32:05.723 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_serverworks.ko 2024-04-17T19:32:05.723 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sil680.ko 2024-04-17T19:32:05.723 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sl82c105.ko 2024-04-17T19:32:05.724 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_triflex.ko 2024-04-17T19:32:05.724 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_via.ko 2024-04-17T19:32:05.724 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pdc_adma.ko 2024-04-17T19:32:05.724 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_inic162x.ko 2024-04-17T19:32:05.724 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 102121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_mv.ko 2024-04-17T19:32:05.724 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 85409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_nv.ko 2024-04-17T19:32:05.724 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 44313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_promise.ko 2024-04-17T19:32:05.724 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_qstor.ko 2024-04-17T19:32:05.724 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil.ko 2024-04-17T19:32:05.724 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 45905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil24.ko 2024-04-17T19:32:05.724 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sis.ko 2024-04-17T19:32:05.724 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_svw.ko 2024-04-17T19:32:05.724 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 53353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sx4.ko 2024-04-17T19:32:05.724 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_uli.ko 2024-04-17T19:32:05.724 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 41049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_via.ko 2024-04-17T19:32:05.725 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_vsc.ko 2024-04-17T19:32:05.725 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ 2024-04-17T19:32:05.725 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ks0108.ko 2024-04-17T19:32:05.725 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/ 2024-04-17T19:32:05.725 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 163969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/bcma.ko 2024-04-17T19:32:05.725 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/ 2024-04-17T19:32:05.725 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/ 2024-04-17T19:32:05.725 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 140865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/aoe.ko 2024-04-17T19:32:05.725 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/ 2024-04-17T19:32:05.725 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1241657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/drbd.ko 2024-04-17T19:32:05.725 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 203289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/floppy.ko 2024-04-17T19:32:05.725 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/ 2024-04-17T19:32:05.725 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 115265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-17T19:32:05.725 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 143433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/nbd.ko 2024-04-17T19:32:05.725 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/ 2024-04-17T19:32:05.726 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 126617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/null_blk.ko 2024-04-17T19:32:05.726 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 119873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/pktcdvd.ko 2024-04-17T19:32:05.726 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 313217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/rbd.ko 2024-04-17T19:32:05.726 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/ 2024-04-17T19:32:05.726 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 117233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-17T19:32:05.726 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/ 2024-04-17T19:32:05.726 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 70385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/zram.ko 2024-04-17T19:32:05.726 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ 2024-04-17T19:32:05.726 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/applicom.ko 2024-04-17T19:32:05.726 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hangcheck-timer.ko 2024-04-17T19:32:05.726 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/ 2024-04-17T19:32:05.726 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-17T19:32:05.726 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-17T19:32:05.726 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-17T19:32:05.727 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/via-rng.ko 2024-04-17T19:32:05.727 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-17T19:32:05.727 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ 2024-04-17T19:32:05.727 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 37257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-17T19:32:05.727 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 186921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-17T19:32:05.727 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-17T19:32:05.727 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 179561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-17T19:32:05.727 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 85441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-17T19:32:05.727 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 65313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-17T19:32:05.727 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 43857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/lp.ko 2024-04-17T19:32:05.727 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/ 2024-04-17T19:32:05.727 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 103409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/mwave.ko 2024-04-17T19:32:05.727 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 31833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/nvram.ko 2024-04-17T19:32:05.727 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 38977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ppdev.ko 2024-04-17T19:32:05.727 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 59241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tlclk.ko 2024-04-17T19:32:05.728 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/ 2024-04-17T19:32:05.728 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-17T19:32:05.728 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2024-04-17T19:32:05.728 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2024-04-17T19:32:05.728 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 39193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2024-04-17T19:32:05.728 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-17T19:32:05.728 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-17T19:32:05.728 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/xen-tpmfront.ko 2024-04-17T19:32:05.728 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/xillybus/ 2024-04-17T19:32:05.728 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/xillybus/xillybus_class.ko 2024-04-17T19:32:05.728 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 62545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/xillybus/xillybus_core.ko 2024-04-17T19:32:05.728 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/xillybus/xillybus_pcie.ko 2024-04-17T19:32:05.728 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/clk/ 2024-04-17T19:32:05.729 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/clk/clk-palmas.ko 2024-04-17T19:32:05.729 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 57289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/clk/clk-si5351.ko 2024-04-17T19:32:05.729 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/clk/clk-twl6040.ko 2024-04-17T19:32:05.729 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/clk/clk-wm831x.ko 2024-04-17T19:32:05.729 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/ 2024-04-17T19:32:05.729 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-17T19:32:05.729 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-17T19:32:05.729 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-17T19:32:05.729 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ 2024-04-17T19:32:05.729 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ 2024-04-17T19:32:05.729 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 89257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-17T19:32:05.729 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 333657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp.ko 2024-04-17T19:32:05.729 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/ 2024-04-17T19:32:05.729 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/ 2024-04-17T19:32:05.730 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/ 2024-04-17T19:32:05.730 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 751713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-17T19:32:05.730 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2024-04-17T19:32:05.730 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-17T19:32:05.730 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-aes.ko 2024-04-17T19:32:05.730 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-sha.ko 2024-04-17T19:32:05.730 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/ 2024-04-17T19:32:05.730 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 90441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-17T19:32:05.730 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/ 2024-04-17T19:32:05.730 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 36105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/dca.ko 2024-04-17T19:32:05.730 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ 2024-04-17T19:32:05.730 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/ 2024-04-17T19:32:05.730 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-17T19:32:05.730 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ 2024-04-17T19:32:05.731 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 146769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-17T19:32:05.731 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ 2024-04-17T19:32:05.731 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 96145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/amd64_edac.ko 2024-04-17T19:32:05.731 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 41993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/e752x_edac.ko 2024-04-17T19:32:05.731 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 51105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/edac_mce_amd.ko 2024-04-17T19:32:05.731 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3000_edac.ko 2024-04-17T19:32:05.731 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3200_edac.ko 2024-04-17T19:32:05.731 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5100_edac.ko 2024-04-17T19:32:05.731 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5400_edac.ko 2024-04-17T19:32:05.731 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7300_edac.ko 2024-04-17T19:32:05.731 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 64161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7core_edac.ko 2024-04-17T19:32:05.731 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i82975x_edac.ko 2024-04-17T19:32:05.731 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ie31200_edac.ko 2024-04-17T19:32:05.731 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 68697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/sb_edac.ko 2024-04-17T19:32:05.732 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/x38_edac.ko 2024-04-17T19:32:05.732 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/ 2024-04-17T19:32:05.732 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-gpio.ko 2024-04-17T19:32:05.732 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 31369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-max14577.ko 2024-04-17T19:32:05.732 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-max77693.ko 2024-04-17T19:32:05.732 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-max8997.ko 2024-04-17T19:32:05.732 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-palmas.ko 2024-04-17T19:32:05.732 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-rt8973a.ko 2024-04-17T19:32:05.732 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 31993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-sm5502.ko 2024-04-17T19:32:05.732 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/ 2024-04-17T19:32:05.732 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 42953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/dmi-sysfs.ko 2024-04-17T19:32:05.732 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/efi/ 2024-04-17T19:32:05.732 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/efi/efi-pstore.ko 2024-04-17T19:32:05.732 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-17T19:32:05.732 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/ 2024-04-17T19:32:05.733 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-adp5520.ko 2024-04-17T19:32:05.733 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-amd8111.ko 2024-04-17T19:32:05.733 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-arizona.ko 2024-04-17T19:32:05.733 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-da9052.ko 2024-04-17T19:32:05.733 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-da9055.ko 2024-04-17T19:32:05.733 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-dln2.ko 2024-04-17T19:32:05.733 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 81073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-f7188x.ko 2024-04-17T19:32:05.733 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 40761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-generic.ko 2024-04-17T19:32:05.733 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ich.ko 2024-04-17T19:32:05.733 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-janz-ttl.ko 2024-04-17T19:32:05.733 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-kempld.ko 2024-04-17T19:32:05.733 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-lp3943.ko 2024-04-17T19:32:05.733 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-max7300.ko 2024-04-17T19:32:05.733 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-max730x.ko 2024-04-17T19:32:05.733 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-max732x.ko 2024-04-17T19:32:05.734 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ml-ioh.ko 2024-04-17T19:32:05.734 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 55505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-pca953x.ko 2024-04-17T19:32:05.734 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 31881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-pcf857x.ko 2024-04-17T19:32:05.734 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-rdc321x.ko 2024-04-17T19:32:05.734 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-sch.ko 2024-04-17T19:32:05.734 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-sch311x.ko 2024-04-17T19:32:05.734 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-tps65912.ko 2024-04-17T19:32:05.734 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-twl4030.ko 2024-04-17T19:32:05.734 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-twl6040.ko 2024-04-17T19:32:05.734 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-17T19:32:05.734 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-vx855.ko 2024-04-17T19:32:05.734 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-wm831x.ko 2024-04-17T19:32:05.734 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-wm8350.ko 2024-04-17T19:32:05.735 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-wm8994.ko 2024-04-17T19:32:05.735 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/ 2024-04-17T19:32:05.735 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-a4tech.ko 2024-04-17T19:32:05.735 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 43553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-apple.ko 2024-04-17T19:32:05.735 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-appleir.ko 2024-04-17T19:32:05.735 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-aureal.ko 2024-04-17T19:32:05.735 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-axff.ko 2024-04-17T19:32:05.735 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-belkin.ko 2024-04-17T19:32:05.735 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-cherry.ko 2024-04-17T19:32:05.735 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-chicony.ko 2024-04-17T19:32:05.735 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 74393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-cp2112.ko 2024-04-17T19:32:05.735 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-cypress.ko 2024-04-17T19:32:05.735 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-dr.ko 2024-04-17T19:32:05.735 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elecom.ko 2024-04-17T19:32:05.735 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elo.ko 2024-04-17T19:32:05.736 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-emsff.ko 2024-04-17T19:32:05.736 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ezkey.ko 2024-04-17T19:32:05.736 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gaff.ko 2024-04-17T19:32:05.736 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-generic.ko 2024-04-17T19:32:05.736 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gt683r.ko 2024-04-17T19:32:05.736 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gyration.ko 2024-04-17T19:32:05.736 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-17T19:32:05.736 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-17T19:32:05.736 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtekff.ko 2024-04-17T19:32:05.736 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-hyperv.ko 2024-04-17T19:32:05.736 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-icade.ko 2024-04-17T19:32:05.736 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kensington.ko 2024-04-17T19:32:05.736 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-keytouch.ko 2024-04-17T19:32:05.736 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kye.ko 2024-04-17T19:32:05.737 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lcpower.ko 2024-04-17T19:32:05.737 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-led.ko 2024-04-17T19:32:05.737 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 54217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lenovo.ko 2024-04-17T19:32:05.737 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 37553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lg-g15.ko 2024-04-17T19:32:05.737 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 68081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-17T19:32:05.737 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 121369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-17T19:32:05.737 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 97953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech.ko 2024-04-17T19:32:05.737 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 36689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-magicmouse.ko 2024-04-17T19:32:05.737 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-microsoft.ko 2024-04-17T19:32:05.737 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-monterey.ko 2024-04-17T19:32:05.737 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 59001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-multitouch.ko 2024-04-17T19:32:05.737 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 39697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ntrig.ko 2024-04-17T19:32:05.737 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ortek.ko 2024-04-17T19:32:05.737 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-penmount.ko 2024-04-17T19:32:05.738 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-petalynx.ko 2024-04-17T19:32:05.738 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 91241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-picolcd.ko 2024-04-17T19:32:05.738 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-pl.ko 2024-04-17T19:32:05.738 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-plantronics.ko 2024-04-17T19:32:05.738 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-primax.ko 2024-04-17T19:32:05.738 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 42889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-rmi.ko 2024-04-17T19:32:05.738 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-17T19:32:05.738 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-common.ko 2024-04-17T19:32:05.738 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 38865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-17T19:32:05.738 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 36305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-17T19:32:05.738 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 38073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-17T19:32:05.738 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-17T19:32:05.739 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-17T19:32:05.739 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-17T19:32:05.739 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-17T19:32:05.739 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-17T19:32:05.739 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-17T19:32:05.739 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat.ko 2024-04-17T19:32:05.739 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-saitek.ko 2024-04-17T19:32:05.739 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 42305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-samsung.ko 2024-04-17T19:32:05.739 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 43609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-17T19:32:05.739 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sjoy.ko 2024-04-17T19:32:05.739 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 62129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sony.ko 2024-04-17T19:32:05.739 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-speedlink.ko 2024-04-17T19:32:05.739 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-steelseries.ko 2024-04-17T19:32:05.739 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sunplus.ko 2024-04-17T19:32:05.739 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-17T19:32:05.740 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tivo.ko 2024-04-17T19:32:05.740 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tmff.ko 2024-04-17T19:32:05.740 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-topseed.ko 2024-04-17T19:32:05.740 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-twinhan.ko 2024-04-17T19:32:05.740 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 81113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-uclogic.ko 2024-04-17T19:32:05.740 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-waltop.ko 2024-04-17T19:32:05.740 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 184745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-wiimote.ko 2024-04-17T19:32:05.740 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-xinmo.ko 2024-04-17T19:32:05.740 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zpff.ko 2024-04-17T19:32:05.740 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zydacron.ko 2024-04-17T19:32:05.740 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 319921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid.ko 2024-04-17T19:32:05.740 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 39529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/uhid.ko 2024-04-17T19:32:05.740 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/usbhid/ 2024-04-17T19:32:05.740 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 153633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/usbhid/usbhid.ko 2024-04-17T19:32:05.741 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/usbhid/usbkbd.ko 2024-04-17T19:32:05.741 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/usbhid/usbmouse.ko 2024-04-17T19:32:05.741 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 261945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/wacom.ko 2024-04-17T19:32:05.741 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/ 2024-04-17T19:32:05.741 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 75065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_balloon.ko 2024-04-17T19:32:05.741 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 110681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_utils.ko 2024-04-17T19:32:05.741 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 382033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_vmbus.ko 2024-04-17T19:32:05.741 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ 2024-04-17T19:32:05.741 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 77153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru.ko 2024-04-17T19:32:05.741 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 67009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru3.ko 2024-04-17T19:32:05.741 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 42249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-17T19:32:05.741 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7414.ko 2024-04-17T19:32:05.741 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7418.ko 2024-04-17T19:32:05.741 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adc128d818.ko 2024-04-17T19:32:05.742 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 38097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1025.ko 2024-04-17T19:32:05.742 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 115057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1026.ko 2024-04-17T19:32:05.742 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1029.ko 2024-04-17T19:32:05.742 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 55225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1031.ko 2024-04-17T19:32:05.742 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 31441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm9240.ko 2024-04-17T19:32:05.742 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ads7828.ko 2024-04-17T19:32:05.742 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7410.ko 2024-04-17T19:32:05.742 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7411.ko 2024-04-17T19:32:05.742 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 89025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7462.ko 2024-04-17T19:32:05.742 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 49153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7470.ko 2024-04-17T19:32:05.742 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 81473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7475.ko 2024-04-17T19:32:05.742 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7x10.ko 2024-04-17T19:32:05.742 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 48177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/amc6821.ko 2024-04-17T19:32:05.742 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 68865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/applesmc.ko 2024-04-17T19:32:05.742 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 60193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asb100.ko 2024-04-17T19:32:05.743 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 69009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asc7621.ko 2024-04-17T19:32:05.743 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 47977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-17T19:32:05.743 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/atxp1.ko 2024-04-17T19:32:05.743 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/coretemp.ko 2024-04-17T19:32:05.743 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/da9052-hwmon.ko 2024-04-17T19:32:05.743 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/da9055-hwmon.ko 2024-04-17T19:32:05.743 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 55713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-17T19:32:05.743 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 113201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dme1737.ko 2024-04-17T19:32:05.743 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds1621.ko 2024-04-17T19:32:05.743 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds620.ko 2024-04-17T19:32:05.743 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc1403.ko 2024-04-17T19:32:05.743 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 40177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc2103.ko 2024-04-17T19:32:05.743 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc6w201.ko 2024-04-17T19:32:05.743 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 78921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71805f.ko 2024-04-17T19:32:05.743 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 108961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71882fg.ko 2024-04-17T19:32:05.744 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 50697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f75375s.ko 2024-04-17T19:32:05.744 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fam15h_power.ko 2024-04-17T19:32:05.744 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 65769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fschmd.ko 2024-04-17T19:32:05.744 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g760a.ko 2024-04-17T19:32:05.744 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g762.ko 2024-04-17T19:32:05.744 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 59761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl518sm.ko 2024-04-17T19:32:05.744 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 56313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl520sm.ko 2024-04-17T19:32:05.744 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hih6130.ko 2024-04-17T19:32:05.744 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-17T19:32:05.744 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5500_temp.ko 2024-04-17T19:32:05.744 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5k_amb.ko 2024-04-17T19:32:05.744 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 42585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmaem.ko 2024-04-17T19:32:05.744 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmpex.ko 2024-04-17T19:32:05.744 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina209.ko 2024-04-17T19:32:05.745 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina2xx.ko 2024-04-17T19:32:05.745 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 156465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/it87.ko 2024-04-17T19:32:05.745 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/jc42.ko 2024-04-17T19:32:05.745 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k10temp.ko 2024-04-17T19:32:05.745 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k8temp.ko 2024-04-17T19:32:05.745 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lineage-pem.ko 2024-04-17T19:32:05.745 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 57769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm63.ko 2024-04-17T19:32:05.745 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm73.ko 2024-04-17T19:32:05.745 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 44121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm75.ko 2024-04-17T19:32:05.745 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm77.ko 2024-04-17T19:32:05.745 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 41473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm78.ko 2024-04-17T19:32:05.745 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 37049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm80.ko 2024-04-17T19:32:05.745 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm83.ko 2024-04-17T19:32:05.745 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 86497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm85.ko 2024-04-17T19:32:05.745 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 55641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm87.ko 2024-04-17T19:32:05.746 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 42249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm90.ko 2024-04-17T19:32:05.746 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm92.ko 2024-04-17T19:32:05.746 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 131505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm93.ko 2024-04-17T19:32:05.746 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 40225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95234.ko 2024-04-17T19:32:05.746 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95241.ko 2024-04-17T19:32:05.746 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95245.ko 2024-04-17T19:32:05.746 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc2945.ko 2024-04-17T19:32:05.746 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4151.ko 2024-04-17T19:32:05.746 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4215.ko 2024-04-17T19:32:05.746 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4222.ko 2024-04-17T19:32:05.746 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4245.ko 2024-04-17T19:32:05.746 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4260.ko 2024-04-17T19:32:05.746 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4261.ko 2024-04-17T19:32:05.746 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 36769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max16065.ko 2024-04-17T19:32:05.746 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1619.ko 2024-04-17T19:32:05.747 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1668.ko 2024-04-17T19:32:05.747 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max197.ko 2024-04-17T19:32:05.747 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6639.ko 2024-04-17T19:32:05.747 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6650.ko 2024-04-17T19:32:05.747 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 39729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6697.ko 2024-04-17T19:32:05.747 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mc13783-adc.ko 2024-04-17T19:32:05.747 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mcp3021.ko 2024-04-17T19:32:05.747 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2024-04-17T19:32:05.747 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 56505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6683.ko 2024-04-17T19:32:05.747 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 141353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775-core.ko 2024-04-17T19:32:05.747 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 72737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775.ko 2024-04-17T19:32:05.747 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 67841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct7802.ko 2024-04-17T19:32:05.747 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 97777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87360.ko 2024-04-17T19:32:05.747 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 61921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87427.ko 2024-04-17T19:32:05.747 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pcf8591.ko 2024-04-17T19:32:05.748 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ 2024-04-17T19:32:05.748 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-17T19:32:05.748 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-17T19:32:05.748 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-17T19:32:05.748 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-17T19:32:05.748 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-17T19:32:05.748 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-17T19:32:05.748 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-17T19:32:05.748 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 101233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-17T19:32:05.748 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-17T19:32:05.748 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 36177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-17T19:32:05.748 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-17T19:32:05.748 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-17T19:32:05.748 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/powr1220.ko 2024-04-17T19:32:05.748 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5627.ko 2024-04-17T19:32:05.749 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5636.ko 2024-04-17T19:32:05.749 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-17T19:32:05.749 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 40849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht15.ko 2024-04-17T19:32:05.749 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht21.ko 2024-04-17T19:32:05.749 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/shtc1.ko 2024-04-17T19:32:05.749 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 48713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sis5595.ko 2024-04-17T19:32:05.749 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47b397.ko 2024-04-17T19:32:05.749 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 46473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m1.ko 2024-04-17T19:32:05.749 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 44313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m192.ko 2024-04-17T19:32:05.749 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 31417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/thmc50.ko 2024-04-17T19:32:05.749 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp102.ko 2024-04-17T19:32:05.749 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp103.ko 2024-04-17T19:32:05.749 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp401.ko 2024-04-17T19:32:05.749 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp421.ko 2024-04-17T19:32:05.750 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via-cputemp.ko 2024-04-17T19:32:05.750 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 50217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via686a.ko 2024-04-17T19:32:05.750 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 59665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt1211.ko 2024-04-17T19:32:05.750 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 62737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt8231.ko 2024-04-17T19:32:05.750 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 71169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627ehf.ko 2024-04-17T19:32:05.750 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 96577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627hf.ko 2024-04-17T19:32:05.750 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 85913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83781d.ko 2024-04-17T19:32:05.750 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 75129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83791d.ko 2024-04-17T19:32:05.750 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 83425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83792d.ko 2024-04-17T19:32:05.750 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 116777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83793.ko 2024-04-17T19:32:05.750 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 134441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83795.ko 2024-04-17T19:32:05.750 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l785ts.ko 2024-04-17T19:32:05.750 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 43385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l786ng.ko 2024-04-17T19:32:05.750 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/wm831x-hwmon.ko 2024-04-17T19:32:05.751 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/wm8350-hwmon.ko 2024-04-17T19:32:05.751 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/ 2024-04-17T19:32:05.751 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/ 2024-04-17T19:32:05.751 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-17T19:32:05.751 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-17T19:32:05.751 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/ 2024-04-17T19:32:05.751 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ali1535.ko 2024-04-17T19:32:05.751 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ali1563.ko 2024-04-17T19:32:05.751 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2024-04-17T19:32:05.751 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-17T19:32:05.751 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-17T19:32:05.751 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-17T19:32:05.751 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2024-04-17T19:32:05.751 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 7729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-17T19:32:05.752 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2024-04-17T19:32:05.752 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 64233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-17T19:32:05.752 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-17T19:32:05.752 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-17T19:32:05.752 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-17T19:32:05.752 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-dln2.ko 2024-04-17T19:32:05.752 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-gpio.ko 2024-04-17T19:32:05.752 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 76969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-17T19:32:05.752 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-17T19:32:05.752 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 52937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-17T19:32:05.752 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-kempld.ko 2024-04-17T19:32:05.752 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-17T19:32:05.752 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-17T19:32:05.752 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ocores.ko 2024-04-17T19:32:05.752 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-17T19:32:05.753 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-17T19:32:05.753 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 57217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-17T19:32:05.753 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2024-04-17T19:32:05.753 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-17T19:32:05.753 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-17T19:32:05.753 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis5595.ko 2024-04-17T19:32:05.753 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis630.ko 2024-04-17T19:32:05.753 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-17T19:32:05.753 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2024-04-17T19:32:05.753 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-17T19:32:05.753 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-17T19:32:05.753 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-17T19:32:05.753 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-17T19:32:05.753 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 59193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-xiic.ko 2024-04-17T19:32:05.754 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-mux.ko 2024-04-17T19:32:05.754 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-smbus.ko 2024-04-17T19:32:05.754 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-stub.ko 2024-04-17T19:32:05.754 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/muxes/ 2024-04-17T19:32:05.754 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2024-04-17T19:32:05.754 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2024-04-17T19:32:05.754 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 36569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2024-04-17T19:32:05.754 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ 2024-04-17T19:32:05.754 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ 2024-04-17T19:32:05.754 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 329697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-17T19:32:05.754 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1119913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_core.ko 2024-04-17T19:32:05.754 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 83905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-17T19:32:05.754 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 408433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-17T19:32:05.754 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 120921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-17T19:32:05.755 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 347217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-17T19:32:05.755 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 84945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-17T19:32:05.755 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/ 2024-04-17T19:32:05.755 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/ 2024-04-17T19:32:05.755 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 590929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-17T19:32:05.755 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/ 2024-04-17T19:32:05.755 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 514641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-17T19:32:05.755 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/ 2024-04-17T19:32:05.755 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 911665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-17T19:32:05.755 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ 2024-04-17T19:32:05.755 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 375545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-17T19:32:05.755 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/ocrdma/ 2024-04-17T19:32:05.755 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 197121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2024-04-17T19:32:05.755 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/usnic/ 2024-04-17T19:32:05.755 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 212553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2024-04-17T19:32:05.756 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ 2024-04-17T19:32:05.756 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ 2024-04-17T19:32:05.756 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 334401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-17T19:32:05.756 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ 2024-04-17T19:32:05.756 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 120673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-17T19:32:05.756 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ 2024-04-17T19:32:05.756 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 132585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-17T19:32:05.756 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ 2024-04-17T19:32:05.756 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 157937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-17T19:32:05.756 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ 2024-04-17T19:32:05.756 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 156673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-17T19:32:05.756 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/ 2024-04-17T19:32:05.756 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/evbug.ko 2024-04-17T19:32:05.756 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/ff-memless.ko 2024-04-17T19:32:05.756 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/ 2024-04-17T19:32:05.757 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-17T19:32:05.757 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-17T19:32:05.757 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 50713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/gameport.ko 2024-04-17T19:32:05.757 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 49441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/joydev.ko 2024-04-17T19:32:05.757 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/ 2024-04-17T19:32:05.757 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/adp5520-keys.ko 2024-04-17T19:32:05.757 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/adp5588-keys.ko 2024-04-17T19:32:05.757 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/adp5589-keys.ko 2024-04-17T19:32:05.757 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2024-04-17T19:32:05.757 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 45473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-17T19:32:05.757 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2024-04-17T19:32:05.757 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/lkkbd.ko 2024-04-17T19:32:05.757 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 31857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/lm8323.ko 2024-04-17T19:32:05.757 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/lm8333.ko 2024-04-17T19:32:05.757 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/matrix_keypad.ko 2024-04-17T19:32:05.758 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/max7359_keypad.ko 2024-04-17T19:32:05.758 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/mcs_touchkey.ko 2024-04-17T19:32:05.758 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2024-04-17T19:32:05.758 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/newtonkbd.ko 2024-04-17T19:32:05.758 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/opencores-kbd.ko 2024-04-17T19:32:05.758 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/qt1070.ko 2024-04-17T19:32:05.758 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/qt2160.ko 2024-04-17T19:32:05.758 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/samsung-keypad.ko 2024-04-17T19:32:05.758 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/stowaway.ko 2024-04-17T19:32:05.758 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/sunkbd.ko 2024-04-17T19:32:05.758 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/tca6416-keypad.ko 2024-04-17T19:32:05.758 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/tca8418_keypad.ko 2024-04-17T19:32:05.758 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/twl4030_keypad.ko 2024-04-17T19:32:05.758 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/xtkbd.ko 2024-04-17T19:32:05.759 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/matrix-keymap.ko 2024-04-17T19:32:05.759 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/ 2024-04-17T19:32:05.759 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 40617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/appletouch.ko 2024-04-17T19:32:05.759 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 39553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/bcm5974.ko 2024-04-17T19:32:05.759 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 160017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/cyapatp.ko 2024-04-17T19:32:05.759 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 114273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-17T19:32:05.759 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/gpio_mouse.ko 2024-04-17T19:32:05.759 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 428185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/psmouse.ko 2024-04-17T19:32:05.759 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/sermouse.ko 2024-04-17T19:32:05.759 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-17T19:32:05.759 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-17T19:32:05.759 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-17T19:32:05.759 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/ 2024-04-17T19:32:05.759 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 140969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-17T19:32:05.760 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/ 2024-04-17T19:32:05.760 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/altera_ps2.ko 2024-04-17T19:32:05.760 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/arc_ps2.ko 2024-04-17T19:32:05.760 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/ct82c710.ko 2024-04-17T19:32:05.760 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-17T19:32:05.760 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/parkbd.ko 2024-04-17T19:32:05.760 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/pcips2.ko 2024-04-17T19:32:05.760 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/ps2mult.ko 2024-04-17T19:32:05.760 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serio_raw.ko 2024-04-17T19:32:05.760 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serport.ko 2024-04-17T19:32:05.760 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/sparse-keymap.ko 2024-04-17T19:32:05.760 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/ 2024-04-17T19:32:05.760 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-88pm860x.ko 2024-04-17T19:32:05.760 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-adp5520.ko 2024-04-17T19:32:05.760 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 77265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-bd2802.ko 2024-04-17T19:32:05.760 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 31097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-blinkm.ko 2024-04-17T19:32:05.761 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-da903x.ko 2024-04-17T19:32:05.761 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-da9052.ko 2024-04-17T19:32:05.761 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-gpio.ko 2024-04-17T19:32:05.761 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3530.ko 2024-04-17T19:32:05.761 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 36705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3533.ko 2024-04-17T19:32:05.761 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm355x.ko 2024-04-17T19:32:05.761 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3642.ko 2024-04-17T19:32:05.761 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp3944.ko 2024-04-17T19:32:05.761 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp8788.ko 2024-04-17T19:32:05.761 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-max8997.ko 2024-04-17T19:32:05.761 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-mc13783.ko 2024-04-17T19:32:05.761 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-menf21bmc.ko 2024-04-17T19:32:05.761 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-pca9532.ko 2024-04-17T19:32:05.761 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-pca955x.ko 2024-04-17T19:32:05.762 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-pca963x.ko 2024-04-17T19:32:05.762 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 38113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-ss4200.ko 2024-04-17T19:32:05.762 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-tca6507.ko 2024-04-17T19:32:05.762 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-wm831x-status.ko 2024-04-17T19:32:05.762 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-wm8350.ko 2024-04-17T19:32:05.762 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ 2024-04-17T19:32:05.762 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-17T19:32:05.762 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-17T19:32:05.762 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-17T19:32:05.762 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-17T19:32:05.762 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-17T19:32:05.762 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-17T19:32:05.762 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-17T19:32:05.762 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-17T19:32:05.762 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-17T19:32:05.763 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/ 2024-04-17T19:32:05.763 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/ 2024-04-17T19:32:05.763 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 790753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/bcache.ko 2024-04-17T19:32:05.763 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 49081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bio-prison.ko 2024-04-17T19:32:05.763 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 116825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bufio.ko 2024-04-17T19:32:05.763 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 58281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache-smq.ko 2024-04-17T19:32:05.763 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 199041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache.ko 2024-04-17T19:32:05.763 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 134945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-crypt.ko 2024-04-17T19:32:05.763 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-delay.ko 2024-04-17T19:32:05.763 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 67497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-era.ko 2024-04-17T19:32:05.763 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-flakey.ko 2024-04-17T19:32:05.763 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 50457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log-userspace.ko 2024-04-17T19:32:05.763 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 41145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log.ko 2024-04-17T19:32:05.763 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 60977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-mirror.ko 2024-04-17T19:32:05.764 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 97209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-multipath.ko 2024-04-17T19:32:05.764 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-queue-length.ko 2024-04-17T19:32:05.764 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 85689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-raid.ko 2024-04-17T19:32:05.764 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 44289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-region-hash.ko 2024-04-17T19:32:05.764 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-round-robin.ko 2024-04-17T19:32:05.764 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-service-time.ko 2024-04-17T19:32:05.764 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 139817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-snapshot.ko 2024-04-17T19:32:05.764 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-switch.ko 2024-04-17T19:32:05.764 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 234457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-thin-pool.ko 2024-04-17T19:32:05.764 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 66425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-verity.ko 2024-04-17T19:32:05.764 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-zero.ko 2024-04-17T19:32:05.764 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/ 2024-04-17T19:32:05.764 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 264329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-17T19:32:05.764 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 43137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid0.ko 2024-04-17T19:32:05.764 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 125009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid1.ko 2024-04-17T19:32:05.765 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 157681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid10.ko 2024-04-17T19:32:05.765 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 444665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid456.ko 2024-04-17T19:32:05.765 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/ 2024-04-17T19:32:05.765 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/ 2024-04-17T19:32:05.765 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 214841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptbase.ko 2024-04-17T19:32:05.765 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 77985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptctl.ko 2024-04-17T19:32:05.765 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 54785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptfc.ko 2024-04-17T19:32:05.765 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 44633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptlan.ko 2024-04-17T19:32:05.765 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 150761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptsas.ko 2024-04-17T19:32:05.765 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 97281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptscsih.ko 2024-04-17T19:32:05.765 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 63873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptspi.ko 2024-04-17T19:32:05.765 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/ 2024-04-17T19:32:05.765 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/88pm800.ko 2024-04-17T19:32:05.766 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/88pm805.ko 2024-04-17T19:32:05.766 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/88pm80x.ko 2024-04-17T19:32:05.766 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/arizona-i2c.ko 2024-04-17T19:32:05.766 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 148633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/arizona.ko 2024-04-17T19:32:05.766 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/bcm590xx.ko 2024-04-17T19:32:05.766 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/cros_ec_dev.ko 2024-04-17T19:32:05.766 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 41009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/dln2.ko 2024-04-17T19:32:05.766 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/janz-cmodio.ko 2024-04-17T19:32:05.766 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 59089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/kempld-core.ko 2024-04-17T19:32:05.766 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 31897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lm3533-core.ko 2024-04-17T19:32:05.766 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lm3533-ctrlbank.ko 2024-04-17T19:32:05.766 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lp3943.ko 2024-04-17T19:32:05.766 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 60497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_ich.ko 2024-04-17T19:32:05.766 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_sch.ko 2024-04-17T19:32:05.767 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/max8907.ko 2024-04-17T19:32:05.767 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mc13xxx-core.ko 2024-04-17T19:32:05.767 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mc13xxx-i2c.ko 2024-04-17T19:32:05.767 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/menf21bmc.ko 2024-04-17T19:32:05.767 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/pcf50633-adc.ko 2024-04-17T19:32:05.767 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/pcf50633-gpio.ko 2024-04-17T19:32:05.767 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 42297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/pcf50633.ko 2024-04-17T19:32:05.767 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/rdc321x-southbridge.ko 2024-04-17T19:32:05.767 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/retu-mfd.ko 2024-04-17T19:32:05.767 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 99425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/si476x-core.ko 2024-04-17T19:32:05.767 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 58057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/sm501.ko 2024-04-17T19:32:05.767 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 60681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/tps65010.ko 2024-04-17T19:32:05.767 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/tps6507x.ko 2024-04-17T19:32:05.767 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/viperboard.ko 2024-04-17T19:32:05.767 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/vx855.ko 2024-04-17T19:32:05.768 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wl1273-core.ko 2024-04-17T19:32:05.768 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 52361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wm8994.ko 2024-04-17T19:32:05.768 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ 2024-04-17T19:32:05.768 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ad525x_dpot-i2c.ko 2024-04-17T19:32:05.768 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 64265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ad525x_dpot.ko 2024-04-17T19:32:05.768 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/ 2024-04-17T19:32:05.768 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 57521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-17T19:32:05.768 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds9802als.ko 2024-04-17T19:32:05.768 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 47689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds990x.ko 2024-04-17T19:32:05.768 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 58633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/bh1770glc.ko 2024-04-17T19:32:05.768 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/c2port/ 2024-04-17T19:32:05.768 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2024-04-17T19:32:05.768 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 47449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/c2port/core.ko 2024-04-17T19:32:05.768 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/ 2024-04-17T19:32:05.768 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/cb710.ko 2024-04-17T19:32:05.769 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ds1682.ko 2024-04-17T19:32:05.769 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/dummy-irq.ko 2024-04-17T19:32:05.769 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/echo/ 2024-04-17T19:32:05.769 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/echo/echo.ko 2024-04-17T19:32:05.769 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/ 2024-04-17T19:32:05.769 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 43041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/at24.ko 2024-04-17T19:32:05.769 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-17T19:32:05.769 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/max6875.ko 2024-04-17T19:32:05.769 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 41185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/enclosure.ko 2024-04-17T19:32:05.769 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/genwqe/ 2024-04-17T19:32:05.769 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 191649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/genwqe/genwqe_card.ko 2024-04-17T19:32:05.769 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hmc6352.ko 2024-04-17T19:32:05.770 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 37761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hpilo.ko 2024-04-17T19:32:05.770 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ibmasm/ 2024-04-17T19:32:05.770 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 92345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ibmasm/ibmasm.ko 2024-04-17T19:32:05.770 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ics932s401.ko 2024-04-17T19:32:05.770 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29003.ko 2024-04-17T19:32:05.770 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29020.ko 2024-04-17T19:32:05.770 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/ 2024-04-17T19:32:05.770 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 54865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-17T19:32:05.770 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-17T19:32:05.770 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/ 2024-04-17T19:32:05.770 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 154169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-me.ko 2024-04-17T19:32:05.770 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 69809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-txe.ko 2024-04-17T19:32:05.770 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 389033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei.ko 2024-04-17T19:32:05.771 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 37961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/phantom.ko 2024-04-17T19:32:05.771 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ti-st/ 2024-04-17T19:32:05.771 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 91145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ti-st/st_drv.ko 2024-04-17T19:32:05.771 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_7xx1.ko 2024-04-17T19:32:05.771 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_core.ko 2024-04-17T19:32:05.771 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tsl2550.ko 2024-04-17T19:32:05.771 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 55793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_balloon.ko 2024-04-17T19:32:05.771 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/ 2024-04-17T19:32:05.771 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 234905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-17T19:32:05.771 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ 2024-04-17T19:32:05.771 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/ 2024-04-17T19:32:05.771 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 620961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/bonding.ko 2024-04-17T19:32:05.771 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/ 2024-04-17T19:32:05.772 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/ 2024-04-17T19:32:05.772 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 139817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-17T19:32:05.772 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 49545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-17T19:32:05.772 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-17T19:32:05.772 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/ 2024-04-17T19:32:05.772 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 587065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-17T19:32:05.772 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dummy.ko 2024-04-17T19:32:05.772 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/eql.ko 2024-04-17T19:32:05.772 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ 2024-04-17T19:32:05.772 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/ 2024-04-17T19:32:05.772 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 53753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-17T19:32:05.772 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 49201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-17T19:32:05.772 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 158649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-17T19:32:05.772 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 85953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-17T19:32:05.772 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ 2024-04-17T19:32:05.773 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 44161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-17T19:32:05.773 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 73865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-17T19:32:05.773 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-17T19:32:05.773 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 109521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-17T19:32:05.773 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/ 2024-04-17T19:32:05.773 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 64705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-17T19:32:05.773 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/ 2024-04-17T19:32:05.773 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 84329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-17T19:32:05.773 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/ 2024-04-17T19:32:05.773 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 70737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-17T19:32:05.773 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/ 2024-04-17T19:32:05.773 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 52705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-17T19:32:05.773 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/ 2024-04-17T19:32:05.773 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 65857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-17T19:32:05.774 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 56057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-17T19:32:05.774 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 115881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-17T19:32:05.774 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/ 2024-04-17T19:32:05.774 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/ 2024-04-17T19:32:05.774 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 124497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-17T19:32:05.774 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/ 2024-04-17T19:32:05.774 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 128409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-17T19:32:05.775 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/ 2024-04-17T19:32:05.775 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 110345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-17T19:32:05.775 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/ 2024-04-17T19:32:05.775 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 101857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-17T19:32:05.775 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 88953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-17T19:32:05.775 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/ 2024-04-17T19:32:05.775 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 105633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-17T19:32:05.775 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 228985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-17T19:32:05.775 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2024-04-17T19:32:05.775 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1733353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-17T19:32:05.776 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 167185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-17T19:32:05.776 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/ 2024-04-17T19:32:05.776 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 161417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-17T19:32:05.776 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 440625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-17T19:32:05.776 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/ 2024-04-17T19:32:05.776 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/ 2024-04-17T19:32:05.776 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 432201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-17T19:32:05.776 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/ 2024-04-17T19:32:05.776 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/ 2024-04-17T19:32:05.776 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 257201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-17T19:32:05.776 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2024-04-17T19:32:05.777 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 446177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-17T19:32:05.777 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2024-04-17T19:32:05.777 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1311433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-17T19:32:05.777 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2024-04-17T19:32:05.777 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 206689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-17T19:32:05.777 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2024-04-17T19:32:05.777 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 40161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-17T19:32:05.777 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/ 2024-04-17T19:32:05.777 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/ 2024-04-17T19:32:05.777 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 269313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-17T19:32:05.778 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/ 2024-04-17T19:32:05.778 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/ 2024-04-17T19:32:05.778 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 85385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-17T19:32:05.778 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 77457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-17T19:32:05.778 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 169089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-17T19:32:05.778 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 65913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-17T19:32:05.778 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 76209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-17T19:32:05.778 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 38065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-17T19:32:05.778 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/ 2024-04-17T19:32:05.778 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 75945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-17T19:32:05.778 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 78705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-17T19:32:05.778 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 52313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dnet.ko 2024-04-17T19:32:05.778 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ec_bhf.ko 2024-04-17T19:32:05.779 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/ 2024-04-17T19:32:05.779 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/ 2024-04-17T19:32:05.779 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 356377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-17T19:32:05.779 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 61737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ethoc.ko 2024-04-17T19:32:05.779 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 63185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fealnx.ko 2024-04-17T19:32:05.779 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fujitsu/ 2024-04-17T19:32:05.779 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 51513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2024-04-17T19:32:05.779 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ 2024-04-17T19:32:05.779 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 115569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-17T19:32:05.779 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/ 2024-04-17T19:32:05.779 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 366217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-17T19:32:05.779 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/ 2024-04-17T19:32:05.779 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 755793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-17T19:32:05.779 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/ 2024-04-17T19:32:05.779 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 324577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-17T19:32:05.780 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/ 2024-04-17T19:32:05.780 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1196297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-17T19:32:05.780 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/ 2024-04-17T19:32:05.780 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 362593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-17T19:32:05.780 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/ 2024-04-17T19:32:05.780 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 671801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-17T19:32:05.780 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/ 2024-04-17T19:32:05.780 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 121537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-17T19:32:05.780 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ 2024-04-17T19:32:05.780 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 966705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-17T19:32:05.780 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ 2024-04-17T19:32:05.780 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 193585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-17T19:32:05.780 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 108633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/jme.ko 2024-04-17T19:32:05.780 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/ 2024-04-17T19:32:05.781 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-17T19:32:05.781 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 126881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-17T19:32:05.781 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 150777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-17T19:32:05.781 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/ 2024-04-17T19:32:05.781 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/ 2024-04-17T19:32:05.781 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 905033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-17T19:32:05.781 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 344145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-17T19:32:05.781 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/ 2024-04-17T19:32:05.781 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2024-04-17T19:32:05.781 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1412417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-17T19:32:05.781 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ 2024-04-17T19:32:05.781 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 65913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ks8842.ko 2024-04-17T19:32:05.782 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 54497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2024-04-17T19:32:05.782 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2024-04-17T19:32:05.782 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 117353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-17T19:32:05.782 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/ 2024-04-17T19:32:05.782 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/ 2024-04-17T19:32:05.782 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 131425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-17T19:32:05.782 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ 2024-04-17T19:32:05.782 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 91465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-17T19:32:05.782 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 56689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-17T19:32:05.782 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/ 2024-04-17T19:32:05.782 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 181081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-17T19:32:05.782 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/ 2024-04-17T19:32:05.782 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 171417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-17T19:32:05.783 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/ 2024-04-17T19:32:05.783 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 58249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-17T19:32:05.783 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 50121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-17T19:32:05.783 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/ 2024-04-17T19:32:05.783 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/ 2024-04-17T19:32:05.783 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 300817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-17T19:32:05.783 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 105689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-17T19:32:05.784 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2024-04-17T19:32:05.784 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 885129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-17T19:32:05.784 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/ 2024-04-17T19:32:05.784 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 52305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-17T19:32:05.784 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/ 2024-04-17T19:32:05.784 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 81641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-17T19:32:05.784 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 100689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-17T19:32:05.784 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-17T19:32:05.784 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 264161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-17T19:32:05.784 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/samsung/ 2024-04-17T19:32:05.784 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/samsung/sxgbe/ 2024-04-17T19:32:05.784 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 142449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2024-04-17T19:32:05.784 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/ 2024-04-17T19:32:05.784 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1350385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-17T19:32:05.785 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/ 2024-04-17T19:32:05.785 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 57953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-17T19:32:05.785 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/ 2024-04-17T19:32:05.785 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 66505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-17T19:32:05.785 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 80065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-17T19:32:05.785 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/ 2024-04-17T19:32:05.785 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 74289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-17T19:32:05.785 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 79457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-17T19:32:05.785 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 100785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-17T19:32:05.785 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 83441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-17T19:32:05.785 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/ 2024-04-17T19:32:05.785 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/ 2024-04-17T19:32:05.785 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2024-04-17T19:32:05.785 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 46985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-17T19:32:05.785 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2024-04-17T19:32:05.786 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 644537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-17T19:32:05.786 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/ 2024-04-17T19:32:05.786 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 128497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-17T19:32:05.786 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 207993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-17T19:32:05.786 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 82633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-17T19:32:05.786 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 91353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-17T19:32:05.786 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/ 2024-04-17T19:32:05.786 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 59865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-17T19:32:05.786 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/ 2024-04-17T19:32:05.786 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 110729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-17T19:32:05.786 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/ 2024-04-17T19:32:05.786 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 98689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-17T19:32:05.786 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 102513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-17T19:32:05.786 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/ 2024-04-17T19:32:05.787 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 61497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-17T19:32:05.787 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 40609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-17T19:32:05.787 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/ 2024-04-17T19:32:05.787 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 69913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-17T19:32:05.787 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/fddi/ 2024-04-17T19:32:05.787 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 49449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/fddi/defxx.ko 2024-04-17T19:32:05.787 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/fddi/skfp/ 2024-04-17T19:32:05.787 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 275801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/fddi/skfp/skfp.ko 2024-04-17T19:32:05.787 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 91105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/geneve.ko 2024-04-17T19:32:05.787 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/ 2024-04-17T19:32:05.787 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 290441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-17T19:32:05.787 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/ 2024-04-17T19:32:05.787 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-17T19:32:05.787 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ifb.ko 2024-04-17T19:32:05.787 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ 2024-04-17T19:32:05.788 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 108329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-17T19:32:05.788 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 78065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvlan.ko 2024-04-17T19:32:05.788 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvtap.ko 2024-04-17T19:32:05.788 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/ 2024-04-17T19:32:05.788 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-17T19:32:05.788 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-17T19:32:05.788 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-gpio.ko 2024-04-17T19:32:05.788 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-17T19:32:05.788 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio.ko 2024-04-17T19:32:05.788 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mii.ko 2024-04-17T19:32:05.788 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 62681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/netconsole.ko 2024-04-17T19:32:05.788 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/nlmon.ko 2024-04-17T19:32:05.788 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/ 2024-04-17T19:32:05.788 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-17T19:32:05.789 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 48313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-17T19:32:05.789 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ 2024-04-17T19:32:05.789 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/amd.ko 2024-04-17T19:32:05.789 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 61985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-17T19:32:05.789 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 52865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-17T19:32:05.789 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm87xx.ko 2024-04-17T19:32:05.789 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 53409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/broadcom.ko 2024-04-17T19:32:05.789 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/cicada.ko 2024-04-17T19:32:05.789 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/davicom.ko 2024-04-17T19:32:05.790 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/et1011c.ko 2024-04-17T19:32:05.790 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 31313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/icplus.ko 2024-04-17T19:32:05.791 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/lxt.ko 2024-04-17T19:32:05.791 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 103377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/marvell.ko 2024-04-17T19:32:05.791 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 96025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/micrel.ko 2024-04-17T19:32:05.791 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/national.ko 2024-04-17T19:32:05.791 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 121105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/phylink.ko 2024-04-17T19:32:05.791 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/qsemi.ko 2024-04-17T19:32:05.791 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 62113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/realtek.ko 2024-04-17T19:32:05.791 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/smsc.ko 2024-04-17T19:32:05.792 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ste10Xp.ko 2024-04-17T19:32:05.792 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/vitesse.ko 2024-04-17T19:32:05.792 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/plip/ 2024-04-17T19:32:05.792 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 55353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/plip/plip.ko 2024-04-17T19:32:05.792 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ 2024-04-17T19:32:05.792 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-17T19:32:05.792 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 36273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_async.ko 2024-04-17T19:32:05.792 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-17T19:32:05.792 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-17T19:32:05.792 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-17T19:32:05.792 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 53233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppoe.ko 2024-04-17T19:32:05.792 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppox.ko 2024-04-17T19:32:05.792 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 40081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pptp.ko 2024-04-17T19:32:05.792 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sb1000.ko 2024-04-17T19:32:05.793 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/ 2024-04-17T19:32:05.793 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 52073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slip.ko 2024-04-17T19:32:05.793 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 42409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sungem_phy.ko 2024-04-17T19:32:05.793 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 80161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tap.ko 2024-04-17T19:32:05.793 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/ 2024-04-17T19:32:05.793 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 137321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team.ko 2024-04-17T19:32:05.793 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-17T19:32:05.793 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-17T19:32:05.793 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 36209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-17T19:32:05.793 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_random.ko 2024-04-17T19:32:05.793 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-17T19:32:05.793 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 108089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/veth.ko 2024-04-17T19:32:05.793 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/ 2024-04-17T19:32:05.793 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 176497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-17T19:32:05.793 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/ 2024-04-17T19:32:05.794 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 383593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/vxlan.ko 2024-04-17T19:32:05.794 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/ 2024-04-17T19:32:05.794 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 175537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-17T19:32:05.794 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ 2024-04-17T19:32:05.794 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ntb.ko 2024-04-17T19:32:05.794 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/ 2024-04-17T19:32:05.794 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/ 2024-04-17T19:32:05.794 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 335857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme-core.ko 2024-04-17T19:32:05.794 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 126553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme.ko 2024-04-17T19:32:05.794 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/ 2024-04-17T19:32:05.794 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 168145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport.ko 2024-04-17T19:32:05.794 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_cs.ko 2024-04-17T19:32:05.794 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 103889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_pc.ko 2024-04-17T19:32:05.794 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_serial.ko 2024-04-17T19:32:05.795 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/ 2024-04-17T19:32:05.795 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/ 2024-04-17T19:32:05.795 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-17T19:32:05.795 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2024-04-17T19:32:05.795 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2024-04-17T19:32:05.795 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pci-stub.ko 2024-04-17T19:32:05.795 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 55329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/xen-pcifront.ko 2024-04-17T19:32:05.795 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/ 2024-04-17T19:32:05.795 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 31529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/i82092.ko 2024-04-17T19:32:05.795 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 179961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pcmcia.ko 2024-04-17T19:32:05.795 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 78161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pcmcia_core.ko 2024-04-17T19:32:05.795 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 41945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2024-04-17T19:32:05.795 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 40457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pd6729.ko 2024-04-17T19:32:05.795 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 133753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-17T19:32:05.796 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/phy/ 2024-04-17T19:32:05.796 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/phy/broadcom/ 2024-04-17T19:32:05.796 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2024-04-17T19:32:05.796 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/phy/samsung/ 2024-04-17T19:32:05.796 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2024-04-17T19:32:05.796 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/ 2024-04-17T19:32:05.796 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/ 2024-04-17T19:32:05.796 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 78193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-17T19:32:05.796 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/ 2024-04-17T19:32:05.796 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/ 2024-04-17T19:32:05.796 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 61041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2024-04-17T19:32:05.796 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros_ec.ko 2024-04-17T19:32:05.796 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2024-04-17T19:32:05.796 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2024-04-17T19:32:05.796 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 31801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2024-04-17T19:32:05.797 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2024-04-17T19:32:05.797 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2024-04-17T19:32:05.797 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/ 2024-04-17T19:32:05.797 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 87481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-17T19:32:05.797 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 44105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acerhdf.ko 2024-04-17T19:32:05.797 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 48257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-17T19:32:05.797 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 73865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-17T19:32:05.797 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-17T19:32:05.797 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 164769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-17T19:32:05.797 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 46185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-17T19:32:05.797 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 62073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-17T19:32:05.798 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-17T19:32:05.798 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 51881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-17T19:32:05.798 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-17T19:32:05.798 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 42761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/hdaps.ko 2024-04-17T19:32:05.798 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/ibm_rtl.ko 2024-04-17T19:32:05.798 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/ 2024-04-17T19:32:05.798 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-17T19:32:05.798 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2024-04-17T19:32:05.798 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 54001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel_ips.ko 2024-04-17T19:32:05.798 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-17T19:32:05.798 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-17T19:32:05.798 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 43753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-17T19:32:05.798 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 51897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-17T19:32:05.798 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-17T19:32:05.799 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-17T19:32:05.799 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-17T19:32:05.799 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-17T19:32:05.799 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-17T19:32:05.799 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 61417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi.ko 2024-04-17T19:32:05.799 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/ 2024-04-17T19:32:05.799 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/ 2024-04-17T19:32:05.799 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 36513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/88pm860x_battery.ko 2024-04-17T19:32:05.799 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 40425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/88pm860x_charger.ko 2024-04-17T19:32:05.799 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 68833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/bq2415x_charger.ko 2024-04-17T19:32:05.799 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 73073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/bq24190_charger.ko 2024-04-17T19:32:05.799 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/bq24735-charger.ko 2024-04-17T19:32:05.799 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/cros_peripheral_charger.ko 2024-04-17T19:32:05.799 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/da9030_battery.ko 2024-04-17T19:32:05.800 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/da9052-battery.ko 2024-04-17T19:32:05.800 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/ds2760_battery.ko 2024-04-17T19:32:05.800 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/ds2780_battery.ko 2024-04-17T19:32:05.800 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/ds2781_battery.ko 2024-04-17T19:32:05.800 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/ds2782_battery.ko 2024-04-17T19:32:05.800 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/gpio-charger.ko 2024-04-17T19:32:05.800 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/isp1704_charger.ko 2024-04-17T19:32:05.800 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/lp8727_charger.ko 2024-04-17T19:32:05.800 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/max14577_charger.ko 2024-04-17T19:32:05.800 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 37833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/max17042_battery.ko 2024-04-17T19:32:05.800 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/max8903_charger.ko 2024-04-17T19:32:05.800 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/max8925_power.ko 2024-04-17T19:32:05.800 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/pcf50633-charger.ko 2024-04-17T19:32:05.800 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 44841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/sbs-battery.ko 2024-04-17T19:32:05.800 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/test_power.ko 2024-04-17T19:32:05.801 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/tps65090-charger.ko 2024-04-17T19:32:05.801 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/wm831x_backup.ko 2024-04-17T19:32:05.801 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/wm831x_power.ko 2024-04-17T19:32:05.801 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/wm8350_power.ko 2024-04-17T19:32:05.801 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/ 2024-04-17T19:32:05.801 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 73297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-17T19:32:05.801 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-17T19:32:05.801 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/ 2024-04-17T19:32:05.801 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/ 2024-04-17T19:32:05.801 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-17T19:32:05.801 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-17T19:32:05.801 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps_parport.ko 2024-04-17T19:32:05.801 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 49817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/pps_core.ko 2024-04-17T19:32:05.801 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ 2024-04-17T19:32:05.801 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 91393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp.ko 2024-04-17T19:32:05.802 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp_kvm.ko 2024-04-17T19:32:05.802 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/ 2024-04-17T19:32:05.802 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/ 2024-04-17T19:32:05.802 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/ 2024-04-17T19:32:05.802 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 67513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-17T19:32:05.802 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 47137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/fmpm.ko 2024-04-17T19:32:05.802 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/ 2024-04-17T19:32:05.802 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-88pm80x.ko 2024-04-17T19:32:05.802 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-88pm860x.ko 2024-04-17T19:32:05.802 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-17T19:32:05.802 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-da9052.ko 2024-04-17T19:32:05.802 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-da9055.ko 2024-04-17T19:32:05.802 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-da9063.ko 2024-04-17T19:32:05.802 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-17T19:32:05.803 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 82225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-17T19:32:05.803 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-17T19:32:05.803 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-17T19:32:05.803 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-17T19:32:05.803 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-17T19:32:05.803 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-17T19:32:05.803 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-17T19:32:05.803 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-17T19:32:05.803 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-em3027.ko 2024-04-17T19:32:05.803 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-17T19:32:05.803 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-17T19:32:05.803 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 44129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-17T19:32:05.803 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-lp8788.ko 2024-04-17T19:32:05.803 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 46913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-17T19:32:05.803 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-17T19:32:05.804 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-17T19:32:05.804 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t86.ko 2024-04-17T19:32:05.804 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max6900.ko 2024-04-17T19:32:05.804 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max8907.ko 2024-04-17T19:32:05.804 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max8925.ko 2024-04-17T19:32:05.804 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 31337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max8997.ko 2024-04-17T19:32:05.804 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max8998.ko 2024-04-17T19:32:05.804 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-mc13xxx.ko 2024-04-17T19:32:05.804 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-17T19:32:05.804 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-palmas.ko 2024-04-17T19:32:05.804 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 56777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-17T19:32:05.804 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf50633.ko 2024-04-17T19:32:05.804 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-17T19:32:05.804 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-17T19:32:05.805 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 36065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-17T19:32:05.805 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-17T19:32:05.805 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rc5t583.ko 2024-04-17T19:32:05.805 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-17T19:32:05.805 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 40081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-17T19:32:05.805 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-17T19:32:05.805 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-17T19:32:05.805 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-17T19:32:05.805 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-s35390a.ko 2024-04-17T19:32:05.805 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-17T19:32:05.805 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-tps6586x.ko 2024-04-17T19:32:05.805 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-tps65910.ko 2024-04-17T19:32:05.805 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-wm831x.ko 2024-04-17T19:32:05.805 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-wm8350.ko 2024-04-17T19:32:05.805 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-x1205.ko 2024-04-17T19:32:05.806 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ 2024-04-17T19:32:05.806 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 90377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-9xxx.ko 2024-04-17T19:32:05.806 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 69593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-sas.ko 2024-04-17T19:32:05.806 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 65945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-xxxx.ko 2024-04-17T19:32:05.806 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 164569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/BusLogic.ko 2024-04-17T19:32:05.806 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 31617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/a100u2w.ko 2024-04-17T19:32:05.806 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/ 2024-04-17T19:32:05.806 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 340721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-17T19:32:05.806 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 120337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/advansys.ko 2024-04-17T19:32:05.806 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/ 2024-04-17T19:32:05.806 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 468081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-17T19:32:05.806 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 380673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-17T19:32:05.806 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic94xx/ 2024-04-17T19:32:05.806 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 243705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic94xx/aic94xx.ko 2024-04-17T19:32:05.806 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/am53c974.ko 2024-04-17T19:32:05.807 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/ 2024-04-17T19:32:05.807 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 135617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-17T19:32:05.807 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 44289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/atp870u.ko 2024-04-17T19:32:05.807 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/ 2024-04-17T19:32:05.807 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 304993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-17T19:32:05.807 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/ 2024-04-17T19:32:05.807 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1527113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/bfa.ko 2024-04-17T19:32:05.807 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/ 2024-04-17T19:32:05.807 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 297801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-17T19:32:05.807 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/ 2024-04-17T19:32:05.807 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 144401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-17T19:32:05.807 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 47457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ch.ko 2024-04-17T19:32:05.807 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/ 2024-04-17T19:32:05.807 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 390713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-17T19:32:05.807 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/ 2024-04-17T19:32:05.808 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/ 2024-04-17T19:32:05.808 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 88881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-17T19:32:05.808 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/ 2024-04-17T19:32:05.808 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 117481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-17T19:32:05.808 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 173017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-17T19:32:05.808 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 91569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dc395x.ko 2024-04-17T19:32:05.808 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 45401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dmx3191d.ko 2024-04-17T19:32:05.808 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/ 2024-04-17T19:32:05.808 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 267577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-17T19:32:05.808 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 78969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esp_scsi.ko 2024-04-17T19:32:05.808 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/ 2024-04-17T19:32:05.808 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 95329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-17T19:32:05.808 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 194017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-17T19:32:05.808 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fdomain.ko 2024-04-17T19:32:05.809 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/ 2024-04-17T19:32:05.809 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 366529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/fnic.ko 2024-04-17T19:32:05.809 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 251505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hpsa.ko 2024-04-17T19:32:05.809 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 65761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hptiop.ko 2024-04-17T19:32:05.809 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 57097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hv_storvsc.ko 2024-04-17T19:32:05.809 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 44689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/imm.ko 2024-04-17T19:32:05.809 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 55153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/initio.ko 2024-04-17T19:32:05.809 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 327841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ipr.ko 2024-04-17T19:32:05.809 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 129785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ips.ko 2024-04-17T19:32:05.809 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/ 2024-04-17T19:32:05.809 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 397785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/isci.ko 2024-04-17T19:32:05.809 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-17T19:32:05.809 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 63121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-17T19:32:05.809 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/ 2024-04-17T19:32:05.810 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 386849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/libfc.ko 2024-04-17T19:32:05.810 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 178177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi.ko 2024-04-17T19:32:05.810 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 64809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-17T19:32:05.810 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/ 2024-04-17T19:32:05.810 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 274009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/libsas.ko 2024-04-17T19:32:05.810 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/ 2024-04-17T19:32:05.810 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 2736001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-17T19:32:05.810 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/ 2024-04-17T19:32:05.810 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 88529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-17T19:32:05.810 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 40145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-17T19:32:05.810 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 389177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-17T19:32:05.810 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 104769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid.ko 2024-04-17T19:32:05.810 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/ 2024-04-17T19:32:05.810 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 836945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-17T19:32:05.811 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/ 2024-04-17T19:32:05.811 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 158729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-17T19:32:05.811 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 96537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvumi.ko 2024-04-17T19:32:05.811 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pcmcia/ 2024-04-17T19:32:05.811 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 101625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2024-04-17T19:32:05.811 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2024-04-17T19:32:05.811 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2024-04-17T19:32:05.811 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2024-04-17T19:32:05.811 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/ 2024-04-17T19:32:05.811 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 443745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-17T19:32:05.811 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 152033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pmcraid.ko 2024-04-17T19:32:05.811 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 41625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ppa.ko 2024-04-17T19:32:05.811 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 66977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla1280.ko 2024-04-17T19:32:05.811 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/ 2024-04-17T19:32:05.812 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 2324809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-17T19:32:05.812 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 103601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-17T19:32:05.812 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/ 2024-04-17T19:32:05.812 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 697337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-17T19:32:05.812 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qlogicfas408.ko 2024-04-17T19:32:05.812 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/raid_class.ko 2024-04-17T19:32:05.812 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 322865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_debug.ko 2024-04-17T19:32:05.812 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 219857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-17T19:32:05.812 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 357153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-17T19:32:05.812 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 121185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-17T19:32:05.812 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 93481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-17T19:32:05.812 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 55401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-17T19:32:05.812 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ses.ko 2024-04-17T19:32:05.812 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 138121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/st.ko 2024-04-17T19:32:05.812 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 57401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/stex.ko 2024-04-17T19:32:05.813 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/ 2024-04-17T19:32:05.813 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 205025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-17T19:32:05.813 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 55425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/virtio_scsi.ko 2024-04-17T19:32:05.813 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 59185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-17T19:32:05.813 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 48777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/wd719x.ko 2024-04-17T19:32:05.813 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 59417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/xen-scsifront.ko 2024-04-17T19:32:05.813 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ 2024-04-17T19:32:05.813 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 174913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ssb.ko 2024-04-17T19:32:05.813 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/ 2024-04-17T19:32:05.813 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/ 2024-04-17T19:32:05.813 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 920169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-17T19:32:05.813 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/ 2024-04-17T19:32:05.813 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 60321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-17T19:32:05.813 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 48129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_file.ko 2024-04-17T19:32:05.813 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 54233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_iblock.ko 2024-04-17T19:32:05.814 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1170033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_mod.ko 2024-04-17T19:32:05.814 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 51897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_pscsi.ko 2024-04-17T19:32:05.814 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 149721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_user.ko 2024-04-17T19:32:05.814 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/ 2024-04-17T19:32:05.814 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 90057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-17T19:32:05.814 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/ 2024-04-17T19:32:05.814 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/ 2024-04-17T19:32:05.814 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/ 2024-04-17T19:32:05.814 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-17T19:32:05.814 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-17T19:32:05.814 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-17T19:32:05.814 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-17T19:32:05.814 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-17T19:32:05.814 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-17T19:32:05.815 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 36729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-17T19:32:05.815 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-17T19:32:05.815 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-17T19:32:05.815 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-17T19:32:05.815 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-17T19:32:05.815 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-17T19:32:05.815 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 46065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-17T19:32:05.815 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-17T19:32:05.815 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-17T19:32:05.815 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_powerclamp.ko 2024-04-17T19:32:05.815 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-17T19:32:05.815 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-17T19:32:05.815 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-17T19:32:05.815 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ 2024-04-17T19:32:05.816 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ 2024-04-17T19:32:05.816 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 99313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-17T19:32:05.816 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 69729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/moxa.ko 2024-04-17T19:32:05.816 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 61649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/mxser.ko 2024-04-17T19:32:05.816 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 134401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_gsm.ko 2024-04-17T19:32:05.816 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 37905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_hdlc.ko 2024-04-17T19:32:05.816 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 78969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/nozomi.ko 2024-04-17T19:32:05.816 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/ 2024-04-17T19:32:05.816 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/ 2024-04-17T19:32:05.816 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 44057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/8250_dw.ko 2024-04-17T19:32:05.816 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 65121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-17T19:32:05.816 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/altera_jtaguart.ko 2024-04-17T19:32:05.816 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/altera_uart.ko 2024-04-17T19:32:05.816 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/arc_uart.ko 2024-04-17T19:32:05.816 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 91665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/fsl_lpuart.ko 2024-04-17T19:32:05.817 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/ 2024-04-17T19:32:05.817 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 120577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-17T19:32:05.817 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/rp2.ko 2024-04-17T19:32:05.817 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 65641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/sc16is7xx.ko 2024-04-17T19:32:05.817 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/ 2024-04-17T19:32:05.817 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ 2024-04-17T19:32:05.817 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 693081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-17T19:32:05.817 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ 2024-04-17T19:32:05.817 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-17T19:32:05.817 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2024-04-17T19:32:05.817 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/ 2024-04-17T19:32:05.817 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 55617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio.ko 2024-04-17T19:32:05.817 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_aec.ko 2024-04-17T19:32:05.817 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_cif.ko 2024-04-17T19:32:05.817 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_dmem_genirq.ko 2024-04-17T19:32:05.818 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_mf624.ko 2024-04-17T19:32:05.818 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_netx.ko 2024-04-17T19:32:05.818 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pci_generic.ko 2024-04-17T19:32:05.818 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pdrv_genirq.ko 2024-04-17T19:32:05.818 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_sercos3.ko 2024-04-17T19:32:05.818 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/ 2024-04-17T19:32:05.818 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/c67x00/ 2024-04-17T19:32:05.818 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 90321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/c67x00/c67x00.ko 2024-04-17T19:32:05.818 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/chipidea/ 2024-04-17T19:32:05.818 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 189209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/chipidea/ci_hdrc.ko 2024-04-17T19:32:05.818 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2024-04-17T19:32:05.818 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/chipidea/ci_hdrc_npcm.ko 2024-04-17T19:32:05.818 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2024-04-17T19:32:05.818 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2024-04-17T19:32:05.819 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/ 2024-04-17T19:32:05.819 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 101545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-acm.ko 2024-04-17T19:32:05.819 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 63073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-17T19:32:05.819 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 56721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usblp.ko 2024-04-17T19:32:05.819 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 107417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usbtmc.ko 2024-04-17T19:32:05.819 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/common/ 2024-04-17T19:32:05.819 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/common/ulpi.ko 2024-04-17T19:32:05.819 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/dwc2/ 2024-04-17T19:32:05.819 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/dwc2/dwc2_pci.ko 2024-04-17T19:32:05.819 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/dwc3/ 2024-04-17T19:32:05.819 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/dwc3/dwc3-haps.ko 2024-04-17T19:32:05.819 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/dwc3/dwc3-pci.ko 2024-04-17T19:32:05.819 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 574593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/dwc3/dwc3.ko 2024-04-17T19:32:05.819 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/ 2024-04-17T19:32:05.819 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/ 2024-04-17T19:32:05.820 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 72249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/u_ether.ko 2024-04-17T19:32:05.820 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 61953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/u_serial.ko 2024-04-17T19:32:05.820 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2024-04-17T19:32:05.820 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 49737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2024-04-17T19:32:05.820 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2024-04-17T19:32:05.820 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 43553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2024-04-17T19:32:05.820 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 131345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2024-04-17T19:32:05.820 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 62321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2024-04-17T19:32:05.820 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 146617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2024-04-17T19:32:05.820 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 70689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2024-04-17T19:32:05.820 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2024-04-17T19:32:05.820 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 37649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2024-04-17T19:32:05.820 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 70017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2024-04-17T19:32:05.820 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 101249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2024-04-17T19:32:05.820 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2024-04-17T19:32:05.821 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 71257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2024-04-17T19:32:05.821 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 84937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2024-04-17T19:32:05.821 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/ 2024-04-17T19:32:05.821 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2024-04-17T19:32:05.821 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2024-04-17T19:32:05.821 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2024-04-17T19:32:05.821 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_ether.ko 2024-04-17T19:32:05.821 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2024-04-17T19:32:05.821 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_hid.ko 2024-04-17T19:32:05.821 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2024-04-17T19:32:05.821 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2024-04-17T19:32:05.821 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2024-04-17T19:32:05.821 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_printer.ko 2024-04-17T19:32:05.821 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_serial.ko 2024-04-17T19:32:05.821 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_zero.ko 2024-04-17T19:32:05.822 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 78073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2024-04-17T19:32:05.822 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2024-04-17T19:32:05.822 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 204049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/libcomposite.ko 2024-04-17T19:32:05.822 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/ 2024-04-17T19:32:05.822 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2024-04-17T19:32:05.822 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/bdc/ 2024-04-17T19:32:05.822 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 182561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2024-04-17T19:32:05.822 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 45545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/goku_udc.ko 2024-04-17T19:32:05.822 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 66513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/gr_udc.ko 2024-04-17T19:32:05.822 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 85217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2024-04-17T19:32:05.822 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 69313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/mv_udc.ko 2024-04-17T19:32:05.822 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 90313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/net2272.ko 2024-04-17T19:32:05.822 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 90361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/net2280.ko 2024-04-17T19:32:05.822 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 94841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/pch_udc.ko 2024-04-17T19:32:05.823 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 80521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2024-04-17T19:32:05.895 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 87465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2024-04-17T19:32:05.895 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 95441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2024-04-17T19:32:05.895 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 195969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/udc-core.ko 2024-04-17T19:32:05.895 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/ 2024-04-17T19:32:05.895 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/bcma-hcd.ko 2024-04-17T19:32:05.895 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 108321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/isp116x-hcd.ko 2024-04-17T19:32:05.895 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 104377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/oxu210hp-hcd.ko 2024-04-17T19:32:05.895 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 102273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/r8a66597-hcd.ko 2024-04-17T19:32:05.895 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 60105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-17T19:32:05.895 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811_cs.ko 2024-04-17T19:32:05.895 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/ssb-hcd.ko 2024-04-17T19:32:05.896 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/xhci-plat-hcd.ko 2024-04-17T19:32:05.896 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/ 2024-04-17T19:32:05.896 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 40409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/mdc800.ko 2024-04-17T19:32:05.896 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/microtek.ko 2024-04-17T19:32:05.896 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ 2024-04-17T19:32:05.896 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 47353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/adutux.ko 2024-04-17T19:32:05.896 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/appledisplay.ko 2024-04-17T19:32:05.896 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/cypress_cy7c63.ko 2024-04-17T19:32:05.896 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/cytherm.ko 2024-04-17T19:32:05.896 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ehset.ko 2024-04-17T19:32:05.896 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi26.ko 2024-04-17T19:32:05.896 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi62.ko 2024-04-17T19:32:05.896 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ezusb.ko 2024-04-17T19:32:05.896 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/idmouse.ko 2024-04-17T19:32:05.897 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 42729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/iowarrior.ko 2024-04-17T19:32:05.897 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-17T19:32:05.897 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 42241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ldusb.ko 2024-04-17T19:32:05.897 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 42233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/legousbtower.ko 2024-04-17T19:32:05.897 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/lvstest.ko 2024-04-17T19:32:05.897 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/ 2024-04-17T19:32:05.897 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 85313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-17T19:32:05.897 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-17T19:32:05.897 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usb3503.ko 2024-04-17T19:32:05.897 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usblcd.ko 2024-04-17T19:32:05.897 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-17T19:32:05.897 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 84265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbtest.ko 2024-04-17T19:32:05.897 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 52065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/uss720.ko 2024-04-17T19:32:05.897 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/yurex.ko 2024-04-17T19:32:05.897 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/mon/ 2024-04-17T19:32:05.898 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 91529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/mon/usbmon.ko 2024-04-17T19:32:05.898 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/musb/ 2024-04-17T19:32:05.898 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 396209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/musb/musb_hdrc.ko 2024-04-17T19:32:05.898 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/ 2024-04-17T19:32:05.898 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-generic.ko 2024-04-17T19:32:05.898 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2024-04-17T19:32:05.898 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-isp1301.ko 2024-04-17T19:32:05.898 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-tahvo.ko 2024-04-17T19:32:05.898 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ 2024-04-17T19:32:05.898 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/aircable.ko 2024-04-17T19:32:05.898 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 38209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ark3116.ko 2024-04-17T19:32:05.898 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-17T19:32:05.898 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 44329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ch341.ko 2024-04-17T19:32:05.899 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 79681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cp210x.ko 2024-04-17T19:32:05.899 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cyberjack.ko 2024-04-17T19:32:05.899 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 66873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-17T19:32:05.899 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 58329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-17T19:32:05.899 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/empeg.ko 2024-04-17T19:32:05.899 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 53009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/f81232.ko 2024-04-17T19:32:05.899 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 188609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-17T19:32:05.899 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 61489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-17T19:32:05.899 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 152305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-17T19:32:05.899 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 142833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_ti.ko 2024-04-17T19:32:05.899 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 69217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipaq.ko 2024-04-17T19:32:05.899 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipw.ko 2024-04-17T19:32:05.899 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ir-usb.ko 2024-04-17T19:32:05.899 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 66361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-17T19:32:05.900 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 122329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan.ko 2024-04-17T19:32:05.900 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 41297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-17T19:32:05.900 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-17T19:32:05.900 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 38049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-17T19:32:05.900 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 38041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mct_u232.ko 2024-04-17T19:32:05.900 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/metro-usb.ko 2024-04-17T19:32:05.900 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 79681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7720.ko 2024-04-17T19:32:05.900 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 89969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7840.ko 2024-04-17T19:32:05.900 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 65073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mxuport.ko 2024-04-17T19:32:05.900 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/navman.ko 2024-04-17T19:32:05.900 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/omninet.ko 2024-04-17T19:32:05.900 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/opticon.ko 2024-04-17T19:32:05.900 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 193857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/option.ko 2024-04-17T19:32:05.900 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 45337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/oti6858.ko 2024-04-17T19:32:05.901 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 61761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/pl2303.ko 2024-04-17T19:32:05.901 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcaux.ko 2024-04-17T19:32:05.901 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 39985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcserial.ko 2024-04-17T19:32:05.901 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 40529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/quatech2.ko 2024-04-17T19:32:05.901 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/safe_serial.ko 2024-04-17T19:32:05.901 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 59345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/sierra.ko 2024-04-17T19:32:05.901 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-17T19:32:05.901 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ssu100.ko 2024-04-17T19:32:05.901 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/symbolserial.ko 2024-04-17T19:32:05.901 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 70497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-17T19:32:05.901 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-17T19:32:05.901 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_debug.ko 2024-04-17T19:32:05.901 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 39321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-17T19:32:05.901 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 151137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usbserial.ko 2024-04-17T19:32:05.901 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/visor.ko 2024-04-17T19:32:05.902 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 43577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/whiteheat.ko 2024-04-17T19:32:05.902 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/wishbone-serial.ko 2024-04-17T19:32:05.902 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-17T19:32:05.902 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ 2024-04-17T19:32:05.902 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 64049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/uas.ko 2024-04-17T19:32:05.902 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-17T19:32:05.902 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-17T19:32:05.902 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-17T19:32:05.902 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 41073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-17T19:32:05.902 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-17T19:32:05.902 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-17T19:32:05.902 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-17T19:32:05.902 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-karma.ko 2024-04-17T19:32:05.903 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-17T19:32:05.903 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-17T19:32:05.903 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-17T19:32:05.903 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-17T19:32:05.903 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 41785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-17T19:32:05.903 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 201193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/usb-storage.ko 2024-04-17T19:32:05.903 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/ 2024-04-17T19:32:05.903 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 76041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-17T19:32:05.903 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 85257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-17T19:32:05.903 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 122321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-17T19:32:05.904 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/ 2024-04-17T19:32:05.904 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/ 2024-04-17T19:32:05.904 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 206921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-17T19:32:05.904 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-17T19:32:05.904 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 134265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio.ko 2024-04-17T19:32:05.904 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 93673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-17T19:32:05.904 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/ 2024-04-17T19:32:05.904 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 138721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost.ko 2024-04-17T19:32:05.904 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-17T19:32:05.904 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 71281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_net.ko 2024-04-17T19:32:05.904 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 107905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_scsi.ko 2024-04-17T19:32:05.904 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/ 2024-04-17T19:32:05.904 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/ 2024-04-17T19:32:05.904 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/88pm860x_bl.ko 2024-04-17T19:32:05.905 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/aat2870_bl.ko 2024-04-17T19:32:05.905 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/adp5520_bl.ko 2024-04-17T19:32:05.905 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 41169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/adp8860_bl.ko 2024-04-17T19:32:05.905 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 51745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/adp8870_bl.ko 2024-04-17T19:32:05.905 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/apple_bl.ko 2024-04-17T19:32:05.905 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/as3711_bl.ko 2024-04-17T19:32:05.905 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/bd6107.ko 2024-04-17T19:32:05.905 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/da903x_bl.ko 2024-04-17T19:32:05.905 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/da9052_bl.ko 2024-04-17T19:32:05.905 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/gpio_backlight.ko 2024-04-17T19:32:05.905 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/kb3886_bl.ko 2024-04-17T19:32:05.905 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lcd.ko 2024-04-17T19:32:05.905 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lm3533_bl.ko 2024-04-17T19:32:05.905 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lm3639_bl.ko 2024-04-17T19:32:05.906 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lv5207lp.ko 2024-04-17T19:32:05.906 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/max8925_bl.ko 2024-04-17T19:32:05.906 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/pandora_bl.ko 2024-04-17T19:32:05.906 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/pcf50633-backlight.ko 2024-04-17T19:32:05.906 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-17T19:32:05.906 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/wm831x_bl.ko 2024-04-17T19:32:05.906 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/ 2024-04-17T19:32:05.906 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ 2024-04-17T19:32:05.906 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2482.ko 2024-04-17T19:32:05.906 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 46801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2490.ko 2024-04-17T19:32:05.907 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/matrox_w1.ko 2024-04-17T19:32:05.908 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/w1-gpio.ko 2024-04-17T19:32:05.908 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/ 2024-04-17T19:32:05.908 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-17T19:32:05.908 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-17T19:32:05.908 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-17T19:32:05.908 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-17T19:32:05.908 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-17T19:32:05.908 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-17T19:32:05.908 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-17T19:32:05.908 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-17T19:32:05.908 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-17T19:32:05.908 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-17T19:32:05.908 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 70369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-17T19:32:05.909 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 129433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/wire.ko 2024-04-17T19:32:05.909 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ 2024-04-17T19:32:05.909 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/acquirewdt.ko 2024-04-17T19:32:05.909 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/advantechwdt.ko 2024-04-17T19:32:05.909 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-17T19:32:05.909 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-17T19:32:05.909 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/cpu5wdt.ko 2024-04-17T19:32:05.909 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/da9052_wdt.ko 2024-04-17T19:32:05.909 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/da9055_wdt.ko 2024-04-17T19:32:05.909 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/da9063_wdt.ko 2024-04-17T19:32:05.909 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/dw_wdt.ko 2024-04-17T19:32:05.909 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/eurotechwdt.ko 2024-04-17T19:32:05.909 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-17T19:32:05.909 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/hpwdt.ko 2024-04-17T19:32:05.910 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/i6300esb.ko 2024-04-17T19:32:05.910 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-17T19:32:05.910 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-17T19:32:05.910 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ib700wdt.ko 2024-04-17T19:32:05.910 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ibmasr.ko 2024-04-17T19:32:05.910 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-17T19:32:05.910 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-17T19:32:05.910 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it87_wdt.ko 2024-04-17T19:32:05.910 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/kempld_wdt.ko 2024-04-17T19:32:05.910 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/machzwd.ko 2024-04-17T19:32:05.910 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/mena21_wdt.ko 2024-04-17T19:32:05.910 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/menf21bmc_wdt.ko 2024-04-17T19:32:05.910 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/nv_tco.ko 2024-04-17T19:32:05.911 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/of_xilinx_wdt.ko 2024-04-17T19:32:05.911 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pc87413_wdt.ko 2024-04-17T19:32:05.911 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 46953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-17T19:32:05.911 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 37193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-17T19:32:05.911 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/retu_wdt.ko 2024-04-17T19:32:05.911 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc60xxwdt.ko 2024-04-17T19:32:05.911 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_epx_c3.ko 2024-04-17T19:32:05.911 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-17T19:32:05.911 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sc1200wdt.ko 2024-04-17T19:32:05.911 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-17T19:32:05.911 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/smsc37b787_wdt.ko 2024-04-17T19:32:05.911 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/softdog.ko 2024-04-17T19:32:05.911 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-17T19:32:05.911 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/twl4030_wdt.ko 2024-04-17T19:32:05.912 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/via_wdt.ko 2024-04-17T19:32:05.912 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-17T19:32:05.912 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-17T19:32:05.912 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-17T19:32:05.912 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wafer5823wdt.ko 2024-04-17T19:32:05.912 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wdt_pci.ko 2024-04-17T19:32:05.912 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wm831x_wdt.ko 2024-04-17T19:32:05.912 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wm8350_wdt.ko 2024-04-17T19:32:05.912 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/xen_wdt.ko 2024-04-17T19:32:05.912 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/ 2024-04-17T19:32:05.912 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-evtchn.ko 2024-04-17T19:32:05.912 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntalloc.ko 2024-04-17T19:32:05.912 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 55713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntdev.ko 2024-04-17T19:32:05.912 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/ 2024-04-17T19:32:05.913 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 213913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-17T19:32:05.913 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 47689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-privcmd.ko 2024-04-17T19:32:05.913 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 85713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-scsiback.ko 2024-04-17T19:32:05.913 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/ 2024-04-17T19:32:05.913 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-17T19:32:05.913 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ 2024-04-17T19:32:05.913 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/ 2024-04-17T19:32:05.913 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 192617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/9p.ko 2024-04-17T19:32:05.913 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/adfs/ 2024-04-17T19:32:05.913 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 78153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/adfs/adfs.ko 2024-04-17T19:32:05.913 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/ 2024-04-17T19:32:05.913 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 197849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/affs.ko 2024-04-17T19:32:05.913 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/afs/ 2024-04-17T19:32:05.913 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1261113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/afs/kafs.ko 2024-04-17T19:32:05.914 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/autofs/ 2024-04-17T19:32:05.914 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 127377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/autofs/autofs4.ko 2024-04-17T19:32:05.914 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/ 2024-04-17T19:32:05.914 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 80521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/befs.ko 2024-04-17T19:32:05.914 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/bfs/ 2024-04-17T19:32:05.914 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 50473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/bfs/bfs.ko 2024-04-17T19:32:05.914 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 45561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/binfmt_misc.ko 2024-04-17T19:32:05.914 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/ 2024-04-17T19:32:05.914 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 4489593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/btrfs.ko 2024-04-17T19:32:05.914 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/ 2024-04-17T19:32:05.914 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 376857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/cachefiles.ko 2024-04-17T19:32:05.914 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ 2024-04-17T19:32:05.914 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1586089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ceph.ko 2024-04-17T19:32:05.915 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/ 2024-04-17T19:32:05.915 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 130073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/coda.ko 2024-04-17T19:32:05.915 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/configfs/ 2024-04-17T19:32:05.915 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 139081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/configfs/configfs.ko 2024-04-17T19:32:05.915 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/ 2024-04-17T19:32:05.915 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/cramfs.ko 2024-04-17T19:32:05.915 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/ 2024-04-17T19:32:05.915 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 813225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/dlm.ko 2024-04-17T19:32:05.915 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/efs/ 2024-04-17T19:32:05.915 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 46409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/efs/efs.ko 2024-04-17T19:32:05.915 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/ 2024-04-17T19:32:05.915 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1906817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/f2fs.ko 2024-04-17T19:32:05.915 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/ 2024-04-17T19:32:05.915 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/msdos.ko 2024-04-17T19:32:05.916 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/freevxfs/ 2024-04-17T19:32:05.916 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 52313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/freevxfs/freevxfs.ko 2024-04-17T19:32:05.916 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/ 2024-04-17T19:32:05.916 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/cuse.ko 2024-04-17T19:32:05.916 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/ 2024-04-17T19:32:05.916 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1486521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/gfs2.ko 2024-04-17T19:32:05.916 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/ 2024-04-17T19:32:05.916 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 183305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/hfs.ko 2024-04-17T19:32:05.916 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/ 2024-04-17T19:32:05.916 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 282097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/hfsplus.ko 2024-04-17T19:32:05.916 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hpfs/ 2024-04-17T19:32:05.916 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 249009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hpfs/hpfs.ko 2024-04-17T19:32:05.916 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/ 2024-04-17T19:32:05.917 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 93033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/isofs.ko 2024-04-17T19:32:05.917 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/ 2024-04-17T19:32:05.917 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 533073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/jfs.ko 2024-04-17T19:32:05.917 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/ 2024-04-17T19:32:05.917 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 361369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/lockd.ko 2024-04-17T19:32:05.917 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/ 2024-04-17T19:32:05.917 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 112609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/minix.ko 2024-04-17T19:32:05.917 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/ 2024-04-17T19:32:05.917 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 591585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/netfs.ko 2024-04-17T19:32:05.917 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/ 2024-04-17T19:32:05.917 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/ 2024-04-17T19:32:05.917 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 102561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-17T19:32:05.917 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/ 2024-04-17T19:32:05.917 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 74353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-17T19:32:05.917 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/ 2024-04-17T19:32:05.918 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 147689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-17T19:32:05.918 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1320249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfs.ko 2024-04-17T19:32:05.918 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 82105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv2.ko 2024-04-17T19:32:05.918 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 151641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv3.ko 2024-04-17T19:32:05.918 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 2518505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv4.ko 2024-04-17T19:32:05.918 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/ 2024-04-17T19:32:05.918 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/grace.ko 2024-04-17T19:32:05.918 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/nfs_acl.ko 2024-04-17T19:32:05.918 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/ 2024-04-17T19:32:05.918 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 2024049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/nfsd.ko 2024-04-17T19:32:05.918 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/ 2024-04-17T19:32:05.918 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 734025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/nilfs2.ko 2024-04-17T19:32:05.919 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/ 2024-04-17T19:32:05.919 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-celtic.ko 2024-04-17T19:32:05.919 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-centeuro.ko 2024-04-17T19:32:05.919 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-croatian.ko 2024-04-17T19:32:05.919 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-cyrillic.ko 2024-04-17T19:32:05.919 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-gaelic.ko 2024-04-17T19:32:05.919 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-greek.ko 2024-04-17T19:32:05.919 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-iceland.ko 2024-04-17T19:32:05.919 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-inuit.ko 2024-04-17T19:32:05.919 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-roman.ko 2024-04-17T19:32:05.920 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-romanian.ko 2024-04-17T19:32:05.920 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-turkish.ko 2024-04-17T19:32:05.920 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ascii.ko 2024-04-17T19:32:05.920 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1250.ko 2024-04-17T19:32:05.920 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1251.ko 2024-04-17T19:32:05.920 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1255.ko 2024-04-17T19:32:05.920 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp737.ko 2024-04-17T19:32:05.920 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp775.ko 2024-04-17T19:32:05.920 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp850.ko 2024-04-17T19:32:05.920 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp852.ko 2024-04-17T19:32:05.920 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp855.ko 2024-04-17T19:32:05.920 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp857.ko 2024-04-17T19:32:05.920 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp860.ko 2024-04-17T19:32:05.920 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp861.ko 2024-04-17T19:32:05.921 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp862.ko 2024-04-17T19:32:05.921 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp863.ko 2024-04-17T19:32:05.921 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp864.ko 2024-04-17T19:32:05.921 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp865.ko 2024-04-17T19:32:05.921 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp866.ko 2024-04-17T19:32:05.921 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp869.ko 2024-04-17T19:32:05.921 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp874.ko 2024-04-17T19:32:05.921 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 95625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp932.ko 2024-04-17T19:32:05.921 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 144209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp936.ko 2024-04-17T19:32:05.921 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 167137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp949.ko 2024-04-17T19:32:05.921 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 119513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp950.ko 2024-04-17T19:32:05.921 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_euc-jp.ko 2024-04-17T19:32:05.921 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-1.ko 2024-04-17T19:32:05.921 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-13.ko 2024-04-17T19:32:05.922 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-14.ko 2024-04-17T19:32:05.922 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-15.ko 2024-04-17T19:32:05.922 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-2.ko 2024-04-17T19:32:05.922 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-3.ko 2024-04-17T19:32:05.922 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-4.ko 2024-04-17T19:32:05.922 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-5.ko 2024-04-17T19:32:05.922 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-6.ko 2024-04-17T19:32:05.922 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-7.ko 2024-04-17T19:32:05.922 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-9.ko 2024-04-17T19:32:05.922 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-r.ko 2024-04-17T19:32:05.922 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-ru.ko 2024-04-17T19:32:05.922 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-u.ko 2024-04-17T19:32:05.922 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 7521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-17T19:32:05.923 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_utf8.ko 2024-04-17T19:32:05.923 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ 2024-04-17T19:32:05.923 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ 2024-04-17T19:32:05.923 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 302073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-17T19:32:05.923 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ 2024-04-17T19:32:05.923 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 548169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-17T19:32:05.923 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ 2024-04-17T19:32:05.923 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 66361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-17T19:32:05.923 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 3245617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2.ko 2024-04-17T19:32:05.923 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 22113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-17T19:32:05.923 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 39313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-17T19:32:05.923 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 43073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-17T19:32:05.923 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/omfs/ 2024-04-17T19:32:05.923 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 67489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/omfs/omfs.ko 2024-04-17T19:32:05.924 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/ 2024-04-17T19:32:05.924 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 462873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/overlay.ko 2024-04-17T19:32:05.924 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ 2024-04-17T19:32:05.924 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 67977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ramoops.ko 2024-04-17T19:32:05.924 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/qnx4/ 2024-04-17T19:32:05.924 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/qnx4/qnx4.ko 2024-04-17T19:32:05.924 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/qnx6/ 2024-04-17T19:32:05.924 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 56097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/qnx6/qnx6.ko 2024-04-17T19:32:05.924 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/quota/ 2024-04-17T19:32:05.924 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 40313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/quota/quota_tree.ko 2024-04-17T19:32:05.924 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/quota/quota_v1.ko 2024-04-17T19:32:05.924 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/quota/quota_v2.ko 2024-04-17T19:32:05.924 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/ 2024-04-17T19:32:05.924 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 684841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/reiserfs.ko 2024-04-17T19:32:05.925 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/ 2024-04-17T19:32:05.925 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/romfs.ko 2024-04-17T19:32:05.925 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/ 2024-04-17T19:32:05.925 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/ 2024-04-17T19:32:05.925 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 3319041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/cifs.ko 2024-04-17T19:32:05.925 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/ 2024-04-17T19:32:05.925 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_arc4.ko 2024-04-17T19:32:05.925 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_md4.ko 2024-04-17T19:32:05.925 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/ 2024-04-17T19:32:05.925 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 164185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/squashfs.ko 2024-04-17T19:32:05.925 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/ 2024-04-17T19:32:05.925 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 115977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/sysv.ko 2024-04-17T19:32:05.925 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/ 2024-04-17T19:32:05.925 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 310017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/udf.ko 2024-04-17T19:32:05.926 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ 2024-04-17T19:32:05.926 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 198233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ufs.ko 2024-04-17T19:32:05.926 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/ 2024-04-17T19:32:05.926 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 5192705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/xfs.ko 2024-04-17T19:32:05.926 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/ 2024-04-17T19:32:05.926 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/locking/ 2024-04-17T19:32:05.926 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 97625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/locking/locktorture.ko 2024-04-17T19:32:05.926 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/time/ 2024-04-17T19:32:05.926 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/time/test_udelay.ko 2024-04-17T19:32:05.926 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 72665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/torture.ko 2024-04-17T19:32:05.926 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ 2024-04-17T19:32:05.926 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 7313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc-itu-t.ko 2024-04-17T19:32:05.926 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 7081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc7.ko 2024-04-17T19:32:05.926 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 7953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc8.ko 2024-04-17T19:32:05.926 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/ 2024-04-17T19:32:05.927 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libarc4.ko 2024-04-17T19:32:05.927 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 38985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libdes.ko 2024-04-17T19:32:05.927 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/interval_tree_test.ko 2024-04-17T19:32:05.927 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/libcrc32c.ko 2024-04-17T19:32:05.927 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lru_cache.ko 2024-04-17T19:32:05.927 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/ 2024-04-17T19:32:05.927 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 36169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4_compress.ko 2024-04-17T19:32:05.927 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4hc_compress.ko 2024-04-17T19:32:05.927 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/ 2024-04-17T19:32:05.927 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 7281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/cordic.ko 2024-04-17T19:32:05.927 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/memory-notifier-error-inject.ko 2024-04-17T19:32:05.927 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/notifier-error-inject.ko 2024-04-17T19:32:05.927 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/percpu_test.ko 2024-04-17T19:32:05.928 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/pm-notifier-error-inject.ko 2024-04-17T19:32:05.928 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/ 2024-04-17T19:32:05.928 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 152025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/raid6_pq.ko 2024-04-17T19:32:05.928 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/rbtree_test.ko 2024-04-17T19:32:05.928 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/ 2024-04-17T19:32:05.928 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-17T19:32:05.928 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 40969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/test-kstrtox.ko 2024-04-17T19:32:05.928 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 4912441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/test_bpf.ko 2024-04-17T19:32:05.928 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 61169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/test_firmware.ko 2024-04-17T19:32:05.928 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 7121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/test_module.ko 2024-04-17T19:32:05.928 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/test_user_copy.ko 2024-04-17T19:32:05.928 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_bm.ko 2024-04-17T19:32:05.928 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_fsm.ko 2024-04-17T19:32:05.928 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_kmp.ko 2024-04-17T19:32:05.929 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/xz/ 2024-04-17T19:32:05.929 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/xz/xz_dec_test.ko 2024-04-17T19:32:05.929 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/ 2024-04-17T19:32:05.929 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/hwpoison-inject.ko 2024-04-17T19:32:05.929 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ 2024-04-17T19:32:05.929 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/ 2024-04-17T19:32:05.929 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 66969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/6lowpan.ko 2024-04-17T19:32:05.929 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 7577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_dest.ko 2024-04-17T19:32:05.929 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 7529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_fragment.ko 2024-04-17T19:32:05.929 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 7513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_hop.ko 2024-04-17T19:32:05.929 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 7513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-17T19:32:05.929 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 7529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_mobility.ko 2024-04-17T19:32:05.929 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 7521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_routing.ko 2024-04-17T19:32:05.929 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_udp.ko 2024-04-17T19:32:05.929 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/ 2024-04-17T19:32:05.930 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 36729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/garp.ko 2024-04-17T19:32:05.930 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 42297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/mrp.ko 2024-04-17T19:32:05.930 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/p8022.ko 2024-04-17T19:32:05.930 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/psnap.ko 2024-04-17T19:32:05.930 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/stp.ko 2024-04-17T19:32:05.930 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/ 2024-04-17T19:32:05.930 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 106793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/8021q.ko 2024-04-17T19:32:05.930 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/ 2024-04-17T19:32:05.930 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 236793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet.ko 2024-04-17T19:32:05.930 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 44449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_fd.ko 2024-04-17T19:32:05.930 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_rdma.ko 2024-04-17T19:32:05.930 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 39513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_virtio.ko 2024-04-17T19:32:05.930 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/ 2024-04-17T19:32:05.931 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 879697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/batman-adv.ko 2024-04-17T19:32:05.931 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/ 2024-04-17T19:32:05.931 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 84761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/br_netfilter.ko 2024-04-17T19:32:05.931 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1030873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/bridge.ko 2024-04-17T19:32:05.931 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ 2024-04-17T19:32:05.931 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-17T19:32:05.931 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-17T19:32:05.931 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-17T19:32:05.931 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-17T19:32:05.931 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-17T19:32:05.931 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-17T19:32:05.931 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-17T19:32:05.931 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-17T19:32:05.931 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-17T19:32:05.931 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-17T19:32:05.932 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-17T19:32:05.932 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-17T19:32:05.932 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-17T19:32:05.932 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-17T19:32:05.932 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-17T19:32:05.932 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-17T19:32:05.932 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-17T19:32:05.932 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-17T19:32:05.932 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-17T19:32:05.932 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-17T19:32:05.932 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 70529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtables.ko 2024-04-17T19:32:05.932 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-17T19:32:05.932 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-17T19:32:05.932 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/ 2024-04-17T19:32:05.933 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1243505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/libceph.ko 2024-04-17T19:32:05.933 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/ 2024-04-17T19:32:05.933 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 153801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/pktgen.ko 2024-04-17T19:32:05.933 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/ 2024-04-17T19:32:05.933 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 244889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp.ko 2024-04-17T19:32:05.933 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_diag.ko 2024-04-17T19:32:05.933 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 71033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv4.ko 2024-04-17T19:32:05.933 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 74993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv6.ko 2024-04-17T19:32:05.933 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/ 2024-04-17T19:32:05.933 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 507705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/dsa_core.ko 2024-04-17T19:32:05.933 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_brcm.ko 2024-04-17T19:32:05.933 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_dsa.ko 2024-04-17T19:32:05.933 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_none.ko 2024-04-17T19:32:05.933 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_trailer.ko 2024-04-17T19:32:05.934 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/hsr/ 2024-04-17T19:32:05.934 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 128873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/hsr/hsr.ko 2024-04-17T19:32:05.934 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ 2024-04-17T19:32:05.934 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ 2024-04-17T19:32:05.934 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 55505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-17T19:32:05.934 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 357425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154.ko 2024-04-17T19:32:05.934 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 59393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-17T19:32:05.934 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ 2024-04-17T19:32:05.934 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ah4.ko 2024-04-17T19:32:05.934 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 46417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/esp4.ko 2024-04-17T19:32:05.934 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 60233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/fou.ko 2024-04-17T19:32:05.934 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/gre.ko 2024-04-17T19:32:05.934 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 65665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/inet_diag.ko 2024-04-17T19:32:05.934 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 75489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_gre.ko 2024-04-17T19:32:05.935 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 77721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_tunnel.ko 2024-04-17T19:32:05.935 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 43705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_vti.ko 2024-04-17T19:32:05.935 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipcomp.ko 2024-04-17T19:32:05.935 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipip.ko 2024-04-17T19:32:05.935 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ 2024-04-17T19:32:05.935 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 45521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-17T19:32:05.935 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-17T19:32:05.935 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-17T19:32:05.935 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 50169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ip_tables.ko 2024-04-17T19:32:05.935 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-17T19:32:05.935 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2024-04-17T19:32:05.935 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-17T19:32:05.935 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-17T19:32:05.935 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-17T19:32:05.936 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_filter.ko 2024-04-17T19:32:05.936 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-17T19:32:05.936 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-17T19:32:05.936 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-17T19:32:05.936 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-17T19:32:05.936 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-17T19:32:05.936 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-17T19:32:05.936 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-17T19:32:05.936 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-17T19:32:05.936 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-17T19:32:05.936 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2024-04-17T19:32:05.936 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-17T19:32:05.936 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-17T19:32:05.936 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-17T19:32:05.937 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-17T19:32:05.937 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-17T19:32:05.937 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_bic.ko 2024-04-17T19:32:05.937 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_dctcp.ko 2024-04-17T19:32:05.937 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_diag.ko 2024-04-17T19:32:05.937 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_highspeed.ko 2024-04-17T19:32:05.937 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_htcp.ko 2024-04-17T19:32:05.937 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_hybla.ko 2024-04-17T19:32:05.937 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_illinois.ko 2024-04-17T19:32:05.937 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_lp.ko 2024-04-17T19:32:05.937 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_scalable.ko 2024-04-17T19:32:05.937 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_vegas.ko 2024-04-17T19:32:05.937 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_veno.ko 2024-04-17T19:32:05.937 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_westwood.ko 2024-04-17T19:32:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_yeah.ko 2024-04-17T19:32:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tunnel4.ko 2024-04-17T19:32:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_diag.ko 2024-04-17T19:32:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 54441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_tunnel.ko 2024-04-17T19:32:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-17T19:32:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ 2024-04-17T19:32:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 31073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ah6.ko 2024-04-17T19:32:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 47433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/esp6.ko 2024-04-17T19:32:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/fou6.ko 2024-04-17T19:32:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 103177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_gre.ko 2024-04-17T19:32:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 118881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_tunnel.ko 2024-04-17T19:32:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-17T19:32:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 71089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_vti.ko 2024-04-17T19:32:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ipcomp6.ko 2024-04-17T19:32:05.938 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/mip6.ko 2024-04-17T19:32:05.939 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ 2024-04-17T19:32:05.939 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 50673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-17T19:32:05.939 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2024-04-17T19:32:05.939 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-17T19:32:05.939 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-17T19:32:05.939 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-17T19:32:05.939 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-17T19:32:05.939 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-17T19:32:05.939 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-17T19:32:05.939 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-17T19:32:05.939 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-17T19:32:05.939 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-17T19:32:05.939 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-17T19:32:05.939 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-17T19:32:05.940 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-17T19:32:05.940 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-17T19:32:05.940 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-17T19:32:05.940 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-17T19:32:05.940 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 46313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-17T19:32:05.940 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-17T19:32:05.940 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 29817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-17T19:32:05.940 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-17T19:32:05.940 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-17T19:32:05.940 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-17T19:32:05.940 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-17T19:32:05.940 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-17T19:32:05.940 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 94089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/sit.ko 2024-04-17T19:32:05.940 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 30297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/tunnel6.ko 2024-04-17T19:32:05.941 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-17T19:32:05.941 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/ 2024-04-17T19:32:05.941 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 106369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/af_key.ko 2024-04-17T19:32:05.941 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/ 2024-04-17T19:32:05.941 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 153217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_core.ko 2024-04-17T19:32:05.941 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-17T19:32:05.941 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 31985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_eth.ko 2024-04-17T19:32:05.941 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 43641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip.ko 2024-04-17T19:32:05.941 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 44993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip6.ko 2024-04-17T19:32:05.941 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 42049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_netlink.ko 2024-04-17T19:32:05.941 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 66241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ppp.ko 2024-04-17T19:32:05.941 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/lapb/ 2024-04-17T19:32:05.941 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 63345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/lapb/lapb.ko 2024-04-17T19:32:05.941 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/ 2024-04-17T19:32:05.942 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc.ko 2024-04-17T19:32:05.942 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 332889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc2.ko 2024-04-17T19:32:05.942 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/ 2024-04-17T19:32:05.942 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 388873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/mac802154.ko 2024-04-17T19:32:05.942 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/ 2024-04-17T19:32:05.942 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/mpls_gso.ko 2024-04-17T19:32:05.942 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ 2024-04-17T19:32:05.942 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ 2024-04-17T19:32:05.942 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 147745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set.ko 2024-04-17T19:32:05.942 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-17T19:32:05.942 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-17T19:32:05.942 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-17T19:32:05.942 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 124041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-17T19:32:05.942 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 124345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-17T19:32:05.943 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 126681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-17T19:32:05.943 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 126473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-17T19:32:05.943 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 137833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-17T19:32:05.943 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 72353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-17T19:32:05.943 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 133681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-17T19:32:05.943 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 139073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-17T19:32:05.943 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 138265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-17T19:32:05.943 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 136177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-17T19:32:05.943 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 139897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-17T19:32:05.943 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 37129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-17T19:32:05.943 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ 2024-04-17T19:32:05.943 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 534113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-17T19:32:05.943 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-17T19:32:05.943 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-17T19:32:05.944 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-17T19:32:05.944 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-17T19:32:05.944 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-17T19:32:05.944 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-17T19:32:05.944 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-17T19:32:05.944 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-17T19:32:05.944 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-17T19:32:05.944 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-17T19:32:05.944 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-17T19:32:05.944 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-17T19:32:05.944 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-17T19:32:05.944 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 55217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conncount.ko 2024-04-17T19:32:05.944 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 496817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack.ko 2024-04-17T19:32:05.944 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-17T19:32:05.944 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-17T19:32:05.945 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 37865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-17T19:32:05.945 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 134377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-17T19:32:05.945 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-17T19:32:05.945 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-17T19:32:05.945 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 140801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-17T19:32:05.945 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 38305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-17T19:32:05.945 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-17T19:32:05.945 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 79873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-17T19:32:05.945 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-17T19:32:05.945 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-17T19:32:05.945 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-17T19:32:05.945 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 117201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table.ko 2024-04-17T19:32:05.945 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-17T19:32:05.945 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 46105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_log_syslog.ko 2024-04-17T19:32:05.945 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 118369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat.ko 2024-04-17T19:32:05.946 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-17T19:32:05.946 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-17T19:32:05.946 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_irc.ko 2024-04-17T19:32:05.946 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_sip.ko 2024-04-17T19:32:05.946 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-17T19:32:05.946 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 53393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-17T19:32:05.946 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 807985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_tables.ko 2024-04-17T19:32:05.946 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 53049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink.ko 2024-04-17T19:32:05.946 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 36481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-17T19:32:05.947 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 37209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_cttimeout.ko 2024-04-17T19:32:05.947 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 49297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_log.ko 2024-04-17T19:32:05.947 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-17T19:32:05.947 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 65145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-17T19:32:05.947 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_chain_nat.ko 2024-04-17T19:32:05.947 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 41769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_compat.ko 2024-04-17T19:32:05.947 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 50657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_ct.ko 2024-04-17T19:32:05.947 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-17T19:32:05.947 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fib.ko 2024-04-17T19:32:05.947 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-17T19:32:05.947 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_hash.ko 2024-04-17T19:32:05.947 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_limit.ko 2024-04-17T19:32:05.947 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_log.ko 2024-04-17T19:32:05.948 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_masq.ko 2024-04-17T19:32:05.948 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_nat.ko 2024-04-17T19:32:05.948 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_numgen.ko 2024-04-17T19:32:05.948 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_osf.ko 2024-04-17T19:32:05.948 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_queue.ko 2024-04-17T19:32:05.948 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_redir.ko 2024-04-17T19:32:05.948 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject.ko 2024-04-17T19:32:05.948 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject_inet.ko 2024-04-17T19:32:05.948 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_socket.ko 2024-04-17T19:32:05.948 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 24993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_synproxy.ko 2024-04-17T19:32:05.948 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_tproxy.ko 2024-04-17T19:32:05.948 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 108393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/x_tables.ko 2024-04-17T19:32:05.948 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_AUDIT.ko 2024-04-17T19:32:05.949 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-17T19:32:05.949 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-17T19:32:05.949 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-17T19:32:05.949 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 21865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CT.ko 2024-04-17T19:32:05.949 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_DSCP.ko 2024-04-17T19:32:05.949 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HL.ko 2024-04-17T19:32:05.949 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HMARK.ko 2024-04-17T19:32:05.949 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 45217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-17T19:32:05.949 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LED.ko 2024-04-17T19:32:05.949 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LOG.ko 2024-04-17T19:32:05.949 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-17T19:32:05.949 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NETMAP.ko 2024-04-17T19:32:05.949 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFLOG.ko 2024-04-17T19:32:05.949 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-17T19:32:05.950 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_RATEEST.ko 2024-04-17T19:32:05.950 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-17T19:32:05.950 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_SECMARK.ko 2024-04-17T19:32:05.950 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-17T19:32:05.950 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-17T19:32:05.950 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TEE.ko 2024-04-17T19:32:05.950 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TPROXY.ko 2024-04-17T19:32:05.950 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TRACE.ko 2024-04-17T19:32:05.950 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_addrtype.ko 2024-04-17T19:32:05.950 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_bpf.ko 2024-04-17T19:32:05.950 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cgroup.ko 2024-04-17T19:32:05.950 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cluster.ko 2024-04-17T19:32:05.950 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_comment.ko 2024-04-17T19:32:05.951 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connbytes.ko 2024-04-17T19:32:05.951 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlabel.ko 2024-04-17T19:32:05.951 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlimit.ko 2024-04-17T19:32:05.951 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connmark.ko 2024-04-17T19:32:05.951 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_conntrack.ko 2024-04-17T19:32:05.951 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cpu.ko 2024-04-17T19:32:05.951 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dccp.ko 2024-04-17T19:32:05.951 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_devgroup.ko 2024-04-17T19:32:05.951 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dscp.ko 2024-04-17T19:32:05.951 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ecn.ko 2024-04-17T19:32:05.951 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_esp.ko 2024-04-17T19:32:05.951 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 52817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hashlimit.ko 2024-04-17T19:32:05.951 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_helper.ko 2024-04-17T19:32:05.951 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hl.ko 2024-04-17T19:32:05.952 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipcomp.ko 2024-04-17T19:32:05.952 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_iprange.ko 2024-04-17T19:32:05.952 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipvs.ko 2024-04-17T19:32:05.952 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_l2tp.ko 2024-04-17T19:32:05.952 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_length.ko 2024-04-17T19:32:05.952 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_limit.ko 2024-04-17T19:32:05.952 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mac.ko 2024-04-17T19:32:05.952 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mark.ko 2024-04-17T19:32:05.952 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_multiport.ko 2024-04-17T19:32:05.952 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 16161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nat.ko 2024-04-17T19:32:05.952 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nfacct.ko 2024-04-17T19:32:05.952 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_osf.ko 2024-04-17T19:32:05.952 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_owner.ko 2024-04-17T19:32:05.952 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_physdev.ko 2024-04-17T19:32:05.953 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_pkttype.ko 2024-04-17T19:32:05.953 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_policy.ko 2024-04-17T19:32:05.953 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_quota.ko 2024-04-17T19:32:05.953 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 11881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_rateest.ko 2024-04-17T19:32:05.953 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_realm.ko 2024-04-17T19:32:05.953 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 43465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_recent.ko 2024-04-17T19:32:05.953 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_sctp.ko 2024-04-17T19:32:05.953 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 36489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_set.ko 2024-04-17T19:32:05.953 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 19777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_socket.ko 2024-04-17T19:32:05.953 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_state.ko 2024-04-17T19:32:05.953 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_statistic.ko 2024-04-17T19:32:05.953 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_string.ko 2024-04-17T19:32:05.953 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpmss.ko 2024-04-17T19:32:05.953 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpudp.ko 2024-04-17T19:32:05.953 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_time.ko 2024-04-17T19:32:05.954 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_u32.ko 2024-04-17T19:32:05.954 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/ 2024-04-17T19:32:05.954 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 17233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/netlink_diag.ko 2024-04-17T19:32:05.954 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/ 2024-04-17T19:32:05.954 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 14073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/nsh.ko 2024-04-17T19:32:05.954 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/ 2024-04-17T19:32:05.954 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 488825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/openvswitch.ko 2024-04-17T19:32:05.954 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-geneve.ko 2024-04-17T19:32:05.954 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 10873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-gre.ko 2024-04-17T19:32:05.954 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-vxlan.ko 2024-04-17T19:32:05.954 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/ 2024-04-17T19:32:05.954 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/af_packet_diag.ko 2024-04-17T19:32:05.954 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/phonet/ 2024-04-17T19:32:05.954 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 142025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/phonet/phonet.ko 2024-04-17T19:32:05.955 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 68817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/phonet/pn_pep.ko 2024-04-17T19:32:05.955 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/ 2024-04-17T19:32:05.955 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 318617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds.ko 2024-04-17T19:32:05.955 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 202305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_rdma.ko 2024-04-17T19:32:05.955 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 85105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_tcp.ko 2024-04-17T19:32:05.955 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rxrpc/ 2024-04-17T19:32:05.955 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1027897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rxrpc/rxrpc.ko 2024-04-17T19:32:05.955 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/ 2024-04-17T19:32:05.955 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 39001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_csum.ko 2024-04-17T19:32:05.955 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 28721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_gact.ko 2024-04-17T19:32:05.955 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 45409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_mirred.ko 2024-04-17T19:32:05.955 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 27593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_nat.ko 2024-04-17T19:32:05.955 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 53465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_pedit.ko 2024-04-17T19:32:05.955 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 38705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_police.ko 2024-04-17T19:32:05.955 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_simple.ko 2024-04-17T19:32:05.956 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 43417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_skbedit.ko 2024-04-17T19:32:05.956 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 45673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_vlan.ko 2024-04-17T19:32:05.956 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_basic.ko 2024-04-17T19:32:05.956 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 41457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_bpf.ko 2024-04-17T19:32:05.956 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_cgroup.ko 2024-04-17T19:32:05.956 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 37305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_flow.ko 2024-04-17T19:32:05.956 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 39601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_fw.ko 2024-04-17T19:32:05.956 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 49321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_route.ko 2024-04-17T19:32:05.956 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 79145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_u32.ko 2024-04-17T19:32:05.956 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_cmp.ko 2024-04-17T19:32:05.956 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 15385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_ipset.ko 2024-04-17T19:32:05.956 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 48785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_meta.ko 2024-04-17T19:32:05.956 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_nbyte.ko 2024-04-17T19:32:05.956 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_text.ko 2024-04-17T19:32:05.957 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 8377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_u32.ko 2024-04-17T19:32:05.957 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_choke.ko 2024-04-17T19:32:05.957 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 25097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_codel.ko 2024-04-17T19:32:05.957 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 43481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_drr.ko 2024-04-17T19:32:05.957 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 43153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq.ko 2024-04-17T19:32:05.957 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 39177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq_codel.ko 2024-04-17T19:32:05.957 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 42889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_gred.ko 2024-04-17T19:32:05.957 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 62817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hfsc.ko 2024-04-17T19:32:05.957 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hhf.ko 2024-04-17T19:32:05.957 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 81833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_htb.ko 2024-04-17T19:32:05.957 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_ingress.ko 2024-04-17T19:32:05.957 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 36625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio.ko 2024-04-17T19:32:05.957 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio_lib.ko 2024-04-17T19:32:05.957 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_multiq.ko 2024-04-17T19:32:05.957 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 44793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_netem.ko 2024-04-17T19:32:05.958 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 32833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_pie.ko 2024-04-17T19:32:05.958 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 13505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_plug.ko 2024-04-17T19:32:05.958 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 31801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_prio.ko 2024-04-17T19:32:05.958 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 60249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_qfq.ko 2024-04-17T19:32:05.958 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 40889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_red.ko 2024-04-17T19:32:05.958 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 33433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfb.ko 2024-04-17T19:32:05.958 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 38329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfq.ko 2024-04-17T19:32:05.958 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 35777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_tbf.ko 2024-04-17T19:32:05.958 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 34161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_teql.ko 2024-04-17T19:32:05.958 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/ 2024-04-17T19:32:05.958 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1187313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp.ko 2024-04-17T19:32:05.958 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp_diag.ko 2024-04-17T19:32:05.958 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/ 2024-04-17T19:32:05.958 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/ 2024-04-17T19:32:05.959 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 432409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-17T19:32:05.959 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 70113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-17T19:32:05.959 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1940889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/sunrpc.ko 2024-04-17T19:32:05.959 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/ 2024-04-17T19:32:05.959 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 890729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-17T19:32:05.959 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/ 2024-04-17T19:32:05.959 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 1122337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc.ko 2024-04-17T19:32:05.959 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc_diag.ko 2024-04-17T19:32:05.959 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/ 2024-04-17T19:32:05.959 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 18801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/unix_diag.ko 2024-04-17T19:32:05.959 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/ 2024-04-17T19:32:05.959 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 105609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-17T19:32:05.959 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 90265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-17T19:32:05.959 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 123769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock.ko 2024-04-17T19:32:05.959 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 12905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-17T19:32:05.960 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 20401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-17T19:32:05.960 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/x25/ 2024-04-17T19:32:05.960 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 165841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/x25/x25.ko 2024-04-17T19:32:05.960 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/ 2024-04-17T19:32:05.960 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 26729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_algo.ko 2024-04-17T19:32:05.961 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-17T19:32:05.961 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 131865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_user.ko 2024-04-17T19:32:05.962 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 644769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias 2024-04-17T19:32:05.962 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 628542 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias.bin 2024-04-17T19:32:05.962 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 7706 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin 2024-04-17T19:32:05.962 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 23609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.alias.bin 2024-04-17T19:32:05.962 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 9941 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.bin 2024-04-17T19:32:05.962 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 70990 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.modinfo 2024-04-17T19:32:05.962 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 146959 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep 2024-04-17T19:32:05.962 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 232835 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep.bin 2024-04-17T19:32:05.962 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 210 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.devname 2024-04-17T19:32:05.962 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 72602 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.order 2024-04-17T19:32:05.962 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 879 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.softdep 2024-04-17T19:32:05.962 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 258450 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols 2024-04-17T19:32:05.962 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 313019 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols.bin 2024-04-17T19:32:05.962 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./usr/ 2024-04-17T19:32:05.963 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./usr/share/ 2024-04-17T19:32:05.963 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./usr/share/doc/ 2024-04-17T19:32:05.963 INFO:teuthology.orchestra.run.smithi029.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./usr/share/doc/linux-image-6.9.0-rc3-g3f60ed14f7c1/ 2024-04-17T19:32:05.963 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 174 2024-04-11 18:50 ./usr/share/doc/linux-image-6.9.0-rc3-g3f60ed14f7c1/changelog.Debian.gz 2024-04-17T19:32:05.963 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r-- root/root 691 2024-04-11 18:50 ./usr/share/doc/linux-image-6.9.0-rc3-g3f60ed14f7c1/copyright 2024-04-17T19:32:05.969 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T19:32:05.969 INFO:teuthology.task.kernel:searching for kernel 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T19:32:05.970 DEBUG:teuthology.orchestra.run.smithi029:> egrep '(submenu|menuentry.*6.9.0-rc3-g3f60ed14f7c1).*{' /boot/grub/grub.cfg 2024-04-17T19:32:05.977 INFO:teuthology.orchestra.run.smithi029.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-04-17T19:32:05.977 INFO:teuthology.orchestra.run.smithi029.stdout: menuentry 'Ubuntu, with Linux 6.9.0-rc3-g3f60ed14f7c1' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.9.0-rc3-g3f60ed14f7c1-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-04-17T19:32:05.977 INFO:teuthology.orchestra.run.smithi029.stdout: menuentry 'Ubuntu, with Linux 6.9.0-rc3-g3f60ed14f7c1 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.9.0-rc3-g3f60ed14f7c1-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-04-17T19:32:05.978 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2024-04-17T19:32:05.978 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T19:32:05.978 DEBUG:teuthology.orchestra.run.smithi029:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2024-04-17T19:32:06.020 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi029.front.sepia.ceph.com to complete... 2024-04-17T19:32:06.609 INFO:teuthology.orchestra.run.smithi029.stderr:Sourcing file `/etc/default/grub' 2024-04-17T19:32:06.618 INFO:teuthology.orchestra.run.smithi029.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-17T19:32:06.633 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2024-04-17T19:32:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:Found linux image: /boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T19:32:06.842 INFO:teuthology.orchestra.run.smithi029.stderr:Found initrd image: /boot/initrd.img-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T19:32:07.073 INFO:teuthology.orchestra.run.smithi029.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-17T19:32:07.079 INFO:teuthology.orchestra.run.smithi029.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-17T19:32:07.121 INFO:teuthology.orchestra.run.smithi029.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-17T19:32:07.126 INFO:teuthology.orchestra.run.smithi029.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-17T19:32:07.265 INFO:teuthology.orchestra.run.smithi029.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-17T19:32:07.265 INFO:teuthology.orchestra.run.smithi029.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-17T19:32:07.265 INFO:teuthology.orchestra.run.smithi029.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-17T19:32:07.291 INFO:teuthology.orchestra.run.smithi029.stderr:done 2024-04-17T19:32:38.478 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-17T19:32:38.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-04-17T19:32:38.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T19:32:56.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2024-04-17T19:33:05.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-04-17T19:33:05.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T19:34:05.998 DEBUG:teuthology.orchestra.remote:timed out 2024-04-17T19:34:17.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-04-17T19:34:18.001 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T19:34:18.424 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-04-17T19:34:19.460 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-04-17T19:34:19.460 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g3f60ed14f7c1"... 2024-04-17T19:34:19.460 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-04-17T19:34:19.512 INFO:teuthology.orchestra.run.smithi029.stdout:6.9.0-rc3-g3f60ed14f7c1 2024-04-17T19:34:19.512 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g3f60ed14f7c1 vs 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T19:34:19.512 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-17T19:34:19.512 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-17T19:34:20.513 DEBUG:teuthology.parallel:result is None 2024-04-17T19:34:20.514 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-17T19:34:20.525 INFO:teuthology.task.internal:Creating test directory... 2024-04-17T19:34:20.525 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-17T19:34:20.529 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-17T19:34:20.536 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-17T19:34:20.559 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-17T19:34:20.625 INFO:teuthology.task.internal:Creating archive directory... 2024-04-17T19:34:20.626 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-17T19:34:20.635 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-17T19:34:20.655 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-17T19:34:20.670 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-17T19:34:20.671 DEBUG:teuthology.orchestra.run.smithi037:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-17T19:34:20.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T19:34:20.699 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-17T19:34:20.703 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-17T19:34:20.795 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-17T19:34:20.810 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-17T19:34:20.827 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-17T19:34:20.876 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-17T19:34:20.880 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-17T19:34:20.892 INFO:teuthology.task.internal:Configuring sudo... 2024-04-17T19:34:20.892 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-17T19:34:20.897 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-17T19:34:20.937 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-17T19:34:20.948 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-17T19:34:20.948 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-17T19:34:20.971 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-17T19:34:20.984 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-17T19:34:21.028 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-17T19:34:21.080 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:34:21.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-17T19:34:21.152 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-17T19:34:21.161 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-17T19:34:21.211 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:34:21.211 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-17T19:34:21.310 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-04-17T19:34:21.321 DEBUG:teuthology.orchestra.run.smithi037:> sudo service rsyslog restart 2024-04-17T19:34:21.417 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-17T19:34:21.426 INFO:teuthology.task.internal:Starting timer... 2024-04-17T19:34:21.426 INFO:teuthology.run_tasks:Running task pcp... 2024-04-17T19:34:21.509 INFO:teuthology.run_tasks:Running task selinux... 2024-04-17T19:34:21.596 DEBUG:teuthology.task.selinux:Excluding smithi029: OS 'ubuntu' does not support SELinux 2024-04-17T19:34:21.596 DEBUG:teuthology.task.selinux:Excluding smithi037: OS 'ubuntu' does not support SELinux 2024-04-17T19:34:21.596 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-17T19:34:21.596 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-17T19:34:21.596 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-17T19:34:21.596 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-17T19:34:21.653 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-17T19:34:21.697 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-17T19:34:21.699 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi037.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-17T19:43:38.284 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi037.front.sepia.ceph.com')] 2024-04-17T19:43:38.285 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-04-17T19:43:38.286 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T19:43:38.369 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-04-17T19:43:38.497 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-04-17T19:43:38.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-17T19:43:38.498 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T19:43:38.580 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-04-17T19:43:38.686 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-04-17T19:43:38.686 INFO:teuthology.run_tasks:Running task clock... 2024-04-17T19:43:38.698 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-17T19:43:38.698 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-17T19:43:38.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-17T19:43:38.702 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-17T19:43:38.702 DEBUG:teuthology.orchestra.run.smithi037:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-17T19:43:38.739 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-17T19:43:38.739 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: Command line: ntpd -gq 2024-04-17T19:43:38.739 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: ---------------------------------------------------- 2024-04-17T19:43:38.739 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: ntp-4 is maintained by Network Time Foundation, 2024-04-17T19:43:38.739 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-17T19:43:38.739 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: corporation. Support and training for ntp-4 are 2024-04-17T19:43:38.739 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: available at https://www.nwtime.org/support 2024-04-17T19:43:38.739 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: ---------------------------------------------------- 2024-04-17T19:43:38.740 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: proto: precision = 0.045 usec (-24) 2024-04-17T19:43:38.740 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: basedate set to 2022-02-04 2024-04-17T19:43:38.740 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: gps base set to 2022-02-06 (week 2196) 2024-04-17T19:43:38.741 INFO:teuthology.orchestra.run.smithi029.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-17T19:43:38.741 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-17T19:43:38.741 INFO:teuthology.orchestra.run.smithi029.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-17T19:43:38.742 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: restrict ::: KOD does nothing without LIMITED. 2024-04-17T19:43:38.742 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: Listen and drop on 0 v6wildcard [::]:123 2024-04-17T19:43:38.742 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-17T19:43:38.743 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: Listen normally on 2 lo 127.0.0.1:123 2024-04-17T19:43:38.743 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: Listen normally on 3 ens1f0 172.21.15.29:123 2024-04-17T19:43:38.743 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: Listen normally on 4 lo [::1]:123 2024-04-17T19:43:38.743 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:125a%3]:123 2024-04-17T19:43:38.743 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:38 ntpd[17131]: Listening on routing socket on fd #22 for interface updates 2024-04-17T19:43:38.755 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-17T19:43:38.755 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: Command line: ntpd -gq 2024-04-17T19:43:38.755 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: ---------------------------------------------------- 2024-04-17T19:43:38.755 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: ntp-4 is maintained by Network Time Foundation, 2024-04-17T19:43:38.755 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-17T19:43:38.755 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: corporation. Support and training for ntp-4 are 2024-04-17T19:43:38.755 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: available at https://www.nwtime.org/support 2024-04-17T19:43:38.756 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: ---------------------------------------------------- 2024-04-17T19:43:38.756 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: proto: precision = 0.046 usec (-24) 2024-04-17T19:43:38.756 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: basedate set to 2022-02-04 2024-04-17T19:43:38.757 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: gps base set to 2022-02-06 (week 2196) 2024-04-17T19:43:38.757 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-17T19:43:38.757 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: restrict ::: KOD does nothing without LIMITED. 2024-04-17T19:43:38.757 INFO:teuthology.orchestra.run.smithi037.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-17T19:43:38.757 INFO:teuthology.orchestra.run.smithi037.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-17T19:43:38.758 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: Listen and drop on 0 v6wildcard [::]:123 2024-04-17T19:43:38.758 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-17T19:43:38.758 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: Listen normally on 2 lo 127.0.0.1:123 2024-04-17T19:43:38.758 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: Listen normally on 3 ens1f0 172.21.15.37:123 2024-04-17T19:43:38.758 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: Listen normally on 4 lo [::1]:123 2024-04-17T19:43:38.758 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:fe88:7794%4]:123 2024-04-17T19:43:38.758 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:38 ntpd[17726]: Listening on routing socket on fd #22 for interface updates 2024-04-17T19:43:39.739 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:39 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:39.739 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:39 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:39.740 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:39 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:39.740 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:39 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:39.740 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:39 ntpd[17131]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:39.740 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:39 ntpd[17131]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:39.756 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:39 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:39.756 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:39 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:40.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:40 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:40.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:40 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:40.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:40 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:40.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:40 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:40.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:40 ntpd[17726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:40.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:40 ntpd[17726]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:41.740 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:41 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:41.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:41 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:41.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:41 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:41.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:41 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:41.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:41 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:41.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:41 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:41.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:41 ntpd[17131]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:41.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:41 ntpd[17131]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:41.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:41 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:41.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:41 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:41.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:41 ntpd[17131]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:41.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:41 ntpd[17131]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:41.756 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:41 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:41.756 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:41 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:41.756 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:41 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:41.756 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:41 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:41.756 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:41 ntpd[17726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:41.756 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:41 ntpd[17726]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:42 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:42 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:42 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:42 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:42 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:42.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:42 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:42.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:42 ntpd[17131]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:42.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:42 ntpd[17131]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:42.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:42 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:42.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:42 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:42.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:42 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:42.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:42 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:42.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:42 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:42.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:42 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:42.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:42 ntpd[17726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:42.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:42 ntpd[17726]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:42.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:42 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:42.758 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:42 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:42.758 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:42 ntpd[17726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:42.758 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:42 ntpd[17726]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:43.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:43 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:43.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:43 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:43.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:43 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:43.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:43 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:43.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:43 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:43.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:43 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:43.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:43 ntpd[17131]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:43.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:43 ntpd[17131]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:43.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:43 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:43.742 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:43 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:43.742 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:43 ntpd[17131]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:43.742 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:43 ntpd[17131]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:43.756 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:43 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:43.756 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:43 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:43.756 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:43 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:43.756 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:43 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:43.756 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:43 ntpd[17726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:43.756 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:43 ntpd[17726]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:44.740 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:44 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:44.740 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:44 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:44.740 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:44 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:44.740 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:44 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:44.740 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:44 ntpd[17131]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:44.740 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:44 ntpd[17131]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:44.756 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:44 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:44.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:44 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:44.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:44 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:44.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:44 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:44.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:44 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:44.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:44 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:44.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:44 ntpd[17726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:44.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:44 ntpd[17726]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:44.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:44 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:44.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:44 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:44.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:44 ntpd[17726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:44.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:44 ntpd[17726]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:45.740 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:45 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:45.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:45 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:45.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:45 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:45.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:45 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:45.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:45 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:45.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:45 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:45.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:45 ntpd[17131]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:45.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:45 ntpd[17131]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:45.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:45 ntpd[17131]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:45.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:45 ntpd[17131]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:45.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:45 ntpd[17131]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:45.741 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:45 ntpd[17131]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:45.742 INFO:teuthology.orchestra.run.smithi029.stdout:17 Apr 19:43:45 ntpd[17131]: ntpd: time slew -0.004060 s 2024-04-17T19:43:45.742 INFO:teuthology.orchestra.run.smithi029.stdout:ntpd: time slew -0.004060s 2024-04-17T19:43:45.742 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:45 ntpd[17131]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-17T19:43:45.742 INFO:teuthology.orchestra.run.smithi029.stderr:17 Apr 19:43:45 ntpd[17131]: can't open /var/log/ntpstats/loopstats.20240417: Permission denied 2024-04-17T19:43:45.758 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:45 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:45.759 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:45 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:45.759 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:45 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:45.759 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:45 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:45.759 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:45 ntpd[17726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:45.759 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:45 ntpd[17726]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:45.855 INFO:teuthology.orchestra.run.smithi029.stdout: remote refid st t when poll reach delay offset jitter 2024-04-17T19:43:45.855 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================== 2024-04-17T19:43:45.855 INFO:teuthology.orchestra.run.smithi029.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-17T19:43:45.855 INFO:teuthology.orchestra.run.smithi029.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-17T19:43:45.855 INFO:teuthology.orchestra.run.smithi029.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-17T19:43:45.855 INFO:teuthology.orchestra.run.smithi029.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-17T19:43:46.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:46 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:46.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:46 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:46.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:46 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:46.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:46 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:46.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:46 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:46.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:46 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:46.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:46 ntpd[17726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:46.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:46 ntpd[17726]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:46.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:46 ntpd[17726]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T19:43:46.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:46 ntpd[17726]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T19:43:46.757 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:46 ntpd[17726]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T19:43:46.758 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:46 ntpd[17726]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T19:43:46.758 INFO:teuthology.orchestra.run.smithi037.stdout:17 Apr 19:43:46 ntpd[17726]: ntpd: time slew -0.000573 s 2024-04-17T19:43:46.758 INFO:teuthology.orchestra.run.smithi037.stdout:ntpd: time slew -0.000573s 2024-04-17T19:43:46.758 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:46 ntpd[17726]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-17T19:43:46.758 INFO:teuthology.orchestra.run.smithi037.stderr:17 Apr 19:43:46 ntpd[17726]: can't open /var/log/ntpstats/loopstats.20240417: Permission denied 2024-04-17T19:43:46.831 INFO:teuthology.orchestra.run.smithi037.stdout: remote refid st t when poll reach delay offset jitter 2024-04-17T19:43:46.831 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================== 2024-04-17T19:43:46.831 INFO:teuthology.orchestra.run.smithi037.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-17T19:43:46.831 INFO:teuthology.orchestra.run.smithi037.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-17T19:43:46.831 INFO:teuthology.orchestra.run.smithi037.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-17T19:43:46.831 INFO:teuthology.orchestra.run.smithi037.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-17T19:43:46.832 INFO:teuthology.run_tasks:Running task install... 2024-04-17T19:43:46.841 DEBUG:teuthology.task.install:project ceph 2024-04-17T19:43:46.841 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f3a98d3f110319c567ec86bf70955c141495ccf6'}} 2024-04-17T19:43:46.842 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'f3a98d3f110319c567ec86bf70955c141495ccf6'} 2024-04-17T19:43:46.842 INFO:teuthology.task.install:Using flavor: default 2024-04-17T19:43:46.847 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-17T19:43:46.847 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-17T19:43:46.848 DEBUG:teuthology.orchestra.run.smithi029:> sudo apt-key list | grep Ceph 2024-04-17T19:43:46.848 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-key list | grep Ceph 2024-04-17T19:43:46.935 INFO:teuthology.orchestra.run.smithi037.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-17T19:43:46.955 INFO:teuthology.orchestra.run.smithi029.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-17T19:43:46.965 INFO:teuthology.orchestra.run.smithi037.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-17T19:43:46.965 INFO:teuthology.orchestra.run.smithi037.stdout:uid [ unknown] Ceph.com (release key) 2024-04-17T19:43:46.966 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-04-17T19:43:46.966 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-04-17T19:43:46.966 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f3a98d3f110319c567ec86bf70955c141495ccf6 2024-04-17T19:43:47.008 INFO:teuthology.orchestra.run.smithi029.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-17T19:43:47.008 INFO:teuthology.orchestra.run.smithi029.stdout:uid [ unknown] Ceph.com (release key) 2024-04-17T19:43:47.009 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-04-17T19:43:47.009 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-04-17T19:43:47.009 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f3a98d3f110319c567ec86bf70955c141495ccf6 2024-04-17T19:43:47.115 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default/ 2024-04-17T19:43:47.115 INFO:teuthology.task.install.deb:Package version is 19.0.0-3146-gf3a98d3f-1jammy 2024-04-17T19:43:47.227 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:43:47.227 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-17T19:43:47.242 DEBUG:teuthology.orchestra.run.smithi037:> sudo apt-get update 2024-04-17T19:43:47.291 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default/ 2024-04-17T19:43:47.291 INFO:teuthology.task.install.deb:Package version is 19.0.0-3146-gf3a98d3f-1jammy 2024-04-17T19:43:47.400 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:43:47.400 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-17T19:43:47.420 DEBUG:teuthology.orchestra.run.smithi029:> sudo apt-get update 2024-04-17T19:43:47.441 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-17T19:43:47.462 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-17T19:43:47.467 INFO:teuthology.orchestra.run.smithi037.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy InRelease 2024-04-17T19:43:47.489 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-17T19:43:47.499 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-17T19:43:47.510 INFO:teuthology.orchestra.run.smithi037.stdout:Ign:6 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-17T19:43:47.532 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-17T19:43:47.559 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,418 B] 2024-04-17T19:43:47.574 INFO:teuthology.orchestra.run.smithi037.stdout:Hit:9 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-17T19:43:47.622 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-17T19:43:47.631 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-17T19:43:47.651 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-17T19:43:47.719 INFO:teuthology.orchestra.run.smithi029.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy InRelease 2024-04-17T19:43:47.726 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-17T19:43:47.758 INFO:teuthology.orchestra.run.smithi029.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-17T19:43:47.808 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 29.2 kB in 0s (76.4 kB/s) 2024-04-17T19:43:47.813 INFO:teuthology.orchestra.run.smithi029.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-17T19:43:47.876 INFO:teuthology.orchestra.run.smithi029.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,418 B] 2024-04-17T19:43:47.948 INFO:teuthology.orchestra.run.smithi029.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-04-17T19:43:48.239 INFO:teuthology.orchestra.run.smithi029.stdout:Fetched 29.2 kB in 1s (49.5 kB/s) 2024-04-17T19:43:48.796 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-04-17T19:43:48.812 DEBUG:teuthology.orchestra.run.smithi037:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3146-gf3a98d3f-1jammy cephadm=19.0.0-3146-gf3a98d3f-1jammy ceph-mds=19.0.0-3146-gf3a98d3f-1jammy ceph-mgr=19.0.0-3146-gf3a98d3f-1jammy ceph-common=19.0.0-3146-gf3a98d3f-1jammy ceph-fuse=19.0.0-3146-gf3a98d3f-1jammy ceph-test=19.0.0-3146-gf3a98d3f-1jammy ceph-volume=19.0.0-3146-gf3a98d3f-1jammy radosgw=19.0.0-3146-gf3a98d3f-1jammy python3-rados=19.0.0-3146-gf3a98d3f-1jammy python3-rgw=19.0.0-3146-gf3a98d3f-1jammy python3-cephfs=19.0.0-3146-gf3a98d3f-1jammy python3-rbd=19.0.0-3146-gf3a98d3f-1jammy libcephfs2=19.0.0-3146-gf3a98d3f-1jammy libcephfs-dev=19.0.0-3146-gf3a98d3f-1jammy librados2=19.0.0-3146-gf3a98d3f-1jammy librbd1=19.0.0-3146-gf3a98d3f-1jammy rbd-fuse=19.0.0-3146-gf3a98d3f-1jammy python3-cephfs=19.0.0-3146-gf3a98d3f-1jammy cephfs-shell=19.0.0-3146-gf3a98d3f-1jammy cephfs-top=19.0.0-3146-gf3a98d3f-1jammy cephfs-mirror=19.0.0-3146-gf3a98d3f-1jammy 2024-04-17T19:43:48.865 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-04-17T19:43:49.043 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2024-04-17T19:43:49.044 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2024-04-17T19:43:49.221 INFO:teuthology.orchestra.run.smithi037.stdout:The following packages were automatically installed and are no longer required: 2024-04-17T19:43:49.221 INFO:teuthology.orchestra.run.smithi037.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-17T19:43:49.222 INFO:teuthology.orchestra.run.smithi037.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-17T19:43:49.222 INFO:teuthology.orchestra.run.smithi037.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-17T19:43:49.222 INFO:teuthology.orchestra.run.smithi037.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-17T19:43:49.223 INFO:teuthology.orchestra.run.smithi037.stdout: python2.7-minimal 2024-04-17T19:43:49.223 INFO:teuthology.orchestra.run.smithi037.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-17T19:43:49.223 INFO:teuthology.orchestra.run.smithi037.stdout:The following additional packages will be installed: 2024-04-17T19:43:49.223 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-17T19:43:49.223 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-17T19:43:49.224 INFO:teuthology.orchestra.run.smithi037.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-17T19:43:49.224 INFO:teuthology.orchestra.run.smithi037.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-17T19:43:49.224 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-17T19:43:49.224 INFO:teuthology.orchestra.run.smithi037.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-17T19:43:49.224 INFO:teuthology.orchestra.run.smithi037.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-17T19:43:49.225 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-17T19:43:49.225 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-17T19:43:49.225 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-04-17T19:43:49.225 INFO:teuthology.orchestra.run.smithi037.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-17T19:43:49.225 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-17T19:43:49.225 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-17T19:43:49.225 INFO:teuthology.orchestra.run.smithi037.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-17T19:43:49.225 INFO:teuthology.orchestra.run.smithi037.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-17T19:43:49.225 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-17T19:43:49.225 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-17T19:43:49.225 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-04-17T19:43:49.225 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-04-17T19:43:49.225 INFO:teuthology.orchestra.run.smithi037.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-04-17T19:43:49.225 INFO:teuthology.orchestra.run.smithi037.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-04-17T19:43:49.225 INFO:teuthology.orchestra.run.smithi037.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-04-17T19:43:49.226 INFO:teuthology.orchestra.run.smithi037.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-04-17T19:43:49.226 INFO:teuthology.orchestra.run.smithi037.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-04-17T19:43:49.226 INFO:teuthology.orchestra.run.smithi037.stdout:Suggested packages: 2024-04-17T19:43:49.226 INFO:teuthology.orchestra.run.smithi037.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-17T19:43:49.226 INFO:teuthology.orchestra.run.smithi037.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-17T19:43:49.226 INFO:teuthology.orchestra.run.smithi037.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-17T19:43:49.226 INFO:teuthology.orchestra.run.smithi037.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-17T19:43:49.226 INFO:teuthology.orchestra.run.smithi037.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-17T19:43:49.226 INFO:teuthology.orchestra.run.smithi037.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-17T19:43:49.226 INFO:teuthology.orchestra.run.smithi037.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-17T19:43:49.227 INFO:teuthology.orchestra.run.smithi037.stdout:Recommended packages: 2024-04-17T19:43:49.227 INFO:teuthology.orchestra.run.smithi037.stdout: btrfs-tools 2024-04-17T19:43:49.244 INFO:teuthology.orchestra.run.smithi029.stdout:Reading package lists... 2024-04-17T19:43:49.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3146-gf3a98d3f-1jammy cephadm=19.0.0-3146-gf3a98d3f-1jammy ceph-mds=19.0.0-3146-gf3a98d3f-1jammy ceph-mgr=19.0.0-3146-gf3a98d3f-1jammy ceph-common=19.0.0-3146-gf3a98d3f-1jammy ceph-fuse=19.0.0-3146-gf3a98d3f-1jammy ceph-test=19.0.0-3146-gf3a98d3f-1jammy ceph-volume=19.0.0-3146-gf3a98d3f-1jammy radosgw=19.0.0-3146-gf3a98d3f-1jammy python3-rados=19.0.0-3146-gf3a98d3f-1jammy python3-rgw=19.0.0-3146-gf3a98d3f-1jammy python3-cephfs=19.0.0-3146-gf3a98d3f-1jammy python3-rbd=19.0.0-3146-gf3a98d3f-1jammy libcephfs2=19.0.0-3146-gf3a98d3f-1jammy libcephfs-dev=19.0.0-3146-gf3a98d3f-1jammy librados2=19.0.0-3146-gf3a98d3f-1jammy librbd1=19.0.0-3146-gf3a98d3f-1jammy rbd-fuse=19.0.0-3146-gf3a98d3f-1jammy python3-cephfs=19.0.0-3146-gf3a98d3f-1jammy cephfs-shell=19.0.0-3146-gf3a98d3f-1jammy cephfs-top=19.0.0-3146-gf3a98d3f-1jammy cephfs-mirror=19.0.0-3146-gf3a98d3f-1jammy 2024-04-17T19:43:49.335 INFO:teuthology.orchestra.run.smithi037.stdout:The following NEW packages will be installed: 2024-04-17T19:43:49.335 INFO:teuthology.orchestra.run.smithi037.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-17T19:43:49.335 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-17T19:43:49.335 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-04-17T19:43:49.335 INFO:teuthology.orchestra.run.smithi037.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-04-17T19:43:49.337 INFO:teuthology.orchestra.run.smithi037.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-17T19:43:49.337 INFO:teuthology.orchestra.run.smithi037.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-04-17T19:43:49.337 INFO:teuthology.orchestra.run.smithi037.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-04-17T19:43:49.337 INFO:teuthology.orchestra.run.smithi037.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-04-17T19:43:49.337 INFO:teuthology.orchestra.run.smithi037.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-04-17T19:43:49.337 INFO:teuthology.orchestra.run.smithi037.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-17T19:43:49.337 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-17T19:43:49.337 INFO:teuthology.orchestra.run.smithi037.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-04-17T19:43:49.337 INFO:teuthology.orchestra.run.smithi037.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-04-17T19:43:49.337 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-17T19:43:49.337 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-17T19:43:49.337 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-17T19:43:49.337 INFO:teuthology.orchestra.run.smithi037.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-17T19:43:49.337 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-17T19:43:49.338 INFO:teuthology.orchestra.run.smithi037.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-17T19:43:49.338 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-04-17T19:43:49.338 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-17T19:43:49.338 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-17T19:43:49.338 INFO:teuthology.orchestra.run.smithi037.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-17T19:43:49.338 INFO:teuthology.orchestra.run.smithi037.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-17T19:43:49.338 INFO:teuthology.orchestra.run.smithi037.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-17T19:43:49.338 INFO:teuthology.orchestra.run.smithi037.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-17T19:43:49.338 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-17T19:43:49.338 INFO:teuthology.orchestra.run.smithi037.stdout: socat unzip xmlstarlet zip 2024-04-17T19:43:49.338 INFO:teuthology.orchestra.run.smithi029.stdout:Reading package lists... 2024-04-17T19:43:49.512 INFO:teuthology.orchestra.run.smithi029.stdout:Building dependency tree... 2024-04-17T19:43:49.513 INFO:teuthology.orchestra.run.smithi029.stdout:Reading state information... 2024-04-17T19:43:49.556 INFO:teuthology.orchestra.run.smithi037.stdout:0 upgraded, 124 newly installed, 0 to remove and 305 not upgraded. 2024-04-17T19:43:49.556 INFO:teuthology.orchestra.run.smithi037.stdout:Need to get 175 MB of archives. 2024-04-17T19:43:49.556 INFO:teuthology.orchestra.run.smithi037.stdout:After this operation, 887 MB of additional disk space will be used. 2024-04-17T19:43:49.556 INFO:teuthology.orchestra.run.smithi037.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-04-17T19:43:49.560 INFO:teuthology.orchestra.run.smithi037.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3146-gf3a98d3f-1jammy [3,523 kB] 2024-04-17T19:43:49.638 INFO:teuthology.orchestra.run.smithi037.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-17T19:43:49.684 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-04-17T19:43:49.690 INFO:teuthology.orchestra.run.smithi029.stdout:The following packages were automatically installed and are no longer required: 2024-04-17T19:43:49.690 INFO:teuthology.orchestra.run.smithi029.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-17T19:43:49.690 INFO:teuthology.orchestra.run.smithi029.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-17T19:43:49.691 INFO:teuthology.orchestra.run.smithi029.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-17T19:43:49.691 INFO:teuthology.orchestra.run.smithi029.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-17T19:43:49.691 INFO:teuthology.orchestra.run.smithi029.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-17T19:43:49.691 INFO:teuthology.orchestra.run.smithi029.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-17T19:43:49.692 INFO:teuthology.orchestra.run.smithi029.stdout:The following additional packages will be installed: 2024-04-17T19:43:49.692 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-17T19:43:49.692 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-17T19:43:49.692 INFO:teuthology.orchestra.run.smithi029.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-17T19:43:49.692 INFO:teuthology.orchestra.run.smithi029.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-17T19:43:49.694 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-17T19:43:49.694 INFO:teuthology.orchestra.run.smithi029.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-17T19:43:49.694 INFO:teuthology.orchestra.run.smithi029.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-17T19:43:49.694 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-17T19:43:49.694 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-17T19:43:49.694 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-04-17T19:43:49.694 INFO:teuthology.orchestra.run.smithi029.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-17T19:43:49.694 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-17T19:43:49.694 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-17T19:43:49.694 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-17T19:43:49.695 INFO:teuthology.orchestra.run.smithi029.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-17T19:43:49.695 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-17T19:43:49.695 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-17T19:43:49.695 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-04-17T19:43:49.695 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-04-17T19:43:49.695 INFO:teuthology.orchestra.run.smithi029.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-04-17T19:43:49.695 INFO:teuthology.orchestra.run.smithi029.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-04-17T19:43:49.695 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-04-17T19:43:49.695 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-04-17T19:43:49.695 INFO:teuthology.orchestra.run.smithi029.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-04-17T19:43:49.696 INFO:teuthology.orchestra.run.smithi029.stdout:Suggested packages: 2024-04-17T19:43:49.696 INFO:teuthology.orchestra.run.smithi029.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-17T19:43:49.696 INFO:teuthology.orchestra.run.smithi029.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-17T19:43:49.696 INFO:teuthology.orchestra.run.smithi029.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-17T19:43:49.696 INFO:teuthology.orchestra.run.smithi029.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-17T19:43:49.696 INFO:teuthology.orchestra.run.smithi029.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-17T19:43:49.696 INFO:teuthology.orchestra.run.smithi029.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-17T19:43:49.696 INFO:teuthology.orchestra.run.smithi029.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-17T19:43:49.696 INFO:teuthology.orchestra.run.smithi029.stdout:Recommended packages: 2024-04-17T19:43:49.696 INFO:teuthology.orchestra.run.smithi029.stdout: btrfs-tools 2024-04-17T19:43:49.852 INFO:teuthology.orchestra.run.smithi029.stdout:The following NEW packages will be installed: 2024-04-17T19:43:49.853 INFO:teuthology.orchestra.run.smithi029.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-17T19:43:49.853 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-17T19:43:49.853 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-04-17T19:43:49.853 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-04-17T19:43:49.853 INFO:teuthology.orchestra.run.smithi029.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-17T19:43:49.853 INFO:teuthology.orchestra.run.smithi029.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-04-17T19:43:49.853 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-04-17T19:43:49.853 INFO:teuthology.orchestra.run.smithi029.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-04-17T19:43:49.854 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-04-17T19:43:49.855 INFO:teuthology.orchestra.run.smithi029.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-17T19:43:49.855 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-17T19:43:49.855 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-04-17T19:43:49.855 INFO:teuthology.orchestra.run.smithi029.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-04-17T19:43:49.855 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-17T19:43:49.855 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-17T19:43:49.855 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-17T19:43:49.855 INFO:teuthology.orchestra.run.smithi029.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-17T19:43:49.855 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-17T19:43:49.855 INFO:teuthology.orchestra.run.smithi029.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-17T19:43:49.855 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-04-17T19:43:49.855 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-17T19:43:49.855 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-17T19:43:49.855 INFO:teuthology.orchestra.run.smithi029.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-17T19:43:49.855 INFO:teuthology.orchestra.run.smithi029.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-17T19:43:49.856 INFO:teuthology.orchestra.run.smithi029.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-17T19:43:49.856 INFO:teuthology.orchestra.run.smithi029.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-17T19:43:49.856 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-17T19:43:49.856 INFO:teuthology.orchestra.run.smithi029.stdout: socat unzip xmlstarlet zip 2024-04-17T19:43:49.886 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-17T19:43:49.929 INFO:teuthology.orchestra.run.smithi029.stdout:0 upgraded, 124 newly installed, 0 to remove and 305 not upgraded. 2024-04-17T19:43:49.929 INFO:teuthology.orchestra.run.smithi029.stdout:Need to get 175 MB of archives. 2024-04-17T19:43:49.930 INFO:teuthology.orchestra.run.smithi029.stdout:After this operation, 887 MB of additional disk space will be used. 2024-04-17T19:43:49.930 INFO:teuthology.orchestra.run.smithi029.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-04-17T19:43:49.951 INFO:teuthology.orchestra.run.smithi029.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-17T19:43:49.963 INFO:teuthology.orchestra.run.smithi029.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-04-17T19:43:49.981 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-17T19:43:49.994 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-17T19:43:50.002 INFO:teuthology.orchestra.run.smithi029.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3146-gf3a98d3f-1jammy [3,523 kB] 2024-04-17T19:43:50.012 INFO:teuthology.orchestra.run.smithi029.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-17T19:43:50.037 INFO:teuthology.orchestra.run.smithi029.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-17T19:43:50.043 INFO:teuthology.orchestra.run.smithi029.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-17T19:43:50.046 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-17T19:43:50.055 INFO:teuthology.orchestra.run.smithi029.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-17T19:43:50.068 INFO:teuthology.orchestra.run.smithi037.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3146-gf3a98d3f-1jammy [3,286 kB] 2024-04-17T19:43:50.108 INFO:teuthology.orchestra.run.smithi029.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-17T19:43:50.111 INFO:teuthology.orchestra.run.smithi029.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-17T19:43:50.121 INFO:teuthology.orchestra.run.smithi029.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-17T19:43:50.125 INFO:teuthology.orchestra.run.smithi029.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-17T19:43:50.125 INFO:teuthology.orchestra.run.smithi029.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-17T19:43:50.133 INFO:teuthology.orchestra.run.smithi029.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-17T19:43:50.135 INFO:teuthology.orchestra.run.smithi029.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-17T19:43:50.146 INFO:teuthology.orchestra.run.smithi029.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-17T19:43:50.146 INFO:teuthology.orchestra.run.smithi029.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-17T19:43:50.146 INFO:teuthology.orchestra.run.smithi029.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-17T19:43:50.166 INFO:teuthology.orchestra.run.smithi029.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-17T19:43:50.168 INFO:teuthology.orchestra.run.smithi029.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-17T19:43:50.170 INFO:teuthology.orchestra.run.smithi029.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-17T19:43:50.170 INFO:teuthology.orchestra.run.smithi029.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-17T19:43:50.171 INFO:teuthology.orchestra.run.smithi029.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-17T19:43:50.172 INFO:teuthology.orchestra.run.smithi029.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-17T19:43:50.186 INFO:teuthology.orchestra.run.smithi029.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-17T19:43:50.186 INFO:teuthology.orchestra.run.smithi029.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-17T19:43:50.187 INFO:teuthology.orchestra.run.smithi029.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-17T19:43:50.187 INFO:teuthology.orchestra.run.smithi029.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-17T19:43:50.206 INFO:teuthology.orchestra.run.smithi029.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-17T19:43:50.207 INFO:teuthology.orchestra.run.smithi029.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-17T19:43:50.210 INFO:teuthology.orchestra.run.smithi029.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-17T19:43:50.210 INFO:teuthology.orchestra.run.smithi029.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-17T19:43:50.211 INFO:teuthology.orchestra.run.smithi029.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-17T19:43:50.211 INFO:teuthology.orchestra.run.smithi029.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-17T19:43:50.220 INFO:teuthology.orchestra.run.smithi037.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-17T19:43:50.229 INFO:teuthology.orchestra.run.smithi037.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-17T19:43:50.239 INFO:teuthology.orchestra.run.smithi029.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-17T19:43:50.240 INFO:teuthology.orchestra.run.smithi029.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-17T19:43:50.240 INFO:teuthology.orchestra.run.smithi029.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-17T19:43:50.242 INFO:teuthology.orchestra.run.smithi029.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-17T19:43:50.260 INFO:teuthology.orchestra.run.smithi029.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-17T19:43:50.260 INFO:teuthology.orchestra.run.smithi029.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-17T19:43:50.261 INFO:teuthology.orchestra.run.smithi037.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-17T19:43:50.266 INFO:teuthology.orchestra.run.smithi029.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-17T19:43:50.266 INFO:teuthology.orchestra.run.smithi029.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-17T19:43:50.267 INFO:teuthology.orchestra.run.smithi029.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-17T19:43:50.267 INFO:teuthology.orchestra.run.smithi029.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-17T19:43:50.269 INFO:teuthology.orchestra.run.smithi037.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-17T19:43:50.281 INFO:teuthology.orchestra.run.smithi029.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-17T19:43:50.283 INFO:teuthology.orchestra.run.smithi029.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-17T19:43:50.284 INFO:teuthology.orchestra.run.smithi029.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-17T19:43:50.285 INFO:teuthology.orchestra.run.smithi029.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-17T19:43:50.302 INFO:teuthology.orchestra.run.smithi029.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-17T19:43:50.302 INFO:teuthology.orchestra.run.smithi029.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3146-gf3a98d3f-1jammy [3,286 kB] 2024-04-17T19:43:50.304 INFO:teuthology.orchestra.run.smithi029.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-17T19:43:50.304 INFO:teuthology.orchestra.run.smithi029.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-17T19:43:50.305 INFO:teuthology.orchestra.run.smithi029.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-17T19:43:50.308 INFO:teuthology.orchestra.run.smithi029.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-17T19:43:50.331 INFO:teuthology.orchestra.run.smithi037.stdout:Get:14 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3146-gf3a98d3f-1jammy [726 kB] 2024-04-17T19:43:50.386 INFO:teuthology.orchestra.run.smithi029.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-17T19:43:50.387 INFO:teuthology.orchestra.run.smithi029.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-17T19:43:50.389 INFO:teuthology.orchestra.run.smithi029.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-17T19:43:50.389 INFO:teuthology.orchestra.run.smithi029.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-17T19:43:50.391 INFO:teuthology.orchestra.run.smithi029.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-17T19:43:50.404 INFO:teuthology.orchestra.run.smithi037.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3146-gf3a98d3f-1jammy [357 kB] 2024-04-17T19:43:50.418 INFO:teuthology.orchestra.run.smithi029.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-17T19:43:50.420 INFO:teuthology.orchestra.run.smithi029.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-17T19:43:50.420 INFO:teuthology.orchestra.run.smithi029.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-17T19:43:50.434 INFO:teuthology.orchestra.run.smithi037.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-17T19:43:50.443 INFO:teuthology.orchestra.run.smithi029.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-17T19:43:50.444 INFO:teuthology.orchestra.run.smithi029.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-17T19:43:50.444 INFO:teuthology.orchestra.run.smithi037.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3146-gf3a98d3f-1jammy [32.8 kB] 2024-04-17T19:43:50.445 INFO:teuthology.orchestra.run.smithi029.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-17T19:43:50.445 INFO:teuthology.orchestra.run.smithi029.stdout:Get:66 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3146-gf3a98d3f-1jammy [726 kB] 2024-04-17T19:43:50.445 INFO:teuthology.orchestra.run.smithi029.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-17T19:43:50.446 INFO:teuthology.orchestra.run.smithi037.stdout:Get:18 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3146-gf3a98d3f-1jammy [180 kB] 2024-04-17T19:43:50.446 INFO:teuthology.orchestra.run.smithi029.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-17T19:43:50.446 INFO:teuthology.orchestra.run.smithi029.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-17T19:43:50.450 INFO:teuthology.orchestra.run.smithi037.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3146-gf3a98d3f-1jammy [66.9 kB] 2024-04-17T19:43:50.452 INFO:teuthology.orchestra.run.smithi037.stdout:Get:20 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3146-gf3a98d3f-1jammy [330 kB] 2024-04-17T19:43:50.462 INFO:teuthology.orchestra.run.smithi029.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-17T19:43:50.463 INFO:teuthology.orchestra.run.smithi029.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-17T19:43:50.467 INFO:teuthology.orchestra.run.smithi029.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-17T19:43:50.483 INFO:teuthology.orchestra.run.smithi029.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-17T19:43:50.484 INFO:teuthology.orchestra.run.smithi029.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-17T19:43:50.485 INFO:teuthology.orchestra.run.smithi029.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-17T19:43:50.487 INFO:teuthology.orchestra.run.smithi037.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3146-gf3a98d3f-1jammy [7,186 kB] 2024-04-17T19:43:50.489 INFO:teuthology.orchestra.run.smithi029.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3146-gf3a98d3f-1jammy [357 kB] 2024-04-17T19:43:50.490 INFO:teuthology.orchestra.run.smithi029.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-17T19:43:50.493 INFO:teuthology.orchestra.run.smithi029.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-04-17T19:43:50.493 INFO:teuthology.orchestra.run.smithi029.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-17T19:43:50.494 INFO:teuthology.orchestra.run.smithi029.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-17T19:43:50.503 INFO:teuthology.orchestra.run.smithi029.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-17T19:43:50.508 INFO:teuthology.orchestra.run.smithi029.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3146-gf3a98d3f-1jammy [32.8 kB] 2024-04-17T19:43:50.509 INFO:teuthology.orchestra.run.smithi029.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-17T19:43:50.510 INFO:teuthology.orchestra.run.smithi029.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3146-gf3a98d3f-1jammy [180 kB] 2024-04-17T19:43:50.510 INFO:teuthology.orchestra.run.smithi029.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-17T19:43:50.519 INFO:teuthology.orchestra.run.smithi029.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3146-gf3a98d3f-1jammy [66.9 kB] 2024-04-17T19:43:50.522 INFO:teuthology.orchestra.run.smithi029.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3146-gf3a98d3f-1jammy [330 kB] 2024-04-17T19:43:50.532 INFO:teuthology.orchestra.run.smithi029.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-17T19:43:50.532 INFO:teuthology.orchestra.run.smithi029.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-17T19:43:50.533 INFO:teuthology.orchestra.run.smithi029.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-17T19:43:50.533 INFO:teuthology.orchestra.run.smithi029.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-17T19:43:50.535 INFO:teuthology.orchestra.run.smithi029.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-17T19:43:50.537 INFO:teuthology.orchestra.run.smithi029.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-17T19:43:50.544 INFO:teuthology.orchestra.run.smithi029.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3146-gf3a98d3f-1jammy [7,186 kB] 2024-04-17T19:43:50.544 INFO:teuthology.orchestra.run.smithi029.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-17T19:43:50.552 INFO:teuthology.orchestra.run.smithi029.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-17T19:43:50.553 INFO:teuthology.orchestra.run.smithi029.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-17T19:43:50.556 INFO:teuthology.orchestra.run.smithi029.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-17T19:43:50.573 INFO:teuthology.orchestra.run.smithi029.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-17T19:43:50.954 INFO:teuthology.orchestra.run.smithi029.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3146-gf3a98d3f-1jammy [112 kB] 2024-04-17T19:43:50.958 INFO:teuthology.orchestra.run.smithi029.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3146-gf3a98d3f-1jammy [471 kB] 2024-04-17T19:43:50.986 INFO:teuthology.orchestra.run.smithi029.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3146-gf3a98d3f-1jammy [25.6 MB] 2024-04-17T19:43:51.006 INFO:teuthology.orchestra.run.smithi037.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-17T19:43:51.021 INFO:teuthology.orchestra.run.smithi037.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-17T19:43:51.037 INFO:teuthology.orchestra.run.smithi037.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-17T19:43:51.038 INFO:teuthology.orchestra.run.smithi037.stdout:Get:25 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3146-gf3a98d3f-1jammy [112 kB] 2024-04-17T19:43:51.044 INFO:teuthology.orchestra.run.smithi037.stdout:Get:26 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3146-gf3a98d3f-1jammy [471 kB] 2024-04-17T19:43:51.046 INFO:teuthology.orchestra.run.smithi037.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-17T19:43:51.046 INFO:teuthology.orchestra.run.smithi037.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-17T19:43:51.062 INFO:teuthology.orchestra.run.smithi037.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-17T19:43:51.083 INFO:teuthology.orchestra.run.smithi037.stdout:Get:30 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3146-gf3a98d3f-1jammy [25.6 MB] 2024-04-17T19:43:51.100 INFO:teuthology.orchestra.run.smithi037.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-17T19:43:51.114 INFO:teuthology.orchestra.run.smithi037.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-17T19:43:51.118 INFO:teuthology.orchestra.run.smithi037.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-17T19:43:51.118 INFO:teuthology.orchestra.run.smithi037.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-17T19:43:51.290 INFO:teuthology.orchestra.run.smithi037.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-17T19:43:51.373 INFO:teuthology.orchestra.run.smithi037.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-17T19:43:51.380 INFO:teuthology.orchestra.run.smithi037.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-17T19:43:51.454 INFO:teuthology.orchestra.run.smithi037.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-17T19:43:51.480 INFO:teuthology.orchestra.run.smithi037.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-17T19:43:51.496 INFO:teuthology.orchestra.run.smithi037.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-17T19:43:51.513 INFO:teuthology.orchestra.run.smithi037.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-17T19:43:51.704 INFO:teuthology.orchestra.run.smithi037.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-17T19:43:51.720 INFO:teuthology.orchestra.run.smithi037.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-17T19:43:51.728 INFO:teuthology.orchestra.run.smithi037.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-17T19:43:51.732 INFO:teuthology.orchestra.run.smithi037.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-17T19:43:51.917 INFO:teuthology.orchestra.run.smithi037.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-17T19:43:52.001 INFO:teuthology.orchestra.run.smithi037.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-17T19:43:52.016 INFO:teuthology.orchestra.run.smithi037.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-17T19:43:52.203 INFO:teuthology.orchestra.run.smithi037.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-17T19:43:52.254 INFO:teuthology.orchestra.run.smithi037.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-17T19:43:52.268 INFO:teuthology.orchestra.run.smithi037.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-17T19:43:52.422 INFO:teuthology.orchestra.run.smithi029.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3146-gf3a98d3f-1jammy [4,916 kB] 2024-04-17T19:43:52.426 INFO:teuthology.orchestra.run.smithi037.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-17T19:43:52.427 INFO:teuthology.orchestra.run.smithi037.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-17T19:43:52.432 INFO:teuthology.orchestra.run.smithi037.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-17T19:43:52.437 INFO:teuthology.orchestra.run.smithi037.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-17T19:43:52.451 INFO:teuthology.orchestra.run.smithi037.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-17T19:43:52.636 INFO:teuthology.orchestra.run.smithi037.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-17T19:43:52.692 INFO:teuthology.orchestra.run.smithi029.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3146-gf3a98d3f-1jammy [244 kB] 2024-04-17T19:43:52.705 INFO:teuthology.orchestra.run.smithi029.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3146-gf3a98d3f-1jammy [125 kB] 2024-04-17T19:43:52.712 INFO:teuthology.orchestra.run.smithi029.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3146-gf3a98d3f-1jammy [1,519 kB] 2024-04-17T19:43:52.798 INFO:teuthology.orchestra.run.smithi029.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3146-gf3a98d3f-1jammy [6,148 kB] 2024-04-17T19:43:52.893 INFO:teuthology.orchestra.run.smithi037.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-17T19:43:53.007 INFO:teuthology.orchestra.run.smithi037.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-17T19:43:53.050 INFO:teuthology.orchestra.run.smithi037.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-17T19:43:53.144 INFO:teuthology.orchestra.run.smithi029.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3146-gf3a98d3f-1jammy [22.5 MB] 2024-04-17T19:43:53.190 INFO:teuthology.orchestra.run.smithi037.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-17T19:43:53.192 INFO:teuthology.orchestra.run.smithi037.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-17T19:43:53.232 INFO:teuthology.orchestra.run.smithi037.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-17T19:43:53.233 INFO:teuthology.orchestra.run.smithi037.stdout:Get:64 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3146-gf3a98d3f-1jammy [4,916 kB] 2024-04-17T19:43:53.237 INFO:teuthology.orchestra.run.smithi037.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-17T19:43:53.497 INFO:teuthology.orchestra.run.smithi037.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-17T19:43:53.499 INFO:teuthology.orchestra.run.smithi037.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-17T19:43:53.502 INFO:teuthology.orchestra.run.smithi037.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-17T19:43:53.504 INFO:teuthology.orchestra.run.smithi037.stdout:Get:69 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3146-gf3a98d3f-1jammy [244 kB] 2024-04-17T19:43:53.531 INFO:teuthology.orchestra.run.smithi037.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3146-gf3a98d3f-1jammy [125 kB] 2024-04-17T19:43:53.538 INFO:teuthology.orchestra.run.smithi037.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3146-gf3a98d3f-1jammy [1,519 kB] 2024-04-17T19:43:53.650 INFO:teuthology.orchestra.run.smithi037.stdout:Get:72 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3146-gf3a98d3f-1jammy [6,148 kB] 2024-04-17T19:43:53.666 INFO:teuthology.orchestra.run.smithi037.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-17T19:43:53.992 INFO:teuthology.orchestra.run.smithi037.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-17T19:43:54.129 INFO:teuthology.orchestra.run.smithi037.stdout:Get:75 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3146-gf3a98d3f-1jammy [22.5 MB] 2024-04-17T19:43:54.398 INFO:teuthology.orchestra.run.smithi037.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-17T19:43:54.407 INFO:teuthology.orchestra.run.smithi037.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-17T19:43:54.409 INFO:teuthology.orchestra.run.smithi037.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-17T19:43:54.420 INFO:teuthology.orchestra.run.smithi029.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3146-gf3a98d3f-1jammy [14.0 kB] 2024-04-17T19:43:54.421 INFO:teuthology.orchestra.run.smithi029.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3146-gf3a98d3f-1jammy [856 kB] 2024-04-17T19:43:54.464 INFO:teuthology.orchestra.run.smithi029.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3146-gf3a98d3f-1jammy [2,369 kB] 2024-04-17T19:43:54.478 INFO:teuthology.orchestra.run.smithi037.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-17T19:43:54.478 INFO:teuthology.orchestra.run.smithi037.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-17T19:43:54.480 INFO:teuthology.orchestra.run.smithi037.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-17T19:43:54.481 INFO:teuthology.orchestra.run.smithi037.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-17T19:43:54.482 INFO:teuthology.orchestra.run.smithi037.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-17T19:43:54.598 INFO:teuthology.orchestra.run.smithi029.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3146-gf3a98d3f-1jammy [791 kB] 2024-04-17T19:43:54.644 INFO:teuthology.orchestra.run.smithi029.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3146-gf3a98d3f-1jammy [148 kB] 2024-04-17T19:43:54.648 INFO:teuthology.orchestra.run.smithi037.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-17T19:43:54.651 INFO:teuthology.orchestra.run.smithi029.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3146-gf3a98d3f-1jammy [2,297 kB] 2024-04-17T19:43:54.734 INFO:teuthology.orchestra.run.smithi037.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-17T19:43:54.783 INFO:teuthology.orchestra.run.smithi029.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3146-gf3a98d3f-1jammy [8,625 kB] 2024-04-17T19:43:54.830 INFO:teuthology.orchestra.run.smithi037.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-17T19:43:55.074 INFO:teuthology.orchestra.run.smithi037.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-17T19:43:55.118 INFO:teuthology.orchestra.run.smithi037.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-17T19:43:55.142 INFO:teuthology.orchestra.run.smithi037.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-17T19:43:55.151 INFO:teuthology.orchestra.run.smithi037.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-17T19:43:55.195 INFO:teuthology.orchestra.run.smithi037.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-17T19:43:55.220 INFO:teuthology.orchestra.run.smithi037.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-04-17T19:43:55.221 INFO:teuthology.orchestra.run.smithi037.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-17T19:43:55.229 INFO:teuthology.orchestra.run.smithi037.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-17T19:43:55.276 INFO:teuthology.orchestra.run.smithi029.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3146-gf3a98d3f-1jammy [14.2 kB] 2024-04-17T19:43:55.276 INFO:teuthology.orchestra.run.smithi029.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3146-gf3a98d3f-1jammy [44.9 MB] 2024-04-17T19:43:55.396 INFO:teuthology.orchestra.run.smithi037.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-17T19:43:55.938 INFO:teuthology.orchestra.run.smithi037.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-17T19:43:55.959 INFO:teuthology.orchestra.run.smithi037.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-17T19:43:55.961 INFO:teuthology.orchestra.run.smithi037.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3146-gf3a98d3f-1jammy [14.0 kB] 2024-04-17T19:43:55.962 INFO:teuthology.orchestra.run.smithi037.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3146-gf3a98d3f-1jammy [856 kB] 2024-04-17T19:43:55.980 INFO:teuthology.orchestra.run.smithi037.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3146-gf3a98d3f-1jammy [2,369 kB] 2024-04-17T19:43:56.019 INFO:teuthology.orchestra.run.smithi037.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-17T19:43:56.019 INFO:teuthology.orchestra.run.smithi037.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-17T19:43:56.021 INFO:teuthology.orchestra.run.smithi037.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-17T19:43:56.022 INFO:teuthology.orchestra.run.smithi037.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-17T19:43:56.033 INFO:teuthology.orchestra.run.smithi037.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-17T19:43:56.042 INFO:teuthology.orchestra.run.smithi037.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-17T19:43:56.120 INFO:teuthology.orchestra.run.smithi037.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-17T19:43:56.122 INFO:teuthology.orchestra.run.smithi037.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-17T19:43:56.160 INFO:teuthology.orchestra.run.smithi037.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3146-gf3a98d3f-1jammy [791 kB] 2024-04-17T19:43:56.222 INFO:teuthology.orchestra.run.smithi037.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3146-gf3a98d3f-1jammy [148 kB] 2024-04-17T19:43:56.234 INFO:teuthology.orchestra.run.smithi037.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3146-gf3a98d3f-1jammy [2,297 kB] 2024-04-17T19:43:56.287 INFO:teuthology.orchestra.run.smithi037.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-17T19:43:56.416 INFO:teuthology.orchestra.run.smithi037.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3146-gf3a98d3f-1jammy [8,625 kB] 2024-04-17T19:43:56.720 INFO:teuthology.orchestra.run.smithi037.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-17T19:43:56.735 INFO:teuthology.orchestra.run.smithi037.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-17T19:43:57.100 INFO:teuthology.orchestra.run.smithi037.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3146-gf3a98d3f-1jammy [14.2 kB] 2024-04-17T19:43:57.102 INFO:teuthology.orchestra.run.smithi037.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3146-gf3a98d3f-1jammy [44.9 MB] 2024-04-17T19:43:57.793 INFO:teuthology.orchestra.run.smithi029.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3146-gf3a98d3f-1jammy [124 kB] 2024-04-17T19:43:57.794 INFO:teuthology.orchestra.run.smithi029.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-3146-gf3a98d3f-1jammy [236 kB] 2024-04-17T19:43:57.799 INFO:teuthology.orchestra.run.smithi029.stdout:Get:120 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-3146-gf3a98d3f-1jammy [33.9 kB] 2024-04-17T19:43:57.799 INFO:teuthology.orchestra.run.smithi029.stdout:Get:121 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-3146-gf3a98d3f-1jammy [23.4 kB] 2024-04-17T19:43:57.801 INFO:teuthology.orchestra.run.smithi029.stdout:Get:122 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3146-gf3a98d3f-1jammy [38.9 kB] 2024-04-17T19:43:57.802 INFO:teuthology.orchestra.run.smithi029.stdout:Get:123 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3146-gf3a98d3f-1jammy [14.1 MB] 2024-04-17T19:43:58.591 INFO:teuthology.orchestra.run.smithi029.stdout:Get:124 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3146-gf3a98d3f-1jammy [92.6 kB] 2024-04-17T19:43:59.261 INFO:teuthology.orchestra.run.smithi029.stdout:Fetched 175 MB in 9s (20.0 MB/s) 2024-04-17T19:43:59.464 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pyperclip. 2024-04-17T19:43:59.535 INFO:teuthology.orchestra.run.smithi029.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132922 files and directories currently installed.) 2024-04-17T19:43:59.539 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-04-17T19:43:59.596 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-04-17T19:43:59.816 INFO:teuthology.orchestra.run.smithi037.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3146-gf3a98d3f-1jammy [124 kB] 2024-04-17T19:43:59.819 INFO:teuthology.orchestra.run.smithi037.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-3146-gf3a98d3f-1jammy [236 kB] 2024-04-17T19:43:59.824 INFO:teuthology.orchestra.run.smithi037.stdout:Get:120 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-3146-gf3a98d3f-1jammy [33.9 kB] 2024-04-17T19:43:59.825 INFO:teuthology.orchestra.run.smithi037.stdout:Get:121 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-3146-gf3a98d3f-1jammy [23.4 kB] 2024-04-17T19:43:59.826 INFO:teuthology.orchestra.run.smithi037.stdout:Get:122 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3146-gf3a98d3f-1jammy [38.9 kB] 2024-04-17T19:43:59.827 INFO:teuthology.orchestra.run.smithi037.stdout:Get:123 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3146-gf3a98d3f-1jammy [14.1 MB] 2024-04-17T19:44:00.026 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-17T19:44:00.040 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-17T19:44:00.101 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-17T19:44:00.279 INFO:teuthology.orchestra.run.smithi037.stdout:Get:124 https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240417.021458-debug/f3a98d3f110319c567ec86bf70955c141495ccf6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3146-gf3a98d3f-1jammy [92.6 kB] 2024-04-17T19:44:00.491 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-cmd2. 2024-04-17T19:44:00.506 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-04-17T19:44:00.566 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-04-17T19:44:00.922 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 175 MB in 11s (16.0 MB/s) 2024-04-17T19:44:00.948 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-17T19:44:00.964 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-17T19:44:01.040 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-17T19:44:01.085 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pyperclip. 2024-04-17T19:44:01.172 INFO:teuthology.orchestra.run.smithi037.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 130504 files and directories currently installed.) 2024-04-17T19:44:01.176 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-04-17T19:44:01.217 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-04-17T19:44:01.406 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-17T19:44:01.422 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-17T19:44:01.465 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-17T19:44:01.512 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-17T19:44:01.532 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-17T19:44:01.595 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-17T19:44:01.854 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-17T19:44:01.864 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cmd2. 2024-04-17T19:44:01.869 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-17T19:44:01.876 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-04-17T19:44:01.905 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-04-17T19:44:01.912 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-17T19:44:02.259 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-17T19:44:02.279 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-17T19:44:02.342 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-17T19:44:02.395 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-17T19:44:02.411 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-17T19:44:02.461 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T19:44:02.687 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-17T19:44:02.708 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-17T19:44:02.745 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-17T19:44:02.927 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-17T19:44:02.943 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-17T19:44:02.986 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T19:44:03.124 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-17T19:44:03.144 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-17T19:44:03.182 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-17T19:44:03.366 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-17T19:44:03.382 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-17T19:44:03.425 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T19:44:03.610 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-17T19:44:03.631 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-17T19:44:03.668 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T19:44:03.790 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-17T19:44:03.806 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-17T19:44:03.840 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-17T19:44:04.113 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-17T19:44:04.134 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-17T19:44:04.171 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T19:44:04.525 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-17T19:44:04.545 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-17T19:44:04.600 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T19:44:04.739 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package librados2. 2024-04-17T19:44:04.755 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../010-librados2_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:04.797 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking librados2 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:04.919 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-17T19:44:04.940 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-17T19:44:04.969 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-17T19:44:05.331 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librados2. 2024-04-17T19:44:05.351 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../010-librados2_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:05.389 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librados2 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:05.394 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package librbd1. 2024-04-17T19:44:05.409 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../011-librbd1_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:05.452 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking librbd1 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:05.927 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librbd1. 2024-04-17T19:44:05.948 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../011-librbd1_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:05.977 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librbd1 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:06.001 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libcephfs2. 2024-04-17T19:44:06.016 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:06.059 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libcephfs2 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:06.382 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-rados. 2024-04-17T19:44:06.388 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libcephfs2. 2024-04-17T19:44:06.397 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../013-python3-rados_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:06.409 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:06.440 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-rados (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:06.446 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libcephfs2 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:06.787 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-17T19:44:06.802 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:06.845 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-ceph-argparse (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:07.032 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rados. 2024-04-17T19:44:07.044 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../013-python3-rados_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:07.101 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-cephfs. 2024-04-17T19:44:07.115 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rados (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:07.117 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:07.160 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-cephfs (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:07.410 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-17T19:44:07.431 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:07.468 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-ceph-argparse (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:07.481 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-17T19:44:07.496 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:07.540 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-ceph-common (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:07.771 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cephfs. 2024-04-17T19:44:07.791 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:07.829 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cephfs (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:07.863 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-prettytable. 2024-04-17T19:44:07.879 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-04-17T19:44:07.914 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-17T19:44:08.149 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-17T19:44:08.169 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:08.186 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-rbd. 2024-04-17T19:44:08.202 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:08.207 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-ceph-common (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:08.245 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-rbd (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:08.601 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-prettytable. 2024-04-17T19:44:08.617 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-17T19:44:08.622 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-04-17T19:44:08.632 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-17T19:44:08.659 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-17T19:44:08.675 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-17T19:44:08.946 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rbd. 2024-04-17T19:44:08.966 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:09.004 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rbd (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:09.015 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-17T19:44:09.030 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-17T19:44:09.073 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-17T19:44:09.332 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-17T19:44:09.353 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-17T19:44:09.390 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-17T19:44:09.395 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-17T19:44:09.410 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-17T19:44:09.454 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-17T19:44:09.701 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-17T19:44:09.713 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-17T19:44:09.750 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-17T19:44:09.811 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package lua5.1. 2024-04-17T19:44:09.826 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-17T19:44:09.869 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-17T19:44:10.012 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-17T19:44:10.023 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-17T19:44:10.059 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-17T19:44:10.202 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package lua-any. 2024-04-17T19:44:10.217 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-04-17T19:44:10.260 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-17T19:44:10.390 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package lua5.1. 2024-04-17T19:44:10.412 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-17T19:44:10.449 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-17T19:44:10.523 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package zip. 2024-04-17T19:44:10.538 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-04-17T19:44:10.581 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking zip (3.0-12build2) ... 2024-04-17T19:44:10.801 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package lua-any. 2024-04-17T19:44:10.823 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-04-17T19:44:10.860 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-17T19:44:10.939 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package unzip. 2024-04-17T19:44:10.954 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-17T19:44:10.997 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-17T19:44:11.088 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package zip. 2024-04-17T19:44:11.100 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-04-17T19:44:11.129 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking zip (3.0-12build2) ... 2024-04-17T19:44:11.414 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package luarocks. 2024-04-17T19:44:11.429 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-17T19:44:11.441 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package unzip. 2024-04-17T19:44:11.462 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-17T19:44:11.472 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-17T19:44:11.491 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-17T19:44:11.881 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package librgw2. 2024-04-17T19:44:11.896 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../027-librgw2_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:11.939 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking librgw2 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:12.153 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package luarocks. 2024-04-17T19:44:12.174 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-17T19:44:12.211 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-17T19:44:12.561 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package librgw2. 2024-04-17T19:44:12.582 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../027-librgw2_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:12.627 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking librgw2 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:12.760 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-rgw. 2024-04-17T19:44:12.776 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:12.819 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-rgw (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:13.184 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-17T19:44:13.199 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-17T19:44:13.242 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-17T19:44:13.288 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rgw. 2024-04-17T19:44:13.302 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:13.363 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rgw (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:13.572 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libradosstriper1. 2024-04-17T19:44:13.588 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:13.633 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libradosstriper1 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:13.708 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-17T19:44:13.729 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-17T19:44:13.766 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-17T19:44:14.035 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libradosstriper1. 2024-04-17T19:44:14.056 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:14.085 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libradosstriper1 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:14.088 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-common. 2024-04-17T19:44:14.103 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../031-ceph-common_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:14.147 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-common (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:14.536 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-common. 2024-04-17T19:44:14.557 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../031-ceph-common_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:14.605 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-common (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:16.016 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-base. 2024-04-17T19:44:16.032 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../032-ceph-base_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:16.101 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-base (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:16.186 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-base. 2024-04-17T19:44:16.207 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../032-ceph-base_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:16.261 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-base (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:16.792 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-17T19:44:16.808 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-17T19:44:16.851 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-17T19:44:16.898 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-17T19:44:16.920 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-17T19:44:16.965 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-17T19:44:17.141 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-cheroot. 2024-04-17T19:44:17.156 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-17T19:44:17.199 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-17T19:44:17.368 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cheroot. 2024-04-17T19:44:17.389 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-17T19:44:17.434 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-17T19:44:17.523 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-17T19:44:17.538 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-17T19:44:17.581 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-17T19:44:17.829 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-17T19:44:17.851 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-17T19:44:17.878 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-17T19:44:17.894 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-17T19:44:17.896 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-17T19:44:17.936 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-17T19:44:18.243 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-17T19:44:18.259 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-17T19:44:18.265 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-17T19:44:18.286 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-17T19:44:18.301 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-17T19:44:18.332 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-17T19:44:18.616 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-tempora. 2024-04-17T19:44:18.632 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-04-17T19:44:18.675 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-17T19:44:18.701 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-17T19:44:18.722 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-17T19:44:18.768 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-17T19:44:18.972 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-portend. 2024-04-17T19:44:18.987 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-04-17T19:44:19.030 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-17T19:44:19.163 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-tempora. 2024-04-17T19:44:19.184 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-04-17T19:44:19.229 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-17T19:44:19.337 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-17T19:44:19.352 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-17T19:44:19.395 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-17T19:44:19.574 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-portend. 2024-04-17T19:44:19.595 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-04-17T19:44:19.632 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-17T19:44:19.742 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-17T19:44:19.758 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-17T19:44:19.800 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-17T19:44:20.010 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-17T19:44:20.031 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-17T19:44:20.101 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-17T19:44:20.133 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-natsort. 2024-04-17T19:44:20.148 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-04-17T19:44:20.183 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-17T19:44:20.488 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-17T19:44:20.499 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-logutils. 2024-04-17T19:44:20.509 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-17T19:44:20.513 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-04-17T19:44:20.554 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-17T19:44:20.557 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-17T19:44:20.847 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-17T19:44:20.863 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-17T19:44:20.906 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-17T19:44:20.925 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-natsort. 2024-04-17T19:44:20.946 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-04-17T19:44:20.983 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-17T19:44:21.211 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-mako. 2024-04-17T19:44:21.226 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-17T19:44:21.269 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-17T19:44:21.294 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-logutils. 2024-04-17T19:44:21.316 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-04-17T19:44:21.369 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-17T19:44:21.576 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-17T19:44:21.592 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-17T19:44:21.626 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-17T19:44:21.672 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-17T19:44:21.694 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-17T19:44:21.730 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-17T19:44:22.050 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-mako. 2024-04-17T19:44:22.063 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-17T19:44:22.100 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-17T19:44:22.395 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-17T19:44:22.416 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-17T19:44:22.453 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-17T19:44:23.417 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-17T19:44:23.432 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-17T19:44:23.476 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-17T19:44:23.616 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-17T19:44:23.636 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-17T19:44:23.673 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-17T19:44:23.772 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-webob. 2024-04-17T19:44:23.788 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-17T19:44:23.830 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-17T19:44:23.968 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-webob. 2024-04-17T19:44:23.988 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-17T19:44:24.026 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-17T19:44:24.205 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-waitress. 2024-04-17T19:44:24.219 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-17T19:44:24.265 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-17T19:44:24.312 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-waitress. 2024-04-17T19:44:24.333 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-17T19:44:24.365 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-17T19:44:24.568 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-tempita. 2024-04-17T19:44:24.583 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-17T19:44:24.623 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-tempita. 2024-04-17T19:44:24.628 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-17T19:44:24.645 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-17T19:44:24.673 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-17T19:44:24.996 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-paste. 2024-04-17T19:44:25.017 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-17T19:44:25.068 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-17T19:44:25.242 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-paste. 2024-04-17T19:44:25.257 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-17T19:44:25.300 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-17T19:44:25.397 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-17T19:44:25.418 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-17T19:44:25.454 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-17T19:44:25.615 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-17T19:44:25.630 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-17T19:44:25.674 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-17T19:44:25.799 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-17T19:44:25.820 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-17T19:44:25.857 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-17T19:44:25.954 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-17T19:44:25.969 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-17T19:44:26.004 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-17T19:44:26.118 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-webtest. 2024-04-17T19:44:26.140 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-04-17T19:44:26.168 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-17T19:44:26.311 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-webtest. 2024-04-17T19:44:26.326 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-04-17T19:44:26.369 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-17T19:44:26.496 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pecan. 2024-04-17T19:44:26.518 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-17T19:44:26.554 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-17T19:44:26.660 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pecan. 2024-04-17T19:44:26.675 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-17T19:44:26.718 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-17T19:44:26.934 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-certifi. 2024-04-17T19:44:26.956 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-04-17T19:44:27.001 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-17T19:44:27.136 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-certifi. 2024-04-17T19:44:27.152 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-04-17T19:44:27.195 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-17T19:44:27.337 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-idna. 2024-04-17T19:44:27.359 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-04-17T19:44:27.395 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-17T19:44:27.508 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-idna. 2024-04-17T19:44:27.524 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-04-17T19:44:27.566 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-17T19:44:27.907 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-urllib3. 2024-04-17T19:44:27.922 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-17T19:44:27.940 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-urllib3. 2024-04-17T19:44:27.961 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-17T19:44:27.965 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-17T19:44:27.998 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-17T19:44:28.281 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-requests. 2024-04-17T19:44:28.296 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-17T19:44:28.318 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-requests. 2024-04-17T19:44:28.331 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-17T19:44:28.339 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-17T19:44:28.368 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-17T19:44:28.644 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-17T19:44:28.660 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-17T19:44:28.696 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-17T19:44:28.702 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-17T19:44:28.717 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-17T19:44:28.746 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-17T19:44:29.001 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-17T19:44:29.016 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:29.040 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-17T19:44:29.051 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:29.053 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:29.090 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:29.445 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-17T19:44:29.460 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:29.495 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-17T19:44:29.504 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:29.516 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:29.562 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:29.827 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-17T19:44:29.842 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-17T19:44:29.885 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-17T19:44:29.906 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-17T19:44:29.928 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-17T19:44:29.964 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-17T19:44:30.182 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-openssl. 2024-04-17T19:44:30.197 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-04-17T19:44:30.240 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-17T19:44:30.242 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-openssl. 2024-04-17T19:44:30.263 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-04-17T19:44:30.300 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-17T19:44:30.612 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr. 2024-04-17T19:44:30.633 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:30.655 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-mgr. 2024-04-17T19:44:30.670 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:30.671 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:30.714 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-mgr (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:31.115 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mon. 2024-04-17T19:44:31.137 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:31.154 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-mon. 2024-04-17T19:44:31.170 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:31.181 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mon (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:31.205 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-mon (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:31.794 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-osd. 2024-04-17T19:44:31.815 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:31.851 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-osd (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:31.919 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-osd. 2024-04-17T19:44:31.935 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:31.979 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-osd (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:32.972 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph. 2024-04-17T19:44:32.993 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../068-ceph_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:33.030 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:33.310 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph. 2024-04-17T19:44:33.326 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../068-ceph_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:33.349 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-fuse. 2024-04-17T19:44:33.368 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:33.370 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:33.408 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-fuse (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:33.699 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-fuse. 2024-04-17T19:44:33.715 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:33.758 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-fuse (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:33.828 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mds. 2024-04-17T19:44:33.849 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:33.885 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mds (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:34.197 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-mds. 2024-04-17T19:44:34.212 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:34.255 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-mds (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:34.322 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package cephadm. 2024-04-17T19:44:34.343 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../071-cephadm_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:34.380 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking cephadm (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:34.712 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package cephadm. 2024-04-17T19:44:34.727 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../071-cephadm_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:34.734 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-17T19:44:34.755 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-17T19:44:34.770 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking cephadm (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:34.800 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-17T19:44:35.110 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-17T19:44:35.125 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-17T19:44:35.127 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-17T19:44:35.146 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-17T19:44:35.168 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-17T19:44:35.185 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-17T19:44:35.557 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-17T19:44:35.572 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-17T19:44:35.615 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-17T19:44:36.291 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-babel. 2024-04-17T19:44:36.304 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-17T19:44:36.341 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-17T19:44:36.617 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-babel. 2024-04-17T19:44:36.632 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-17T19:44:36.675 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-17T19:44:36.702 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-jinja2. 2024-04-17T19:44:36.723 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-17T19:44:36.760 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-17T19:44:37.016 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-jinja2. 2024-04-17T19:44:37.030 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-17T19:44:37.074 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-17T19:44:37.105 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-17T19:44:37.126 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:37.163 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:37.439 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-17T19:44:37.455 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:37.498 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:37.500 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-17T19:44:37.521 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-04-17T19:44:37.558 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-17T19:44:37.894 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-routes. 2024-04-17T19:44:37.895 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-17T19:44:37.913 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-04-17T19:44:37.916 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-17T19:44:37.954 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-17T19:44:37.994 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-17T19:44:38.252 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-routes. 2024-04-17T19:44:38.268 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-17T19:44:38.310 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-17T19:44:38.323 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-17T19:44:38.344 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:38.381 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:38.660 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-17T19:44:38.675 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:38.710 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:39.862 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-17T19:44:39.883 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-17T19:44:39.928 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-17T19:44:40.370 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-17T19:44:40.386 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-17T19:44:40.420 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-17T19:44:40.449 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-joblib. 2024-04-17T19:44:40.470 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-17T19:44:40.515 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-17T19:44:40.902 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-17T19:44:40.923 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-17T19:44:40.928 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-joblib. 2024-04-17T19:44:40.944 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-17T19:44:40.978 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-17T19:44:41.062 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-17T19:44:41.357 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-sklearn. 2024-04-17T19:44:41.376 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-17T19:44:41.378 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-17T19:44:41.392 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-17T19:44:41.415 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-17T19:44:41.557 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-17T19:44:41.854 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-sklearn. 2024-04-17T19:44:41.868 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-17T19:44:41.913 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-17T19:44:42.174 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-17T19:44:42.196 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:42.240 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:42.574 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-17T19:44:42.587 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:42.625 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:43.536 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-cachetools. 2024-04-17T19:44:43.558 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-04-17T19:44:43.602 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-17T19:44:43.913 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-17T19:44:43.935 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-17T19:44:43.979 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-17T19:44:44.222 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-cachetools. 2024-04-17T19:44:44.237 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-04-17T19:44:44.281 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-17T19:44:44.333 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-17T19:44:44.355 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-17T19:44:44.399 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-17T19:44:44.596 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-17T19:44:44.610 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-17T19:44:44.654 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-17T19:44:44.744 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-rsa. 2024-04-17T19:44:44.766 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-04-17T19:44:44.811 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-17T19:44:45.037 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-17T19:44:45.052 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-17T19:44:45.095 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-17T19:44:45.198 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-google-auth. 2024-04-17T19:44:45.220 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-04-17T19:44:45.289 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-17T19:44:45.392 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-rsa. 2024-04-17T19:44:45.407 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-04-17T19:44:45.442 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-17T19:44:45.676 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-17T19:44:45.698 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-17T19:44:45.751 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-17T19:44:45.751 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-google-auth. 2024-04-17T19:44:45.766 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-04-17T19:44:45.809 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-17T19:44:46.071 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-websocket. 2024-04-17T19:44:46.093 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-04-17T19:44:46.100 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-17T19:44:46.115 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-17T19:44:46.137 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-17T19:44:46.150 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-17T19:44:46.447 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-websocket. 2024-04-17T19:44:46.462 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-04-17T19:44:46.507 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-17T19:44:46.534 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-17T19:44:46.556 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-17T19:44:46.609 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-17T19:44:46.813 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-17T19:44:46.828 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-17T19:44:46.871 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-17T19:44:47.348 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-17T19:44:47.361 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:47.406 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:47.591 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-17T19:44:47.605 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:47.641 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:47.755 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-17T19:44:47.777 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-17T19:44:47.817 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-17T19:44:47.947 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-17T19:44:47.961 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-17T19:44:48.006 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-17T19:44:48.154 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-17T19:44:48.176 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-17T19:44:48.220 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-17T19:44:48.413 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-17T19:44:48.428 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-17T19:44:48.471 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-17T19:44:48.514 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package jq. 2024-04-17T19:44:48.537 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-17T19:44:48.581 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-17T19:44:48.743 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package jq. 2024-04-17T19:44:48.758 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-17T19:44:48.801 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-17T19:44:48.842 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package socat. 2024-04-17T19:44:48.864 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-17T19:44:48.908 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-17T19:44:49.057 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package socat. 2024-04-17T19:44:49.072 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-17T19:44:49.116 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-17T19:44:49.236 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package xmlstarlet. 2024-04-17T19:44:49.250 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-17T19:44:49.286 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-17T19:44:49.446 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package xmlstarlet. 2024-04-17T19:44:49.461 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-17T19:44:49.506 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-17T19:44:49.658 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-test. 2024-04-17T19:44:49.680 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../099-ceph-test_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:49.724 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-test (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:49.881 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-test. 2024-04-17T19:44:49.895 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../099-ceph-test_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:49.939 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-test (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:52.078 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ceph-volume. 2024-04-17T19:44:52.100 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:52.145 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ceph-volume (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:52.532 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package cephfs-mirror. 2024-04-17T19:44:52.554 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:52.590 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking cephfs-mirror (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:52.927 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-colorama. 2024-04-17T19:44:52.949 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-04-17T19:44:52.993 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-04-17T19:44:53.254 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package cephfs-shell. 2024-04-17T19:44:53.276 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:53.354 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking cephfs-shell (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:53.623 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package cephfs-top. 2024-04-17T19:44:53.645 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:53.689 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking cephfs-top (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:53.923 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-volume. 2024-04-17T19:44:53.930 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:53.967 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-17T19:44:53.973 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-volume (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:53.990 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:54.034 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libcephfs-dev (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:54.295 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-17T19:44:54.308 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-17T19:44:54.340 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package cephfs-mirror. 2024-04-17T19:44:54.345 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-17T19:44:54.348 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:54.390 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking cephfs-mirror (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:54.708 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-17T19:44:54.730 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-04-17T19:44:54.730 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-colorama. 2024-04-17T19:44:54.738 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-04-17T19:44:54.774 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-17T19:44:54.780 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-04-17T19:44:55.051 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package cephfs-shell. 2024-04-17T19:44:55.059 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:55.102 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking cephfs-shell (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:55.212 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package nvme-cli. 2024-04-17T19:44:55.234 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-17T19:44:55.278 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-17T19:44:55.391 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package cephfs-top. 2024-04-17T19:44:55.399 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-3146-gf3a98d3f-1jammy_all.deb ... 2024-04-17T19:44:55.441 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking cephfs-top (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:55.687 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-17T19:44:55.692 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package pkg-config. 2024-04-17T19:44:55.694 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:44:55.714 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-17T19:44:55.730 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libcephfs-dev (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:44:55.783 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-17T19:44:56.019 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-17T19:44:56.038 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-17T19:44:56.077 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-17T19:44:56.078 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-17T19:44:56.084 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-17T19:44:56.127 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-17T19:44:56.495 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-17T19:44:56.503 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-04-17T19:44:56.545 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-17T19:44:56.547 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-17T19:44:56.570 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-17T19:44:56.614 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-17T19:44:56.917 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pastescript. 2024-04-17T19:44:56.939 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-04-17T19:44:56.995 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package nvme-cli. 2024-04-17T19:44:57.006 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-17T19:44:57.008 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-17T19:44:57.045 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-17T19:44:57.337 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pluggy. 2024-04-17T19:44:57.359 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-17T19:44:57.403 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-17T19:44:57.482 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package pkg-config. 2024-04-17T19:44:57.490 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-17T19:44:57.531 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-17T19:44:57.706 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-psutil. 2024-04-17T19:44:57.728 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-17T19:44:57.772 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-17T19:44:57.797 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-17T19:44:57.805 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-17T19:44:57.847 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-17T19:44:58.109 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-py. 2024-04-17T19:44:58.131 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-04-17T19:44:58.208 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-17T19:44:58.327 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-17T19:44:58.336 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-17T19:44:58.377 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-17T19:44:58.537 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pygments. 2024-04-17T19:44:58.560 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-17T19:44:58.603 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-17T19:44:58.684 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pastescript. 2024-04-17T19:44:58.693 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-04-17T19:44:58.734 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-17T19:44:59.125 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-17T19:44:59.147 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-17T19:44:59.182 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-17T19:44:59.375 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pluggy. 2024-04-17T19:44:59.384 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-17T19:44:59.425 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-17T19:44:59.518 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-toml. 2024-04-17T19:44:59.541 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-04-17T19:44:59.585 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-17T19:44:59.713 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-psutil. 2024-04-17T19:44:59.722 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-17T19:44:59.763 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-17T19:44:59.905 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-pytest. 2024-04-17T19:44:59.927 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-17T19:44:59.970 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-17T19:45:00.120 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-py. 2024-04-17T19:45:00.128 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-04-17T19:45:00.161 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-17T19:45:00.324 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package python3-simplejson. 2024-04-17T19:45:00.347 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-17T19:45:00.399 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-17T19:45:00.652 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pygments. 2024-04-17T19:45:00.659 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-17T19:45:00.702 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-17T19:45:00.720 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-17T19:45:00.725 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-17T19:45:00.768 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-17T19:45:01.186 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-17T19:45:01.195 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-17T19:45:01.237 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-17T19:45:01.531 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package radosgw. 2024-04-17T19:45:01.553 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../122-radosgw_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:45:01.567 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-toml. 2024-04-17T19:45:01.575 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-04-17T19:45:01.597 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking radosgw (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:01.617 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-17T19:45:01.915 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pytest. 2024-04-17T19:45:01.923 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-17T19:45:01.965 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-17T19:45:02.306 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-simplejson. 2024-04-17T19:45:02.316 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-17T19:45:02.356 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-17T19:45:02.611 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-17T19:45:02.619 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-17T19:45:02.634 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package rbd-fuse. 2024-04-17T19:45:02.647 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:45:02.661 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-17T19:45:02.692 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking rbd-fuse (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:03.003 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-17T19:45:03.272 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-17T19:45:03.373 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-17T19:45:03.494 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package radosgw. 2024-04-17T19:45:03.502 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../122-radosgw_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:45:03.545 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking radosgw (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:03.612 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-17T19:45:03.962 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-17T19:45:04.428 INFO:teuthology.orchestra.run.smithi037.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-17T19:45:04.533 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up cephadm (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:04.760 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package rbd-fuse. 2024-04-17T19:45:04.769 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-3146-gf3a98d3f-1jammy_amd64.deb ... 2024-04-17T19:45:04.818 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking rbd-fuse (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:05.091 INFO:teuthology.orchestra.run.smithi037.stdout:Adding system user cephadm....done 2024-04-17T19:45:05.148 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-17T19:45:05.264 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-17T19:45:05.422 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-17T19:45:05.548 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-17T19:45:05.548 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-17T19:45:05.808 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-17T19:45:05.820 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-17T19:45:05.934 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-17T19:45:06.194 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-17T19:45:06.225 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-17T19:45:06.553 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-17T19:45:06.713 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-17T19:45:06.851 INFO:teuthology.orchestra.run.smithi029.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-17T19:45:06.908 INFO:teuthology.orchestra.run.smithi029.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2024-04-17T19:45:06.954 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up cephadm (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:07.015 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-17T19:45:07.366 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-17T19:45:07.528 INFO:teuthology.orchestra.run.smithi029.stdout:Adding system user cephadm....done 2024-04-17T19:45:07.668 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-04-17T19:45:07.785 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-17T19:45:07.952 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-17T19:45:08.184 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-17T19:45:08.228 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-17T19:45:08.517 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-17T19:45:08.659 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-17T19:45:08.773 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-17T19:45:08.997 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-17T19:45:09.315 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-17T19:45:09.390 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-17T19:45:09.583 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-ceph-argparse (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:09.669 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-17T19:45:09.795 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-17T19:45:09.910 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-17T19:45:10.035 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-17T19:45:10.107 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-17T19:45:10.161 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-04-17T19:45:10.429 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-17T19:45:10.487 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-17T19:45:10.580 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-17T19:45:10.756 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-04-17T19:45:10.822 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-17T19:45:11.034 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-17T19:45:11.219 INFO:teuthology.orchestra.run.smithi037.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-17T19:45:11.279 INFO:teuthology.orchestra.run.smithi037.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-17T19:45:11.312 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-17T19:45:11.342 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-17T19:45:11.469 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-17T19:45:11.565 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-17T19:45:11.837 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-17T19:45:11.876 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-17T19:45:11.962 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-17T19:45:12.153 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-17T19:45:12.238 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-17T19:45:12.423 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-ceph-argparse (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:12.514 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-17T19:45:12.700 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-17T19:45:12.827 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-17T19:45:12.849 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up zip (3.0-12build2) ... 2024-04-17T19:45:12.954 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-04-17T19:45:13.009 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-17T19:45:13.216 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-17T19:45:13.342 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-17T19:45:13.611 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-17T19:45:13.685 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-17T19:45:13.953 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-17T19:45:14.041 INFO:teuthology.orchestra.run.smithi029.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-17T19:45:14.078 INFO:teuthology.orchestra.run.smithi029.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-17T19:45:14.079 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-17T19:45:14.152 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-17T19:45:14.222 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-17T19:45:14.278 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-17T19:45:14.498 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-17T19:45:14.664 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-17T19:45:14.791 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-17T19:45:14.874 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-17T19:45:15.070 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-17T19:45:15.125 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-17T19:45:15.347 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-17T19:45:15.552 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-17T19:45:15.674 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up zip (3.0-12build2) ... 2024-04-17T19:45:15.801 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-17T19:45:15.853 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-04-17T19:45:16.171 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-17T19:45:16.505 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-17T19:45:16.523 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-17T19:45:16.773 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-17T19:45:16.800 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-17T19:45:16.899 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-ceph-common (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:16.918 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-17T19:45:17.047 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-17T19:45:17.226 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-idna (3.3-1) ... 2024-04-17T19:45:17.374 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-17T19:45:17.543 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-17T19:45:17.785 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-17T19:45:18.053 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-17T19:45:18.449 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-17T19:45:18.774 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-17T19:45:18.802 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-04-17T19:45:19.051 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T19:45:19.129 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-17T19:45:19.185 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-17T19:45:19.453 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-17T19:45:19.492 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-17T19:45:19.579 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-17T19:45:19.722 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-17T19:45:19.752 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-17T19:45:19.879 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-ceph-common (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:20.006 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-17T19:45:20.190 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-idna (3.3-1) ... 2024-04-17T19:45:20.274 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-17T19:45:20.525 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T19:45:20.543 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-17T19:45:20.651 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-17T19:45:20.969 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-17T19:45:21.228 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-17T19:45:21.505 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-17T19:45:21.748 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-17T19:45:21.831 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-17T19:45:22.007 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-17T19:45:22.092 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T19:45:22.228 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-17T19:45:22.316 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-17T19:45:22.442 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-17T19:45:22.514 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-17T19:45:22.641 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-17T19:45:22.727 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-17T19:45:22.767 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-17T19:45:22.853 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-17T19:45:23.072 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-17T19:45:23.146 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-17T19:45:23.341 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-17T19:45:23.447 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-17T19:45:23.610 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T19:45:23.690 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-17T19:45:23.736 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-17T19:45:23.816 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-17T19:45:23.983 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-17T19:45:24.057 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-17T19:45:24.072 INFO:teuthology.orchestra.run.smithi037.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-17T19:45:24.326 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-17T19:45:24.327 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-17T19:45:24.453 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-17T19:45:24.603 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-17T19:45:24.871 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-17T19:45:24.872 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-17T19:45:25.134 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-17T19:45:25.139 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-17T19:45:25.256 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-17T19:45:25.419 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-17T19:45:25.524 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T19:45:25.546 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-17T19:45:25.650 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-17T19:45:25.840 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-17T19:45:26.001 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-17T19:45:26.010 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-17T19:45:26.312 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-17T19:45:26.361 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-17T19:45:26.622 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-17T19:45:26.645 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-17T19:45:26.865 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-17T19:45:26.891 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-17T19:45:27.019 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-17T19:45:27.146 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-17T19:45:27.166 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-17T19:45:27.227 INFO:teuthology.orchestra.run.smithi029.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-17T19:45:27.442 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-17T19:45:27.474 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-17T19:45:27.601 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-17T19:45:27.744 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-17T19:45:28.014 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-17T19:45:28.170 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-17T19:45:28.283 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-17T19:45:28.409 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-17T19:45:28.455 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-17T19:45:28.678 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T19:45:28.790 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-17T19:45:28.798 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-17T19:45:28.917 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librados2 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:29.043 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:29.134 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-17T19:45:29.169 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-17T19:45:29.452 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-17T19:45:29.539 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libcephfs2 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:29.639 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libradosstriper1 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:29.765 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-17T19:45:30.059 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librbd1 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:30.185 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-modules-core (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:30.310 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-fuse (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:30.317 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-17T19:45:30.547 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-17T19:45:30.547 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-17T19:45:30.579 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-17T19:45:30.865 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-17T19:45:31.112 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libcephfs-dev (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:31.143 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-17T19:45:31.246 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rados (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:31.347 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up librgw2 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:31.421 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-17T19:45:31.448 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up cephfs-top (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:31.574 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-17T19:45:31.818 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-17T19:45:32.120 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-17T19:45:32.516 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-17T19:45:32.645 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up librados2 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:32.772 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:32.899 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-17T19:45:32.920 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rbd (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:33.071 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up rbd-fuse (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:33.188 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-rgw (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:33.264 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libcephfs2 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:33.289 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up python3-cephfs (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:33.391 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libradosstriper1 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:33.415 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up cephfs-shell (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:33.510 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-17T19:45:33.541 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-common (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:33.814 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up librbd1 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:33.941 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-mgr-modules-core (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:34.051 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-fuse (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:34.126 INFO:teuthology.orchestra.run.smithi037.stdout:Adding group ceph....done 2024-04-17T19:45:34.272 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-17T19:45:34.272 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-17T19:45:34.544 INFO:teuthology.orchestra.run.smithi037.stdout:Adding system user ceph....done 2024-04-17T19:45:34.715 INFO:teuthology.orchestra.run.smithi037.stdout:Setting system user ceph properties....done 2024-04-17T19:45:34.728 INFO:teuthology.orchestra.run.smithi037.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-17T19:45:34.871 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-17T19:45:34.895 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libcephfs-dev (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:35.023 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-rados (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:35.150 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up librgw2 (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:35.238 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-17T19:45:35.277 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up cephfs-top (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:35.429 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-17T19:45:35.844 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up cephfs-mirror (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:36.058 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-17T19:45:36.058 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-17T19:45:36.737 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-test (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:36.780 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-rbd (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:36.855 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up radosgw (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:36.898 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up rbd-fuse (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:37.026 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-rgw (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:37.153 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-cephfs (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:37.280 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up cephfs-shell (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:37.311 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-17T19:45:37.311 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-17T19:45:37.407 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-common (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:37.848 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-base (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:37.991 INFO:teuthology.orchestra.run.smithi029.stdout:Adding group ceph....done 2024-04-17T19:45:38.254 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-17T19:45:38.463 INFO:teuthology.orchestra.run.smithi029.stdout:Adding system user ceph....done 2024-04-17T19:45:38.684 INFO:teuthology.orchestra.run.smithi029.stdout:Setting system user ceph properties....done 2024-04-17T19:45:38.707 INFO:teuthology.orchestra.run.smithi029.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-17T19:45:38.865 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-17T19:45:38.909 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mds (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:39.148 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-17T19:45:39.148 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-17T19:45:39.273 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-17T19:45:39.685 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:39.996 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up cephfs-mirror (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:40.145 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-17T19:45:40.145 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-17T19:45:40.205 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-17T19:45:40.205 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-17T19:45:40.671 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-osd (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:40.841 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-test (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:40.968 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-17T19:45:40.968 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-17T19:45:40.968 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up radosgw (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:41.514 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:41.522 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-17T19:45:41.522 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-17T19:45:41.673 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:41.816 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mon (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:42.020 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-17T19:45:42.020 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-17T19:45:42.197 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-base (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:42.601 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-cephadm (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:42.629 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-17T19:45:42.752 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:42.903 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-mgr-dashboard (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:43.054 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ceph-volume (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:43.406 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-17T19:45:43.524 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-17T19:45:43.663 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-mds (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:43.668 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-17T19:45:43.884 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-17T19:45:43.884 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-17T19:45:44.785 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-17T19:45:44.952 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-mgr (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:45.180 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-17T19:45:45.180 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-17T19:45:45.442 INFO:teuthology.orchestra.run.smithi037.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-17T19:45:45.446 DEBUG:teuthology.orchestra.run.smithi037:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-04-17T19:45:45.547 INFO:teuthology.orchestra.run.smithi037.stdout:Reading package lists... 2024-04-17T19:45:45.756 INFO:teuthology.orchestra.run.smithi037.stdout:Building dependency tree... 2024-04-17T19:45:45.757 INFO:teuthology.orchestra.run.smithi037.stdout:Reading state information... 2024-04-17T19:45:45.904 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-osd (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:45.959 INFO:teuthology.orchestra.run.smithi037.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-04-17T19:45:45.959 INFO:teuthology.orchestra.run.smithi037.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-04-17T19:45:45.960 INFO:teuthology.orchestra.run.smithi037.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-04-17T19:45:45.960 INFO:teuthology.orchestra.run.smithi037.stdout:The following packages were automatically installed and are no longer required: 2024-04-17T19:45:45.960 INFO:teuthology.orchestra.run.smithi037.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-17T19:45:45.960 INFO:teuthology.orchestra.run.smithi037.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-17T19:45:45.960 INFO:teuthology.orchestra.run.smithi037.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-17T19:45:45.960 INFO:teuthology.orchestra.run.smithi037.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-17T19:45:45.961 INFO:teuthology.orchestra.run.smithi037.stdout: python2.7-minimal 2024-04-17T19:45:45.961 INFO:teuthology.orchestra.run.smithi037.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-17T19:45:45.961 INFO:teuthology.orchestra.run.smithi037.stdout:The following additional packages will be installed: 2024-04-17T19:45:45.961 INFO:teuthology.orchestra.run.smithi037.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-04-17T19:45:45.962 INFO:teuthology.orchestra.run.smithi037.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-04-17T19:45:45.962 INFO:teuthology.orchestra.run.smithi037.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-04-17T19:45:45.962 INFO:teuthology.orchestra.run.smithi037.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-04-17T19:45:45.963 INFO:teuthology.orchestra.run.smithi037.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-17T19:45:45.963 INFO:teuthology.orchestra.run.smithi037.stdout:Suggested packages: 2024-04-17T19:45:45.963 INFO:teuthology.orchestra.run.smithi037.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-04-17T19:45:45.964 INFO:teuthology.orchestra.run.smithi037.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-04-17T19:45:45.964 INFO:teuthology.orchestra.run.smithi037.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-04-17T19:45:46.005 INFO:teuthology.orchestra.run.smithi037.stdout:The following NEW packages will be installed: 2024-04-17T19:45:46.005 INFO:teuthology.orchestra.run.smithi037.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-04-17T19:45:46.006 INFO:teuthology.orchestra.run.smithi037.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-04-17T19:45:46.006 INFO:teuthology.orchestra.run.smithi037.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-04-17T19:45:46.006 INFO:teuthology.orchestra.run.smithi037.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-04-17T19:45:46.007 INFO:teuthology.orchestra.run.smithi037.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-04-17T19:45:46.007 INFO:teuthology.orchestra.run.smithi037.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-04-17T19:45:46.007 INFO:teuthology.orchestra.run.smithi037.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-17T19:45:46.007 INFO:teuthology.orchestra.run.smithi037.stdout:The following packages will be upgraded: 2024-04-17T19:45:46.008 INFO:teuthology.orchestra.run.smithi037.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-04-17T19:45:46.190 INFO:teuthology.orchestra.run.smithi037.stdout:4 upgraded, 39 newly installed, 0 to remove and 301 not upgraded. 2024-04-17T19:45:46.191 INFO:teuthology.orchestra.run.smithi037.stdout:Need to get 55.7 MB of archives. 2024-04-17T19:45:46.191 INFO:teuthology.orchestra.run.smithi037.stdout:After this operation, 202 MB of additional disk space will be used. 2024-04-17T19:45:46.191 INFO:teuthology.orchestra.run.smithi037.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-04-17T19:45:46.216 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-17T19:45:46.216 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-17T19:45:46.824 INFO:teuthology.orchestra.run.smithi037.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-04-17T19:45:46.840 INFO:teuthology.orchestra.run.smithi037.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-04-17T19:45:46.866 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:46.967 INFO:teuthology.orchestra.run.smithi037.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-04-17T19:45:46.968 INFO:teuthology.orchestra.run.smithi037.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-04-17T19:45:46.970 INFO:teuthology.orchestra.run.smithi037.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-04-17T19:45:46.986 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:46.997 INFO:teuthology.orchestra.run.smithi037.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-04-17T19:45:46.999 INFO:teuthology.orchestra.run.smithi037.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-04-17T19:45:47.003 INFO:teuthology.orchestra.run.smithi037.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-04-17T19:45:47.006 INFO:teuthology.orchestra.run.smithi037.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-04-17T19:45:47.007 INFO:teuthology.orchestra.run.smithi037.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-04-17T19:45:47.138 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-mon (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:47.173 INFO:teuthology.orchestra.run.smithi037.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-04-17T19:45:47.357 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-17T19:45:47.357 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-17T19:45:47.456 INFO:teuthology.orchestra.run.smithi037.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-04-17T19:45:47.466 INFO:teuthology.orchestra.run.smithi037.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-04-17T19:45:47.473 INFO:teuthology.orchestra.run.smithi037.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-04-17T19:45:47.554 INFO:teuthology.orchestra.run.smithi037.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-04-17T19:45:48.033 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-mgr-cephadm (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:48.168 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:48.296 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-mgr-dashboard (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:48.448 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-volume (19.0.0-3146-gf3a98d3f-1jammy) ... 2024-04-17T19:45:48.675 INFO:teuthology.orchestra.run.smithi037.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-04-17T19:45:48.681 INFO:teuthology.orchestra.run.smithi037.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-04-17T19:45:48.681 INFO:teuthology.orchestra.run.smithi037.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-04-17T19:45:48.687 INFO:teuthology.orchestra.run.smithi037.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-04-17T19:45:48.688 INFO:teuthology.orchestra.run.smithi037.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-04-17T19:45:48.698 INFO:teuthology.orchestra.run.smithi037.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-04-17T19:45:48.819 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-17T19:45:48.865 INFO:teuthology.orchestra.run.smithi037.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-04-17T19:45:48.946 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-17T19:45:49.101 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-17T19:45:49.463 INFO:teuthology.orchestra.run.smithi037.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-04-17T19:45:49.463 INFO:teuthology.orchestra.run.smithi037.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-04-17T19:45:49.646 INFO:teuthology.orchestra.run.smithi037.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-04-17T19:45:49.646 INFO:teuthology.orchestra.run.smithi037.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-04-17T19:45:49.684 INFO:teuthology.orchestra.run.smithi037.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-04-17T19:45:49.744 INFO:teuthology.orchestra.run.smithi037.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-04-17T19:45:49.751 INFO:teuthology.orchestra.run.smithi037.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-04-17T19:45:49.752 INFO:teuthology.orchestra.run.smithi037.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-04-17T19:45:49.754 INFO:teuthology.orchestra.run.smithi037.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-04-17T19:45:49.755 INFO:teuthology.orchestra.run.smithi037.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-04-17T19:45:49.969 INFO:teuthology.orchestra.run.smithi037.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-04-17T19:45:50.381 INFO:teuthology.orchestra.run.smithi037.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-04-17T19:45:50.437 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-17T19:45:51.234 INFO:teuthology.orchestra.run.smithi037.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-04-17T19:45:51.239 INFO:teuthology.orchestra.run.smithi029.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-17T19:45:51.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-04-17T19:45:51.293 INFO:teuthology.orchestra.run.smithi037.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-04-17T19:45:51.293 INFO:teuthology.orchestra.run.smithi037.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-04-17T19:45:51.294 INFO:teuthology.orchestra.run.smithi037.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-04-17T19:45:51.294 INFO:teuthology.orchestra.run.smithi037.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-04-17T19:45:51.295 INFO:teuthology.orchestra.run.smithi037.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-04-17T19:45:51.320 INFO:teuthology.orchestra.run.smithi029.stdout:Reading package lists... 2024-04-17T19:45:51.323 INFO:teuthology.orchestra.run.smithi037.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-04-17T19:45:51.324 INFO:teuthology.orchestra.run.smithi037.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-04-17T19:45:51.497 INFO:teuthology.orchestra.run.smithi029.stdout:Building dependency tree... 2024-04-17T19:45:51.498 INFO:teuthology.orchestra.run.smithi029.stdout:Reading state information... 2024-04-17T19:45:51.678 INFO:teuthology.orchestra.run.smithi029.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-04-17T19:45:51.678 INFO:teuthology.orchestra.run.smithi029.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-04-17T19:45:51.678 INFO:teuthology.orchestra.run.smithi029.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-04-17T19:45:51.678 INFO:teuthology.orchestra.run.smithi029.stdout:The following packages were automatically installed and are no longer required: 2024-04-17T19:45:51.678 INFO:teuthology.orchestra.run.smithi029.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-17T19:45:51.679 INFO:teuthology.orchestra.run.smithi029.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-17T19:45:51.679 INFO:teuthology.orchestra.run.smithi029.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-17T19:45:51.679 INFO:teuthology.orchestra.run.smithi029.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-17T19:45:51.680 INFO:teuthology.orchestra.run.smithi029.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-17T19:45:51.680 INFO:teuthology.orchestra.run.smithi029.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-17T19:45:51.680 INFO:teuthology.orchestra.run.smithi029.stdout:The following additional packages will be installed: 2024-04-17T19:45:51.680 INFO:teuthology.orchestra.run.smithi029.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-04-17T19:45:51.680 INFO:teuthology.orchestra.run.smithi029.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-04-17T19:45:51.681 INFO:teuthology.orchestra.run.smithi029.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-04-17T19:45:51.681 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-04-17T19:45:51.681 INFO:teuthology.orchestra.run.smithi029.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-17T19:45:51.681 INFO:teuthology.orchestra.run.smithi029.stdout:Suggested packages: 2024-04-17T19:45:51.682 INFO:teuthology.orchestra.run.smithi029.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-04-17T19:45:51.682 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-04-17T19:45:51.682 INFO:teuthology.orchestra.run.smithi029.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-04-17T19:45:51.738 INFO:teuthology.orchestra.run.smithi029.stdout:The following NEW packages will be installed: 2024-04-17T19:45:51.739 INFO:teuthology.orchestra.run.smithi029.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-04-17T19:45:51.739 INFO:teuthology.orchestra.run.smithi029.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-04-17T19:45:51.739 INFO:teuthology.orchestra.run.smithi029.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-04-17T19:45:51.739 INFO:teuthology.orchestra.run.smithi029.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-04-17T19:45:51.739 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-04-17T19:45:51.739 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-04-17T19:45:51.740 INFO:teuthology.orchestra.run.smithi029.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-17T19:45:51.740 INFO:teuthology.orchestra.run.smithi029.stdout:The following packages will be upgraded: 2024-04-17T19:45:51.741 INFO:teuthology.orchestra.run.smithi029.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-04-17T19:45:51.747 INFO:teuthology.orchestra.run.smithi037.stdout:Preconfiguring packages ... 2024-04-17T19:45:51.910 INFO:teuthology.orchestra.run.smithi037.stdout:Fetched 55.7 MB in 5s (10.5 MB/s) 2024-04-17T19:45:51.926 INFO:teuthology.orchestra.run.smithi029.stdout:4 upgraded, 39 newly installed, 0 to remove and 301 not upgraded. 2024-04-17T19:45:51.926 INFO:teuthology.orchestra.run.smithi029.stdout:Need to get 55.7 MB of archives. 2024-04-17T19:45:51.926 INFO:teuthology.orchestra.run.smithi029.stdout:After this operation, 202 MB of additional disk space will be used. 2024-04-17T19:45:51.926 INFO:teuthology.orchestra.run.smithi029.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-04-17T19:45:52.236 INFO:teuthology.orchestra.run.smithi037.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138504 files and directories currently installed.) 2024-04-17T19:45:52.241 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-17T19:45:52.401 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-17T19:45:52.568 INFO:teuthology.orchestra.run.smithi029.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-04-17T19:45:52.585 INFO:teuthology.orchestra.run.smithi029.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-04-17T19:45:52.710 INFO:teuthology.orchestra.run.smithi029.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-04-17T19:45:52.711 INFO:teuthology.orchestra.run.smithi029.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-04-17T19:45:52.712 INFO:teuthology.orchestra.run.smithi029.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-04-17T19:45:52.741 INFO:teuthology.orchestra.run.smithi029.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-04-17T19:45:52.742 INFO:teuthology.orchestra.run.smithi029.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-04-17T19:45:52.745 INFO:teuthology.orchestra.run.smithi029.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-04-17T19:45:52.748 INFO:teuthology.orchestra.run.smithi029.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-04-17T19:45:52.748 INFO:teuthology.orchestra.run.smithi029.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-04-17T19:45:52.914 INFO:teuthology.orchestra.run.smithi029.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-04-17T19:45:53.070 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-04-17T19:45:53.208 INFO:teuthology.orchestra.run.smithi029.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-04-17T19:45:53.218 INFO:teuthology.orchestra.run.smithi029.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-04-17T19:45:53.223 INFO:teuthology.orchestra.run.smithi029.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-04-17T19:45:53.307 INFO:teuthology.orchestra.run.smithi029.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-04-17T19:45:54.090 INFO:teuthology.orchestra.run.smithi037.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-04-17T19:45:54.277 INFO:teuthology.orchestra.run.smithi029.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-04-17T19:45:54.279 INFO:teuthology.orchestra.run.smithi029.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-04-17T19:45:54.279 INFO:teuthology.orchestra.run.smithi029.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-04-17T19:45:54.282 INFO:teuthology.orchestra.run.smithi029.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-04-17T19:45:54.282 INFO:teuthology.orchestra.run.smithi029.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-04-17T19:45:54.329 INFO:teuthology.orchestra.run.smithi029.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-04-17T19:45:54.398 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package flex. 2024-04-17T19:45:54.429 INFO:teuthology.orchestra.run.smithi029.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-04-17T19:45:54.490 INFO:teuthology.orchestra.run.smithi037.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138504 files and directories currently installed.) 2024-04-17T19:45:54.496 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-04-17T19:45:54.512 INFO:teuthology.orchestra.run.smithi029.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-04-17T19:45:54.534 INFO:teuthology.orchestra.run.smithi029.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-04-17T19:45:54.538 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking flex (2.6.4-8build2) ... 2024-04-17T19:45:54.602 INFO:teuthology.orchestra.run.smithi029.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-04-17T19:45:54.623 INFO:teuthology.orchestra.run.smithi029.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-04-17T19:45:54.661 INFO:teuthology.orchestra.run.smithi029.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-04-17T19:45:54.707 INFO:teuthology.orchestra.run.smithi029.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-04-17T19:45:54.732 INFO:teuthology.orchestra.run.smithi029.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-04-17T19:45:54.753 INFO:teuthology.orchestra.run.smithi029.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-04-17T19:45:54.775 INFO:teuthology.orchestra.run.smithi029.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-04-17T19:45:54.796 INFO:teuthology.orchestra.run.smithi029.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-04-17T19:45:54.830 INFO:teuthology.orchestra.run.smithi029.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-04-17T19:45:54.852 INFO:teuthology.orchestra.run.smithi029.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-04-17T19:45:55.039 INFO:teuthology.orchestra.run.smithi029.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-04-17T19:45:55.056 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-17T19:45:55.060 INFO:teuthology.orchestra.run.smithi029.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-04-17T19:45:55.081 INFO:teuthology.orchestra.run.smithi029.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-04-17T19:45:55.103 INFO:teuthology.orchestra.run.smithi029.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-04-17T19:45:55.124 INFO:teuthology.orchestra.run.smithi029.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-04-17T19:45:55.145 INFO:teuthology.orchestra.run.smithi029.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-04-17T19:45:55.181 INFO:teuthology.orchestra.run.smithi029.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-04-17T19:45:55.202 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-04-17T19:45:55.202 INFO:teuthology.orchestra.run.smithi029.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-04-17T19:45:55.594 INFO:teuthology.orchestra.run.smithi029.stdout:Preconfiguring packages ... 2024-04-17T19:45:55.745 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-17T19:45:55.791 INFO:teuthology.orchestra.run.smithi029.stdout:Fetched 55.7 MB in 3s (16.1 MB/s) 2024-04-17T19:45:56.058 INFO:teuthology.orchestra.run.smithi029.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140922 files and directories currently installed.) 2024-04-17T19:45:56.062 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-17T19:45:56.184 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-17T19:45:56.288 INFO:teuthology.orchestra.run.smithi037.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138577 files and directories currently installed.) 2024-04-17T19:45:56.292 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-17T19:45:56.426 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-17T19:45:56.835 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-04-17T19:45:56.863 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-17T19:45:56.983 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-17T19:45:57.308 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-17T19:45:57.725 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package bison. 2024-04-17T19:45:57.817 INFO:teuthology.orchestra.run.smithi037.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 138577 files and directories currently installed.) 2024-04-17T19:45:57.822 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-04-17T19:45:57.868 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-04-17T19:45:58.027 INFO:teuthology.orchestra.run.smithi029.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-04-17T19:45:58.311 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-04-17T19:45:58.334 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-04-17T19:45:58.372 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package flex. 2024-04-17T19:45:58.378 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-04-17T19:45:58.446 INFO:teuthology.orchestra.run.smithi029.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140922 files and directories currently installed.) 2024-04-17T19:45:58.450 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-04-17T19:45:58.488 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking flex (2.6.4-8build2) ... 2024-04-17T19:45:58.798 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package dump. 2024-04-17T19:45:58.820 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-04-17T19:45:58.864 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking dump (0.4b47-3) ... 2024-04-17T19:45:59.072 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-17T19:45:59.142 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package indent. 2024-04-17T19:45:59.156 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-04-17T19:45:59.184 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-17T19:45:59.242 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-04-17T19:45:59.487 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-04-17T19:45:59.501 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-04-17T19:45:59.545 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-17T19:45:59.798 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-17T19:45:59.907 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-04-17T19:45:59.930 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-04-17T19:45:59.974 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-17T19:46:00.227 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-04-17T19:46:00.240 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-04-17T19:46:00.268 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-04-17T19:46:00.484 INFO:teuthology.orchestra.run.smithi029.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140995 files and directories currently installed.) 2024-04-17T19:46:00.488 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-17T19:46:00.610 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-17T19:46:00.663 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libfl2:amd64. 2024-04-17T19:46:00.686 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-04-17T19:46:00.772 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-04-17T19:46:01.025 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-04-17T19:46:01.047 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-04-17T19:46:01.117 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-17T19:46:01.378 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libjson-perl. 2024-04-17T19:46:01.391 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-04-17T19:46:01.428 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-04-17T19:46:01.637 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-17T19:46:01.789 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libllvm14:amd64. 2024-04-17T19:46:01.812 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-04-17T19:46:01.864 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-17T19:46:01.992 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-17T19:46:02.337 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-17T19:46:02.742 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package bison. 2024-04-17T19:46:02.824 INFO:teuthology.orchestra.run.smithi029.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140995 files and directories currently installed.) 2024-04-17T19:46:02.828 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-04-17T19:46:02.895 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-04-17T19:46:03.460 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-04-17T19:46:03.476 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-04-17T19:46:03.535 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-04-17T19:46:03.700 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-04-17T19:46:03.723 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-17T19:46:03.775 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-17T19:46:04.033 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package dump. 2024-04-17T19:46:04.050 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-04-17T19:46:04.095 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libmbim-proxy. 2024-04-17T19:46:04.117 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking dump (0.4b47-3) ... 2024-04-17T19:46:04.117 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-17T19:46:04.161 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-17T19:46:04.514 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-04-17T19:46:04.517 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package indent. 2024-04-17T19:46:04.533 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-04-17T19:46:04.536 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-17T19:46:04.589 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-17T19:46:04.592 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-17T19:46:04.864 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-04-17T19:46:04.880 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-04-17T19:46:04.923 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-17T19:46:05.025 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libndp0:amd64. 2024-04-17T19:46:05.048 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-04-17T19:46:05.100 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-17T19:46:05.272 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-04-17T19:46:05.289 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-04-17T19:46:05.331 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-17T19:46:05.520 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libnm0:amd64. 2024-04-17T19:46:05.542 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-04-17T19:46:05.594 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-17T19:46:05.662 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-04-17T19:46:05.678 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-04-17T19:46:05.721 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-04-17T19:46:06.078 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libfl2:amd64. 2024-04-17T19:46:06.095 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-04-17T19:46:06.137 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-04-17T19:46:06.316 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libpq5:amd64. 2024-04-17T19:46:06.339 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-17T19:46:06.382 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T19:46:06.383 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-04-17T19:46:06.400 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-04-17T19:46:06.442 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-17T19:46:06.707 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libjson-perl. 2024-04-17T19:46:06.723 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-04-17T19:46:06.765 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-04-17T19:46:06.810 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-04-17T19:46:06.832 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-17T19:46:06.885 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-17T19:46:07.075 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libllvm14:amd64. 2024-04-17T19:46:07.089 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-04-17T19:46:07.131 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-17T19:46:07.229 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libqmi-proxy. 2024-04-17T19:46:07.252 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-17T19:46:07.304 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-17T19:46:07.540 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-04-17T19:46:07.554 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-17T19:46:07.607 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-17T19:46:08.285 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-04-17T19:46:08.308 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-04-17T19:46:08.360 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-04-17T19:46:08.763 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package modemmanager. 2024-04-17T19:46:08.786 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-17T19:46:08.830 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-17T19:46:09.236 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-04-17T19:46:09.253 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-17T19:46:09.295 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-17T19:46:09.511 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package network-manager. 2024-04-17T19:46:09.534 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-04-17T19:46:09.579 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-04-17T19:46:09.601 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libmbim-proxy. 2024-04-17T19:46:09.617 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-17T19:46:09.660 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-17T19:46:09.991 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-04-17T19:46:10.008 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-17T19:46:10.058 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-17T19:46:10.219 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package ppp. 2024-04-17T19:46:10.242 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-04-17T19:46:10.430 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libndp0:amd64. 2024-04-17T19:46:10.446 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-04-17T19:46:10.488 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-17T19:46:10.524 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-04-17T19:46:10.828 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libnm0:amd64. 2024-04-17T19:46:10.832 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package pptp-linux. 2024-04-17T19:46:10.845 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-04-17T19:46:10.855 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-04-17T19:46:10.887 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-17T19:46:10.890 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-04-17T19:46:11.176 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package network-manager-pptp. 2024-04-17T19:46:11.199 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-04-17T19:46:11.242 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-04-17T19:46:11.276 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libpq5:amd64. 2024-04-17T19:46:11.292 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-17T19:46:11.336 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T19:46:11.707 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-04-17T19:46:11.724 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-17T19:46:11.766 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-17T19:46:12.021 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package postgresql-client-common. 2024-04-17T19:46:12.044 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-04-17T19:46:12.087 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking postgresql-client-common (238) ... 2024-04-17T19:46:12.089 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libqmi-proxy. 2024-04-17T19:46:12.105 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-17T19:46:12.148 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-17T19:46:12.393 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-04-17T19:46:12.409 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-17T19:46:12.424 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package postgresql-client-14. 2024-04-17T19:46:12.447 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-17T19:46:12.452 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-17T19:46:12.491 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T19:46:13.120 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package postgresql-common. 2024-04-17T19:46:13.126 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-04-17T19:46:13.143 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-04-17T19:46:13.143 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-04-17T19:46:13.185 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-04-17T19:46:13.206 INFO:teuthology.orchestra.run.smithi037.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-04-17T19:46:13.235 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking postgresql-common (238) ... 2024-04-17T19:46:13.542 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package modemmanager. 2024-04-17T19:46:13.558 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-17T19:46:13.600 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-17T19:46:13.669 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package postgresql-14. 2024-04-17T19:46:13.691 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-17T19:46:13.738 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T19:46:14.306 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package network-manager. 2024-04-17T19:46:14.322 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-04-17T19:46:14.372 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-04-17T19:46:15.072 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ppp. 2024-04-17T19:46:15.089 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-04-17T19:46:15.196 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package postgresql. 2024-04-17T19:46:15.211 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-04-17T19:46:15.255 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking postgresql (14+238) ... 2024-04-17T19:46:15.449 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-04-17T19:46:15.498 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package postgresql-client. 2024-04-17T19:46:15.512 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-04-17T19:46:15.548 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking postgresql-client (14+238) ... 2024-04-17T19:46:15.792 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package postgresql-contrib. 2024-04-17T19:46:15.799 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package pptp-linux. 2024-04-17T19:46:15.814 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-04-17T19:46:15.815 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-04-17T19:46:15.858 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-04-17T19:46:15.859 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking postgresql-contrib (14+238) ... 2024-04-17T19:46:16.156 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package network-manager-pptp. 2024-04-17T19:46:16.171 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-04-17T19:46:16.215 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-04-17T19:46:16.220 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package tcl8.6. 2024-04-17T19:46:16.243 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-04-17T19:46:16.286 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-17T19:46:16.511 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package postgresql-client-common. 2024-04-17T19:46:16.528 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-04-17T19:46:16.570 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking postgresql-client-common (238) ... 2024-04-17T19:46:16.597 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package tcl. 2024-04-17T19:46:16.620 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-04-17T19:46:16.668 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-04-17T19:46:16.887 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package postgresql-client-14. 2024-04-17T19:46:16.903 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-17T19:46:16.945 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T19:46:17.033 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package wpasupplicant. 2024-04-17T19:46:17.056 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-04-17T19:46:17.350 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-17T19:46:17.871 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-04-17T19:46:17.885 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-04-17T19:46:17.929 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-04-17T19:46:17.980 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package postgresql-common. 2024-04-17T19:46:17.997 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-04-17T19:46:18.058 INFO:teuthology.orchestra.run.smithi029.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-04-17T19:46:18.106 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking postgresql-common (238) ... 2024-04-17T19:46:18.291 INFO:teuthology.orchestra.run.smithi037.stdout:Selecting previously unselected package usb-modeswitch. 2024-04-17T19:46:18.313 INFO:teuthology.orchestra.run.smithi037.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-04-17T19:46:18.366 INFO:teuthology.orchestra.run.smithi037.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-17T19:46:18.597 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package postgresql-14. 2024-04-17T19:46:18.614 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-17T19:46:18.658 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T19:46:18.668 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up postgresql-client-common (238) ... 2024-04-17T19:46:18.888 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up flex (2.6.4-8build2) ... 2024-04-17T19:46:19.056 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-17T19:46:19.207 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T19:46:19.349 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-04-17T19:46:19.475 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-17T19:46:19.601 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T19:46:20.021 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package postgresql. 2024-04-17T19:46:20.037 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-04-17T19:46:20.080 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking postgresql (14+238) ... 2024-04-17T19:46:20.208 INFO:teuthology.orchestra.run.smithi037.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-04-17T19:46:20.310 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package postgresql-client. 2024-04-17T19:46:20.324 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-17T19:46:20.325 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-04-17T19:46:20.368 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking postgresql-client (14+238) ... 2024-04-17T19:46:20.631 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package postgresql-contrib. 2024-04-17T19:46:20.646 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-04-17T19:46:20.689 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking postgresql-contrib (14+238) ... 2024-04-17T19:46:20.713 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-17T19:46:20.713 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-17T19:46:21.020 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package tcl8.6. 2024-04-17T19:46:21.037 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-04-17T19:46:21.079 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-17T19:46:21.276 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-17T19:46:21.375 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package tcl. 2024-04-17T19:46:21.391 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-04-17T19:46:21.394 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-04-17T19:46:21.438 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-04-17T19:46:21.554 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-17T19:46:21.679 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-04-17T19:46:21.781 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package wpasupplicant. 2024-04-17T19:46:21.798 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-04-17T19:46:21.806 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-04-17T19:46:21.840 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-17T19:46:21.932 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-04-17T19:46:22.057 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-17T19:46:22.225 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-04-17T19:46:22.241 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-04-17T19:46:22.257 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-04-17T19:46:22.299 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-04-17T19:46:23.016 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-17T19:46:23.142 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up dump (0.4b47-3) ... 2024-04-17T19:46:23.235 INFO:teuthology.orchestra.run.smithi037.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-04-17T19:46:23.327 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-17T19:46:23.452 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-17T19:46:23.457 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package usb-modeswitch. 2024-04-17T19:46:23.474 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-04-17T19:46:23.524 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-17T19:46:23.578 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up tcl (8.6.11+1build2) ... 2024-04-17T19:46:23.704 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libjson-perl (4.04000-1) ... 2024-04-17T19:46:23.839 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-04-17T19:46:23.840 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up postgresql-client-common (238) ... 2024-04-17T19:46:23.945 INFO:teuthology.orchestra.run.smithi037.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-04-17T19:46:24.015 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-17T19:46:24.053 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up flex (2.6.4-8build2) ... 2024-04-17T19:46:24.141 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-17T19:46:24.180 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-17T19:46:24.308 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T19:46:24.317 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-17T19:46:24.435 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-17T19:46:24.436 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-04-17T19:46:24.562 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-17T19:46:24.578 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-04-17T19:46:24.689 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T19:46:24.746 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up postgresql-client (14+238) ... 2024-04-17T19:46:24.872 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-04-17T19:46:25.345 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:25.345 INFO:teuthology.orchestra.run.smithi037.stdout:The following network interfaces were found in /etc/network/interfaces 2024-04-17T19:46:25.345 INFO:teuthology.orchestra.run.smithi037.stdout:which means they are currently configured by ifupdown: 2024-04-17T19:46:25.345 INFO:teuthology.orchestra.run.smithi037.stdout:- ens1f0 2024-04-17T19:46:25.345 INFO:teuthology.orchestra.run.smithi037.stdout:If you want to manage those interfaces with NetworkManager instead 2024-04-17T19:46:25.345 INFO:teuthology.orchestra.run.smithi037.stdout:remove their configuration from /etc/network/interfaces. 2024-04-17T19:46:25.345 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:25.469 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-04-17T19:46:25.663 INFO:teuthology.orchestra.run.smithi029.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-04-17T19:46:25.769 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-17T19:46:25.829 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-04-17T19:46:26.157 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-17T19:46:26.157 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-17T19:46:26.180 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-04-17T19:46:26.758 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-17T19:46:26.873 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-17T19:46:26.893 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-17T19:46:27.002 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-04-17T19:46:27.069 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-04-17T19:46:27.195 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up postgresql-common (238) ... 2024-04-17T19:46:27.255 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-17T19:46:27.382 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-04-17T19:46:27.509 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-04-17T19:46:27.669 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-04-17T19:46:27.798 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-17T19:46:27.924 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-04-17T19:46:28.117 INFO:teuthology.orchestra.run.smithi037.stdout:Adding user postgres to group ssl-cert 2024-04-17T19:46:28.341 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:28.341 INFO:teuthology.orchestra.run.smithi037.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-04-17T19:46:28.513 INFO:teuthology.orchestra.run.smithi037.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-04-17T19:46:28.513 INFO:teuthology.orchestra.run.smithi037.stdout:Removing obsolete dictionary files: 2024-04-17T19:46:28.594 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-17T19:46:28.721 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up dump (0.4b47-3) ... 2024-04-17T19:46:28.817 INFO:teuthology.orchestra.run.smithi029.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-04-17T19:46:28.884 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-17T19:46:29.011 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-17T19:46:29.115 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-04-17T19:46:29.138 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up tcl (8.6.11+1build2) ... 2024-04-17T19:46:29.265 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libjson-perl (4.04000-1) ... 2024-04-17T19:46:29.391 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-04-17T19:46:29.483 INFO:teuthology.orchestra.run.smithi029.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-04-17T19:46:29.546 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T19:46:29.563 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-17T19:46:29.698 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-17T19:46:29.875 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-17T19:46:29.888 INFO:teuthology.orchestra.run.smithi037.stdout:Creating new PostgreSQL cluster 14/main ... 2024-04-17T19:46:29.911 INFO:teuthology.orchestra.run.smithi037.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-04-17T19:46:29.975 INFO:teuthology.orchestra.run.smithi037.stdout:The files belonging to this database system will be owned by user "postgres". 2024-04-17T19:46:29.975 INFO:teuthology.orchestra.run.smithi037.stdout:This user must also own the server process. 2024-04-17T19:46:29.975 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:29.975 INFO:teuthology.orchestra.run.smithi037.stdout:The database cluster will be initialized with locale "en_US". 2024-04-17T19:46:29.975 INFO:teuthology.orchestra.run.smithi037.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-04-17T19:46:29.975 INFO:teuthology.orchestra.run.smithi037.stdout:The default text search configuration will be set to "english". 2024-04-17T19:46:29.975 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:29.975 INFO:teuthology.orchestra.run.smithi037.stdout:Data page checksums are disabled. 2024-04-17T19:46:29.975 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:29.975 INFO:teuthology.orchestra.run.smithi037.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-04-17T19:46:29.975 INFO:teuthology.orchestra.run.smithi037.stdout:creating subdirectories ... ok 2024-04-17T19:46:29.975 INFO:teuthology.orchestra.run.smithi037.stdout:selecting dynamic shared memory implementation ... posix 2024-04-17T19:46:29.992 INFO:teuthology.orchestra.run.smithi037.stdout:selecting default max_connections ... 100 2024-04-17T19:46:30.004 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-17T19:46:30.015 INFO:teuthology.orchestra.run.smithi037.stdout:selecting default shared_buffers ... 128MB 2024-04-17T19:46:30.025 INFO:teuthology.orchestra.run.smithi037.stdout:selecting default time zone ... Etc/UTC 2024-04-17T19:46:30.026 INFO:teuthology.orchestra.run.smithi037.stdout:creating configuration files ... ok 2024-04-17T19:46:30.147 INFO:teuthology.orchestra.run.smithi037.stdout:running bootstrap script ... ok 2024-04-17T19:46:30.147 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-04-17T19:46:30.316 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up postgresql-client (14+238) ... 2024-04-17T19:46:30.493 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-04-17T19:46:30.764 INFO:teuthology.orchestra.run.smithi037.stdout:performing post-bootstrap initialization ... ok 2024-04-17T19:46:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:31.089 INFO:teuthology.orchestra.run.smithi029.stdout:The following network interfaces were found in /etc/network/interfaces 2024-04-17T19:46:31.089 INFO:teuthology.orchestra.run.smithi029.stdout:which means they are currently configured by ifupdown: 2024-04-17T19:46:31.089 INFO:teuthology.orchestra.run.smithi029.stdout:- ens1f0 2024-04-17T19:46:31.089 INFO:teuthology.orchestra.run.smithi029.stdout:If you want to manage those interfaces with NetworkManager instead 2024-04-17T19:46:31.089 INFO:teuthology.orchestra.run.smithi029.stdout:remove their configuration from /etc/network/interfaces. 2024-04-17T19:46:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:31.228 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-04-17T19:46:31.590 INFO:teuthology.orchestra.run.smithi037.stdout:syncing data to disk ... ok 2024-04-17T19:46:31.642 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-04-17T19:46:32.000 INFO:teuthology.orchestra.run.smithi037.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-04-17T19:46:32.062 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-04-17T19:46:32.809 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-17T19:46:32.961 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-17T19:46:33.089 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-04-17T19:46:33.216 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up postgresql-common (238) ... 2024-04-17T19:46:34.211 INFO:teuthology.orchestra.run.smithi029.stdout:Adding user postgres to group ssl-cert 2024-04-17T19:46:34.472 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:34.472 INFO:teuthology.orchestra.run.smithi029.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-04-17T19:46:34.742 INFO:teuthology.orchestra.run.smithi029.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-04-17T19:46:34.742 INFO:teuthology.orchestra.run.smithi029.stdout:Removing obsolete dictionary files: 2024-04-17T19:46:34.810 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-17T19:46:34.937 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up postgresql-contrib (14+238) ... 2024-04-17T19:46:35.096 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up postgresql (14+238) ... 2024-04-17T19:46:35.248 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-17T19:46:35.398 INFO:teuthology.orchestra.run.smithi037.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-17T19:46:35.470 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-04-17T19:46:35.681 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-04-17T19:46:35.681 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-04-17T19:46:36.179 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T19:46:36.268 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-17T19:46:36.393 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-04-17T19:46:36.529 INFO:teuthology.orchestra.run.smithi029.stdout:Creating new PostgreSQL cluster 14/main ... 2024-04-17T19:46:36.543 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-04-17T19:46:36.612 INFO:teuthology.orchestra.run.smithi029.stdout:The files belonging to this database system will be owned by user "postgres". 2024-04-17T19:46:36.612 INFO:teuthology.orchestra.run.smithi029.stdout:This user must also own the server process. 2024-04-17T19:46:36.612 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:36.612 INFO:teuthology.orchestra.run.smithi029.stdout:The database cluster will be initialized with locale "en_US". 2024-04-17T19:46:36.612 INFO:teuthology.orchestra.run.smithi029.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-04-17T19:46:36.612 INFO:teuthology.orchestra.run.smithi029.stdout:The default text search configuration will be set to "english". 2024-04-17T19:46:36.612 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:36.612 INFO:teuthology.orchestra.run.smithi029.stdout:Data page checksums are disabled. 2024-04-17T19:46:36.612 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:36.612 INFO:teuthology.orchestra.run.smithi029.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-04-17T19:46:36.613 INFO:teuthology.orchestra.run.smithi029.stdout:creating subdirectories ... ok 2024-04-17T19:46:36.613 INFO:teuthology.orchestra.run.smithi029.stdout:selecting dynamic shared memory implementation ... posix 2024-04-17T19:46:36.638 INFO:teuthology.orchestra.run.smithi029.stdout:selecting default max_connections ... 100 2024-04-17T19:46:36.672 INFO:teuthology.orchestra.run.smithi029.stdout:selecting default shared_buffers ... 128MB 2024-04-17T19:46:36.686 INFO:teuthology.orchestra.run.smithi029.stdout:selecting default time zone ... Etc/UTC 2024-04-17T19:46:36.687 INFO:teuthology.orchestra.run.smithi029.stdout:creating configuration files ... ok 2024-04-17T19:46:36.728 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-17T19:46:36.863 INFO:teuthology.orchestra.run.smithi029.stdout:running bootstrap script ... ok 2024-04-17T19:46:36.908 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-17T19:46:37.575 INFO:teuthology.orchestra.run.smithi029.stdout:performing post-bootstrap initialization ... ok 2024-04-17T19:46:38.334 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-04-17T19:46:38.458 INFO:teuthology.orchestra.run.smithi037.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-17T19:46:38.482 INFO:teuthology.orchestra.run.smithi029.stdout:syncing data to disk ... ok 2024-04-17T19:46:39.017 INFO:teuthology.orchestra.run.smithi037.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-17T19:46:39.017 INFO:teuthology.orchestra.run.smithi029.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-04-17T19:46:39.023 DEBUG:teuthology.parallel:result is None 2024-04-17T19:46:42.112 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-17T19:46:42.238 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up postgresql-contrib (14+238) ... 2024-04-17T19:46:42.373 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up postgresql (14+238) ... 2024-04-17T19:46:42.500 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-17T19:46:42.627 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-17T19:46:42.898 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-04-17T19:46:42.898 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-04-17T19:46:43.658 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-17T19:46:43.759 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-04-17T19:46:44.131 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-17T19:46:44.326 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-17T19:46:45.908 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-04-17T19:46:46.017 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-17T19:46:46.626 INFO:teuthology.orchestra.run.smithi029.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-17T19:46:46.632 DEBUG:teuthology.parallel:result is None 2024-04-17T19:46:46.632 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f3a98d3f110319c567ec86bf70955c141495ccf6 2024-04-17T19:46:46.778 DEBUG:teuthology.orchestra.run.smithi029:> dpkg-query -W -f '${Version}' ceph 2024-04-17T19:46:46.800 INFO:teuthology.orchestra.run.smithi029.stdout:19.0.0-3146-gf3a98d3f-1jammy 2024-04-17T19:46:46.801 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3146-gf3a98d3f-1jammy 2024-04-17T19:46:46.801 INFO:teuthology.task.install:The correct ceph version 19.0.0-3146-gf3a98d3f-1jammy is installed. 2024-04-17T19:46:46.803 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f3a98d3f110319c567ec86bf70955c141495ccf6 2024-04-17T19:46:47.027 DEBUG:teuthology.orchestra.run.smithi037:> dpkg-query -W -f '${Version}' ceph 2024-04-17T19:46:47.047 INFO:teuthology.orchestra.run.smithi037.stdout:19.0.0-3146-gf3a98d3f-1jammy 2024-04-17T19:46:47.047 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3146-gf3a98d3f-1jammy 2024-04-17T19:46:47.047 INFO:teuthology.task.install:The correct ceph version 19.0.0-3146-gf3a98d3f-1jammy is installed. 2024-04-17T19:46:47.049 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-17T19:46:47.049 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:46:47.049 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-17T19:46:47.070 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:46:47.070 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-17T19:46:47.103 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-17T19:46:47.103 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:46:47.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2024-04-17T19:46:47.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-17T19:46:47.199 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:46:47.199 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/daemon-helper 2024-04-17T19:46:47.213 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-17T19:46:47.270 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-17T19:46:47.270 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:46:47.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-17T19:46:47.289 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-17T19:46:47.355 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:46:47.355 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-17T19:46:47.369 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-17T19:46:47.425 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-17T19:46:47.426 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:46:47.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2024-04-17T19:46:47.445 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-17T19:46:47.511 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:46:47.511 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/stdin-killer 2024-04-17T19:46:47.525 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-17T19:46:47.582 INFO:teuthology.run_tasks:Running task ceph... 2024-04-17T19:46:47.665 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-17T19:46:47.665 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /var/log/ceph 2024-04-17T19:46:47.668 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 777 /var/log/ceph 2024-04-17T19:46:47.685 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-17T19:46:47.685 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-17T19:46:47.735 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-17T19:46:47.751 INFO:tasks.ceph:Creating extra log directories... 2024-04-17T19:46:47.752 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-17T19:46:47.795 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-17T19:46:47.819 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-17T19:46:47.819 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-17T19:46:47.819 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-17T19:46:47.819 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:46:47.819 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-17T19:46:47.868 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-17T19:46:47.964 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:46:47.965 DEBUG:teuthology.orchestra.run.smithi037:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-17T19:46:47.970 DEBUG:teuthology.orchestra.run.smithi037:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-17T19:46:48.051 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-17T19:46:48.052 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-17T19:46:48.052 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-04-17_17:06:06-fs-wip-pdonnell-testing-20240417.021458-debug-distro-default-smithi/7660566', 'branch': 'wip-pdonnell-testing-20240417.021458-debug', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7660566', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'pdonnell-2024-04-17_17:06:06-fs-wip-pdonnell-testing-20240417.021458-debug-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20240417.021458-debug'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'Replacing daemon mds'], 'sha1': 'f3a98d3f110319c567ec86bf70955c141495ccf6'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f3a98d3f110319c567ec86bf70955c141495ccf6'}}, 'kclient': {'syntax': 'v1'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20240417.021458-debug', 'sha1': 'f3a98d3f110319c567ec86bf70955c141495ccf6'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 80, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 4494, 'sha1': 'f3a98d3f110319c567ec86bf70955c141495ccf6', 'sleep_before_teardown': 0, 'subset': '496/512', 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20240417.021458-debug', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_f3a98d3f110319c567ec86bf70955c141495ccf6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'f3a98d3f110319c567ec86bf70955c141495ccf6', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi029.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDm42Cu+zPyGmJHkLKzD0ySzoi04P1QY9nWBJ94YKWJ7BQFVm0GmklFrtakxdR+DcokzKXLG5p7f3ZJusv/XSSQ=', 'smithi037.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKJdJX3KscOjByZ1RIC3C9USJdBQlNeoTTCYVqNOfcu/LA1THBL7/idc3lx9iF9y2GKJOFUAL4DeUNUORXpiBuk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'f3a98d3f110319c567ec86bf70955c141495ccf6'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'Replacing daemon mds'], 'sha1': 'f3a98d3f110319c567ec86bf70955c141495ccf6', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-17_17:06:06', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-17T19:46:48.052 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-17T19:46:48.056 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-17T19:46:48.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/run/ceph 2024-04-17T19:46:48.107 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m0777 -- /var/run/ceph 2024-04-17T19:46:48.154 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:46:48.155 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-04-17T19:46:48.176 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-17T19:46:48.177 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2024-04-17T19:46:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-17T19:46:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T19:46:48.229 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 6h/6d Inode: 892 Links: 1 2024-04-17T19:46:48.229 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T19:46:48.229 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-17 19:38:59.878657173 +0000 2024-04-17T19:46:48.229 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-17 19:38:59.874657208 +0000 2024-04-17T19:46:48.229 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-17 19:38:59.874657208 +0000 2024-04-17T19:46:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-17 19:38:59.874657208 +0000 2024-04-17T19:46:48.229 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-17T19:46:48.290 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-17T19:46:48.291 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-17T19:46:48.291 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000250701 s, 2.0 MB/s 2024-04-17T19:46:48.292 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-17T19:46:48.346 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2024-04-17T19:46:48.401 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-17T19:46:48.401 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T19:46:48.401 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 6h/6d Inode: 909 Links: 1 2024-04-17T19:46:48.401 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T19:46:48.401 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-17 19:39:00.666650134 +0000 2024-04-17T19:46:48.401 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-17 19:39:00.418652350 +0000 2024-04-17T19:46:48.401 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-17 19:39:00.418652350 +0000 2024-04-17T19:46:48.401 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-17 19:39:00.418652350 +0000 2024-04-17T19:46:48.401 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-17T19:46:48.462 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-17T19:46:48.463 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-17T19:46:48.463 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000247015 s, 2.1 MB/s 2024-04-17T19:46:48.464 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-17T19:46:48.522 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2024-04-17T19:46:48.576 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-17T19:46:48.577 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T19:46:48.577 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 6h/6d Inode: 924 Links: 1 2024-04-17T19:46:48.577 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T19:46:48.577 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-17 19:39:01.286644597 +0000 2024-04-17T19:46:48.577 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-17 19:39:01.006647098 +0000 2024-04-17T19:46:48.577 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-17 19:39:01.006647098 +0000 2024-04-17T19:46:48.577 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-17 19:39:01.006647098 +0000 2024-04-17T19:46:48.577 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-17T19:46:48.643 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-17T19:46:48.643 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-17T19:46:48.643 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000261413 s, 2.0 MB/s 2024-04-17T19:46:48.645 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-17T19:46:48.703 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2024-04-17T19:46:48.757 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-17T19:46:48.757 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T19:46:48.757 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 6h/6d Inode: 941 Links: 1 2024-04-17T19:46:48.757 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T19:46:48.757 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-04-17 19:39:01.866639416 +0000 2024-04-17T19:46:48.758 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-17 19:39:01.574642024 +0000 2024-04-17T19:46:48.758 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-17 19:39:01.574642024 +0000 2024-04-17T19:46:48.758 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-17 19:39:01.574642024 +0000 2024-04-17T19:46:48.758 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-17T19:46:48.823 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-04-17T19:46:48.823 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-04-17T19:46:48.823 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000262656 s, 1.9 MB/s 2024-04-17T19:46:48.825 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-17T19:46:48.883 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-17T19:46:48.883 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:46:48.883 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/scratch_devs of=/dev/stdout 2024-04-17T19:46:48.889 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-17T19:46:48.889 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_1 2024-04-17T19:46:48.938 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-17T19:46:48.938 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T19:46:48.939 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 887 Links: 1 2024-04-17T19:46:48.939 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T19:46:48.939 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-04-17 19:38:38.628998349 +0000 2024-04-17T19:46:48.939 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-04-17 19:38:38.365001442 +0000 2024-04-17T19:46:48.939 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-04-17 19:38:38.365001442 +0000 2024-04-17T19:46:48.939 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-04-17T19:46:48.939 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-17T19:46:48.993 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-04-17T19:46:48.994 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-04-17T19:46:48.994 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000536077 s, 955 kB/s 2024-04-17T19:46:48.995 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-17T19:46:49.048 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_2 2024-04-17T19:46:49.095 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-17T19:46:49.095 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T19:46:49.096 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 902 Links: 1 2024-04-17T19:46:49.096 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T19:46:49.096 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-04-17 19:38:39.144992304 +0000 2024-04-17T19:46:49.096 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-04-17 19:38:38.864995583 +0000 2024-04-17T19:46:49.096 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-04-17 19:38:38.864995583 +0000 2024-04-17T19:46:49.096 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-04-17T19:46:49.097 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-17T19:46:49.149 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-04-17T19:46:49.149 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-04-17T19:46:49.150 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000518337 s, 988 kB/s 2024-04-17T19:46:49.151 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-17T19:46:49.199 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_3 2024-04-17T19:46:49.251 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-17T19:46:49.251 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T19:46:49.251 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 913 Links: 1 2024-04-17T19:46:49.251 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T19:46:49.252 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-04-17 19:38:39.672986123 +0000 2024-04-17T19:46:49.252 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-04-17 19:38:39.404989261 +0000 2024-04-17T19:46:49.252 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-04-17 19:38:39.404989261 +0000 2024-04-17T19:46:49.252 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-04-17T19:46:49.252 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-17T19:46:49.308 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-04-17T19:46:49.309 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-04-17T19:46:49.309 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000473893 s, 1.1 MB/s 2024-04-17T19:46:49.310 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-17T19:46:49.360 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_4 2024-04-17T19:46:49.406 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-17T19:46:49.406 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T19:46:49.407 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 938 Links: 1 2024-04-17T19:46:49.407 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T19:46:49.407 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-04-17 19:38:40.180980180 +0000 2024-04-17T19:46:49.407 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-04-17 19:38:39.900983455 +0000 2024-04-17T19:46:49.407 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-04-17 19:38:39.900983455 +0000 2024-04-17T19:46:49.407 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: - 2024-04-17T19:46:49.407 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-17T19:46:49.462 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-04-17T19:46:49.462 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-04-17T19:46:49.462 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000410899 s, 1.2 MB/s 2024-04-17T19:46:49.463 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-17T19:46:49.511 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-17T19:46:49.511 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi037.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-17T19:46:49.511 INFO:tasks.ceph:Generating config... 2024-04-17T19:46:49.512 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-17T19:46:49.512 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-04-17T19:46:49.512 INFO:tasks.ceph:[client] debug client = 20 2024-04-17T19:46:49.513 INFO:tasks.ceph:[client] debug ms = 1 2024-04-17T19:46:49.513 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-17T19:46:49.513 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-17T19:46:49.513 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-04-17T19:46:49.513 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-17T19:46:49.513 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-17T19:46:49.513 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-17T19:46:49.513 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2024-04-17T19:46:49.513 INFO:tasks.ceph:[mds] mds bal merge size = 5 2024-04-17T19:46:49.513 INFO:tasks.ceph:[mds] mds bal split bits = 3 2024-04-17T19:46:49.513 INFO:tasks.ceph:[mds] mds bal split size = 100 2024-04-17T19:46:49.513 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-17T19:46:49.513 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-17T19:46:49.513 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2024-04-17T19:46:49.513 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-17T19:46:49.514 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-17T19:46:49.514 INFO:tasks.ceph:[mds] mds_dir_prefetch = True 2024-04-17T19:46:49.514 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2024-04-17T19:46:49.514 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2024-04-17T19:46:49.514 INFO:tasks.ceph:[mds] ms inject delay max = 1 2024-04-17T19:46:49.514 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2024-04-17T19:46:49.514 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2024-04-17T19:46:49.514 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2024-04-17T19:46:49.514 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-17T19:46:49.514 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-17T19:46:49.514 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-17T19:46:49.514 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-17T19:46:49.514 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-17T19:46:49.514 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-17T19:46:49.514 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-17T19:46:49.514 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-17T19:46:49.515 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-17T19:46:49.515 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-17T19:46:49.515 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-17T19:46:49.515 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-17T19:46:49.515 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-17T19:46:49.515 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-17T19:46:49.515 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-17T19:46:49.515 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-17T19:46:49.515 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-17T19:46:49.515 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-17T19:46:49.515 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2024-04-17T19:46:49.515 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-17T19:46:49.515 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-17T19:46:49.515 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-17T19:46:49.515 INFO:tasks.ceph:[osd] ms inject delay max = 1 2024-04-17T19:46:49.516 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2024-04-17T19:46:49.516 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2024-04-17T19:46:49.516 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2024-04-17T19:46:49.516 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-17T19:46:49.516 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-17T19:46:49.516 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-17T19:46:49.516 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-17T19:46:49.516 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-17T19:46:49.516 INFO:tasks.ceph:Setting up mon.a... 2024-04-17T19:46:49.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-17T19:46:49.576 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.keyring 2024-04-17T19:46:49.589 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-17T19:46:49.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-17T19:46:49.670 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.29'), ('mon.b', '172.21.15.37'), ('mon.c', '[v2:172.21.15.37:3301,v1:172.21.15.37:6790]')] 2024-04-17T19:46:49.670 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.29,172.21.15.37,[v2:172.21.15.37:3301,v1:172.21.15.37:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-17T19:46:49.671 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:46:49.671 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-17T19:46:49.724 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.29 --add b 172.21.15.37 --addv c '[v2:172.21.15.37:3301,v1:172.21.15.37:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-17T19:46:49.827 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-17T19:46:49.827 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: generated fsid a680ee90-875c-4283-a901-86d7638393c3 2024-04-17T19:46:49.827 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = quincy 2024-04-17T19:46:49.827 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 0 2024-04-17T19:46:49.827 INFO:teuthology.orchestra.run.smithi029.stdout:fsid a680ee90-875c-4283-a901-86d7638393c3 2024-04-17T19:46:49.827 INFO:teuthology.orchestra.run.smithi029.stdout:last_changed 2024-04-17T19:46:49.841084+0000 2024-04-17T19:46:49.827 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-04-17T19:46:49.841084+0000 2024-04-17T19:46:49.827 INFO:teuthology.orchestra.run.smithi029.stdout:min_mon_release 17 (quincy) 2024-04-17T19:46:49.827 INFO:teuthology.orchestra.run.smithi029.stdout:election_strategy: 1 2024-04-17T19:46:49.827 INFO:teuthology.orchestra.run.smithi029.stdout:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2024-04-17T19:46:49.827 INFO:teuthology.orchestra.run.smithi029.stdout:1: [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] mon.b 2024-04-17T19:46:49.828 INFO:teuthology.orchestra.run.smithi029.stdout:2: [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] mon.c 2024-04-17T19:46:49.828 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-17T19:46:49.831 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-17T19:46:49.839 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a680ee90-875c-4283-a901-86d7638393c3... 2024-04-17T19:46:49.839 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-17T19:46:49.883 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-17T19:46:49.915 INFO:teuthology.orchestra.run.smithi037.stdout:[global] 2024-04-17T19:46:49.917 INFO:teuthology.orchestra.run.smithi037.stdout: chdir = "" 2024-04-17T19:46:49.917 INFO:teuthology.orchestra.run.smithi037.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-17T19:46:49.917 INFO:teuthology.orchestra.run.smithi037.stdout: auth supported = cephx 2024-04-17T19:46:49.918 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.918 INFO:teuthology.orchestra.run.smithi037.stdout: filestore xattr use omap = true 2024-04-17T19:46:49.918 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.918 INFO:teuthology.orchestra.run.smithi037.stdout: mon clock drift allowed = 1.000 2024-04-17T19:46:49.918 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.918 INFO:teuthology.orchestra.run.smithi037.stdout: osd crush chooseleaf type = 0 2024-04-17T19:46:49.918 INFO:teuthology.orchestra.run.smithi037.stdout: auth debug = true 2024-04-17T19:46:49.918 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.918 INFO:teuthology.orchestra.run.smithi037.stdout: ms die on old message = true 2024-04-17T19:46:49.918 INFO:teuthology.orchestra.run.smithi037.stdout: ms die on bug = true 2024-04-17T19:46:49.918 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.918 INFO:teuthology.orchestra.run.smithi037.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-17T19:46:49.918 INFO:teuthology.orchestra.run.smithi037.stdout: mon pg warn max object skew = 0 2024-04-17T19:46:49.918 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.919 INFO:teuthology.orchestra.run.smithi037.stdout: # disable pg_autoscaler by default for new pools 2024-04-17T19:46:49.919 INFO:teuthology.orchestra.run.smithi037.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-17T19:46:49.919 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.919 INFO:teuthology.orchestra.run.smithi037.stdout: osd pool default size = 2 2024-04-17T19:46:49.919 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.919 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd allow primary affinity = true 2024-04-17T19:46:49.919 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd allow pg remap = true 2024-04-17T19:46:49.919 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on legacy crush tunables = false 2024-04-17T19:46:49.919 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on crush straw calc version zero = false 2024-04-17T19:46:49.919 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on no sortbitwise = false 2024-04-17T19:46:49.919 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on osd down out interval zero = false 2024-04-17T19:46:49.919 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on too few osds = false 2024-04-17T19:46:49.919 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-17T19:46:49.919 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-17T19:46:49.919 INFO:teuthology.orchestra.run.smithi037.stdout: mon_allow_pool_size_one = true 2024-04-17T19:46:49.920 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.920 INFO:teuthology.orchestra.run.smithi037.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-17T19:46:49.920 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.920 INFO:teuthology.orchestra.run.smithi037.stdout: osd default data pool replay window = 5 2024-04-17T19:46:49.920 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.920 INFO:teuthology.orchestra.run.smithi037.stdout: mon allow pool delete = true 2024-04-17T19:46:49.920 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.920 INFO:teuthology.orchestra.run.smithi037.stdout: mon cluster log file level = debug 2024-04-17T19:46:49.920 INFO:teuthology.orchestra.run.smithi037.stdout: debug asserts on shutdown = true 2024-04-17T19:46:49.920 INFO:teuthology.orchestra.run.smithi037.stdout: mon health detail to clog = false 2024-04-17T19:46:49.920 INFO:teuthology.orchestra.run.smithi037.stdout: mon host = "172.21.15.29,172.21.15.37,[v2:172.21.15.37:3301,v1:172.21.15.37:6790]" 2024-04-17T19:46:49.920 INFO:teuthology.orchestra.run.smithi037.stdout: ms die on skipped message = False 2024-04-17T19:46:49.920 INFO:teuthology.orchestra.run.smithi037.stdout: fsid = a680ee90-875c-4283-a901-86d7638393c3 2024-04-17T19:46:49.921 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.921 INFO:teuthology.orchestra.run.smithi037.stdout:[osd] 2024-04-17T19:46:49.921 INFO:teuthology.orchestra.run.smithi037.stdout: osd journal size = 100 2024-04-17T19:46:49.921 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.921 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub load threshold = 5.0 2024-04-17T19:46:49.921 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub max interval = 600 2024-04-17T19:46:49.921 INFO:teuthology.orchestra.run.smithi037.stdout: osd mclock profile = high_recovery_ops 2024-04-17T19:46:49.921 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.921 INFO:teuthology.orchestra.run.smithi037.stdout: osd recover clone overlap = true 2024-04-17T19:46:49.921 INFO:teuthology.orchestra.run.smithi037.stdout: osd recovery max chunk = 1048576 2024-04-17T19:46:49.921 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.921 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug shutdown = true 2024-04-17T19:46:49.921 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug op order = true 2024-04-17T19:46:49.921 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify stray on activate = true 2024-04-17T19:46:49.922 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug trim objects = true 2024-04-17T19:46:49.922 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.922 INFO:teuthology.orchestra.run.smithi037.stdout: osd open classes on start = true 2024-04-17T19:46:49.922 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug pg log writeout = true 2024-04-17T19:46:49.922 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.922 INFO:teuthology.orchestra.run.smithi037.stdout: osd deep scrub update digest min age = 30 2024-04-17T19:46:49.922 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.922 INFO:teuthology.orchestra.run.smithi037.stdout: osd map max advance = 10 2024-04-17T19:46:49.922 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.922 INFO:teuthology.orchestra.run.smithi037.stdout: journal zero on create = true 2024-04-17T19:46:49.922 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.922 INFO:teuthology.orchestra.run.smithi037.stdout: filestore ondisk finisher threads = 3 2024-04-17T19:46:49.923 INFO:teuthology.orchestra.run.smithi037.stdout: filestore apply finisher threads = 3 2024-04-17T19:46:49.923 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.923 INFO:teuthology.orchestra.run.smithi037.stdout: bdev debug aio = true 2024-04-17T19:46:49.923 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug misdirected ops = true 2024-04-17T19:46:49.923 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore block size = 96636764160 2024-04-17T19:46:49.923 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore fsck on mount = True 2024-04-17T19:46:49.923 INFO:teuthology.orchestra.run.smithi037.stdout: debug bluefs = 20 2024-04-17T19:46:49.923 INFO:teuthology.orchestra.run.smithi037.stdout: debug bluestore = 20 2024-04-17T19:46:49.923 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-04-17T19:46:49.923 INFO:teuthology.orchestra.run.smithi037.stdout: debug osd = 20 2024-04-17T19:46:49.923 INFO:teuthology.orchestra.run.smithi037.stdout: debug rocksdb = 10 2024-04-17T19:46:49.923 INFO:teuthology.orchestra.run.smithi037.stdout: mon client directed command retry = 5 2024-04-17T19:46:49.923 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd backfillfull_ratio = 0.85 2024-04-17T19:46:49.923 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd full ratio = 0.9 2024-04-17T19:46:49.923 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd nearfull ratio = 0.8 2024-04-17T19:46:49.924 INFO:teuthology.orchestra.run.smithi037.stdout: ms inject delay max = 1 2024-04-17T19:46:49.924 INFO:teuthology.orchestra.run.smithi037.stdout: ms inject delay probability = 0.005 2024-04-17T19:46:49.924 INFO:teuthology.orchestra.run.smithi037.stdout: ms inject delay type = client mds 2024-04-17T19:46:49.924 INFO:teuthology.orchestra.run.smithi037.stdout: ms inject socket failures = 2500 2024-04-17T19:46:49.924 INFO:teuthology.orchestra.run.smithi037.stdout: osd blocked scrub grace period = 3600 2024-04-17T19:46:49.924 INFO:teuthology.orchestra.run.smithi037.stdout: osd failsafe full ratio = 0.95 2024-04-17T19:46:49.924 INFO:teuthology.orchestra.run.smithi037.stdout: osd max markdown count = 1000 2024-04-17T19:46:49.924 INFO:teuthology.orchestra.run.smithi037.stdout: osd objectstore = bluestore 2024-04-17T19:46:49.924 INFO:teuthology.orchestra.run.smithi037.stdout: osd op complaint time = 180 2024-04-17T19:46:49.924 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.924 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr] 2024-04-17T19:46:49.924 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-04-17T19:46:49.924 INFO:teuthology.orchestra.run.smithi037.stdout: debug mgr = 20 2024-04-17T19:46:49.924 INFO:teuthology.orchestra.run.smithi037.stdout: debug mon = 20 2024-04-17T19:46:49.925 INFO:teuthology.orchestra.run.smithi037.stdout: debug auth = 20 2024-04-17T19:46:49.925 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min pgs per osd = 4 2024-04-17T19:46:49.925 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min bytes per osd = 10 2024-04-17T19:46:49.925 INFO:teuthology.orchestra.run.smithi037.stdout: mgr/telemetry/nag = false 2024-04-17T19:46:49.925 INFO:teuthology.orchestra.run.smithi037.stdout: debug client = 20 2024-04-17T19:46:49.925 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.925 INFO:teuthology.orchestra.run.smithi037.stdout:[mon] 2024-04-17T19:46:49.925 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-04-17T19:46:49.925 INFO:teuthology.orchestra.run.smithi037.stdout: debug mon = 20 2024-04-17T19:46:49.925 INFO:teuthology.orchestra.run.smithi037.stdout: debug paxos = 20 2024-04-17T19:46:49.925 INFO:teuthology.orchestra.run.smithi037.stdout: debug auth = 20 2024-04-17T19:46:49.925 INFO:teuthology.orchestra.run.smithi037.stdout: mon data avail warn = 5 2024-04-17T19:46:49.925 INFO:teuthology.orchestra.run.smithi037.stdout: mon mgr mkfs grace = 240 2024-04-17T19:46:49.925 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min pgs per osd = 4 2024-04-17T19:46:49.925 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd reporter subtree level = osd 2024-04-17T19:46:49.926 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd prime pg temp = true 2024-04-17T19:46:49.926 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min bytes per osd = 10 2024-04-17T19:46:49.926 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.926 INFO:teuthology.orchestra.run.smithi037.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-17T19:46:49.926 INFO:teuthology.orchestra.run.smithi037.stdout: auth mon ticket ttl = 660 # 11m 2024-04-17T19:46:49.926 INFO:teuthology.orchestra.run.smithi037.stdout: auth service ticket ttl = 240 # 4m 2024-04-17T19:46:49.926 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.926 INFO:teuthology.orchestra.run.smithi037.stdout: # don't complain about insecure global_id in the test suite 2024-04-17T19:46:49.926 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-17T19:46:49.926 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-17T19:46:49.926 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.926 INFO:teuthology.orchestra.run.smithi037.stdout: # 1m isn't quite enough 2024-04-17T19:46:49.926 INFO:teuthology.orchestra.run.smithi037.stdout: mon_down_mkfs_grace = 2m 2024-04-17T19:46:49.926 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.927 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_filestore_osds = false 2024-04-17T19:46:49.927 INFO:teuthology.orchestra.run.smithi037.stdout: mon down mkfs grace = 300 2024-04-17T19:46:49.927 INFO:teuthology.orchestra.run.smithi037.stdout: mon op complaint time = 120 2024-04-17T19:46:49.927 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-04-17T19:46:49.927 INFO:teuthology.orchestra.run.smithi037.stdout:[client] 2024-04-17T19:46:49.927 INFO:teuthology.orchestra.run.smithi037.stdout: rgw cache enabled = true 2024-04-17T19:46:49.927 INFO:teuthology.orchestra.run.smithi037.stdout: rgw enable ops log = true 2024-04-17T19:46:49.927 INFO:teuthology.orchestra.run.smithi037.stdout: rgw enable usage log = true 2024-04-17T19:46:49.927 INFO:teuthology.orchestra.run.smithi037.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-17T19:46:49.927 INFO:teuthology.orchestra.run.smithi037.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-17T19:46:49.927 INFO:teuthology.orchestra.run.smithi037.stdout: client mount timeout = 600 2024-04-17T19:46:49.927 INFO:teuthology.orchestra.run.smithi037.stdout: client_shutdown_timeout = 120 2024-04-17T19:46:49.927 INFO:teuthology.orchestra.run.smithi037.stdout: debug client = 20 2024-04-17T19:46:49.927 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-04-17T19:46:49.927 INFO:teuthology.orchestra.run.smithi037.stdout: rados mon op timeout = 900 2024-04-17T19:46:49.928 INFO:teuthology.orchestra.run.smithi037.stdout: rados osd op timeout = 900 2024-04-17T19:46:49.928 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.a] 2024-04-17T19:46:49.928 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.b] 2024-04-17T19:46:49.928 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.c] 2024-04-17T19:46:49.928 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.a] 2024-04-17T19:46:49.928 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.c] 2024-04-17T19:46:49.928 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.e] 2024-04-17T19:46:49.928 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.b] 2024-04-17T19:46:49.928 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.d] 2024-04-17T19:46:49.928 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.f] 2024-04-17T19:46:49.928 INFO:teuthology.orchestra.run.smithi037.stdout:[mds] 2024-04-17T19:46:49.928 INFO:teuthology.orchestra.run.smithi037.stdout: debug mds = 20 2024-04-17T19:46:49.928 INFO:teuthology.orchestra.run.smithi037.stdout: debug mds balancer = 20 2024-04-17T19:46:49.928 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-04-17T19:46:49.928 INFO:teuthology.orchestra.run.smithi037.stdout: mds bal fragment size max = 10000 2024-04-17T19:46:49.928 INFO:teuthology.orchestra.run.smithi037.stdout: mds bal merge size = 5 2024-04-17T19:46:49.929 INFO:teuthology.orchestra.run.smithi037.stdout: mds bal split bits = 3 2024-04-17T19:46:49.929 INFO:teuthology.orchestra.run.smithi037.stdout: mds bal split size = 100 2024-04-17T19:46:49.929 INFO:teuthology.orchestra.run.smithi037.stdout: mds debug frag = True 2024-04-17T19:46:49.929 INFO:teuthology.orchestra.run.smithi037.stdout: mds debug scatterstat = True 2024-04-17T19:46:49.929 INFO:teuthology.orchestra.run.smithi037.stdout: mds oft prefetch dirfrags = True 2024-04-17T19:46:49.929 INFO:teuthology.orchestra.run.smithi037.stdout: mds op complaint time = 180 2024-04-17T19:46:49.929 INFO:teuthology.orchestra.run.smithi037.stdout: mds verify scatter = True 2024-04-17T19:46:49.929 INFO:teuthology.orchestra.run.smithi037.stdout: mds_dir_prefetch = True 2024-04-17T19:46:49.929 INFO:teuthology.orchestra.run.smithi037.stdout: mds_sleep_rank_change = 5000000.0 2024-04-17T19:46:49.929 INFO:teuthology.orchestra.run.smithi037.stdout: mon client directed command retry = 5 2024-04-17T19:46:49.929 INFO:teuthology.orchestra.run.smithi037.stdout: ms inject delay max = 1 2024-04-17T19:46:49.929 INFO:teuthology.orchestra.run.smithi037.stdout: ms inject delay probability = 0.005 2024-04-17T19:46:49.929 INFO:teuthology.orchestra.run.smithi037.stdout: ms inject delay type = client mds osd 2024-04-17T19:46:49.929 INFO:teuthology.orchestra.run.smithi037.stdout: ms inject socket failures = 2500 2024-04-17T19:46:49.929 INFO:teuthology.orchestra.run.smithi037.stdout: osd op complaint time = 180 2024-04-17T19:46:49.929 INFO:teuthology.orchestra.run.smithi037.stdout: rados mon op timeout = 900 2024-04-17T19:46:49.930 INFO:teuthology.orchestra.run.smithi037.stdout: rados osd op timeout = 900 2024-04-17T19:46:49.930 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2024-04-17T19:46:49.930 INFO:teuthology.orchestra.run.smithi029.stdout: chdir = "" 2024-04-17T19:46:49.930 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-17T19:46:49.930 INFO:teuthology.orchestra.run.smithi029.stdout: auth supported = cephx 2024-04-17T19:46:49.930 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.930 INFO:teuthology.orchestra.run.smithi029.stdout: filestore xattr use omap = true 2024-04-17T19:46:49.931 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.931 INFO:teuthology.orchestra.run.smithi029.stdout: mon clock drift allowed = 1.000 2024-04-17T19:46:49.931 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.931 INFO:teuthology.orchestra.run.smithi029.stdout: osd crush chooseleaf type = 0 2024-04-17T19:46:49.931 INFO:teuthology.orchestra.run.smithi029.stdout: auth debug = true 2024-04-17T19:46:49.931 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.931 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on old message = true 2024-04-17T19:46:49.931 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on bug = true 2024-04-17T19:46:49.931 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.931 INFO:teuthology.orchestra.run.smithi029.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-17T19:46:49.931 INFO:teuthology.orchestra.run.smithi029.stdout: mon pg warn max object skew = 0 2024-04-17T19:46:49.931 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.931 INFO:teuthology.orchestra.run.smithi029.stdout: # disable pg_autoscaler by default for new pools 2024-04-17T19:46:49.931 INFO:teuthology.orchestra.run.smithi029.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-17T19:46:49.931 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.932 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default size = 2 2024-04-17T19:46:49.932 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.932 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow primary affinity = true 2024-04-17T19:46:49.932 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow pg remap = true 2024-04-17T19:46:49.932 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on legacy crush tunables = false 2024-04-17T19:46:49.932 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on crush straw calc version zero = false 2024-04-17T19:46:49.932 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on no sortbitwise = false 2024-04-17T19:46:49.932 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on osd down out interval zero = false 2024-04-17T19:46:49.932 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on too few osds = false 2024-04-17T19:46:49.932 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-17T19:46:49.932 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-17T19:46:49.932 INFO:teuthology.orchestra.run.smithi029.stdout: mon_allow_pool_size_one = true 2024-04-17T19:46:49.932 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.932 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-17T19:46:49.932 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.932 INFO:teuthology.orchestra.run.smithi029.stdout: osd default data pool replay window = 5 2024-04-17T19:46:49.933 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.933 INFO:teuthology.orchestra.run.smithi029.stdout: mon allow pool delete = true 2024-04-17T19:46:49.933 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.933 INFO:teuthology.orchestra.run.smithi029.stdout: mon cluster log file level = debug 2024-04-17T19:46:49.933 INFO:teuthology.orchestra.run.smithi029.stdout: debug asserts on shutdown = true 2024-04-17T19:46:49.933 INFO:teuthology.orchestra.run.smithi029.stdout: mon health detail to clog = false 2024-04-17T19:46:49.933 INFO:teuthology.orchestra.run.smithi029.stdout: mon host = "172.21.15.29,172.21.15.37,[v2:172.21.15.37:3301,v1:172.21.15.37:6790]" 2024-04-17T19:46:49.933 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on skipped message = False 2024-04-17T19:46:49.933 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = a680ee90-875c-4283-a901-86d7638393c3 2024-04-17T19:46:49.933 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.933 INFO:teuthology.orchestra.run.smithi029.stdout:[osd] 2024-04-17T19:46:49.933 INFO:teuthology.orchestra.run.smithi029.stdout: osd journal size = 100 2024-04-17T19:46:49.933 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.933 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub load threshold = 5.0 2024-04-17T19:46:49.933 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub max interval = 600 2024-04-17T19:46:49.933 INFO:teuthology.orchestra.run.smithi029.stdout: osd mclock profile = high_recovery_ops 2024-04-17T19:46:49.934 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.934 INFO:teuthology.orchestra.run.smithi029.stdout: osd recover clone overlap = true 2024-04-17T19:46:49.934 INFO:teuthology.orchestra.run.smithi029.stdout: osd recovery max chunk = 1048576 2024-04-17T19:46:49.934 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.934 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug shutdown = true 2024-04-17T19:46:49.934 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug op order = true 2024-04-17T19:46:49.934 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify stray on activate = true 2024-04-17T19:46:49.934 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug trim objects = true 2024-04-17T19:46:49.934 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.934 INFO:teuthology.orchestra.run.smithi029.stdout: osd open classes on start = true 2024-04-17T19:46:49.934 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug pg log writeout = true 2024-04-17T19:46:49.934 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.934 INFO:teuthology.orchestra.run.smithi029.stdout: osd deep scrub update digest min age = 30 2024-04-17T19:46:49.934 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.935 INFO:teuthology.orchestra.run.smithi029.stdout: osd map max advance = 10 2024-04-17T19:46:49.935 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.935 INFO:teuthology.orchestra.run.smithi029.stdout: journal zero on create = true 2024-04-17T19:46:49.935 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.935 INFO:teuthology.orchestra.run.smithi029.stdout: filestore ondisk finisher threads = 3 2024-04-17T19:46:49.935 INFO:teuthology.orchestra.run.smithi029.stdout: filestore apply finisher threads = 3 2024-04-17T19:46:49.935 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.935 INFO:teuthology.orchestra.run.smithi029.stdout: bdev debug aio = true 2024-04-17T19:46:49.935 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug misdirected ops = true 2024-04-17T19:46:49.935 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore block size = 96636764160 2024-04-17T19:46:49.935 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore fsck on mount = True 2024-04-17T19:46:49.935 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluefs = 20 2024-04-17T19:46:49.935 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluestore = 20 2024-04-17T19:46:49.935 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2024-04-17T19:46:49.935 INFO:teuthology.orchestra.run.smithi029.stdout: debug osd = 20 2024-04-17T19:46:49.935 INFO:teuthology.orchestra.run.smithi029.stdout: debug rocksdb = 10 2024-04-17T19:46:49.936 INFO:teuthology.orchestra.run.smithi029.stdout: mon client directed command retry = 5 2024-04-17T19:46:49.936 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd backfillfull_ratio = 0.85 2024-04-17T19:46:49.936 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd full ratio = 0.9 2024-04-17T19:46:49.936 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd nearfull ratio = 0.8 2024-04-17T19:46:49.936 INFO:teuthology.orchestra.run.smithi029.stdout: ms inject delay max = 1 2024-04-17T19:46:49.936 INFO:teuthology.orchestra.run.smithi029.stdout: ms inject delay probability = 0.005 2024-04-17T19:46:49.936 INFO:teuthology.orchestra.run.smithi029.stdout: ms inject delay type = client mds 2024-04-17T19:46:49.936 INFO:teuthology.orchestra.run.smithi029.stdout: ms inject socket failures = 2500 2024-04-17T19:46:49.936 INFO:teuthology.orchestra.run.smithi029.stdout: osd blocked scrub grace period = 3600 2024-04-17T19:46:49.936 INFO:teuthology.orchestra.run.smithi029.stdout: osd failsafe full ratio = 0.95 2024-04-17T19:46:49.936 INFO:teuthology.orchestra.run.smithi029.stdout: osd max markdown count = 1000 2024-04-17T19:46:49.936 INFO:teuthology.orchestra.run.smithi029.stdout: osd objectstore = bluestore 2024-04-17T19:46:49.936 INFO:teuthology.orchestra.run.smithi029.stdout: osd op complaint time = 180 2024-04-17T19:46:49.936 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.936 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr] 2024-04-17T19:46:49.937 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2024-04-17T19:46:49.937 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgr = 20 2024-04-17T19:46:49.937 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2024-04-17T19:46:49.937 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2024-04-17T19:46:49.937 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2024-04-17T19:46:49.937 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2024-04-17T19:46:49.937 INFO:teuthology.orchestra.run.smithi029.stdout: mgr/telemetry/nag = false 2024-04-17T19:46:49.937 INFO:teuthology.orchestra.run.smithi029.stdout: debug client = 20 2024-04-17T19:46:49.937 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.937 INFO:teuthology.orchestra.run.smithi029.stdout:[mon] 2024-04-17T19:46:49.937 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2024-04-17T19:46:49.938 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2024-04-17T19:46:49.938 INFO:teuthology.orchestra.run.smithi029.stdout: debug paxos = 20 2024-04-17T19:46:49.938 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2024-04-17T19:46:49.938 INFO:teuthology.orchestra.run.smithi029.stdout: mon data avail warn = 5 2024-04-17T19:46:49.938 INFO:teuthology.orchestra.run.smithi029.stdout: mon mgr mkfs grace = 240 2024-04-17T19:46:49.938 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2024-04-17T19:46:49.938 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd reporter subtree level = osd 2024-04-17T19:46:49.938 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd prime pg temp = true 2024-04-17T19:46:49.938 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2024-04-17T19:46:49.938 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.938 INFO:teuthology.orchestra.run.smithi029.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-17T19:46:49.938 INFO:teuthology.orchestra.run.smithi029.stdout: auth mon ticket ttl = 660 # 11m 2024-04-17T19:46:49.938 INFO:teuthology.orchestra.run.smithi029.stdout: auth service ticket ttl = 240 # 4m 2024-04-17T19:46:49.938 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.938 INFO:teuthology.orchestra.run.smithi029.stdout: # don't complain about insecure global_id in the test suite 2024-04-17T19:46:49.939 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-17T19:46:49.939 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-17T19:46:49.939 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.939 INFO:teuthology.orchestra.run.smithi029.stdout: # 1m isn't quite enough 2024-04-17T19:46:49.939 INFO:teuthology.orchestra.run.smithi029.stdout: mon_down_mkfs_grace = 2m 2024-04-17T19:46:49.939 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.939 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_filestore_osds = false 2024-04-17T19:46:49.939 INFO:teuthology.orchestra.run.smithi029.stdout: mon down mkfs grace = 300 2024-04-17T19:46:49.939 INFO:teuthology.orchestra.run.smithi029.stdout: mon op complaint time = 120 2024-04-17T19:46:49.939 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:46:49.939 INFO:teuthology.orchestra.run.smithi029.stdout:[client] 2024-04-17T19:46:49.939 INFO:teuthology.orchestra.run.smithi029.stdout: rgw cache enabled = true 2024-04-17T19:46:49.939 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable ops log = true 2024-04-17T19:46:49.939 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable usage log = true 2024-04-17T19:46:49.940 INFO:teuthology.orchestra.run.smithi029.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-17T19:46:49.940 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-17T19:46:49.940 INFO:teuthology.orchestra.run.smithi029.stdout: client mount timeout = 600 2024-04-17T19:46:49.940 INFO:teuthology.orchestra.run.smithi029.stdout: client_shutdown_timeout = 120 2024-04-17T19:46:49.940 INFO:teuthology.orchestra.run.smithi029.stdout: debug client = 20 2024-04-17T19:46:49.940 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2024-04-17T19:46:49.940 INFO:teuthology.orchestra.run.smithi029.stdout: rados mon op timeout = 900 2024-04-17T19:46:49.940 INFO:teuthology.orchestra.run.smithi029.stdout: rados osd op timeout = 900 2024-04-17T19:46:49.940 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.a] 2024-04-17T19:46:49.940 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.b] 2024-04-17T19:46:49.940 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.c] 2024-04-17T19:46:49.940 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.a] 2024-04-17T19:46:49.940 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.c] 2024-04-17T19:46:49.940 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.e] 2024-04-17T19:46:49.941 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.b] 2024-04-17T19:46:49.941 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.d] 2024-04-17T19:46:49.941 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.f] 2024-04-17T19:46:49.941 INFO:teuthology.orchestra.run.smithi029.stdout:[mds] 2024-04-17T19:46:49.941 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds = 20 2024-04-17T19:46:49.941 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds balancer = 20 2024-04-17T19:46:49.941 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2024-04-17T19:46:49.941 INFO:teuthology.orchestra.run.smithi029.stdout: mds bal fragment size max = 10000 2024-04-17T19:46:49.941 INFO:teuthology.orchestra.run.smithi029.stdout: mds bal merge size = 5 2024-04-17T19:46:49.941 INFO:teuthology.orchestra.run.smithi029.stdout: mds bal split bits = 3 2024-04-17T19:46:49.941 INFO:teuthology.orchestra.run.smithi029.stdout: mds bal split size = 100 2024-04-17T19:46:49.941 INFO:teuthology.orchestra.run.smithi029.stdout: mds debug frag = True 2024-04-17T19:46:49.941 INFO:teuthology.orchestra.run.smithi029.stdout: mds debug scatterstat = True 2024-04-17T19:46:49.941 INFO:teuthology.orchestra.run.smithi029.stdout: mds oft prefetch dirfrags = True 2024-04-17T19:46:49.942 INFO:teuthology.orchestra.run.smithi029.stdout: mds op complaint time = 180 2024-04-17T19:46:49.942 INFO:teuthology.orchestra.run.smithi029.stdout: mds verify scatter = True 2024-04-17T19:46:49.942 INFO:teuthology.orchestra.run.smithi029.stdout: mds_dir_prefetch = True 2024-04-17T19:46:49.942 INFO:teuthology.orchestra.run.smithi029.stdout: mds_sleep_rank_change = 5000000.0 2024-04-17T19:46:49.942 INFO:teuthology.orchestra.run.smithi029.stdout: mon client directed command retry = 5 2024-04-17T19:46:49.942 INFO:teuthology.orchestra.run.smithi029.stdout: ms inject delay max = 1 2024-04-17T19:46:49.942 INFO:teuthology.orchestra.run.smithi029.stdout: ms inject delay probability = 0.005 2024-04-17T19:46:49.942 INFO:teuthology.orchestra.run.smithi029.stdout: ms inject delay type = client mds osd 2024-04-17T19:46:49.942 INFO:teuthology.orchestra.run.smithi029.stdout: ms inject socket failures = 2500 2024-04-17T19:46:49.942 INFO:teuthology.orchestra.run.smithi029.stdout: osd op complaint time = 180 2024-04-17T19:46:49.942 INFO:teuthology.orchestra.run.smithi029.stdout: rados mon op timeout = 900 2024-04-17T19:46:49.942 INFO:teuthology.orchestra.run.smithi029.stdout: rados osd op timeout = 900 2024-04-17T19:46:49.944 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-17T19:46:49.944 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-17T19:46:50.051 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-17T19:46:50.051 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:46:50.051 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-17T19:46:50.060 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:46:50.060 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-17T19:46:50.116 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi029.front.sepia.ceph.com 2024-04-17T19:46:50.116 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:46:50.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-17T19:46:50.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-17T19:46:50.188 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:46:50.188 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-17T19:46:50.236 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi037.front.sepia.ceph.com 2024-04-17T19:46:50.236 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:46:50.236 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-17T19:46:50.236 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-17T19:46:50.259 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:46:50.259 DEBUG:teuthology.orchestra.run.smithi037:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-17T19:46:50.306 INFO:tasks.ceph:Setting up mon nodes... 2024-04-17T19:46:50.306 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-17T19:46:50.306 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-17T19:46:50.370 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-17T19:46:50.376 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-17T19:46:50.453 INFO:teuthology.orchestra.run.smithi037.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-17T19:46:50.469 INFO:tasks.ceph:Setting up mds nodes... 2024-04-17T19:46:50.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-17T19:46:50.534 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-17T19:46:50.540 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-17T19:46:50.559 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-17T19:46:50.665 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-17T19:46:50.670 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-17T19:46:50.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-04-17T19:46:50.796 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-04-17T19:46:50.801 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-04-17T19:46:50.821 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-17T19:46:50.899 INFO:teuthology.orchestra.run.smithi037.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-17T19:46:50.906 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-17T19:46:50.917 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-17T19:46:51.029 INFO:teuthology.orchestra.run.smithi037.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-17T19:46:51.036 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-17T19:46:51.047 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-04-17T19:46:51.159 INFO:teuthology.orchestra.run.smithi037.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-04-17T19:46:51.166 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-04-17T19:46:51.177 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-17T19:46:51.177 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-17T19:46:51.232 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-17T19:46:51.250 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-17T19:46:51.250 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi037.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-17T19:46:51.250 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-17T19:46:51.270 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-17T19:46:51.270 INFO:tasks.ceph:role: osd.0 2024-04-17T19:46:51.270 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com 2024-04-17T19:46:51.270 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-17T19:46:51.334 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T19:46:51.335 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T19:46:51.335 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T19:46:51.335 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-17T19:46:51.335 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T19:46:51.335 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-04-17T19:46:51.335 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T19:46:51.335 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-17T19:46:51.335 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T19:46:51.335 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T19:46:51.358 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-04-17T19:46:51.362 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-04-17T19:46:51.362 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-17T19:46:51.429 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-17T19:46:51.492 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: /sbin/restorecon: command not found 2024-04-17T19:46:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T19:46:51.493 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-17T19:46:51.559 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-17T19:46:51.559 INFO:tasks.ceph:role: osd.1 2024-04-17T19:46:51.559 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com 2024-04-17T19:46:51.560 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-17T19:46:51.623 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T19:46:51.623 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T19:46:51.623 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T19:46:51.623 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-17T19:46:51.623 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T19:46:51.623 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-04-17T19:46:51.623 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T19:46:51.623 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-17T19:46:51.623 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T19:46:51.623 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T19:46:51.650 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-04-17T19:46:51.652 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-04-17T19:46:51.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-17T19:46:51.720 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-17T19:46:51.775 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: /sbin/restorecon: command not found 2024-04-17T19:46:51.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T19:46:51.777 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-17T19:46:51.839 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-17T19:46:51.839 INFO:tasks.ceph:role: osd.2 2024-04-17T19:46:51.839 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com 2024-04-17T19:46:51.840 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-17T19:46:51.899 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T19:46:51.899 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T19:46:51.899 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T19:46:51.899 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-17T19:46:51.899 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T19:46:51.899 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-04-17T19:46:51.899 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T19:46:51.899 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-17T19:46:51.900 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T19:46:51.900 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T19:46:51.924 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-04-17T19:46:51.926 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-04-17T19:46:51.926 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-17T19:46:51.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-17T19:46:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: /sbin/restorecon: command not found 2024-04-17T19:46:52.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T19:46:52.053 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-17T19:46:52.115 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-17T19:46:52.115 INFO:tasks.ceph:role: osd.3 2024-04-17T19:46:52.115 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi029.front.sepia.ceph.com 2024-04-17T19:46:52.115 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-17T19:46:52.179 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T19:46:52.179 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T19:46:52.179 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T19:46:52.179 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-17T19:46:52.179 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T19:46:52.180 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-04-17T19:46:52.180 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T19:46:52.180 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-17T19:46:52.180 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T19:46:52.180 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T19:46:52.203 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-04-17T19:46:52.206 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-04-17T19:46:52.206 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-17T19:46:52.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-17T19:46:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: /sbin/restorecon: command not found 2024-04-17T19:46:52.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T19:46:52.328 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T19:46:52.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:46:52.481+0000 7f117fe2b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-17T19:46:52.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:46:52.485+0000 7f117fe2b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-17T19:46:52.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:46:52.485+0000 7f117fe2b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-17T19:46:52.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:46:52.485+0000 7f117fe2b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-17T19:46:52.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:46:52.485+0000 7f117fe2b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-17T19:46:52.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:46:52.485+0000 7f117fe2b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-17T19:46:55.529 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-17T19:46:55.550 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T19:46:55.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:46:55.669+0000 7fb1a585b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-17T19:46:55.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:46:55.669+0000 7fb1a585b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-17T19:46:55.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:46:55.669+0000 7fb1a585b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-17T19:46:55.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:46:55.669+0000 7fb1a585b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-17T19:46:55.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:46:55.669+0000 7fb1a585b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-17T19:46:55.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:46:55.673+0000 7fb1a585b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-17T19:46:58.705 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-17T19:46:58.727 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T19:46:58.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:46:58.845+0000 7f652bce67c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-17T19:46:58.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:46:58.845+0000 7f652bce67c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-17T19:46:58.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:46:58.845+0000 7f652bce67c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-17T19:46:58.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:46:58.845+0000 7f652bce67c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-17T19:46:58.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:46:58.845+0000 7f652bce67c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-17T19:46:58.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:46:58.845+0000 7f652bce67c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-17T19:47:01.901 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-17T19:47:01.922 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T19:47:02.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:02.045+0000 7f6f588b27c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-17T19:47:02.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:02.045+0000 7f6f588b27c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-17T19:47:02.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:02.045+0000 7f6f588b27c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-17T19:47:02.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:02.045+0000 7f6f588b27c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-17T19:47:02.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:02.045+0000 7f6f588b27c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-17T19:47:02.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:02.045+0000 7f6f588b27c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-17T19:47:05.070 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-17T19:47:05.091 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-17T19:47:05.107 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-17T19:47:05.107 INFO:tasks.ceph:role: osd.4 2024-04-17T19:47:05.107 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi037.front.sepia.ceph.com 2024-04-17T19:47:05.107 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-17T19:47:05.170 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T19:47:05.170 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T19:47:05.170 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T19:47:05.170 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-17T19:47:05.170 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T19:47:05.170 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-04-17T19:47:05.170 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T19:47:05.170 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-17T19:47:05.170 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T19:47:05.170 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T19:47:05.653 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-04-17T19:47:05.655 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-04-17T19:47:05.655 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-17T19:47:05.806 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-17T19:47:05.817 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: /sbin/restorecon: command not found 2024-04-17T19:47:05.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T19:47:05.819 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-17T19:47:05.874 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-17T19:47:05.874 INFO:tasks.ceph:role: osd.5 2024-04-17T19:47:05.874 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi037.front.sepia.ceph.com 2024-04-17T19:47:05.874 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-17T19:47:05.938 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T19:47:05.938 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T19:47:05.938 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T19:47:05.938 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-17T19:47:05.938 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T19:47:05.938 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-04-17T19:47:05.938 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T19:47:05.939 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-17T19:47:05.939 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T19:47:05.939 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T19:47:06.378 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-04-17T19:47:06.380 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-04-17T19:47:06.381 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-17T19:47:06.467 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-17T19:47:06.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T19:47:06.521 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: /sbin/restorecon: command not found 2024-04-17T19:47:06.521 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-17T19:47:06.574 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-17T19:47:06.574 INFO:tasks.ceph:role: osd.6 2024-04-17T19:47:06.574 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi037.front.sepia.ceph.com 2024-04-17T19:47:06.574 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-17T19:47:06.639 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T19:47:06.639 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T19:47:06.639 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T19:47:06.639 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-17T19:47:06.639 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T19:47:06.639 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-04-17T19:47:06.639 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T19:47:06.639 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-17T19:47:06.639 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T19:47:06.639 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T19:47:07.095 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-04-17T19:47:07.098 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-04-17T19:47:07.098 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-17T19:47:07.255 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-17T19:47:07.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T19:47:07.268 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: /sbin/restorecon: command not found 2024-04-17T19:47:07.268 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-17T19:47:07.323 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-17T19:47:07.323 INFO:tasks.ceph:role: osd.7 2024-04-17T19:47:07.323 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi037.front.sepia.ceph.com 2024-04-17T19:47:07.323 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-17T19:47:07.385 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T19:47:07.385 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T19:47:07.386 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T19:47:07.386 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-17T19:47:07.386 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T19:47:07.386 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-04-17T19:47:07.386 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T19:47:07.386 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-17T19:47:07.386 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T19:47:07.386 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T19:47:07.872 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-04-17T19:47:07.874 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-04-17T19:47:07.874 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-17T19:47:08.017 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-17T19:47:08.029 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: /sbin/restorecon: command not found 2024-04-17T19:47:08.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T19:47:08.030 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T19:47:08.176 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:08.189+0000 7f4f9f2d87c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-17T19:47:08.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:08.189+0000 7f4f9f2d87c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-17T19:47:08.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:08.189+0000 7f4f9f2d87c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-17T19:47:08.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:08.189+0000 7f4f9f2d87c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-17T19:47:08.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:08.189+0000 7f4f9f2d87c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-17T19:47:08.178 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:08.193+0000 7f4f9f2d87c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-17T19:47:11.174 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-17T19:47:11.189 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T19:47:11.302 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:11.313+0000 7fdb63ade7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-17T19:47:11.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:11.317+0000 7fdb63ade7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-17T19:47:11.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:11.317+0000 7fdb63ade7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-17T19:47:11.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:11.317+0000 7fdb63ade7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-17T19:47:11.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:11.317+0000 7fdb63ade7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-17T19:47:11.303 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:11.317+0000 7fdb63ade7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-17T19:47:14.283 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-17T19:47:14.296 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T19:47:14.413 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:14.425+0000 7f502e8477c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-17T19:47:14.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:14.429+0000 7f502e8477c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-17T19:47:14.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:14.429+0000 7f502e8477c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-17T19:47:14.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:14.429+0000 7f502e8477c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-17T19:47:14.414 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:14.429+0000 7f502e8477c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-17T19:47:14.415 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:14.429+0000 7f502e8477c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-17T19:47:17.412 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-17T19:47:17.426 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T19:47:17.546 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:17.561+0000 7fe9539867c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-17T19:47:17.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:17.561+0000 7fe9539867c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-17T19:47:17.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:17.561+0000 7fe9539867c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-17T19:47:17.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:17.561+0000 7fe9539867c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-17T19:47:17.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:17.561+0000 7fe9539867c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-17T19:47:17.547 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:17.561+0000 7fe9539867c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-17T19:47:18.054 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:47:18.058 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:47:20.531 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-17T19:47:20.546 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-17T19:47:20.546 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:47:20.546 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-17T19:47:20.569 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:47:20.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-17T19:47:20.634 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:47:20.634 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-17T19:47:20.697 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:47:20.697 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-04-17T19:47:20.762 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:47:20.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-17T19:47:20.825 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:47:20.825 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-17T19:47:20.889 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:47:20.889 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-17T19:47:20.953 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:47:20.953 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-17T19:47:21.021 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:47:21.021 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-17T19:47:21.035 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:47:21.035 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-17T19:47:21.090 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:47:21.090 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-17T19:47:21.146 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:47:21.146 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-04-17T19:47:21.202 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:47:21.202 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-17T19:47:21.258 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:47:21.258 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-17T19:47:21.318 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:47:21.319 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-17T19:47:21.374 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:47:21.374 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-17T19:47:21.431 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:47:21.431 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-17T19:47:21.439 INFO:tasks.ceph:Adding keys to all mons... 2024-04-17T19:47:21.440 DEBUG:teuthology.orchestra.run.smithi029:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-17T19:47:21.482 DEBUG:teuthology.orchestra.run.smithi037:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-17T19:47:21.494 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr.y] 2024-04-17T19:47:21.494 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAqJyBm4/XuFhAAB2qG9cJ6IgYB2Xs27yqzVw== 2024-04-17T19:47:21.494 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.a] 2024-04-17T19:47:21.494 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAqJyBmgZ2xIBAAEmICM+FIiSv7ldsvPekVxg== 2024-04-17T19:47:21.494 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.c] 2024-04-17T19:47:21.494 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAqJyBm5xZ4KBAAHr49uG04WfKk6mqFzX7vAQ== 2024-04-17T19:47:21.494 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.e] 2024-04-17T19:47:21.494 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAqJyBmz2pJMBAAnx3D4BGRBRrVY7ROggwQgw== 2024-04-17T19:47:21.494 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.0] 2024-04-17T19:47:21.494 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAsJyBmCp37HBAAD/zTPTMFEvCye9ZUhlwAGA== 2024-04-17T19:47:21.494 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.1] 2024-04-17T19:47:21.495 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAvJyBm+qIVKBAAu0NU3IfmHDp1o8eDCrS0Nw== 2024-04-17T19:47:21.495 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.2] 2024-04-17T19:47:21.495 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAyJyBm982PMhAAW+Nz9YXTeS5zqh39qyX0sA== 2024-04-17T19:47:21.495 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.3] 2024-04-17T19:47:21.495 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQA2JyBmx7vXAhAAh6hYuL3xRna2xW8u19foSg== 2024-04-17T19:47:21.495 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr.x] 2024-04-17T19:47:21.495 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAqJyBmMlv+GxAAm45EYOinbm8Gdif7FpHNqw== 2024-04-17T19:47:21.495 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.b] 2024-04-17T19:47:21.495 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQAqJyBmywmUNhAATlwMXvN5ZRFEX+zcl7GNCQ== 2024-04-17T19:47:21.495 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.d] 2024-04-17T19:47:21.495 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQArJyBmGKTAAhAAbgQ9GPHsI6/TA/Fq12sj4g== 2024-04-17T19:47:21.495 INFO:teuthology.orchestra.run.smithi037.stdout:[mds.f] 2024-04-17T19:47:21.495 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQArJyBmgHZ7ChAAZEJl4GmiWgUdDMhCBlmuMA== 2024-04-17T19:47:21.495 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.4] 2024-04-17T19:47:21.495 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQA8JyBmtB96CxAAx145DHkKg/qdrJW6dEkaJQ== 2024-04-17T19:47:21.496 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.5] 2024-04-17T19:47:21.496 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQA/JyBmFhTyEhAAKA/0pN+98EazJSpaHMa1aA== 2024-04-17T19:47:21.496 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.6] 2024-04-17T19:47:21.496 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBCJyBmMyKaGRAApOhkqdr2XyMtrfOylilR2A== 2024-04-17T19:47:21.496 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.7] 2024-04-17T19:47:21.496 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBFJyBmZV9+IRAAg5YEXlwL1/FyPeYQjXy46g== 2024-04-17T19:47:21.496 INFO:teuthology.orchestra.run.smithi037.stdout:[client.0] 2024-04-17T19:47:21.496 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQArJyBmIkStDhAAPXRJzzvxEZo+vfteVS+Y3Q== 2024-04-17T19:47:21.498 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.y] 2024-04-17T19:47:21.498 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAqJyBm4/XuFhAAB2qG9cJ6IgYB2Xs27yqzVw== 2024-04-17T19:47:21.498 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.a] 2024-04-17T19:47:21.498 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAqJyBmgZ2xIBAAEmICM+FIiSv7ldsvPekVxg== 2024-04-17T19:47:21.499 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.c] 2024-04-17T19:47:21.499 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAqJyBm5xZ4KBAAHr49uG04WfKk6mqFzX7vAQ== 2024-04-17T19:47:21.499 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.e] 2024-04-17T19:47:21.499 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAqJyBmz2pJMBAAnx3D4BGRBRrVY7ROggwQgw== 2024-04-17T19:47:21.499 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.0] 2024-04-17T19:47:21.499 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAsJyBmCp37HBAAD/zTPTMFEvCye9ZUhlwAGA== 2024-04-17T19:47:21.499 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.1] 2024-04-17T19:47:21.499 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAvJyBm+qIVKBAAu0NU3IfmHDp1o8eDCrS0Nw== 2024-04-17T19:47:21.499 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.2] 2024-04-17T19:47:21.499 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAyJyBm982PMhAAW+Nz9YXTeS5zqh39qyX0sA== 2024-04-17T19:47:21.499 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.3] 2024-04-17T19:47:21.499 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQA2JyBmx7vXAhAAh6hYuL3xRna2xW8u19foSg== 2024-04-17T19:47:21.499 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.x] 2024-04-17T19:47:21.499 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAqJyBmMlv+GxAAm45EYOinbm8Gdif7FpHNqw== 2024-04-17T19:47:21.499 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.b] 2024-04-17T19:47:21.500 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAqJyBmywmUNhAATlwMXvN5ZRFEX+zcl7GNCQ== 2024-04-17T19:47:21.500 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.d] 2024-04-17T19:47:21.500 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQArJyBmGKTAAhAAbgQ9GPHsI6/TA/Fq12sj4g== 2024-04-17T19:47:21.500 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.f] 2024-04-17T19:47:21.500 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQArJyBmgHZ7ChAAZEJl4GmiWgUdDMhCBlmuMA== 2024-04-17T19:47:21.500 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.4] 2024-04-17T19:47:21.500 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQA8JyBmtB96CxAAx145DHkKg/qdrJW6dEkaJQ== 2024-04-17T19:47:21.500 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.5] 2024-04-17T19:47:21.500 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQA/JyBmFhTyEhAAKA/0pN+98EazJSpaHMa1aA== 2024-04-17T19:47:21.500 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.6] 2024-04-17T19:47:21.500 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBCJyBmMyKaGRAApOhkqdr2XyMtrfOylilR2A== 2024-04-17T19:47:21.500 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.7] 2024-04-17T19:47:21.500 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBFJyBmZV9+IRAAg5YEXlwL1/FyPeYQjXy46g== 2024-04-17T19:47:21.500 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2024-04-17T19:47:21.501 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQArJyBmIkStDhAAPXRJzzvxEZo+vfteVS+Y3Q== 2024-04-17T19:47:21.501 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-17T19:47:21.547 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-17T19:47:21.628 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T19:47:21.632 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T19:47:21.712 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T19:47:21.715 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T19:47:21.793 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T19:47:21.796 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T19:47:21.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T19:47:21.870 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T19:47:21.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T19:47:21.942 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T19:47:22.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T19:47:22.012 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T19:47:22.086 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T19:47:22.089 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T19:47:22.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-17T19:47:22.161 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-17T19:47:22.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T19:47:22.241 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T19:47:22.313 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T19:47:22.316 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T19:47:22.389 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T19:47:22.392 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T19:47:22.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T19:47:22.467 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T19:47:22.536 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T19:47:22.539 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T19:47:22.610 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T19:47:22.613 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T19:47:22.687 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T19:47:22.691 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T19:47:22.759 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-17T19:47:22.762 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-17T19:47:22.832 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-17T19:47:22.832 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-17T19:47:22.852 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-17T19:47:22.976 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-17T19:47:22.996 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-17T19:47:23.009 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-17T19:47:23.137 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-17T19:47:23.147 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-17T19:47:23.200 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-17T19:47:23.326 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-17T19:47:23.337 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-17T19:47:23.340 DEBUG:teuthology.orchestra.run.smithi037:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-17T19:47:23.382 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-17T19:47:23.383 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-17T19:47:23.383 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-17T19:47:23.390 INFO:tasks.ceph.mon.a:Started 2024-04-17T19:47:23.390 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-17T19:47:23.390 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-17T19:47:23.428 INFO:tasks.ceph.mon.b:Started 2024-04-17T19:47:23.428 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-17T19:47:23.428 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-17T19:47:23.430 INFO:tasks.ceph.mon.c:Started 2024-04-17T19:47:23.430 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-17T19:47:23.430 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-17T19:47:23.430 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-17T19:47:23.433 INFO:tasks.ceph.mgr.y:Started 2024-04-17T19:47:23.433 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-17T19:47:23.434 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-17T19:47:23.435 INFO:tasks.ceph.mgr.x:Started 2024-04-17T19:47:23.435 INFO:tasks.ceph:Setting crush tunables to default 2024-04-17T19:47:23.436 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd crush tunables default 2024-04-17T19:47:23.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.601+0000 7fb8c5e00640 1 Processor -- start 2024-04-17T19:47:23.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.601+0000 7fb8c5e00640 1 -- start start 2024-04-17T19:47:23.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.605+0000 7fb8c5e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb8c0153c10 0x7fb8c0154020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:23.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.605+0000 7fb8c5e00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8c00c5110 con 0x7fb8c015a7f0 2024-04-17T19:47:23.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.605+0000 7fb8c5e00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8c00c84c0 con 0x7fb8c00ca0d0 2024-04-17T19:47:23.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.605+0000 7fb8c5e00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb8c0154560 con 0x7fb8c0153c10 2024-04-17T19:47:23.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.605+0000 7fb8b7e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb8c0153c10 0x7fb8c0154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:23.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.605+0000 7fb8bf400640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fb8c00ca0d0 0x7fb8c00ca4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:43746/0 (socket says 172.21.15.29:43746) 2024-04-17T19:47:23.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.605+0000 7fb8bf400640 1 -- 172.21.15.29:0/1666042758 learned_addr learned my addr 172.21.15.29:0/1666042758 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:47:23.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.605+0000 7fb8b7e00640 1 -- 172.21.15.29:0/1666042758 >> v2:172.21.15.37:3300/0 conn(0x7fb8c0153c10 msgr2=0x7fb8c0154020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-17T19:47:23.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.605+0000 7fb8b7e00640 1 -- 172.21.15.29:0/1666042758 >> v2:172.21.15.37:3300/0 conn(0x7fb8c0153c10 msgr2=0x7fb8c0154020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-17T19:47:23.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.605+0000 7fb8b7e00640 1 --2- 172.21.15.29:0/1666042758 >> v2:172.21.15.37:3300/0 conn(0x7fb8c0153c10 0x7fb8c0154020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-17T19:47:23.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.605+0000 7fb8b7e00640 1 --2- 172.21.15.29:0/1666042758 >> v2:172.21.15.37:3300/0 conn(0x7fb8c0153c10 0x7fb8c0154020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T19:47:23.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.809+0000 7fb8b7e00640 1 --2- 172.21.15.29:0/1666042758 >> v2:172.21.15.37:3300/0 conn(0x7fb8c0153c10 0x7fb8c0154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:23.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.809+0000 7fb8b7e00640 1 -- 172.21.15.29:0/1666042758 >> v2:172.21.15.37:3300/0 conn(0x7fb8c0153c10 msgr2=0x7fb8c0154020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-17T19:47:23.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.809+0000 7fb8b7e00640 1 -- 172.21.15.29:0/1666042758 >> v2:172.21.15.37:3300/0 conn(0x7fb8c0153c10 msgr2=0x7fb8c0154020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-17T19:47:23.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.809+0000 7fb8b7e00640 1 --2- 172.21.15.29:0/1666042758 >> v2:172.21.15.37:3300/0 conn(0x7fb8c0153c10 0x7fb8c0154020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-17T19:47:23.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.809+0000 7fb8b7e00640 1 --2- 172.21.15.29:0/1666042758 >> v2:172.21.15.37:3300/0 conn(0x7fb8c0153c10 0x7fb8c0154020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-17T19:47:23.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.817+0000 7fb8be000640 1 -- 172.21.15.29:0/1666042758 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3230819952 0 0) 0x7fb8c00c5110 con 0x7fb8c015a7f0 2024-04-17T19:47:23.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.817+0000 7fb8be000640 1 -- 172.21.15.29:0/1666042758 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8a4003530 con 0x7fb8c015a7f0 2024-04-17T19:47:23.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.817+0000 7fb8be000640 1 -- 172.21.15.29:0/1666042758 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2462299042 0 0) 0x7fb8a4003530 con 0x7fb8c015a7f0 2024-04-17T19:47:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.817+0000 7fb8be000640 1 -- 172.21.15.29:0/1666042758 >> v2:172.21.15.37:3300/0 conn(0x7fb8c0153c10 msgr2=0x7fb8c0154020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-17T19:47:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.817+0000 7fb8be000640 1 --2- 172.21.15.29:0/1666042758 >> v2:172.21.15.37:3300/0 conn(0x7fb8c0153c10 0x7fb8c0154020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.817+0000 7fb8be000640 1 -- 172.21.15.29:0/1666042758 >> v1:172.21.15.37:6789/0 conn(0x7fb8c00ca0d0 legacy=0x7fb8c00ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.817+0000 7fb8be000640 1 -- 172.21.15.29:0/1666042758 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8c00c87a0 con 0x7fb8c015a7f0 2024-04-17T19:47:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.817+0000 7fb8be000640 1 -- 172.21.15.29:0/1666042758 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fb8b0001680 con 0x7fb8c015a7f0 2024-04-17T19:47:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.817+0000 7fb8be000640 1 -- 172.21.15.29:0/1666042758 >> v1:172.21.15.29:6789/0 conn(0x7fb8c015a7f0 legacy=0x7fb8c00c4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.817+0000 7fb8be000640 1 --2- 172.21.15.29:0/1666042758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8a4003a10 0x7fb8a4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.817+0000 7fb8be000640 1 --2- 172.21.15.29:0/1666042758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8a4004810 0x7fb8a4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.817+0000 7fb8be000640 1 --2- 172.21.15.29:0/1666042758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8a4005360 0x7fb8a400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.817+0000 7fb8be000640 1 -- 172.21.15.29:0/1666042758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c0154560 con 0x7fb8a4003a10 2024-04-17T19:47:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.817+0000 7fb8be000640 1 -- 172.21.15.29:0/1666042758 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a400c260 con 0x7fb8a4004810 2024-04-17T19:47:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.817+0000 7fb8be000640 1 -- 172.21.15.29:0/1666042758 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb8a400c460 con 0x7fb8a4005360 2024-04-17T19:47:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.821+0000 7fb8bea00640 1 --2- 172.21.15.29:0/1666042758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8a4003a10 0x7fb8a4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.821+0000 7fb8b7e00640 1 --2- 172.21.15.29:0/1666042758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8a4004810 0x7fb8a4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:23.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.821+0000 7fb8bf400640 1 --2- 172.21.15.29:0/1666042758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8a4005360 0x7fb8a400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:23.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.821+0000 7fb8bea00640 1 -- 172.21.15.29:0/1666042758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8a4005360 msgr2=0x7fb8a400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:23.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.821+0000 7fb8bea00640 1 --2- 172.21.15.29:0/1666042758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8a4005360 0x7fb8a400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:23.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.821+0000 7fb8bea00640 1 -- 172.21.15.29:0/1666042758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8a4004810 msgr2=0x7fb8a4004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:23.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.821+0000 7fb8bea00640 1 --2- 172.21.15.29:0/1666042758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8a4004810 0x7fb8a4004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:23.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.821+0000 7fb8bea00640 1 -- 172.21.15.29:0/1666042758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8a4003530 con 0x7fb8a4003a10 2024-04-17T19:47:23.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.821+0000 7fb8bf400640 1 -- 172.21.15.29:0/1666042758 reap_dead start 2024-04-17T19:47:23.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.821+0000 7fb8bea00640 1 --2- 172.21.15.29:0/1666042758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8a4003a10 0x7fb8a4003e80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0007f60 tx=0x7fb8b001cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0bbe34e3ba4679d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:23.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.821+0000 7fb8be000640 1 -- 172.21.15.29:0/1666042758 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b00b2070 con 0x7fb8a4003a10 2024-04-17T19:47:23.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.821+0000 7fb8be000640 1 -- 172.21.15.29:0/1666042758 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8b00a58c0 con 0x7fb8a4003a10 2024-04-17T19:47:23.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.821+0000 7fb8be000640 1 -- 172.21.15.29:0/1666042758 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b0007b80 con 0x7fb8a4003a10 2024-04-17T19:47:23.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.825+0000 7fb8c5e00640 1 -- 172.21.15.29:0/1666042758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8a4003a10 msgr2=0x7fb8a4003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:23.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.825+0000 7fb8c5e00640 1 --2- 172.21.15.29:0/1666042758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8a4003a10 0x7fb8a4003e80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0007f60 tx=0x7fb8b001cad0 comp rx=0 tx=0).stop 2024-04-17T19:47:23.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.825+0000 7fb8c5e00640 1 -- 172.21.15.29:0/1666042758 shutdown_connections 2024-04-17T19:47:23.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.825+0000 7fb8c5e00640 1 --2- 172.21.15.29:0/1666042758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8a4003a10 0x7fb8a4003e80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:23.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.825+0000 7fb8c5e00640 1 -- 172.21.15.29:0/1666042758 >> 172.21.15.29:0/1666042758 conn(0x7fb8c006bf90 msgr2=0x7fb8c006c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:23.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.825+0000 7fb8c5e00640 1 -- 172.21.15.29:0/1666042758 shutdown_connections 2024-04-17T19:47:23.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.825+0000 7fb8c5e00640 1 -- 172.21.15.29:0/1666042758 wait complete. 2024-04-17T19:47:23.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.825+0000 7fb8c5e00640 1 Processor -- start 2024-04-17T19:47:23.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.825+0000 7fb8c5e00640 1 -- start start 2024-04-17T19:47:23.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.829+0000 7fb8c5e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8a4003a10 0x7fb8c01f4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:23.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.829+0000 7fb8c5e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8c00cbe50 0x7fb8c014d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:23.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.829+0000 7fb8c5e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8c014d740 0x7fb8c0145070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:23.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.829+0000 7fb8c5e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c015ac00 con 0x7fb8c014d740 2024-04-17T19:47:23.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.829+0000 7fb8c5e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb8c015dc30 con 0x7fb8a4003a10 2024-04-17T19:47:23.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.829+0000 7fb8c5e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb8c015e150 con 0x7fb8c00cbe50 2024-04-17T19:47:23.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.829+0000 7fb8bea00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8a4003a10 0x7fb8c01f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:23.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.829+0000 7fb8bf400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8c014d740 0x7fb8c0145070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:23.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.829+0000 7fb8b7e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8c00cbe50 0x7fb8c014d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:23.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.829+0000 7fb8bf400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8c014d740 0x7fb8c0145070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36786/0 (socket says 172.21.15.29:36786) 2024-04-17T19:47:23.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.829+0000 7fb8bf400640 1 -- 172.21.15.29:0/3274218327 learned_addr learned my addr 172.21.15.29:0/3274218327 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:23.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.829+0000 7fb8bf400640 1 -- 172.21.15.29:0/3274218327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8c00cbe50 msgr2=0x7fb8c014d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:23.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.829+0000 7fb8bf400640 1 --2- 172.21.15.29:0/3274218327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8c00cbe50 0x7fb8c014d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:23.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.829+0000 7fb8bf400640 1 -- 172.21.15.29:0/3274218327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8a4003a10 msgr2=0x7fb8c01f4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:23.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.829+0000 7fb8bf400640 1 --2- 172.21.15.29:0/3274218327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8a4003a10 0x7fb8c01f4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:23.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.829+0000 7fb8bf400640 1 -- 172.21.15.29:0/3274218327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8c01e5fe0 con 0x7fb8c014d740 2024-04-17T19:47:23.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.829+0000 7fb8bf400640 1 --2- 172.21.15.29:0/3274218327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8c014d740 0x7fb8c0145070 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb8c00c9af0 tx=0x7fb8b8007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:23.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.833+0000 7fb8b7400640 1 -- 172.21.15.29:0/3274218327 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b800e8c0 con 0x7fb8c014d740 2024-04-17T19:47:23.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.833+0000 7fb8b7400640 1 -- 172.21.15.29:0/3274218327 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8b8018070 con 0x7fb8c014d740 2024-04-17T19:47:23.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.833+0000 7fb8c5e00640 1 -- 172.21.15.29:0/3274218327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8c0070510 con 0x7fb8c014d740 2024-04-17T19:47:23.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.833+0000 7fb8b7400640 1 -- 172.21.15.29:0/3274218327 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b800ece0 con 0x7fb8c014d740 2024-04-17T19:47:23.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.833+0000 7fb8c5e00640 1 -- 172.21.15.29:0/3274218327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8c01f4960 con 0x7fb8c014d740 2024-04-17T19:47:23.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.833+0000 7fb8b7400640 1 -- 172.21.15.29:0/3274218327 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb8b801d460 con 0x7fb8c014d740 2024-04-17T19:47:23.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.833+0000 7fb8b7400640 1 -- 172.21.15.29:0/3274218327 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb8b80136f0 con 0x7fb8c014d740 2024-04-17T19:47:23.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.833+0000 7fb8c5e00640 1 -- 172.21.15.29:0/3274218327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8c00ccdb0 con 0x7fb8c014d740 2024-04-17T19:47:23.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.849+0000 7fb8b7400640 1 -- 172.21.15.29:0/3274218327 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fb8b80138f0 con 0x7fb8c014d740 2024-04-17T19:47:23.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.909+0000 7fb8c5e00640 1 -- 172.21.15.29:0/3274218327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fb8c0069b90 con 0x7fb8c014d740 2024-04-17T19:47:23.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.913+0000 7fb8b7400640 1 -- 172.21.15.29:0/3274218327 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fb8b8013ad0 con 0x7fb8c014d740 2024-04-17T19:47:23.902 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2024-04-17T19:47:23.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.913+0000 7fb8c5e00640 1 -- 172.21.15.29:0/3274218327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8c014d740 msgr2=0x7fb8c0145070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:23.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.913+0000 7fb8c5e00640 1 --2- 172.21.15.29:0/3274218327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8c014d740 0x7fb8c0145070 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb8c00c9af0 tx=0x7fb8b8007b00 comp rx=0 tx=0).stop 2024-04-17T19:47:23.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.913+0000 7fb8c5e00640 1 -- 172.21.15.29:0/3274218327 shutdown_connections 2024-04-17T19:47:23.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.913+0000 7fb8c5e00640 1 --2- 172.21.15.29:0/3274218327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8c014d740 0x7fb8c0145070 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:23.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.913+0000 7fb8c5e00640 1 --2- 172.21.15.29:0/3274218327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8c00cbe50 0x7fb8c014d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:23.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.913+0000 7fb8c5e00640 1 --2- 172.21.15.29:0/3274218327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb8a4003a10 0x7fb8c01f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:23.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.913+0000 7fb8c5e00640 1 -- 172.21.15.29:0/3274218327 >> 172.21.15.29:0/3274218327 conn(0x7fb8c006bf90 msgr2=0x7fb8c006fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:23.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.913+0000 7fb8c5e00640 1 -- 172.21.15.29:0/3274218327 shutdown_connections 2024-04-17T19:47:23.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:23.913+0000 7fb8c5e00640 1 -- 172.21.15.29:0/3274218327 wait complete. 2024-04-17T19:47:23.923 INFO:tasks.ceph:check_enable_crimson: False 2024-04-17T19:47:23.924 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-17T19:47:23.924 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:47:23.924 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-17T19:47:23.944 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:47:23.944 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-17T19:47:24.007 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:47:24.007 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-17T19:47:24.047 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:24.060+0000 7f80eb593180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T19:47:24.060 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:24.073+0000 7f30ad4e2180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T19:47:24.067 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:47:24.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-17T19:47:24.127 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:47:24.127 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-17T19:47:24.140 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:47:24.140 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-17T19:47:24.194 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:47:24.194 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-17T19:47:24.251 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-04-17T19:47:24.251 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-17T19:47:24.306 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd new a63774a5-4a49-4bae-a850-18ecff670ccb 0 2024-04-17T19:47:24.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.516+0000 7fba24556640 1 Processor -- start 2024-04-17T19:47:24.506 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.516+0000 7fba24556640 1 -- start start 2024-04-17T19:47:24.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.520+0000 7fba24556640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fba1c0732c0 0x7fba1c141bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:24.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.520+0000 7fba24556640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fba1c142130 0x7fba1c0c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:24.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.520+0000 7fba24556640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba1c15c460 con 0x7fba1c072eb0 2024-04-17T19:47:24.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.520+0000 7fba24556640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fba1c142540 con 0x7fba1c142130 2024-04-17T19:47:24.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.520+0000 7fba24556640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fba1c0c8e10 con 0x7fba1c0732c0 2024-04-17T19:47:24.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.520+0000 7fba222f2640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fba1c072eb0 0x7fba1c073970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48784/0 (socket says 172.21.15.37:48784) 2024-04-17T19:47:24.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.520+0000 7fba222f2640 1 -- 172.21.15.37:0/1789047407 learned_addr learned my addr 172.21.15.37:0/1789047407 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-17T19:47:24.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.520+0000 7fba21af1640 1 --2- 172.21.15.37:0/1789047407 >> v2:172.21.15.37:3300/0 conn(0x7fba1c0732c0 0x7fba1c141bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:24.508 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.520+0000 7fba22af3640 1 --2- 172.21.15.37:0/1789047407 >> v2:172.21.15.29:3300/0 conn(0x7fba1c142130 0x7fba1c0c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:24.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.520+0000 7fba212f0640 1 -- 172.21.15.37:0/1789047407 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 925506566 0 0) 0x7fba1c15c460 con 0x7fba1c072eb0 2024-04-17T19:47:24.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.520+0000 7fba212f0640 1 -- 172.21.15.37:0/1789047407 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba08003530 con 0x7fba1c072eb0 2024-04-17T19:47:24.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.520+0000 7fba21af1640 1 -- 172.21.15.37:0/1789047407 >> v2:172.21.15.29:3300/0 conn(0x7fba1c142130 msgr2=0x7fba1c0c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:24.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.520+0000 7fba21af1640 1 --2- 172.21.15.37:0/1789047407 >> v2:172.21.15.29:3300/0 conn(0x7fba1c142130 0x7fba1c0c7c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:24.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.520+0000 7fba21af1640 1 -- 172.21.15.37:0/1789047407 >> v1:172.21.15.37:6789/0 conn(0x7fba1c072eb0 legacy=0x7fba1c073970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:24.509 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.520+0000 7fba21af1640 1 -- 172.21.15.37:0/1789047407 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba1c15c740 con 0x7fba1c0732c0 2024-04-17T19:47:24.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.524+0000 7fba21af1640 1 --2- 172.21.15.37:0/1789047407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba1c0732c0 0x7fba1c141bf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fba1c076670 tx=0x7fba0c01ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b038f2ec14de4f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:24.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.524+0000 7fba212f0640 1 -- 172.21.15.37:0/1789047407 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0c0b5070 con 0x7fba1c0732c0 2024-04-17T19:47:24.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.524+0000 7fba212f0640 1 -- 172.21.15.37:0/1789047407 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba0c0a8970 con 0x7fba1c0732c0 2024-04-17T19:47:24.511 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.524+0000 7fba212f0640 1 -- 172.21.15.37:0/1789047407 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0c0a8d20 con 0x7fba1c0732c0 2024-04-17T19:47:24.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.524+0000 7fba24556640 1 -- 172.21.15.37:0/1789047407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba1c0732c0 msgr2=0x7fba1c141bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:24.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.524+0000 7fba24556640 1 --2- 172.21.15.37:0/1789047407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba1c0732c0 0x7fba1c141bf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fba1c076670 tx=0x7fba0c01ec70 comp rx=0 tx=0).stop 2024-04-17T19:47:24.512 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.524+0000 7fba24556640 1 -- 172.21.15.37:0/1789047407 shutdown_connections 2024-04-17T19:47:24.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.524+0000 7fba24556640 1 --2- 172.21.15.37:0/1789047407 >> v2:172.21.15.29:3300/0 conn(0x7fba1c142130 0x7fba1c0c7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:24.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.524+0000 7fba24556640 1 --2- 172.21.15.37:0/1789047407 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba1c0732c0 0x7fba1c141bf0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:24.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.524+0000 7fba24556640 1 -- 172.21.15.37:0/1789047407 >> 172.21.15.37:0/1789047407 conn(0x7fba1c0cef00 msgr2=0x7fba1c0cad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:24.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.524+0000 7fba24556640 1 -- 172.21.15.37:0/1789047407 shutdown_connections 2024-04-17T19:47:24.513 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.524+0000 7fba24556640 1 -- 172.21.15.37:0/1789047407 wait complete. 2024-04-17T19:47:24.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.524+0000 7fba24556640 1 Processor -- start 2024-04-17T19:47:24.514 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.524+0000 7fba24556640 1 -- start start 2024-04-17T19:47:24.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba24556640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba1c072eb0 0x7fba1c155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:24.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba24556640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba1c0732c0 0x7fba1c1f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:24.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba24556640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba1c142130 0x7fba1c0cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:24.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba24556640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fba1c15cfe0 con 0x7fba1c0732c0 2024-04-17T19:47:24.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba24556640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fba1c15cd00 con 0x7fba1c142130 2024-04-17T19:47:24.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba24556640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fba1c074080 con 0x7fba1c072eb0 2024-04-17T19:47:24.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba222f2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba1c072eb0 0x7fba1c155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:24.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba22af3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba1c142130 0x7fba1c0cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:24.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba222f2640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba1c072eb0 0x7fba1c155f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48608/0 (socket says 172.21.15.37:48608) 2024-04-17T19:47:24.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba222f2640 1 -- 172.21.15.37:0/3071312461 learned_addr learned my addr 172.21.15.37:0/3071312461 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-17T19:47:24.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba21af1640 1 --2- 172.21.15.37:0/3071312461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba1c0732c0 0x7fba1c1f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:24.517 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba222f2640 1 -- 172.21.15.37:0/3071312461 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba1c142130 msgr2=0x7fba1c0cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:24.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba222f2640 1 --2- 172.21.15.37:0/3071312461 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba1c142130 0x7fba1c0cd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:24.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba222f2640 1 -- 172.21.15.37:0/3071312461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba1c0732c0 msgr2=0x7fba1c1f54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:24.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba222f2640 1 --2- 172.21.15.37:0/3071312461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba1c0732c0 0x7fba1c1f54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:24.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba222f2640 1 -- 172.21.15.37:0/3071312461 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba1c1f5a20 con 0x7fba1c072eb0 2024-04-17T19:47:24.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba222f2640 1 --2- 172.21.15.37:0/3071312461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba1c072eb0 0x7fba1c155f80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fba1400c3c0 tx=0x7fba1400c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:24.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba12ffd640 1 -- 172.21.15.37:0/3071312461 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba14017070 con 0x7fba1c072eb0 2024-04-17T19:47:24.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba12ffd640 1 -- 172.21.15.37:0/3071312461 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba14005cf0 con 0x7fba1c072eb0 2024-04-17T19:47:24.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba12ffd640 1 -- 172.21.15.37:0/3071312461 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba140072f0 con 0x7fba1c072eb0 2024-04-17T19:47:24.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba24556640 1 -- 172.21.15.37:0/3071312461 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba1c156700 con 0x7fba1c072eb0 2024-04-17T19:47:24.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba24556640 1 -- 172.21.15.37:0/3071312461 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fba1c1f1fc0 con 0x7fba1c072eb0 2024-04-17T19:47:24.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba12ffd640 1 -- 172.21.15.37:0/3071312461 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fba1401a070 con 0x7fba1c072eb0 2024-04-17T19:47:24.518 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba12ffd640 1 -- 172.21.15.37:0/3071312461 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fba14012740 con 0x7fba1c072eb0 2024-04-17T19:47:24.519 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.528+0000 7fba24556640 1 -- 172.21.15.37:0/3071312461 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba1c04fea0 con 0x7fba1c072eb0 2024-04-17T19:47:24.529 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.540+0000 7fba12ffd640 1 -- 172.21.15.37:0/3071312461 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fba14021020 con 0x7fba1c072eb0 2024-04-17T19:47:24.535 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:24.548+0000 7f80eb593180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T19:47:24.599 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.612+0000 7fba24556640 1 -- 172.21.15.37:0/3071312461 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a63774a5-4a49-4bae-a850-18ecff670ccb", "id": 0} v 0) -- 0x7fba1c1406c0 con 0x7fba1c072eb0 2024-04-17T19:47:24.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.616+0000 7fba12ffd640 1 -- 172.21.15.37:0/3071312461 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a63774a5-4a49-4bae-a850-18ecff670ccb", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fba140075b0 con 0x7fba1c072eb0 2024-04-17T19:47:24.606 INFO:teuthology.orchestra.run.smithi037.stdout:0 2024-04-17T19:47:24.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.620+0000 7fba24556640 1 -- 172.21.15.37:0/3071312461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba1c072eb0 msgr2=0x7fba1c155f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:24.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.620+0000 7fba24556640 1 --2- 172.21.15.37:0/3071312461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba1c072eb0 0x7fba1c155f80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fba1400c3c0 tx=0x7fba1400c880 comp rx=0 tx=0).stop 2024-04-17T19:47:24.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.620+0000 7fba24556640 1 -- 172.21.15.37:0/3071312461 shutdown_connections 2024-04-17T19:47:24.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.620+0000 7fba24556640 1 --2- 172.21.15.37:0/3071312461 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba1c142130 0x7fba1c0cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:24.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.620+0000 7fba24556640 1 --2- 172.21.15.37:0/3071312461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba1c0732c0 0x7fba1c1f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:24.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.620+0000 7fba24556640 1 --2- 172.21.15.37:0/3071312461 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba1c072eb0 0x7fba1c155f80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:24.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.620+0000 7fba24556640 1 -- 172.21.15.37:0/3071312461 >> 172.21.15.37:0/3071312461 conn(0x7fba1c0cef00 msgr2=0x7fba1c0c58b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:24.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.620+0000 7fba24556640 1 -- 172.21.15.37:0/3071312461 shutdown_connections 2024-04-17T19:47:24.608 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.620+0000 7fba24556640 1 -- 172.21.15.37:0/3071312461 wait complete. 2024-04-17T19:47:24.625 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:24.637+0000 7f30ad4e2180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T19:47:24.627 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd new a567ac96-9ee4-495f-a20c-2686d59f1fe2 1 2024-04-17T19:47:24.788 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:24.800+0000 7f80eb593180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T19:47:24.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.820+0000 7f5fa1758640 1 Processor -- start 2024-04-17T19:47:24.808 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.820+0000 7f5fa1758640 1 -- start start 2024-04-17T19:47:24.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.820+0000 7f5fa1758640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5f9c070fb0 0x7f5f9c152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:24.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.820+0000 7f5fa1758640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f9c0c9cf0 0x7f5f9c1636a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:24.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.820+0000 7f5fa1758640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f9c0c85c0 con 0x7f5f9c0c98e0 2024-04-17T19:47:24.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.820+0000 7f5fa1758640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f5f9c1597a0 con 0x7f5f9c070fb0 2024-04-17T19:47:24.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.820+0000 7f5fa1758640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5f9c159a40 con 0x7f5f9c0c9cf0 2024-04-17T19:47:24.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.820+0000 7f5f9affd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5f9c070fb0 0x7f5f9c152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:24.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.820+0000 7f5f9b7fe640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f9c0c9cf0 0x7f5f9c1636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:24.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.820+0000 7f5f9affd640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5f9c070fb0 0x7f5f9c152fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34890/0 (socket says 172.21.15.37:34890) 2024-04-17T19:47:24.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.820+0000 7f5f9affd640 1 -- 172.21.15.37:0/1620909356 learned_addr learned my addr 172.21.15.37:0/1620909356 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-17T19:47:24.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.824+0000 7f5f9affd640 1 -- 172.21.15.37:0/1620909356 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f9c0c9cf0 msgr2=0x7f5f9c1636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:24.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.824+0000 7f5f9affd640 1 --2- 172.21.15.37:0/1620909356 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f9c0c9cf0 0x7f5f9c1636a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:24.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.824+0000 7f5f9affd640 1 -- 172.21.15.37:0/1620909356 >> v1:172.21.15.29:6789/0 conn(0x7f5f9c0c98e0 legacy=0x7f5f9c0c55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:24.811 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.824+0000 7f5f9affd640 1 -- 172.21.15.37:0/1620909356 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f9c0ca100 con 0x7f5f9c070fb0 2024-04-17T19:47:24.812 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.824+0000 7f5f9b7fe640 1 --2- 172.21.15.37:0/1620909356 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f9c0c9cf0 0x7f5f9c1636a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:47:24.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.824+0000 7f5f9affd640 1 --2- 172.21.15.37:0/1620909356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f9c070fb0 0x7f5f9c152fa0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c009870 tx=0x7f5f8c01ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60d70b5988f9bff3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:24.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.824+0000 7f5f99ffb640 1 -- 172.21.15.37:0/1620909356 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c0b5070 con 0x7f5f9c070fb0 2024-04-17T19:47:24.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.824+0000 7f5f99ffb640 1 -- 172.21.15.37:0/1620909356 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f8c0a8a50 con 0x7f5f9c070fb0 2024-04-17T19:47:24.814 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.824+0000 7f5f99ffb640 1 -- 172.21.15.37:0/1620909356 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c0a8e20 con 0x7f5f9c070fb0 2024-04-17T19:47:24.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.828+0000 7f5fa1758640 1 -- 172.21.15.37:0/1620909356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f9c070fb0 msgr2=0x7f5f9c152fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:24.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.828+0000 7f5fa1758640 1 --2- 172.21.15.37:0/1620909356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f9c070fb0 0x7f5f9c152fa0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c009870 tx=0x7f5f8c01ed50 comp rx=0 tx=0).stop 2024-04-17T19:47:24.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.828+0000 7f5fa1758640 1 -- 172.21.15.37:0/1620909356 shutdown_connections 2024-04-17T19:47:24.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.828+0000 7f5fa1758640 1 --2- 172.21.15.37:0/1620909356 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f9c0c9cf0 0x7f5f9c1636a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:24.815 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.828+0000 7f5fa1758640 1 --2- 172.21.15.37:0/1620909356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f9c070fb0 0x7f5f9c152fa0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:24.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.828+0000 7f5fa1758640 1 -- 172.21.15.37:0/1620909356 >> 172.21.15.37:0/1620909356 conn(0x7f5f9c072dc0 msgr2=0x7f5f9c0731d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:24.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.828+0000 7f5fa1758640 1 -- 172.21.15.37:0/1620909356 shutdown_connections 2024-04-17T19:47:24.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.828+0000 7f5fa1758640 1 -- 172.21.15.37:0/1620909356 wait complete. 2024-04-17T19:47:24.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.828+0000 7f5fa1758640 1 Processor -- start 2024-04-17T19:47:24.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.828+0000 7f5fa1758640 1 -- start start 2024-04-17T19:47:24.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.828+0000 7f5fa1758640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f9c070fb0 0x7f5f9c13e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:24.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.828+0000 7f5fa1758640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f9c0c98e0 0x7f5f9c140e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:24.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.828+0000 7f5fa1758640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f9c0c9cf0 0x7f5f9c13cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:24.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.828+0000 7f5fa1758640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5f9c0c7dc0 con 0x7f5f9c0c98e0 2024-04-17T19:47:24.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.828+0000 7f5fa1758640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5f9c0c7ae0 con 0x7f5f9c0c9cf0 2024-04-17T19:47:24.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.828+0000 7f5fa1758640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5f9c0c80a0 con 0x7f5f9c070fb0 2024-04-17T19:47:24.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5f9b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f9c0c9cf0 0x7f5f9c13cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:24.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5f9affd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f9c070fb0 0x7f5f9c13e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:24.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5f9b7fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f9c0c9cf0 0x7f5f9c13cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34898/0 (socket says 172.21.15.37:34898) 2024-04-17T19:47:24.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5f9b7fe640 1 -- 172.21.15.37:0/835104675 learned_addr learned my addr 172.21.15.37:0/835104675 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-17T19:47:24.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5f9a7fc640 1 --2- 172.21.15.37:0/835104675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f9c0c98e0 0x7f5f9c140e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:24.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5f9affd640 1 -- 172.21.15.37:0/835104675 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f9c0c9cf0 msgr2=0x7f5f9c13cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:24.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5f9affd640 1 --2- 172.21.15.37:0/835104675 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f9c0c9cf0 0x7f5f9c13cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:24.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5f9affd640 1 -- 172.21.15.37:0/835104675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f9c0c98e0 msgr2=0x7f5f9c140e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:24.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5f9affd640 1 --2- 172.21.15.37:0/835104675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f9c0c98e0 0x7f5f9c140e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:24.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5f9affd640 1 -- 172.21.15.37:0/835104675 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f9c155800 con 0x7f5f9c070fb0 2024-04-17T19:47:24.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5f9b7fe640 1 --2- 172.21.15.37:0/835104675 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f9c0c9cf0 0x7f5f9c13cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:47:24.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5f9affd640 1 --2- 172.21.15.37:0/835104675 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f9c070fb0 0x7f5f9c13e210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c0af000 tx=0x7f5f8c01baf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:24.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5f837fe640 1 -- 172.21.15.37:0/835104675 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c0c0070 con 0x7f5f9c070fb0 2024-04-17T19:47:24.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5f837fe640 1 -- 172.21.15.37:0/835104675 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f8c0ae550 con 0x7f5f9c070fb0 2024-04-17T19:47:24.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5fa1758640 1 -- 172.21.15.37:0/835104675 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f9c13e750 con 0x7f5f9c070fb0 2024-04-17T19:47:24.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5f837fe640 1 -- 172.21.15.37:0/835104675 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c0b5070 con 0x7f5f9c070fb0 2024-04-17T19:47:24.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5fa1758640 1 -- 172.21.15.37:0/835104675 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f9c152b70 con 0x7f5f9c070fb0 2024-04-17T19:47:24.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5f837fe640 1 -- 172.21.15.37:0/835104675 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5f8c0b9070 con 0x7f5f9c070fb0 2024-04-17T19:47:24.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5f837fe640 1 -- 172.21.15.37:0/835104675 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f5f8c0bbb10 con 0x7f5f9c070fb0 2024-04-17T19:47:24.822 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.832+0000 7f5fa1758640 1 -- 172.21.15.37:0/835104675 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f9c04fea0 con 0x7f5f9c070fb0 2024-04-17T19:47:24.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.836+0000 7f5f837fe640 1 -- 172.21.15.37:0/835104675 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f5f8c0c9080 con 0x7f5f9c070fb0 2024-04-17T19:47:24.884 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:24.893+0000 7f30ad4e2180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T19:47:24.892 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.904+0000 7f5fa1758640 1 -- 172.21.15.37:0/835104675 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a567ac96-9ee4-495f-a20c-2686d59f1fe2", "id": 1} v 0) -- 0x7f5f9c0c8c20 con 0x7f5f9c070fb0 2024-04-17T19:47:24.897 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.908+0000 7f5f837fe640 1 -- 172.21.15.37:0/835104675 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a567ac96-9ee4-495f-a20c-2686d59f1fe2", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f5f8c0188b0 con 0x7f5f9c070fb0 2024-04-17T19:47:24.898 INFO:teuthology.orchestra.run.smithi037.stdout:1 2024-04-17T19:47:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.912+0000 7f5fa1758640 1 -- 172.21.15.37:0/835104675 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f9c070fb0 msgr2=0x7f5f9c13e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.912+0000 7f5fa1758640 1 --2- 172.21.15.37:0/835104675 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f9c070fb0 0x7f5f9c13e210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c0af000 tx=0x7f5f8c01baf0 comp rx=0 tx=0).stop 2024-04-17T19:47:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.912+0000 7f5fa1758640 1 -- 172.21.15.37:0/835104675 shutdown_connections 2024-04-17T19:47:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.912+0000 7f5fa1758640 1 --2- 172.21.15.37:0/835104675 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f9c0c9cf0 0x7f5f9c13cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:24.899 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.912+0000 7f5fa1758640 1 --2- 172.21.15.37:0/835104675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f9c0c98e0 0x7f5f9c140e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.912+0000 7f5fa1758640 1 --2- 172.21.15.37:0/835104675 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f9c070fb0 0x7f5f9c13e210 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.912+0000 7f5fa1758640 1 -- 172.21.15.37:0/835104675 >> 172.21.15.37:0/835104675 conn(0x7f5f9c072dc0 msgr2=0x7f5f9c157800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.912+0000 7f5fa1758640 1 -- 172.21.15.37:0/835104675 shutdown_connections 2024-04-17T19:47:24.900 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:24.912+0000 7f5fa1758640 1 -- 172.21.15.37:0/835104675 wait complete. 2024-04-17T19:47:24.917 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd new eb2aca46-5f62-4cd2-979f-906958ea2cc9 2 2024-04-17T19:47:25.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.076+0000 7f155a192640 1 Processor -- start 2024-04-17T19:47:25.065 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.076+0000 7f155a192640 1 -- start start 2024-04-17T19:47:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f155a192640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1554155fe0 0x7f155415c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f155a192640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f15541648f0 0x7f1554166d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f155a192640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f155406c0d0 con 0x7f15540c8fd0 2024-04-17T19:47:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f155a192640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f155415cf10 con 0x7f1554155fe0 2024-04-17T19:47:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f155a192640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f155415fc60 con 0x7f15541648f0 2024-04-17T19:47:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f15537fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f15540c8fd0 0x7f1554067ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48796/0 (socket says 172.21.15.37:48796) 2024-04-17T19:47:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f15537fe640 1 -- 172.21.15.37:0/2817135115 learned_addr learned my addr 172.21.15.37:0/2817135115 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-17T19:47:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f1553fff640 1 --2- 172.21.15.37:0/2817135115 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f15541648f0 0x7f1554166d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.068 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f1552ffd640 1 --2- 172.21.15.37:0/2817135115 >> v2:172.21.15.29:3300/0 conn(0x7f1554155fe0 0x7f155415c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f15527fc640 1 -- 172.21.15.37:0/2817135115 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2576050329 0 0) 0x7f155406c0d0 con 0x7f15540c8fd0 2024-04-17T19:47:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f15527fc640 1 -- 172.21.15.37:0/2817135115 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f153c003530 con 0x7f15540c8fd0 2024-04-17T19:47:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f1553fff640 1 -- 172.21.15.37:0/2817135115 >> v2:172.21.15.29:3300/0 conn(0x7f1554155fe0 msgr2=0x7f155415c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f1553fff640 1 --2- 172.21.15.37:0/2817135115 >> v2:172.21.15.29:3300/0 conn(0x7f1554155fe0 0x7f155415c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f1553fff640 1 -- 172.21.15.37:0/2817135115 >> v1:172.21.15.37:6789/0 conn(0x7f15540c8fd0 legacy=0x7f1554067ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f1553fff640 1 -- 172.21.15.37:0/2817135115 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f155406c3b0 con 0x7f15541648f0 2024-04-17T19:47:25.069 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f1552ffd640 1 --2- 172.21.15.37:0/2817135115 >> v2:172.21.15.29:3300/0 conn(0x7f1554155fe0 0x7f155415c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:25.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f1553fff640 1 --2- 172.21.15.37:0/2817135115 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f15541648f0 0x7f1554166d10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f154c009d90 tx=0x7f154c01f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf7ad2e7b37080aa server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:25.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f15527fc640 1 -- 172.21.15.37:0/2817135115 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154c0b4070 con 0x7f15541648f0 2024-04-17T19:47:25.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f15527fc640 1 -- 172.21.15.37:0/2817135115 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f154c0af600 con 0x7f15541648f0 2024-04-17T19:47:25.070 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.080+0000 7f15527fc640 1 -- 172.21.15.37:0/2817135115 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154c0af9d0 con 0x7f15541648f0 2024-04-17T19:47:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f155a192640 1 -- 172.21.15.37:0/2817135115 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f15541648f0 msgr2=0x7f1554166d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f155a192640 1 --2- 172.21.15.37:0/2817135115 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f15541648f0 0x7f1554166d10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f154c009d90 tx=0x7f154c01f150 comp rx=0 tx=0).stop 2024-04-17T19:47:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f155a192640 1 -- 172.21.15.37:0/2817135115 shutdown_connections 2024-04-17T19:47:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f155a192640 1 --2- 172.21.15.37:0/2817135115 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f15541648f0 0x7f1554166d10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f155a192640 1 --2- 172.21.15.37:0/2817135115 >> v2:172.21.15.29:3300/0 conn(0x7f1554155fe0 0x7f155415c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f155a192640 1 -- 172.21.15.37:0/2817135115 >> 172.21.15.37:0/2817135115 conn(0x7f15540caec0 msgr2=0x7f15540cb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f155a192640 1 -- 172.21.15.37:0/2817135115 shutdown_connections 2024-04-17T19:47:25.071 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f155a192640 1 -- 172.21.15.37:0/2817135115 wait complete. 2024-04-17T19:47:25.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f155a192640 1 Processor -- start 2024-04-17T19:47:25.073 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f155a192640 1 -- start start 2024-04-17T19:47:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f155a192640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15540c8fd0 0x7f1554143620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f155a192640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1554155fe0 0x7f1554148e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f155a192640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15541648f0 0x7f15541577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f155a192640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f155415f8f0 con 0x7f15540c8fd0 2024-04-17T19:47:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f155a192640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f15540c93e0 con 0x7f15541648f0 2024-04-17T19:47:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f155a192640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f15541563f0 con 0x7f1554155fe0 2024-04-17T19:47:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f1552ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1554155fe0 0x7f1554148e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f1553fff640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15541648f0 0x7f15541577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f1552ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1554155fe0 0x7f1554148e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48636/0 (socket says 172.21.15.37:48636) 2024-04-17T19:47:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f1552ffd640 1 -- 172.21.15.37:0/4181751374 learned_addr learned my addr 172.21.15.37:0/4181751374 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-17T19:47:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.084+0000 7f15537fe640 1 --2- 172.21.15.37:0/4181751374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15540c8fd0 0x7f1554143620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.088+0000 7f1552ffd640 1 -- 172.21.15.37:0/4181751374 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15541648f0 msgr2=0x7f15541577e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.076 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.088+0000 7f1552ffd640 1 --2- 172.21.15.37:0/4181751374 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15541648f0 0x7f15541577e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.088+0000 7f1552ffd640 1 -- 172.21.15.37:0/4181751374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15540c8fd0 msgr2=0x7f1554143620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.088+0000 7f1552ffd640 1 --2- 172.21.15.37:0/4181751374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15540c8fd0 0x7f1554143620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.088+0000 7f1552ffd640 1 -- 172.21.15.37:0/4181751374 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1554149380 con 0x7f1554155fe0 2024-04-17T19:47:25.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.088+0000 7f1553fff640 1 --2- 172.21.15.37:0/4181751374 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15541648f0 0x7f15541577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:47:25.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.088+0000 7f1552ffd640 1 --2- 172.21.15.37:0/4181751374 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1554155fe0 0x7f1554148e40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f154400d860 tx=0x7f154400dd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:25.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.088+0000 7f153bfff640 1 -- 172.21.15.37:0/4181751374 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1544007e30 con 0x7f1554155fe0 2024-04-17T19:47:25.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.088+0000 7f153bfff640 1 -- 172.21.15.37:0/4181751374 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1544019070 con 0x7f1554155fe0 2024-04-17T19:47:25.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.088+0000 7f155a192640 1 -- 172.21.15.37:0/4181751374 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f155414acb0 con 0x7f1554155fe0 2024-04-17T19:47:25.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.088+0000 7f153bfff640 1 -- 172.21.15.37:0/4181751374 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f154400bb10 con 0x7f1554155fe0 2024-04-17T19:47:25.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.088+0000 7f155a192640 1 -- 172.21.15.37:0/4181751374 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1554159280 con 0x7f1554155fe0 2024-04-17T19:47:25.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.088+0000 7f153bfff640 1 -- 172.21.15.37:0/4181751374 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f154400bcb0 con 0x7f1554155fe0 2024-04-17T19:47:25.077 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.088+0000 7f153bfff640 1 -- 172.21.15.37:0/4181751374 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f15440148f0 con 0x7f1554155fe0 2024-04-17T19:47:25.078 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.088+0000 7f155a192640 1 -- 172.21.15.37:0/4181751374 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1554069b90 con 0x7f1554155fe0 2024-04-17T19:47:25.079 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.092+0000 7f153bfff640 1 -- 172.21.15.37:0/4181751374 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f15440144d0 con 0x7f1554155fe0 2024-04-17T19:47:25.148 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.160+0000 7f155a192640 1 -- 172.21.15.37:0/4181751374 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd new", "uuid": "eb2aca46-5f62-4cd2-979f-906958ea2cc9", "id": 2} v 0) -- 0x7f1554159740 con 0x7f1554155fe0 2024-04-17T19:47:25.153 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.164+0000 7f153bfff640 1 -- 172.21.15.37:0/4181751374 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eb2aca46-5f62-4cd2-979f-906958ea2cc9", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f1544014b90 con 0x7f1554155fe0 2024-04-17T19:47:25.153 INFO:teuthology.orchestra.run.smithi037.stdout:2 2024-04-17T19:47:25.154 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.164+0000 7f155a192640 1 -- 172.21.15.37:0/4181751374 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1554155fe0 msgr2=0x7f1554148e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.164+0000 7f155a192640 1 --2- 172.21.15.37:0/4181751374 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1554155fe0 0x7f1554148e40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f154400d860 tx=0x7f154400dd20 comp rx=0 tx=0).stop 2024-04-17T19:47:25.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.164+0000 7f155a192640 1 -- 172.21.15.37:0/4181751374 shutdown_connections 2024-04-17T19:47:25.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.164+0000 7f155a192640 1 --2- 172.21.15.37:0/4181751374 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15541648f0 0x7f15541577e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.164+0000 7f155a192640 1 --2- 172.21.15.37:0/4181751374 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1554155fe0 0x7f1554148e40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.164+0000 7f155a192640 1 --2- 172.21.15.37:0/4181751374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15540c8fd0 0x7f1554143620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.164+0000 7f155a192640 1 -- 172.21.15.37:0/4181751374 >> 172.21.15.37:0/4181751374 conn(0x7f15540caec0 msgr2=0x7f1554067ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:25.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.164+0000 7f155a192640 1 -- 172.21.15.37:0/4181751374 shutdown_connections 2024-04-17T19:47:25.155 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.168+0000 7f155a192640 1 -- 172.21.15.37:0/4181751374 wait complete. 2024-04-17T19:47:25.174 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd new 667781ea-7172-42ab-9a97-d03fa0b273ee 3 2024-04-17T19:47:25.321 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.332+0000 7f16bc4e3640 1 Processor -- start 2024-04-17T19:47:25.322 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.332+0000 7f16bc4e3640 1 -- start start 2024-04-17T19:47:25.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.336+0000 7f16bc4e3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16b40c8dc0 0x7f16b4072e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.336+0000 7f16bc4e3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f16b4073370 0x7f16b4163610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.336+0000 7f16bc4e3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16b40c79c0 con 0x7f16b4141ae0 2024-04-17T19:47:25.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.336+0000 7f16bc4e3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f16b415c9f0 con 0x7f16b4073370 2024-04-17T19:47:25.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.336+0000 7f16bc4e3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f16b4141ef0 con 0x7f16b40c8dc0 2024-04-17T19:47:25.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.336+0000 7f16b9a7e640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16b40c8dc0 0x7f16b4072e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.336+0000 7f16ba27f640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f16b4141ae0 0x7f16b415c2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48802/0 (socket says 172.21.15.37:48802) 2024-04-17T19:47:25.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.336+0000 7f16ba27f640 1 -- 172.21.15.37:0/2422193407 learned_addr learned my addr 172.21.15.37:0/2422193407 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-17T19:47:25.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.336+0000 7f16baa80640 1 --2- 172.21.15.37:0/2422193407 >> v2:172.21.15.29:3300/0 conn(0x7f16b4073370 0x7f16b4163610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.324 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.336+0000 7f16b9a7e640 1 -- 172.21.15.37:0/2422193407 >> v2:172.21.15.29:3300/0 conn(0x7f16b4073370 msgr2=0x7f16b4163610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:25.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.336+0000 7f16b9a7e640 1 --2- 172.21.15.37:0/2422193407 >> v2:172.21.15.29:3300/0 conn(0x7f16b4073370 0x7f16b4163610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.336+0000 7f16b927d640 1 -- 172.21.15.37:0/2422193407 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1304059398 0 0) 0x7f16b40c79c0 con 0x7f16b4141ae0 2024-04-17T19:47:25.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.336+0000 7f16b9a7e640 1 -- 172.21.15.37:0/2422193407 >> v1:172.21.15.37:6789/0 conn(0x7f16b4141ae0 legacy=0x7f16b415c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.325 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.336+0000 7f16b9a7e640 1 -- 172.21.15.37:0/2422193407 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b40c6360 con 0x7f16b40c8dc0 2024-04-17T19:47:25.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.336+0000 7f16b9a7e640 1 --2- 172.21.15.37:0/2422193407 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16b40c8dc0 0x7f16b4072e30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f16a40097c0 tx=0x7f16a401ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a08a7d91464c30c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:25.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.336+0000 7f16b927d640 1 -- 172.21.15.37:0/2422193407 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a40b5070 con 0x7f16b40c8dc0 2024-04-17T19:47:25.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.336+0000 7f16b927d640 1 -- 172.21.15.37:0/2422193407 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16a40a8990 con 0x7f16b40c8dc0 2024-04-17T19:47:25.326 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.336+0000 7f16b927d640 1 -- 172.21.15.37:0/2422193407 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a40a8d60 con 0x7f16b40c8dc0 2024-04-17T19:47:25.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16bc4e3640 1 -- 172.21.15.37:0/2422193407 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16b40c8dc0 msgr2=0x7f16b4072e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16bc4e3640 1 --2- 172.21.15.37:0/2422193407 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16b40c8dc0 0x7f16b4072e30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f16a40097c0 tx=0x7f16a401ec90 comp rx=0 tx=0).stop 2024-04-17T19:47:25.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16bc4e3640 1 -- 172.21.15.37:0/2422193407 shutdown_connections 2024-04-17T19:47:25.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16bc4e3640 1 --2- 172.21.15.37:0/2422193407 >> v2:172.21.15.29:3300/0 conn(0x7f16b4073370 0x7f16b4163610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16bc4e3640 1 --2- 172.21.15.37:0/2422193407 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16b40c8dc0 0x7f16b4072e30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.327 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16bc4e3640 1 -- 172.21.15.37:0/2422193407 >> 172.21.15.37:0/2422193407 conn(0x7f16b40caec0 msgr2=0x7f16b40cb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:25.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16bc4e3640 1 -- 172.21.15.37:0/2422193407 shutdown_connections 2024-04-17T19:47:25.328 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16bc4e3640 1 -- 172.21.15.37:0/2422193407 wait complete. 2024-04-17T19:47:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16bc4e3640 1 Processor -- start 2024-04-17T19:47:25.329 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16bc4e3640 1 -- start start 2024-04-17T19:47:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16bc4e3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16b4073370 0x7f16b4137c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16bc4e3640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16b40c8dc0 0x7f16b41ef8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16bc4e3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16b4141ae0 0x7f16b413f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16bc4e3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f16b40c7c30 con 0x7f16b4141ae0 2024-04-17T19:47:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16bc4e3640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f16b40740e0 con 0x7f16b40c8dc0 2024-04-17T19:47:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16bc4e3640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f16b40c7f10 con 0x7f16b4073370 2024-04-17T19:47:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16ba27f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16b4073370 0x7f16b4137c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16b9a7e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16b40c8dc0 0x7f16b41ef8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16ba27f640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16b4073370 0x7f16b4137c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48648/0 (socket says 172.21.15.37:48648) 2024-04-17T19:47:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16b9a7e640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16b40c8dc0 0x7f16b41ef8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34918/0 (socket says 172.21.15.37:34918) 2024-04-17T19:47:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16ba27f640 1 -- 172.21.15.37:0/914147281 learned_addr learned my addr 172.21.15.37:0/914147281 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-17T19:47:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.340+0000 7f16baa80640 1 --2- 172.21.15.37:0/914147281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16b4141ae0 0x7f16b413f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.332 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.344+0000 7f16ba27f640 1 -- 172.21.15.37:0/914147281 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16b40c8dc0 msgr2=0x7f16b41ef8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.344+0000 7f16ba27f640 1 --2- 172.21.15.37:0/914147281 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16b40c8dc0 0x7f16b41ef8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.344+0000 7f16ba27f640 1 -- 172.21.15.37:0/914147281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16b4141ae0 msgr2=0x7f16b413f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.344+0000 7f16ba27f640 1 --2- 172.21.15.37:0/914147281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16b4141ae0 0x7f16b413f200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.344+0000 7f16ba27f640 1 -- 172.21.15.37:0/914147281 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b41efde0 con 0x7f16b4073370 2024-04-17T19:47:25.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.344+0000 7f16b9a7e640 1 --2- 172.21.15.37:0/914147281 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16b40c8dc0 0x7f16b41ef8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:47:25.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.344+0000 7f16ba27f640 1 --2- 172.21.15.37:0/914147281 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16b4073370 0x7f16b4137c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f16b000ea30 tx=0x7f16b000ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:25.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.344+0000 7f16aaffd640 1 -- 172.21.15.37:0/914147281 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b0004d80 con 0x7f16b4073370 2024-04-17T19:47:25.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.344+0000 7f16aaffd640 1 -- 172.21.15.37:0/914147281 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16b0016020 con 0x7f16b4073370 2024-04-17T19:47:25.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.344+0000 7f16bc4e3640 1 -- 172.21.15.37:0/914147281 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b4138570 con 0x7f16b4073370 2024-04-17T19:47:25.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.344+0000 7f16aaffd640 1 -- 172.21.15.37:0/914147281 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b0001d20 con 0x7f16b4073370 2024-04-17T19:47:25.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.344+0000 7f16bc4e3640 1 -- 172.21.15.37:0/914147281 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b41f2790 con 0x7f16b4073370 2024-04-17T19:47:25.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.344+0000 7f16aaffd640 1 -- 172.21.15.37:0/914147281 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f16b0001ec0 con 0x7f16b4073370 2024-04-17T19:47:25.333 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.344+0000 7f16aaffd640 1 -- 172.21.15.37:0/914147281 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f16b0011b30 con 0x7f16b4073370 2024-04-17T19:47:25.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.344+0000 7f16bc4e3640 1 -- 172.21.15.37:0/914147281 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16b4069b90 con 0x7f16b4073370 2024-04-17T19:47:25.334 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.344+0000 7f16aaffd640 1 -- 172.21.15.37:0/914147281 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f16b0020020 con 0x7f16b4073370 2024-04-17T19:47:25.403 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.416+0000 7f16bc4e3640 1 -- 172.21.15.37:0/914147281 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd new", "uuid": "667781ea-7172-42ab-9a97-d03fa0b273ee", "id": 3} v 0) -- 0x7f16b40c81e0 con 0x7f16b4073370 2024-04-17T19:47:25.409 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.420+0000 7f16aaffd640 1 -- 172.21.15.37:0/914147281 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "667781ea-7172-42ab-9a97-d03fa0b273ee", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f16b41efde0 con 0x7f16b4073370 2024-04-17T19:47:25.409 INFO:teuthology.orchestra.run.smithi037.stdout:3 2024-04-17T19:47:25.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.420+0000 7f16bc4e3640 1 -- 172.21.15.37:0/914147281 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16b4073370 msgr2=0x7f16b4137c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.420+0000 7f16bc4e3640 1 --2- 172.21.15.37:0/914147281 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16b4073370 0x7f16b4137c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f16b000ea30 tx=0x7f16b000ee30 comp rx=0 tx=0).stop 2024-04-17T19:47:25.410 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.420+0000 7f16bc4e3640 1 -- 172.21.15.37:0/914147281 shutdown_connections 2024-04-17T19:47:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.420+0000 7f16bc4e3640 1 --2- 172.21.15.37:0/914147281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16b4141ae0 0x7f16b413f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.420+0000 7f16bc4e3640 1 --2- 172.21.15.37:0/914147281 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16b40c8dc0 0x7f16b41ef8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.420+0000 7f16bc4e3640 1 --2- 172.21.15.37:0/914147281 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16b4073370 0x7f16b4137c40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.420+0000 7f16bc4e3640 1 -- 172.21.15.37:0/914147281 >> 172.21.15.37:0/914147281 conn(0x7f16b40caec0 msgr2=0x7f16b40cb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.420+0000 7f16bc4e3640 1 -- 172.21.15.37:0/914147281 shutdown_connections 2024-04-17T19:47:25.411 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.420+0000 7f16bc4e3640 1 -- 172.21.15.37:0/914147281 wait complete. 2024-04-17T19:47:25.421 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:25.432+0000 7f80eb593180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T19:47:25.428 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd new 1e88ed79-c62d-4257-ac5e-51dcefaa1f94 4 2024-04-17T19:47:25.479 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:25.492+0000 7f80eb593180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T19:47:25.537 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:25.548+0000 7f80eb593180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T19:47:25.542 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:25.553+0000 7f30ad4e2180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T19:47:25.569 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.580+0000 7f556ba58640 1 Processor -- start 2024-04-17T19:47:25.570 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.580+0000 7f556ba58640 1 -- start start 2024-04-17T19:47:25.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.584+0000 7f556ba58640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5564152fa0 0x7f55640c55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.571 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.584+0000 7f556ba58640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f55640c5af0 0x7f55641636a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.584+0000 7f556ba58640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55640c85c0 con 0x7f5564070fb0 2024-04-17T19:47:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.584+0000 7f556ba58640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f55641597a0 con 0x7f55640c5af0 2024-04-17T19:47:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.584+0000 7f556ba58640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5564159a40 con 0x7f5564152fa0 2024-04-17T19:47:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.584+0000 7f55697f4640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f5564070fb0 0x7f55640c98e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48806/0 (socket says 172.21.15.37:48806) 2024-04-17T19:47:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.584+0000 7f5568ff3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5564152fa0 0x7f55640c55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.584+0000 7f55697f4640 1 -- 172.21.15.37:0/2978964631 learned_addr learned my addr 172.21.15.37:0/2978964631 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-17T19:47:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.584+0000 7f5569ff5640 1 --2- 172.21.15.37:0/2978964631 >> v2:172.21.15.37:3300/0 conn(0x7f55640c5af0 0x7f55641636a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.572 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.584+0000 7f5563fff640 1 -- 172.21.15.37:0/2978964631 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1713590965 0 0) 0x7f555c0014a0 con 0x7f5564070fb0 2024-04-17T19:47:25.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.584+0000 7f5568ff3640 1 -- 172.21.15.37:0/2978964631 >> v2:172.21.15.37:3300/0 conn(0x7f55640c5af0 msgr2=0x7f55641636a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:25.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.584+0000 7f5568ff3640 1 --2- 172.21.15.37:0/2978964631 >> v2:172.21.15.37:3300/0 conn(0x7f55640c5af0 0x7f55641636a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.584+0000 7f5568ff3640 1 -- 172.21.15.37:0/2978964631 >> v1:172.21.15.37:6789/0 conn(0x7f5564070fb0 legacy=0x7f55640c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.573 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.584+0000 7f5568ff3640 1 -- 172.21.15.37:0/2978964631 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55640c7800 con 0x7f5564152fa0 2024-04-17T19:47:25.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.584+0000 7f5568ff3640 1 --2- 172.21.15.37:0/2978964631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5564152fa0 0x7f55640c55b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f555800b870 tx=0x7f555801f140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2faea71c2800564d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:25.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.584+0000 7f5563fff640 1 -- 172.21.15.37:0/2978964631 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55580b4070 con 0x7f5564152fa0 2024-04-17T19:47:25.574 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.588+0000 7f5563fff640 1 -- 172.21.15.37:0/2978964631 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55580af600 con 0x7f5564152fa0 2024-04-17T19:47:25.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.588+0000 7f5563fff640 1 -- 172.21.15.37:0/2978964631 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55580af9d0 con 0x7f5564152fa0 2024-04-17T19:47:25.575 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.588+0000 7f556ba58640 1 -- 172.21.15.37:0/2978964631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5564152fa0 msgr2=0x7f55640c55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.588+0000 7f556ba58640 1 --2- 172.21.15.37:0/2978964631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5564152fa0 0x7f55640c55b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f555800b870 tx=0x7f555801f140 comp rx=0 tx=0).stop 2024-04-17T19:47:25.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.588+0000 7f556ba58640 1 -- 172.21.15.37:0/2978964631 shutdown_connections 2024-04-17T19:47:25.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.588+0000 7f556ba58640 1 --2- 172.21.15.37:0/2978964631 >> v2:172.21.15.37:3300/0 conn(0x7f55640c5af0 0x7f55641636a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.588+0000 7f556ba58640 1 --2- 172.21.15.37:0/2978964631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5564152fa0 0x7f55640c55b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.588+0000 7f556ba58640 1 -- 172.21.15.37:0/2978964631 >> 172.21.15.37:0/2978964631 conn(0x7f5564072dc0 msgr2=0x7f55640731d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:25.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.588+0000 7f556ba58640 1 -- 172.21.15.37:0/2978964631 shutdown_connections 2024-04-17T19:47:25.576 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.588+0000 7f556ba58640 1 -- 172.21.15.37:0/2978964631 wait complete. 2024-04-17T19:47:25.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.588+0000 7f556ba58640 1 Processor -- start 2024-04-17T19:47:25.577 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.588+0000 7f556ba58640 1 -- start start 2024-04-17T19:47:25.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f556ba58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5564070fb0 0x7f556413d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f556ba58640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55640c5af0 0x7f5564157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f556ba58640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5564152fa0 0x7f55641f1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f556ba58640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f55640c80a0 con 0x7f5564070fb0 2024-04-17T19:47:25.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f556ba58640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f55640c7dc0 con 0x7f5564152fa0 2024-04-17T19:47:25.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f556ba58640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f55641533b0 con 0x7f55640c5af0 2024-04-17T19:47:25.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f5568ff3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55640c5af0 0x7f5564157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f5568ff3640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55640c5af0 0x7f5564157350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48660/0 (socket says 172.21.15.37:48660) 2024-04-17T19:47:25.580 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f5569ff5640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5564152fa0 0x7f55641f1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f5568ff3640 1 -- 172.21.15.37:0/1127699534 learned_addr learned my addr 172.21.15.37:0/1127699534 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-17T19:47:25.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f55697f4640 1 --2- 172.21.15.37:0/1127699534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5564070fb0 0x7f556413d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f5568ff3640 1 -- 172.21.15.37:0/1127699534 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5564152fa0 msgr2=0x7f55641f1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f5568ff3640 1 --2- 172.21.15.37:0/1127699534 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5564152fa0 0x7f55641f1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f5568ff3640 1 -- 172.21.15.37:0/1127699534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5564070fb0 msgr2=0x7f556413d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f5568ff3640 1 --2- 172.21.15.37:0/1127699534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5564070fb0 0x7f556413d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f5568ff3640 1 -- 172.21.15.37:0/1127699534 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f556415a670 con 0x7f55640c5af0 2024-04-17T19:47:25.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f5569ff5640 1 --2- 172.21.15.37:0/1127699534 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5564152fa0 0x7f55641f1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:47:25.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f55697f4640 1 --2- 172.21.15.37:0/1127699534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5564070fb0 0x7f556413d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:25.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f5568ff3640 1 --2- 172.21.15.37:0/1127699534 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55640c5af0 0x7f5564157350 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f555801d7f0 tx=0x7f555801bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:25.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f5561ffb640 1 -- 172.21.15.37:0/1127699534 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55580b4040 con 0x7f55640c5af0 2024-04-17T19:47:25.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f5561ffb640 1 -- 172.21.15.37:0/1127699534 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55580c1070 con 0x7f55640c5af0 2024-04-17T19:47:25.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f556ba58640 1 -- 172.21.15.37:0/1127699534 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f556415a950 con 0x7f55640c5af0 2024-04-17T19:47:25.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f5561ffb640 1 -- 172.21.15.37:0/1127699534 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55580ba6b0 con 0x7f55640c5af0 2024-04-17T19:47:25.581 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f556ba58640 1 -- 172.21.15.37:0/1127699534 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f556415af10 con 0x7f55640c5af0 2024-04-17T19:47:25.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f5561ffb640 1 -- 172.21.15.37:0/1127699534 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f55580ba850 con 0x7f55640c5af0 2024-04-17T19:47:25.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f5561ffb640 1 -- 172.21.15.37:0/1127699534 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f55580c6ce0 con 0x7f55640c5af0 2024-04-17T19:47:25.582 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.592+0000 7f556ba58640 1 -- 172.21.15.37:0/1127699534 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f556404fea0 con 0x7f55640c5af0 2024-04-17T19:47:25.583 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.596+0000 7f5561ffb640 1 -- 172.21.15.37:0/1127699534 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f5558018ca0 con 0x7f55640c5af0 2024-04-17T19:47:25.608 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:25.621+0000 7f30ad4e2180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T19:47:25.619 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:25.632+0000 7f80eb593180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T19:47:25.649 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.660+0000 7f556ba58640 1 -- 172.21.15.37:0/1127699534 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1e88ed79-c62d-4257-ac5e-51dcefaa1f94", "id": 4} v 0) -- 0x7f55640c8c20 con 0x7f55640c5af0 2024-04-17T19:47:25.654 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.664+0000 7f5561ffb640 1 -- 172.21.15.37:0/1127699534 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1e88ed79-c62d-4257-ac5e-51dcefaa1f94", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f55580bac80 con 0x7f55640c5af0 2024-04-17T19:47:25.655 INFO:teuthology.orchestra.run.smithi037.stdout:4 2024-04-17T19:47:25.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.668+0000 7f556ba58640 1 -- 172.21.15.37:0/1127699534 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55640c5af0 msgr2=0x7f5564157350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.668+0000 7f556ba58640 1 --2- 172.21.15.37:0/1127699534 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55640c5af0 0x7f5564157350 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f555801d7f0 tx=0x7f555801bee0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.668+0000 7f556ba58640 1 -- 172.21.15.37:0/1127699534 shutdown_connections 2024-04-17T19:47:25.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.668+0000 7f556ba58640 1 --2- 172.21.15.37:0/1127699534 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5564152fa0 0x7f55641f1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.668+0000 7f556ba58640 1 --2- 172.21.15.37:0/1127699534 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55640c5af0 0x7f5564157350 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.668+0000 7f556ba58640 1 --2- 172.21.15.37:0/1127699534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5564070fb0 0x7f556413d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.668+0000 7f556ba58640 1 -- 172.21.15.37:0/1127699534 >> 172.21.15.37:0/1127699534 conn(0x7f5564072dc0 msgr2=0x7f55640ccba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:25.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.668+0000 7f556ba58640 1 -- 172.21.15.37:0/1127699534 shutdown_connections 2024-04-17T19:47:25.656 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.668+0000 7f556ba58640 1 -- 172.21.15.37:0/1127699534 wait complete. 2024-04-17T19:47:25.673 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd new 0857a0aa-c148-4ce9-baa5-96c519989c8a 5 2024-04-17T19:47:25.673 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:25.685+0000 7f30ad4e2180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T19:47:25.676 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:25.688+0000 7f80eb593180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T19:47:25.760 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:25.772+0000 7f80eb593180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T19:47:25.767 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:25.777+0000 7f30ad4e2180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T19:47:25.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.824+0000 7fe52026a640 1 Processor -- start 2024-04-17T19:47:25.813 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.824+0000 7fe52026a640 1 -- start start 2024-04-17T19:47:25.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.828+0000 7fe52026a640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe518155fe0 0x7fe51815c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.828+0000 7fe52026a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe51806c0d0 con 0x7fe5181648f0 2024-04-17T19:47:25.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.828+0000 7fe52026a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe51806c3b0 con 0x7fe5180c8fd0 2024-04-17T19:47:25.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.828+0000 7fe52026a640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fe51815cf10 con 0x7fe518155fe0 2024-04-17T19:47:25.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.828+0000 7fe51e006640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe5180c8fd0 0x7fe518067ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48812/0 (socket says 172.21.15.37:48812) 2024-04-17T19:47:25.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.828+0000 7fe51e006640 1 -- 172.21.15.37:0/3868894140 learned_addr learned my addr 172.21.15.37:0/3868894140 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-17T19:47:25.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.828+0000 7fe51d805640 1 --2- 172.21.15.37:0/3868894140 >> v2:172.21.15.37:3300/0 conn(0x7fe518155fe0 0x7fe51815c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.828+0000 7fe51d004640 1 -- 172.21.15.37:0/3868894140 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 170723954 0 0) 0x7fe51806c3b0 con 0x7fe5180c8fd0 2024-04-17T19:47:25.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.828+0000 7fe51d004640 1 -- 172.21.15.37:0/3868894140 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe504003530 con 0x7fe5180c8fd0 2024-04-17T19:47:25.816 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.828+0000 7fe51d805640 1 -- 172.21.15.37:0/3868894140 >> v1:172.21.15.37:6789/0 conn(0x7fe5180c8fd0 legacy=0x7fe518067ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.828+0000 7fe51d805640 1 -- 172.21.15.37:0/3868894140 >> v1:172.21.15.29:6789/0 conn(0x7fe5181648f0 legacy=0x7fe518166d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.817 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.828+0000 7fe51d805640 1 -- 172.21.15.37:0/3868894140 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe51815f610 con 0x7fe518155fe0 2024-04-17T19:47:25.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.828+0000 7fe51d805640 1 --2- 172.21.15.37:0/3868894140 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe518155fe0 0x7fe51815c9d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe5080097c0 tx=0x7fe50801ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7e91b23d248c2c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:25.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.828+0000 7fe51d004640 1 -- 172.21.15.37:0/3868894140 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5080b5070 con 0x7fe518155fe0 2024-04-17T19:47:25.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.828+0000 7fe51d004640 1 -- 172.21.15.37:0/3868894140 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5080a8990 con 0x7fe518155fe0 2024-04-17T19:47:25.818 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.828+0000 7fe51d004640 1 -- 172.21.15.37:0/3868894140 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5080a8d60 con 0x7fe518155fe0 2024-04-17T19:47:25.818 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:25.828+0000 7f80eb593180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T19:47:25.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.832+0000 7fe52026a640 1 -- 172.21.15.37:0/3868894140 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe518155fe0 msgr2=0x7fe51815c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.832+0000 7fe52026a640 1 --2- 172.21.15.37:0/3868894140 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe518155fe0 0x7fe51815c9d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe5080097c0 tx=0x7fe50801ec90 comp rx=0 tx=0).stop 2024-04-17T19:47:25.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.832+0000 7fe52026a640 1 -- 172.21.15.37:0/3868894140 shutdown_connections 2024-04-17T19:47:25.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.832+0000 7fe52026a640 1 --2- 172.21.15.37:0/3868894140 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe518155fe0 0x7fe51815c9d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.832+0000 7fe52026a640 1 -- 172.21.15.37:0/3868894140 >> 172.21.15.37:0/3868894140 conn(0x7fe5180caec0 msgr2=0x7fe5180cb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:25.819 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.832+0000 7fe52026a640 1 -- 172.21.15.37:0/3868894140 shutdown_connections 2024-04-17T19:47:25.820 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.832+0000 7fe52026a640 1 -- 172.21.15.37:0/3868894140 wait complete. 2024-04-17T19:47:25.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.832+0000 7fe52026a640 1 Processor -- start 2024-04-17T19:47:25.821 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.832+0000 7fe52026a640 1 -- start start 2024-04-17T19:47:25.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.832+0000 7fe52026a640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5180c8fd0 0x7fe5181f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.832+0000 7fe52026a640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe518155fe0 0x7fe518143c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.832+0000 7fe52026a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5181648f0 0x7fe518144190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:25.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.832+0000 7fe52026a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe5180c93e0 con 0x7fe5181648f0 2024-04-17T19:47:25.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.832+0000 7fe52026a640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe51815fc60 con 0x7fe518155fe0 2024-04-17T19:47:25.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.832+0000 7fe52026a640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe5181563f0 con 0x7fe5180c8fd0 2024-04-17T19:47:25.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe51d805640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe518155fe0 0x7fe518143c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe51e006640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5180c8fd0 0x7fe5181f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe51e006640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5180c8fd0 0x7fe5181f2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48662/0 (socket says 172.21.15.37:48662) 2024-04-17T19:47:25.824 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe51d805640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe518155fe0 0x7fe518143c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34958/0 (socket says 172.21.15.37:34958) 2024-04-17T19:47:25.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe51e006640 1 -- 172.21.15.37:0/2369728525 learned_addr learned my addr 172.21.15.37:0/2369728525 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-17T19:47:25.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe51e807640 1 --2- 172.21.15.37:0/2369728525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5181648f0 0x7fe518144190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:25.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe51d805640 1 -- 172.21.15.37:0/2369728525 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5180c8fd0 msgr2=0x7fe5181f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe51d805640 1 --2- 172.21.15.37:0/2369728525 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5180c8fd0 0x7fe5181f2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe51d805640 1 -- 172.21.15.37:0/2369728525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5181648f0 msgr2=0x7fe518144190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe51d805640 1 --2- 172.21.15.37:0/2369728525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5181648f0 0x7fe518144190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe51d805640 1 -- 172.21.15.37:0/2369728525 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe518145cc0 con 0x7fe518155fe0 2024-04-17T19:47:25.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe51e006640 1 --2- 172.21.15.37:0/2369728525 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5180c8fd0 0x7fe5181f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:47:25.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe51d805640 1 --2- 172.21.15.37:0/2369728525 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe518155fe0 0x7fe518143c50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe5080af000 tx=0x7fe50801ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:25.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe50effd640 1 -- 172.21.15.37:0/2369728525 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5080b5040 con 0x7fe518155fe0 2024-04-17T19:47:25.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe50effd640 1 -- 172.21.15.37:0/2369728525 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5080bf070 con 0x7fe518155fe0 2024-04-17T19:47:25.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe50effd640 1 -- 172.21.15.37:0/2369728525 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5080b0960 con 0x7fe518155fe0 2024-04-17T19:47:25.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe52026a640 1 -- 172.21.15.37:0/2369728525 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5181f5990 con 0x7fe518155fe0 2024-04-17T19:47:25.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe52026a640 1 -- 172.21.15.37:0/2369728525 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5181f5f50 con 0x7fe518155fe0 2024-04-17T19:47:25.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe50effd640 1 -- 172.21.15.37:0/2369728525 <== mon.1 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe5080b0b00 con 0x7fe518155fe0 2024-04-17T19:47:25.825 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe50effd640 1 -- 172.21.15.37:0/2369728525 <== mon.1 v2:172.21.15.37:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fe5080c4af0 con 0x7fe518155fe0 2024-04-17T19:47:25.826 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.836+0000 7fe52026a640 1 -- 172.21.15.37:0/2369728525 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe518069b90 con 0x7fe518155fe0 2024-04-17T19:47:25.827 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.840+0000 7fe50effd640 1 -- 172.21.15.37:0/2369728525 <== mon.1 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fe5080187f0 con 0x7fe518155fe0 2024-04-17T19:47:25.833 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:25.845+0000 7f30ad4e2180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T19:47:25.903 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.916+0000 7fe52026a640 1 -- 172.21.15.37:0/2369728525 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0857a0aa-c148-4ce9-baa5-96c519989c8a", "id": 5} v 0) -- 0x7fe51806c960 con 0x7fe518155fe0 2024-04-17T19:47:25.909 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.920+0000 7fe50effd640 1 -- 172.21.15.37:0/2369728525 <== mon.1 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0857a0aa-c148-4ce9-baa5-96c519989c8a", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fe5080b0df0 con 0x7fe518155fe0 2024-04-17T19:47:25.910 INFO:teuthology.orchestra.run.smithi037.stdout:5 2024-04-17T19:47:25.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.924+0000 7fe52026a640 1 -- 172.21.15.37:0/2369728525 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe518155fe0 msgr2=0x7fe518143c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:25.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.924+0000 7fe52026a640 1 --2- 172.21.15.37:0/2369728525 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe518155fe0 0x7fe518143c50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe5080af000 tx=0x7fe50801ba30 comp rx=0 tx=0).stop 2024-04-17T19:47:25.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.924+0000 7fe52026a640 1 -- 172.21.15.37:0/2369728525 shutdown_connections 2024-04-17T19:47:25.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.924+0000 7fe52026a640 1 --2- 172.21.15.37:0/2369728525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5181648f0 0x7fe518144190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.924+0000 7fe52026a640 1 --2- 172.21.15.37:0/2369728525 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe518155fe0 0x7fe518143c50 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.924+0000 7fe52026a640 1 --2- 172.21.15.37:0/2369728525 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe5180c8fd0 0x7fe5181f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:25.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.924+0000 7fe52026a640 1 -- 172.21.15.37:0/2369728525 >> 172.21.15.37:0/2369728525 conn(0x7fe5180caec0 msgr2=0x7fe518067ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:25.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.924+0000 7fe52026a640 1 -- 172.21.15.37:0/2369728525 shutdown_connections 2024-04-17T19:47:25.911 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:25.924+0000 7fe52026a640 1 -- 172.21.15.37:0/2369728525 wait complete. 2024-04-17T19:47:25.928 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd new 6ce4a0ee-7778-4238-b1e5-3efaf90c9b81 6 2024-04-17T19:47:25.928 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:25.937+0000 7f30ad4e2180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T19:47:25.991 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:26.004+0000 7f80eb593180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T19:47:25.994 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:26.005+0000 7f30ad4e2180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T19:47:26.060 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:26.072+0000 7f80eb593180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T19:47:26.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.132+0000 7f69b0e9c640 1 Processor -- start 2024-04-17T19:47:26.122 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.132+0000 7f69b0e9c640 1 -- start start 2024-04-17T19:47:26.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69b0e9c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f69ac15c9d0 0x7f69ac168020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:26.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69b0e9c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69ac06c0d0 con 0x7f69ac0c8fd0 2024-04-17T19:47:26.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69b0e9c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69ac06c3b0 con 0x7f69ac155fe0 2024-04-17T19:47:26.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69b0e9c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f69ac15fc60 con 0x7f69ac15c9d0 2024-04-17T19:47:26.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69a9d9b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f69ac155fe0 0x7f69ac1648f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48828/0 (socket says 172.21.15.37:48828) 2024-04-17T19:47:26.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69a9d9b640 1 -- 172.21.15.37:0/3440126917 learned_addr learned my addr 172.21.15.37:0/3440126917 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-17T19:47:26.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69aad9d640 1 --2- 172.21.15.37:0/3440126917 >> v2:172.21.15.29:3300/0 conn(0x7f69ac15c9d0 0x7f69ac168020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:26.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69a959a640 1 -- 172.21.15.37:0/3440126917 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1730556728 0 0) 0x7f69ac06c3b0 con 0x7f69ac155fe0 2024-04-17T19:47:26.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69a959a640 1 -- 172.21.15.37:0/3440126917 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6994003530 con 0x7f69ac155fe0 2024-04-17T19:47:26.125 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69a959a640 1 -- 172.21.15.37:0/3440126917 <== mon.1 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1807524728 0 0) 0x7f6994003530 con 0x7f69ac155fe0 2024-04-17T19:47:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69a959a640 1 -- 172.21.15.37:0/3440126917 >> v2:172.21.15.29:3300/0 conn(0x7f69ac15c9d0 msgr2=0x7f69ac168020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69a959a640 1 --2- 172.21.15.37:0/3440126917 >> v2:172.21.15.29:3300/0 conn(0x7f69ac15c9d0 0x7f69ac168020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69a959a640 1 -- 172.21.15.37:0/3440126917 >> v1:172.21.15.29:6789/0 conn(0x7f69ac0c8fd0 legacy=0x7f69ac067ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69a959a640 1 -- 172.21.15.37:0/3440126917 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69ac15f610 con 0x7f69ac155fe0 2024-04-17T19:47:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69a959a640 1 -- 172.21.15.37:0/3440126917 <== mon.1 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f69a0001680 con 0x7f69ac155fe0 2024-04-17T19:47:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69a959a640 1 -- 172.21.15.37:0/3440126917 >> v1:172.21.15.37:6789/0 conn(0x7f69ac155fe0 legacy=0x7f69ac1648f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69a959a640 1 --2- 172.21.15.37:0/3440126917 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69940039c0 0x7f6994003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69a959a640 1 --2- 172.21.15.37:0/3440126917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6994004810 0x7f6994004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69a959a640 1 --2- 172.21.15.37:0/3440126917 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69940051d0 0x7f699400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69a959a640 1 -- 172.21.15.37:0/3440126917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f69ac15fc60 con 0x7f6994004810 2024-04-17T19:47:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69a959a640 1 -- 172.21.15.37:0/3440126917 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f699400c160 con 0x7f69940039c0 2024-04-17T19:47:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69a959a640 1 -- 172.21.15.37:0/3440126917 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f699400c360 con 0x7f69940051d0 2024-04-17T19:47:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69aa59c640 1 --2- 172.21.15.37:0/3440126917 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69940039c0 0x7f6994003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:26.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69aad9d640 1 --2- 172.21.15.37:0/3440126917 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69940051d0 0x7f699400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:26.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69a9d9b640 1 --2- 172.21.15.37:0/3440126917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6994004810 0x7f6994004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:26.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69aa59c640 1 -- 172.21.15.37:0/3440126917 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69940051d0 msgr2=0x7f699400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:26.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69aa59c640 1 --2- 172.21.15.37:0/3440126917 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69940051d0 0x7f699400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69aa59c640 1 -- 172.21.15.37:0/3440126917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6994004810 msgr2=0x7f6994004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:26.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69aa59c640 1 --2- 172.21.15.37:0/3440126917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6994004810 0x7f6994004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69aa59c640 1 -- 172.21.15.37:0/3440126917 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6994003530 con 0x7f69940039c0 2024-04-17T19:47:26.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.136+0000 7f69aad9d640 1 -- 172.21.15.37:0/3440126917 reap_dead start 2024-04-17T19:47:26.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.140+0000 7f69aa59c640 1 --2- 172.21.15.37:0/3440126917 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69940039c0 0x7f6994003e30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f69a4001e40 tx=0x7f69a401c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7abc2047bf546d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:26.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.140+0000 7f69a959a640 1 -- 172.21.15.37:0/3440126917 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a4001a90 con 0x7f69940039c0 2024-04-17T19:47:26.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.140+0000 7f69a959a640 1 -- 172.21.15.37:0/3440126917 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69a4003b80 con 0x7f69940039c0 2024-04-17T19:47:26.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.140+0000 7f69a959a640 1 -- 172.21.15.37:0/3440126917 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a4005080 con 0x7f69940039c0 2024-04-17T19:47:26.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.140+0000 7f69b0e9c640 1 -- 172.21.15.37:0/3440126917 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69940039c0 msgr2=0x7f6994003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:26.128 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.140+0000 7f69b0e9c640 1 --2- 172.21.15.37:0/3440126917 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69940039c0 0x7f6994003e30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f69a4001e40 tx=0x7f69a401c680 comp rx=0 tx=0).stop 2024-04-17T19:47:26.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.140+0000 7f69b0e9c640 1 -- 172.21.15.37:0/3440126917 shutdown_connections 2024-04-17T19:47:26.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.140+0000 7f69b0e9c640 1 --2- 172.21.15.37:0/3440126917 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69940039c0 0x7f6994003e30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.140+0000 7f69b0e9c640 1 -- 172.21.15.37:0/3440126917 >> 172.21.15.37:0/3440126917 conn(0x7f69ac0caec0 msgr2=0x7f69ac0cb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:26.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.140+0000 7f69b0e9c640 1 -- 172.21.15.37:0/3440126917 shutdown_connections 2024-04-17T19:47:26.129 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.140+0000 7f69b0e9c640 1 -- 172.21.15.37:0/3440126917 wait complete. 2024-04-17T19:47:26.130 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69b0e9c640 1 Processor -- start 2024-04-17T19:47:26.131 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69b0e9c640 1 -- start start 2024-04-17T19:47:26.133 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69b0e9c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69940039c0 0x7f69ac151020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:26.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69b0e9c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69ac0c8560 0x7f69ac1f5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:26.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69b0e9c640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69ac152250 0x7f69ac15ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:26.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69b0e9c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f69ac1563f0 con 0x7f69ac0c8560 2024-04-17T19:47:26.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69b0e9c640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f69ac0c93e0 con 0x7f69ac152250 2024-04-17T19:47:26.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69b0e9c640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f69ac06d6c0 con 0x7f69940039c0 2024-04-17T19:47:26.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69aa59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69940039c0 0x7f69ac151020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:26.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69aad9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69ac152250 0x7f69ac15ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:26.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69aad9d640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69ac152250 0x7f69ac15ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:34978/0 (socket says 172.21.15.37:34978) 2024-04-17T19:47:26.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69aad9d640 1 -- 172.21.15.37:0/429352391 learned_addr learned my addr 172.21.15.37:0/429352391 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-17T19:47:26.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69aa59c640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69940039c0 0x7f69ac151020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.37:48692/0 (socket says 172.21.15.37:48692) 2024-04-17T19:47:26.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69a9d9b640 1 --2- 172.21.15.37:0/429352391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69ac0c8560 0x7f69ac1f5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:26.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69aad9d640 1 -- 172.21.15.37:0/429352391 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69940039c0 msgr2=0x7f69ac151020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:26.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69aad9d640 1 --2- 172.21.15.37:0/429352391 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69940039c0 0x7f69ac151020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.134 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69aad9d640 1 -- 172.21.15.37:0/429352391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69ac0c8560 msgr2=0x7f69ac1f5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:26.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69aad9d640 1 --2- 172.21.15.37:0/429352391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69ac0c8560 0x7f69ac1f5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69aad9d640 1 -- 172.21.15.37:0/429352391 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69ac1f61c0 con 0x7f69ac152250 2024-04-17T19:47:26.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69aa59c640 1 --2- 172.21.15.37:0/429352391 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69940039c0 0x7f69ac151020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:47:26.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69aad9d640 1 --2- 172.21.15.37:0/429352391 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69ac152250 0x7f69ac15ea10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0cbc30 tx=0x7f699c011cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:26.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f699affd640 1 -- 172.21.15.37:0/429352391 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c004240 con 0x7f69ac152250 2024-04-17T19:47:26.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f699affd640 1 -- 172.21.15.37:0/429352391 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f699c01b070 con 0x7f69ac152250 2024-04-17T19:47:26.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69b0e9c640 1 -- 172.21.15.37:0/429352391 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69ac1f2730 con 0x7f69ac152250 2024-04-17T19:47:26.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f699affd640 1 -- 172.21.15.37:0/429352391 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c00f6f0 con 0x7f69ac152250 2024-04-17T19:47:26.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69b0e9c640 1 -- 172.21.15.37:0/429352391 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69ac1f2cf0 con 0x7f69ac152250 2024-04-17T19:47:26.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f699affd640 1 -- 172.21.15.37:0/429352391 <== mon.1 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f699c012040 con 0x7f69ac152250 2024-04-17T19:47:26.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f699affd640 1 -- 172.21.15.37:0/429352391 <== mon.1 v2:172.21.15.37:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f699c003ad0 con 0x7f69ac152250 2024-04-17T19:47:26.135 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.144+0000 7f69b0e9c640 1 -- 172.21.15.37:0/429352391 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69ac069b90 con 0x7f69ac152250 2024-04-17T19:47:26.138 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.148+0000 7f699affd640 1 -- 172.21.15.37:0/429352391 <== mon.1 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f699c003e10 con 0x7f69ac152250 2024-04-17T19:47:26.170 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:26.181+0000 7f30ad4e2180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T19:47:26.206 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.216+0000 7f69b0e9c640 1 -- 172.21.15.37:0/429352391 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6ce4a0ee-7778-4238-b1e5-3efaf90c9b81", "id": 6} v 0) -- 0x7f69ac06c960 con 0x7f69ac152250 2024-04-17T19:47:26.212 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.224+0000 7f699affd640 1 -- 172.21.15.37:0/429352391 <== mon.1 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6ce4a0ee-7778-4238-b1e5-3efaf90c9b81", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f699c0059a0 con 0x7f69ac152250 2024-04-17T19:47:26.212 INFO:teuthology.orchestra.run.smithi037.stdout:6 2024-04-17T19:47:26.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.224+0000 7f69b0e9c640 1 -- 172.21.15.37:0/429352391 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69ac152250 msgr2=0x7f69ac15ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:26.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.224+0000 7f69b0e9c640 1 --2- 172.21.15.37:0/429352391 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69ac152250 0x7f69ac15ea10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0cbc30 tx=0x7f699c011cf0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.224+0000 7f69b0e9c640 1 -- 172.21.15.37:0/429352391 shutdown_connections 2024-04-17T19:47:26.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.224+0000 7f69b0e9c640 1 --2- 172.21.15.37:0/429352391 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f69ac152250 0x7f69ac15ea10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.224+0000 7f69b0e9c640 1 --2- 172.21.15.37:0/429352391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69ac0c8560 0x7f69ac1f5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.224+0000 7f69b0e9c640 1 --2- 172.21.15.37:0/429352391 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69940039c0 0x7f69ac151020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.224+0000 7f69b0e9c640 1 -- 172.21.15.37:0/429352391 >> 172.21.15.37:0/429352391 conn(0x7f69ac0caec0 msgr2=0x7f69ac0ca6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:26.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.224+0000 7f69b0e9c640 1 -- 172.21.15.37:0/429352391 shutdown_connections 2024-04-17T19:47:26.213 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.224+0000 7f69b0e9c640 1 -- 172.21.15.37:0/429352391 wait complete. 2024-04-17T19:47:26.230 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd new 8c4a07d2-5528-4995-a660-b551a61f1289 7 2024-04-17T19:47:26.246 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:26.257+0000 7f30ad4e2180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T19:47:26.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.456+0000 7f833da21640 1 Processor -- start 2024-04-17T19:47:26.445 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.456+0000 7f833da21640 1 -- start start 2024-04-17T19:47:26.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.456+0000 7f833da21640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f833806e260 0x7f833806e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:26.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.456+0000 7f833da21640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f83380c99a0 0x7f83380c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:26.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.456+0000 7f833da21640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83380c8660 con 0x7f833815cc30 2024-04-17T19:47:26.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.456+0000 7f833da21640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f83380c9510 con 0x7f833806e260 2024-04-17T19:47:26.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.456+0000 7f833da21640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f83380c8900 con 0x7f83380c99a0 2024-04-17T19:47:26.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.456+0000 7f83377fe640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f83380c99a0 0x7f83380c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:26.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.456+0000 7f8336ffd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f833815cc30 0x7f8338156380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48830/0 (socket says 172.21.15.37:48830) 2024-04-17T19:47:26.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.456+0000 7f8336ffd640 1 -- 172.21.15.37:0/1358728766 learned_addr learned my addr 172.21.15.37:0/1358728766 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-04-17T19:47:26.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f83367fc640 1 --2- 172.21.15.37:0/1358728766 >> v2:172.21.15.29:3300/0 conn(0x7f833806e260 0x7f833806e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:26.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f8335ffb640 1 -- 172.21.15.37:0/1358728766 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3173557699 0 0) 0x7f83380c8660 con 0x7f833815cc30 2024-04-17T19:47:26.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f8335ffb640 1 -- 172.21.15.37:0/1358728766 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8324003530 con 0x7f833815cc30 2024-04-17T19:47:26.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f83377fe640 1 -- 172.21.15.37:0/1358728766 >> v2:172.21.15.29:3300/0 conn(0x7f833806e260 msgr2=0x7f833806e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:26.447 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f83377fe640 1 --2- 172.21.15.37:0/1358728766 >> v2:172.21.15.29:3300/0 conn(0x7f833806e260 0x7f833806e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f83377fe640 1 -- 172.21.15.37:0/1358728766 >> v1:172.21.15.37:6789/0 conn(0x7f833815cc30 legacy=0x7f8338156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:26.448 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f83377fe640 1 -- 172.21.15.37:0/1358728766 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83380c8be0 con 0x7f83380c99a0 2024-04-17T19:47:26.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f83377fe640 1 --2- 172.21.15.37:0/1358728766 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83380c99a0 0x7f83380c9db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f83380c50c0 tx=0x7f833001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a24d46ea75bfa9f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:26.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f8335ffb640 1 -- 172.21.15.37:0/1358728766 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83300b0070 con 0x7f83380c99a0 2024-04-17T19:47:26.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f8335ffb640 1 -- 172.21.15.37:0/1358728766 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83300ab600 con 0x7f83380c99a0 2024-04-17T19:47:26.449 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f8335ffb640 1 -- 172.21.15.37:0/1358728766 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83300ab9d0 con 0x7f83380c99a0 2024-04-17T19:47:26.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f833da21640 1 -- 172.21.15.37:0/1358728766 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83380c99a0 msgr2=0x7f83380c9db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:26.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f833da21640 1 --2- 172.21.15.37:0/1358728766 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83380c99a0 0x7f83380c9db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f83380c50c0 tx=0x7f833001c480 comp rx=0 tx=0).stop 2024-04-17T19:47:26.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f833da21640 1 -- 172.21.15.37:0/1358728766 shutdown_connections 2024-04-17T19:47:26.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f833da21640 1 --2- 172.21.15.37:0/1358728766 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f83380c99a0 0x7f83380c9db0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f833da21640 1 --2- 172.21.15.37:0/1358728766 >> v2:172.21.15.29:3300/0 conn(0x7f833806e260 0x7f833806e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f833da21640 1 -- 172.21.15.37:0/1358728766 >> 172.21.15.37:0/1358728766 conn(0x7f833806bdc0 msgr2=0x7f833806c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:26.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f833da21640 1 -- 172.21.15.37:0/1358728766 shutdown_connections 2024-04-17T19:47:26.450 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.460+0000 7f833da21640 1 -- 172.21.15.37:0/1358728766 wait complete. 2024-04-17T19:47:26.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f833da21640 1 Processor -- start 2024-04-17T19:47:26.451 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f833da21640 1 -- start start 2024-04-17T19:47:26.453 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f833da21640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f833806e260 0x7f8338159680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f833da21640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83380c99a0 0x7f83381492f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f833da21640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f833815cc30 0x7f833815b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f833da21640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f83380c4d30 con 0x7f83380c99a0 2024-04-17T19:47:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f833da21640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f83380c4bb0 con 0x7f833815cc30 2024-04-17T19:47:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f833da21640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f83381692a0 con 0x7f833806e260 2024-04-17T19:47:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f83377fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f833815cc30 0x7f833815b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f8336ffd640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f833806e260 0x7f8338159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:26.454 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f83377fe640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f833815cc30 0x7f833815b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.37:35002/0 (socket says 172.21.15.37:35002) 2024-04-17T19:47:26.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f83377fe640 1 -- 172.21.15.37:0/2643688310 learned_addr learned my addr 172.21.15.37:0/2643688310 (peer_addr_for_me v2:172.21.15.37:0/0) 2024-04-17T19:47:26.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f83367fc640 1 --2- 172.21.15.37:0/2643688310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83380c99a0 0x7f83381492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:26.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f8336ffd640 1 -- 172.21.15.37:0/2643688310 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f833815cc30 msgr2=0x7f833815b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:26.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f8336ffd640 1 --2- 172.21.15.37:0/2643688310 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f833815cc30 0x7f833815b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f8336ffd640 1 -- 172.21.15.37:0/2643688310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83380c99a0 msgr2=0x7f83381492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:26.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f8336ffd640 1 --2- 172.21.15.37:0/2643688310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83380c99a0 0x7f83381492f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.457 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f8336ffd640 1 -- 172.21.15.37:0/2643688310 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f833815a050 con 0x7f833806e260 2024-04-17T19:47:26.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f83377fe640 1 --2- 172.21.15.37:0/2643688310 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f833815cc30 0x7f833815b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:47:26.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f83367fc640 1 --2- 172.21.15.37:0/2643688310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83380c99a0 0x7f83381492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:26.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f8336ffd640 1 --2- 172.21.15.37:0/2643688310 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f833806e260 0x7f8338159680 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f832800c9f0 tx=0x7f832800ceb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:26.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f832ffff640 1 -- 172.21.15.37:0/2643688310 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8328017070 con 0x7f833806e260 2024-04-17T19:47:26.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f832ffff640 1 -- 172.21.15.37:0/2643688310 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f832800ab70 con 0x7f833806e260 2024-04-17T19:47:26.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f833da21640 1 -- 172.21.15.37:0/2643688310 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8338149830 con 0x7f833806e260 2024-04-17T19:47:26.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f832ffff640 1 -- 172.21.15.37:0/2643688310 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8328012400 con 0x7f833806e260 2024-04-17T19:47:26.458 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f833da21640 1 -- 172.21.15.37:0/2643688310 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8338148810 con 0x7f833806e260 2024-04-17T19:47:26.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f833da21640 1 -- 172.21.15.37:0/2643688310 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f833815a220 con 0x7f833806e260 2024-04-17T19:47:26.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f832ffff640 1 -- 172.21.15.37:0/2643688310 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f832801a070 con 0x7f833806e260 2024-04-17T19:47:26.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.464+0000 7f832ffff640 1 -- 172.21.15.37:0/2643688310 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f83280076d0 con 0x7f833806e260 2024-04-17T19:47:26.459 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.468+0000 7f832ffff640 1 -- 172.21.15.37:0/2643688310 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8328021070 con 0x7f833806e260 2024-04-17T19:47:26.522 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.532+0000 7f833da21640 1 -- 172.21.15.37:0/2643688310 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8c4a07d2-5528-4995-a660-b551a61f1289", "id": 7} v 0) -- 0x7f8338069b90 con 0x7f833806e260 2024-04-17T19:47:26.528 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.540+0000 7f832ffff640 1 -- 172.21.15.37:0/2643688310 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8c4a07d2-5528-4995-a660-b551a61f1289", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f83280079f0 con 0x7f833806e260 2024-04-17T19:47:26.529 INFO:teuthology.orchestra.run.smithi037.stdout:7 2024-04-17T19:47:26.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.540+0000 7f833da21640 1 -- 172.21.15.37:0/2643688310 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f833806e260 msgr2=0x7f8338159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:26.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.540+0000 7f833da21640 1 --2- 172.21.15.37:0/2643688310 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f833806e260 0x7f8338159680 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f832800c9f0 tx=0x7f832800ceb0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.540+0000 7f833da21640 1 -- 172.21.15.37:0/2643688310 shutdown_connections 2024-04-17T19:47:26.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.540+0000 7f833da21640 1 --2- 172.21.15.37:0/2643688310 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f833815cc30 0x7f833815b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.530 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.540+0000 7f833da21640 1 --2- 172.21.15.37:0/2643688310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83380c99a0 0x7f83381492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.540+0000 7f833da21640 1 --2- 172.21.15.37:0/2643688310 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f833806e260 0x7f8338159680 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.540+0000 7f833da21640 1 -- 172.21.15.37:0/2643688310 >> 172.21.15.37:0/2643688310 conn(0x7f833806bdc0 msgr2=0x7f833806c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:26.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.540+0000 7f833da21640 1 -- 172.21.15.37:0/2643688310 shutdown_connections 2024-04-17T19:47:26.531 INFO:teuthology.orchestra.run.smithi037.stderr:2024-04-17T19:47:26.540+0000 7f833da21640 1 -- 172.21.15.37:0/2643688310 wait complete. 2024-04-17T19:47:26.548 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-17T19:47:26.548 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-17T19:47:26.551 INFO:tasks.ceph.osd.0:Started 2024-04-17T19:47:26.551 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-17T19:47:26.551 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-17T19:47:26.555 INFO:tasks.ceph.osd.1:Started 2024-04-17T19:47:26.555 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-17T19:47:26.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-17T19:47:26.558 INFO:tasks.ceph.osd.2:Started 2024-04-17T19:47:26.558 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-17T19:47:26.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-17T19:47:26.561 INFO:tasks.ceph.osd.3:Started 2024-04-17T19:47:26.561 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-17T19:47:26.561 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-17T19:47:26.563 INFO:tasks.ceph.osd.4:Started 2024-04-17T19:47:26.563 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-17T19:47:26.563 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-17T19:47:26.565 INFO:tasks.ceph.osd.5:Started 2024-04-17T19:47:26.565 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-17T19:47:26.565 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-17T19:47:26.567 INFO:tasks.ceph.osd.6:Started 2024-04-17T19:47:26.567 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-17T19:47:26.567 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-17T19:47:26.569 INFO:tasks.ceph.osd.7:Started 2024-04-17T19:47:26.570 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:47:26.649 INFO:tasks.ceph.mgr.x.smithi037.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-17T19:47:26.649 INFO:tasks.ceph.mgr.x.smithi037.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-17T19:47:26.649 INFO:tasks.ceph.mgr.x.smithi037.stderr: from numpy import show_config as show_numpy_config 2024-04-17T19:47:26.690 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:26.700+0000 7f80eb593180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T19:47:26.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.821+0000 7f5918000640 1 Processor -- start 2024-04-17T19:47:26.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.821+0000 7f5918000640 1 -- start start 2024-04-17T19:47:26.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.821+0000 7f5918000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5910159e60 0x7f59100cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:26.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.821+0000 7f5918000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f59101627d0 0x7f5910164bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:26.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.821+0000 7f5918000640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59100c4930 con 0x7f5910153d20 2024-04-17T19:47:26.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.821+0000 7f5918000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f59100d0320 con 0x7f59101627d0 2024-04-17T19:47:26.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.821+0000 7f5918000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f59100cf620 con 0x7f5910159e60 2024-04-17T19:47:26.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.821+0000 7f5914c00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5910153d20 0x7f5910071940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35542/0 (socket says 172.21.15.29:35542) 2024-04-17T19:47:26.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.821+0000 7f5914c00640 1 -- 172.21.15.29:0/2880254756 learned_addr learned my addr 172.21.15.29:0/2880254756 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:47:26.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.821+0000 7f5916000640 1 --2- 172.21.15.29:0/2880254756 >> v2:172.21.15.29:3300/0 conn(0x7f59101627d0 0x7f5910164bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:26.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.821+0000 7f5915600640 1 --2- 172.21.15.29:0/2880254756 >> v2:172.21.15.37:3300/0 conn(0x7f5910159e60 0x7f59100cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:26.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.825+0000 7f5916000640 1 -- 172.21.15.29:0/2880254756 >> v2:172.21.15.37:3300/0 conn(0x7f5910159e60 msgr2=0x7f59100cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:26.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.825+0000 7f5916000640 1 --2- 172.21.15.29:0/2880254756 >> v2:172.21.15.37:3300/0 conn(0x7f5910159e60 0x7f59100cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.825+0000 7f5916000640 1 -- 172.21.15.29:0/2880254756 >> v1:172.21.15.29:6789/0 conn(0x7f5910153d20 legacy=0x7f5910071940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:26.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.825+0000 7f5916000640 1 -- 172.21.15.29:0/2880254756 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59100c5190 con 0x7f59101627d0 2024-04-17T19:47:26.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.825+0000 7f590fe00640 1 -- 172.21.15.29:0/2880254756 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1498866590 0 0) 0x7f59100c4930 con 0x7f5910153d20 2024-04-17T19:47:26.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.825+0000 7f5916000640 1 --2- 172.21.15.29:0/2880254756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59101627d0 0x7f5910164bf0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f59100733a0 tx=0x7f590801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e48d7006715c523 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:26.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.825+0000 7f590fe00640 1 -- 172.21.15.29:0/2880254756 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59080b2070 con 0x7f59101627d0 2024-04-17T19:47:26.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.825+0000 7f590fe00640 1 -- 172.21.15.29:0/2880254756 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59080a57f0 con 0x7f59101627d0 2024-04-17T19:47:26.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.825+0000 7f590fe00640 1 -- 172.21.15.29:0/2880254756 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59080a5bc0 con 0x7f59101627d0 2024-04-17T19:47:26.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.825+0000 7f5918000640 1 -- 172.21.15.29:0/2880254756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59101627d0 msgr2=0x7f5910164bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:26.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.825+0000 7f5918000640 1 --2- 172.21.15.29:0/2880254756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59101627d0 0x7f5910164bf0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f59100733a0 tx=0x7f590801c930 comp rx=0 tx=0).stop 2024-04-17T19:47:26.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.829+0000 7f5918000640 1 -- 172.21.15.29:0/2880254756 shutdown_connections 2024-04-17T19:47:26.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.829+0000 7f5918000640 1 --2- 172.21.15.29:0/2880254756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59101627d0 0x7f5910164bf0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.829+0000 7f5918000640 1 --2- 172.21.15.29:0/2880254756 >> v2:172.21.15.37:3300/0 conn(0x7f5910159e60 0x7f59100cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.829+0000 7f5918000640 1 -- 172.21.15.29:0/2880254756 >> 172.21.15.29:0/2880254756 conn(0x7f5910067eb0 msgr2=0x7f5910070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:26.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.829+0000 7f5918000640 1 -- 172.21.15.29:0/2880254756 shutdown_connections 2024-04-17T19:47:26.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.829+0000 7f5918000640 1 -- 172.21.15.29:0/2880254756 wait complete. 2024-04-17T19:47:26.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.829+0000 7f5918000640 1 Processor -- start 2024-04-17T19:47:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.829+0000 7f5918000640 1 -- start start 2024-04-17T19:47:26.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.829+0000 7f5918000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5910153d20 0x7f59101486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.829+0000 7f5918000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5910159e60 0x7f59101f6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f5918000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59101627d0 0x7f59101f7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f5918000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f59100cbd10 con 0x7f5910159e60 2024-04-17T19:47:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f5918000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f59100cbb90 con 0x7f59101627d0 2024-04-17T19:47:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f5918000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f59100c4510 con 0x7f5910153d20 2024-04-17T19:47:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f5915600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5910153d20 0x7f59101486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f5914c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5910159e60 0x7f59101f6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f5915600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5910153d20 0x7f59101486a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:53218/0 (socket says 172.21.15.29:53218) 2024-04-17T19:47:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f5916000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59101627d0 0x7f59101f7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f5915600640 1 -- 172.21.15.29:0/2958212158 learned_addr learned my addr 172.21.15.29:0/2958212158 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:26.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f5914c00640 1 -- 172.21.15.29:0/2958212158 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5910153d20 msgr2=0x7f59101486a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:26.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f5914c00640 1 --2- 172.21.15.29:0/2958212158 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5910153d20 0x7f59101486a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f5914c00640 1 -- 172.21.15.29:0/2958212158 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59101627d0 msgr2=0x7f59101f7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f5914c00640 1 --2- 172.21.15.29:0/2958212158 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59101627d0 0x7f59101f7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f5914c00640 1 -- 172.21.15.29:0/2958212158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59100ca980 con 0x7f5910159e60 2024-04-17T19:47:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f5914c00640 1 --2- 172.21.15.29:0/2958212158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5910159e60 0x7f59101f6d40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00c410 tx=0x7f58fc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f590d600640 1 -- 172.21.15.29:0/2958212158 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58fc0074e0 con 0x7f5910159e60 2024-04-17T19:47:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f590d600640 1 -- 172.21.15.29:0/2958212158 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58fc0047f0 con 0x7f5910159e60 2024-04-17T19:47:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f590d600640 1 -- 172.21.15.29:0/2958212158 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58fc007680 con 0x7f5910159e60 2024-04-17T19:47:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f5918000640 1 -- 172.21.15.29:0/2958212158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5910141bb0 con 0x7f5910159e60 2024-04-17T19:47:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f5918000640 1 -- 172.21.15.29:0/2958212158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5910146540 con 0x7f5910159e60 2024-04-17T19:47:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f590d600640 1 -- 172.21.15.29:0/2958212158 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f58fc011070 con 0x7f5910159e60 2024-04-17T19:47:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f590d600640 1 -- 172.21.15.29:0/2958212158 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f58fc013a90 con 0x7f5910159e60 2024-04-17T19:47:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.833+0000 7f5918000640 1 -- 172.21.15.29:0/2958212158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5910069b90 con 0x7f5910159e60 2024-04-17T19:47:26.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.837+0000 7f590d600640 1 -- 172.21.15.29:0/2958212158 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f58fc013d70 con 0x7f5910159e60 2024-04-17T19:47:26.830 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:26.840+0000 7f80eb593180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T19:47:26.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.897+0000 7f5918000640 1 -- 172.21.15.29:0/2958212158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f59100b3f30 con 0x7f5910159e60 2024-04-17T19:47:26.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.897+0000 7f590d600640 1 -- 172.21.15.29:0/2958212158 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f58fc0133c0 con 0x7f5910159e60 2024-04-17T19:47:26.885 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:47:26.886 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":10,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:47:26.537699+0000","last_up_change":"0.000000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:47:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.897+0000 7f5918000640 1 -- 172.21.15.29:0/2958212158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5910159e60 msgr2=0x7f59101f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.897+0000 7f5918000640 1 --2- 172.21.15.29:0/2958212158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5910159e60 0x7f59101f6d40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f58fc00c410 tx=0x7f58fc00c8d0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.897+0000 7f5918000640 1 -- 172.21.15.29:0/2958212158 shutdown_connections 2024-04-17T19:47:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.897+0000 7f5918000640 1 --2- 172.21.15.29:0/2958212158 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59101627d0 0x7f59101f7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.897+0000 7f5918000640 1 --2- 172.21.15.29:0/2958212158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5910159e60 0x7f59101f6d40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.897+0000 7f5918000640 1 --2- 172.21.15.29:0/2958212158 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5910153d20 0x7f59101486a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.897+0000 7f5918000640 1 -- 172.21.15.29:0/2958212158 >> 172.21.15.29:0/2958212158 conn(0x7f5910067eb0 msgr2=0x7f59100b4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.897+0000 7f5918000640 1 -- 172.21.15.29:0/2958212158 shutdown_connections 2024-04-17T19:47:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:26.897+0000 7f5918000640 1 -- 172.21.15.29:0/2958212158 wait complete. 2024-04-17T19:47:26.905 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-17T19:47:26.905 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-17T19:47:26.994 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:27.004+0000 7f80eb593180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T19:47:27.002 INFO:tasks.ceph.mgr.y.smithi029.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-17T19:47:27.002 INFO:tasks.ceph.mgr.y.smithi029.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-17T19:47:27.002 INFO:tasks.ceph.mgr.y.smithi029.stderr: from numpy import show_config as show_numpy_config 2024-04-17T19:47:27.060 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:27.072+0000 7f80eb593180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T19:47:27.079 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:27.089+0000 7f30ad4e2180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T19:47:27.192 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-17T19:47:27.204+0000 7f71152ee7c0 -1 Falling back to public interface 2024-04-17T19:47:27.201 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:27.212+0000 7f80eb593180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T19:47:27.208 INFO:tasks.ceph.osd.6.smithi037.stderr:2024-04-17T19:47:27.220+0000 7f91cac6b7c0 -1 Falling back to public interface 2024-04-17T19:47:27.211 INFO:tasks.ceph.osd.7.smithi037.stderr:2024-04-17T19:47:27.224+0000 7f41df9607c0 -1 Falling back to public interface 2024-04-17T19:47:27.219 INFO:tasks.ceph.osd.5.smithi037.stderr:2024-04-17T19:47:27.232+0000 7fd20f4717c0 -1 Falling back to public interface 2024-04-17T19:47:27.236 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:27.245+0000 7f30ad4e2180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T19:47:27.241 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-04-17T19:47:27.253+0000 7f0a5e22b7c0 -1 Falling back to public interface 2024-04-17T19:47:27.253 INFO:tasks.ceph.osd.2.smithi029.stderr:2024-04-17T19:47:27.265+0000 7f98d21537c0 -1 Falling back to public interface 2024-04-17T19:47:27.258 INFO:tasks.ceph.osd.3.smithi029.stderr:2024-04-17T19:47:27.269+0000 7f21479297c0 -1 Falling back to public interface 2024-04-17T19:47:27.297 INFO:tasks.ceph.osd.1.smithi029.stderr:2024-04-17T19:47:27.309+0000 7faf897a77c0 -1 Falling back to public interface 2024-04-17T19:47:27.438 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:27.449+0000 7f30ad4e2180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T19:47:27.525 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:27.537+0000 7f30ad4e2180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T19:47:27.685 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:27.697+0000 7f30ad4e2180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T19:47:27.711 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-17T19:47:27.853 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.865+0000 7f82cc600640 1 Processor -- start 2024-04-17T19:47:27.853 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.865+0000 7f82cc600640 1 -- start start 2024-04-17T19:47:27.855 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:27.868+0000 7f80eb593180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T19:47:27.855 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.865+0000 7f82cc600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82c40c71a0 0x7f82c40ca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:27.856 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.865+0000 7f82cc600640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f82c40c75b0 0x7f82c415da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:27.856 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.865+0000 7f82cc600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82c415a2a0 con 0x7f82c415dfc0 2024-04-17T19:47:27.856 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.865+0000 7f82cc600640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f82c415d5f0 con 0x7f82c40c75b0 2024-04-17T19:47:27.856 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.865+0000 7f82cc600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f82c415a580 con 0x7f82c40c71a0 2024-04-17T19:47:27.856 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.865+0000 7f82c9c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82c40c71a0 0x7f82c40ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:27.856 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.865+0000 7f82c9200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f82c40c75b0 0x7f82c415da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:27.856 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.865+0000 7f82ca600640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f82c415dfc0 0x7f82c40c33c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:43776/0 (socket says 172.21.15.29:43776) 2024-04-17T19:47:27.856 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.865+0000 7f82ca600640 1 -- 172.21.15.29:0/3384620973 learned_addr learned my addr 172.21.15.29:0/3384620973 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:47:27.857 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.869+0000 7f82c9c00640 1 -- 172.21.15.29:0/3384620973 >> v2:172.21.15.37:3300/0 conn(0x7f82c40c75b0 msgr2=0x7f82c415da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:27.857 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.869+0000 7f82c9c00640 1 --2- 172.21.15.29:0/3384620973 >> v2:172.21.15.37:3300/0 conn(0x7f82c40c75b0 0x7f82c415da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:27.857 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.869+0000 7f82c9c00640 1 -- 172.21.15.29:0/3384620973 >> v1:172.21.15.37:6789/0 conn(0x7f82c415dfc0 legacy=0x7f82c40c33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:27.857 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.869+0000 7f82c9c00640 1 -- 172.21.15.29:0/3384620973 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c415a860 con 0x7f82c40c71a0 2024-04-17T19:47:27.857 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.869+0000 7f82c9200640 1 --2- 172.21.15.29:0/3384620973 >> v2:172.21.15.37:3300/0 conn(0x7f82c40c75b0 0x7f82c415da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:47:27.859 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.869+0000 7f82c9c00640 1 --2- 172.21.15.29:0/3384620973 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82c40c71a0 0x7f82c40ca410 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f82b80097c0 tx=0x7f82b801ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=350701ba69e9a91a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:27.859 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.869+0000 7f82c3e00640 1 -- 172.21.15.29:0/3384620973 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b80b5070 con 0x7f82c40c71a0 2024-04-17T19:47:27.859 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.869+0000 7f82c3e00640 1 -- 172.21.15.29:0/3384620973 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82b80a8990 con 0x7f82c40c71a0 2024-04-17T19:47:27.859 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.869+0000 7f82c3e00640 1 -- 172.21.15.29:0/3384620973 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b80a8d60 con 0x7f82c40c71a0 2024-04-17T19:47:27.860 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.873+0000 7f82cc600640 1 -- 172.21.15.29:0/3384620973 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82c40c71a0 msgr2=0x7f82c40ca410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:27.860 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.873+0000 7f82cc600640 1 --2- 172.21.15.29:0/3384620973 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82c40c71a0 0x7f82c40ca410 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f82b80097c0 tx=0x7f82b801ec90 comp rx=0 tx=0).stop 2024-04-17T19:47:27.860 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.873+0000 7f82cc600640 1 -- 172.21.15.29:0/3384620973 shutdown_connections 2024-04-17T19:47:27.860 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.873+0000 7f82cc600640 1 --2- 172.21.15.29:0/3384620973 >> v2:172.21.15.37:3300/0 conn(0x7f82c40c75b0 0x7f82c415da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:27.860 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.873+0000 7f82cc600640 1 --2- 172.21.15.29:0/3384620973 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82c40c71a0 0x7f82c40ca410 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:27.860 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.873+0000 7f82cc600640 1 -- 172.21.15.29:0/3384620973 >> 172.21.15.29:0/3384620973 conn(0x7f82c406c1c0 msgr2=0x7f82c406c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:27.860 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.873+0000 7f82cc600640 1 -- 172.21.15.29:0/3384620973 shutdown_connections 2024-04-17T19:47:27.861 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.873+0000 7f82cc600640 1 -- 172.21.15.29:0/3384620973 wait complete. 2024-04-17T19:47:27.862 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.873+0000 7f82cc600640 1 Processor -- start 2024-04-17T19:47:27.862 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.873+0000 7f82cc600640 1 -- start start 2024-04-17T19:47:27.865 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.873+0000 7f82cc600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82c40c71a0 0x7f82c414eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:27.865 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.873+0000 7f82cc600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82c415dfc0 0x7f82c40c4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:27.865 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.873+0000 7f82cc600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82c40c5740 0x7f82c415c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:27.865 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.873+0000 7f82cc600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f82c40cb140 con 0x7f82c40c71a0 2024-04-17T19:47:27.865 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.873+0000 7f82cc600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f82c415d280 con 0x7f82c415dfc0 2024-04-17T19:47:27.865 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.873+0000 7f82cc600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f82c40c4480 con 0x7f82c40c5740 2024-04-17T19:47:27.865 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82c9c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82c40c71a0 0x7f82c414eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:27.865 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82c9c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82c40c71a0 0x7f82c414eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36850/0 (socket says 172.21.15.29:36850) 2024-04-17T19:47:27.865 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82c9c00640 1 -- 172.21.15.29:0/3794365841 learned_addr learned my addr 172.21.15.29:0/3794365841 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:27.865 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82c9200640 1 --2- 172.21.15.29:0/3794365841 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82c415dfc0 0x7f82c40c4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:27.865 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82ca600640 1 --2- 172.21.15.29:0/3794365841 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82c40c5740 0x7f82c415c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:27.865 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82c9c00640 1 -- 172.21.15.29:0/3794365841 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82c40c5740 msgr2=0x7f82c415c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:27.865 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82c9c00640 1 --2- 172.21.15.29:0/3794365841 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82c40c5740 0x7f82c415c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:27.866 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82c9c00640 1 -- 172.21.15.29:0/3794365841 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82c415dfc0 msgr2=0x7f82c40c4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:27.866 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82c9c00640 1 --2- 172.21.15.29:0/3794365841 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82c415dfc0 0x7f82c40c4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:27.866 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82c9c00640 1 -- 172.21.15.29:0/3794365841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c40c6840 con 0x7f82c40c71a0 2024-04-17T19:47:27.866 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82c9c00640 1 --2- 172.21.15.29:0/3794365841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82c40c71a0 0x7f82c414eb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f82b801c8c0 tx=0x7f82b801ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:27.867 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82c1600640 1 -- 172.21.15.29:0/3794365841 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b80bf070 con 0x7f82c40c71a0 2024-04-17T19:47:27.867 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82c1600640 1 -- 172.21.15.29:0/3794365841 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82b80b0720 con 0x7f82c40c71a0 2024-04-17T19:47:27.867 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82cc600640 1 -- 172.21.15.29:0/3794365841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82c40c5da0 con 0x7f82c40c71a0 2024-04-17T19:47:27.867 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82c1600640 1 -- 172.21.15.29:0/3794365841 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b80b5040 con 0x7f82c40c71a0 2024-04-17T19:47:27.867 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82cc600640 1 -- 172.21.15.29:0/3794365841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82c414ded0 con 0x7f82c40c71a0 2024-04-17T19:47:27.867 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82c1600640 1 -- 172.21.15.29:0/3794365841 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f82b80c2070 con 0x7f82c40c71a0 2024-04-17T19:47:27.867 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82c1600640 1 -- 172.21.15.29:0/3794365841 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f82b80b9ac0 con 0x7f82c40c71a0 2024-04-17T19:47:27.867 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.877+0000 7f82cc600640 1 -- 172.21.15.29:0/3794365841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82c4069b90 con 0x7f82c40c71a0 2024-04-17T19:47:27.869 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.881+0000 7f82c1600640 1 -- 172.21.15.29:0/3794365841 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f82b80187f0 con 0x7f82c40c71a0 2024-04-17T19:47:27.929 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.941+0000 7f82cc600640 1 -- 172.21.15.29:0/3794365841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f82c40cd800 con 0x7f82c40c71a0 2024-04-17T19:47:27.930 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.941+0000 7f82c1600640 1 -- 172.21.15.29:0/3794365841 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f82b80a8910 con 0x7f82c40c71a0 2024-04-17T19:47:27.930 INFO:teuthology.misc.health.smithi029.stdout: 2024-04-17T19:47:27.930 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:47:26.537699+0000","last_up_change":"0.000000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:47:27.931 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.941+0000 7f82cc600640 1 -- 172.21.15.29:0/3794365841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82c40c71a0 msgr2=0x7f82c414eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:27.931 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.941+0000 7f82cc600640 1 --2- 172.21.15.29:0/3794365841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82c40c71a0 0x7f82c414eb10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f82b801c8c0 tx=0x7f82b801ba30 comp rx=0 tx=0).stop 2024-04-17T19:47:27.931 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.941+0000 7f82cc600640 1 -- 172.21.15.29:0/3794365841 shutdown_connections 2024-04-17T19:47:27.931 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.941+0000 7f82cc600640 1 --2- 172.21.15.29:0/3794365841 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82c40c5740 0x7f82c415c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:27.931 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.941+0000 7f82cc600640 1 --2- 172.21.15.29:0/3794365841 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f82c415dfc0 0x7f82c40c4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:27.932 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.941+0000 7f82cc600640 1 --2- 172.21.15.29:0/3794365841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82c40c71a0 0x7f82c414eb10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:27.932 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.941+0000 7f82cc600640 1 -- 172.21.15.29:0/3794365841 >> 172.21.15.29:0/3794365841 conn(0x7f82c406c1c0 msgr2=0x7f82c41526d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:27.932 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.941+0000 7f82cc600640 1 -- 172.21.15.29:0/3794365841 shutdown_connections 2024-04-17T19:47:27.932 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:27.941+0000 7f82cc600640 1 -- 172.21.15.29:0/3794365841 wait complete. 2024-04-17T19:47:27.934 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:27.944+0000 7f80eb593180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T19:47:27.948 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-17T19:47:28.126 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:28.136+0000 7f80eb593180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T19:47:28.203 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:28.216+0000 7f80eb593180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T19:47:28.438 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:28.449+0000 7f30ad4e2180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T19:47:28.510 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:28.520+0000 7f80eb593180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T19:47:28.531 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:28.541+0000 7f30ad4e2180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T19:47:28.574 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:28.584+0000 7f80eb593180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T19:47:28.753 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:28.765+0000 7f30ad4e2180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T19:47:28.844 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:28.853+0000 7f30ad4e2180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T19:47:29.193 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:29.205+0000 7f30ad4e2180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T19:47:29.268 INFO:tasks.ceph.mgr.y.smithi029.stderr:2024-04-17T19:47:29.277+0000 7f30ad4e2180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T19:47:29.622 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:29.632+0000 7f8092741640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-17T19:47:29.622 INFO:tasks.ceph.mgr.x.smithi037.stderr:2024-04-17T19:47:29.632+0000 7f8092741640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-17T19:47:30.926 INFO:tasks.ceph.osd.6.smithi037.stderr:2024-04-17T19:47:30.936+0000 7f91cac6b7c0 -1 osd.6 0 log_to_monitors true 2024-04-17T19:47:30.928 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-17T19:47:30.940+0000 7f71152ee7c0 -1 osd.4 0 log_to_monitors true 2024-04-17T19:47:31.023 INFO:tasks.ceph.osd.7.smithi037.stderr:2024-04-17T19:47:31.032+0000 7f41df9607c0 -1 osd.7 0 log_to_monitors true 2024-04-17T19:47:31.074 INFO:tasks.ceph.osd.2.smithi029.stderr:2024-04-17T19:47:31.085+0000 7f98d21537c0 -1 osd.2 0 log_to_monitors true 2024-04-17T19:47:31.106 INFO:tasks.ceph.osd.5.smithi037.stderr:2024-04-17T19:47:31.116+0000 7fd20f4717c0 -1 osd.5 0 log_to_monitors true 2024-04-17T19:47:31.217 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-04-17T19:47:31.229+0000 7f0a5e22b7c0 -1 osd.0 0 log_to_monitors true 2024-04-17T19:47:31.227 INFO:tasks.ceph.osd.1.smithi029.stderr:2024-04-17T19:47:31.237+0000 7faf897a77c0 -1 osd.1 0 log_to_monitors true 2024-04-17T19:47:31.248 INFO:tasks.ceph.osd.3.smithi029.stderr:2024-04-17T19:47:31.257+0000 7f21479297c0 -1 osd.3 0 log_to_monitors true 2024-04-17T19:47:34.596 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-04-17T19:47:34.605+0000 7f0a59000640 -1 osd.0 0 waiting for initial osdmap 2024-04-17T19:47:34.634 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-04-17T19:47:34.645+0000 7f0a4f000640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T19:47:34.649 INFO:tasks.ceph.osd.3.smithi029.stderr:2024-04-17T19:47:34.661+0000 7f2143200640 -1 osd.3 0 waiting for initial osdmap 2024-04-17T19:47:34.677 INFO:tasks.ceph.osd.3.smithi029.stderr:2024-04-17T19:47:34.689+0000 7f2138800640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T19:47:34.683 INFO:tasks.ceph.osd.1.smithi029.stderr:2024-04-17T19:47:34.693+0000 7faf84600640 -1 osd.1 0 waiting for initial osdmap 2024-04-17T19:47:34.683 INFO:tasks.ceph.osd.2.smithi029.stderr:2024-04-17T19:47:34.693+0000 7f98cd000640 -1 osd.2 0 waiting for initial osdmap 2024-04-17T19:47:34.705 INFO:tasks.ceph.osd.1.smithi029.stderr:2024-04-17T19:47:34.717+0000 7faf7a600640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T19:47:34.717 INFO:tasks.ceph.osd.2.smithi029.stderr:2024-04-17T19:47:34.729+0000 7f98c3000640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T19:47:34.754 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-17T19:47:34.896 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.905+0000 7f5023800640 1 Processor -- start 2024-04-17T19:47:34.897 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.905+0000 7f5023800640 1 -- start start 2024-04-17T19:47:34.909 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.909+0000 7f5023800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f501c0c7420 0x7f501c1526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:34.910 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.909+0000 7f5023800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f501c158480 0x7f501c158890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:34.910 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.909+0000 7f5023800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f501c152be0 con 0x7f501c0cc790 2024-04-17T19:47:34.910 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.909+0000 7f5023800640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f501c0c7ee0 con 0x7f501c0c7420 2024-04-17T19:47:34.910 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.909+0000 7f5023800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f501c0c6180 con 0x7f501c158480 2024-04-17T19:47:34.910 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.909+0000 7f5020e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f501c0c7420 0x7f501c1526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:34.910 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.909+0000 7f5020e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f501c0c7420 0x7f501c1526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50806/0 (socket says 172.21.15.29:50806) 2024-04-17T19:47:34.910 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.909+0000 7f5020e00640 1 -- 172.21.15.29:0/1400459337 learned_addr learned my addr 172.21.15.29:0/1400459337 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:34.910 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.909+0000 7f501be00640 1 --2- 172.21.15.29:0/1400459337 >> v2:172.21.15.37:3300/0 conn(0x7f501c158480 0x7f501c158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:34.910 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.909+0000 7f5020e00640 1 -- 172.21.15.29:0/1400459337 >> v2:172.21.15.37:3300/0 conn(0x7f501c158480 msgr2=0x7f501c158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:34.910 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.909+0000 7f5020e00640 1 --2- 172.21.15.29:0/1400459337 >> v2:172.21.15.37:3300/0 conn(0x7f501c158480 0x7f501c158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:34.911 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.909+0000 7f5020e00640 1 -- 172.21.15.29:0/1400459337 >> v1:172.21.15.37:6789/0 conn(0x7f501c0cc790 legacy=0x7f501c15b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:34.911 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.909+0000 7f5020e00640 1 -- 172.21.15.29:0/1400459337 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f501c0c6460 con 0x7f501c0c7420 2024-04-17T19:47:34.911 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.913+0000 7f5020e00640 1 --2- 172.21.15.29:0/1400459337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c0c7420 0x7f501c1526a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f501c06dfa0 tx=0x7f501001e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ec394e399cc5ca server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:34.911 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.913+0000 7f501b400640 1 -- 172.21.15.29:0/1400459337 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50100ad070 con 0x7f501c0c7420 2024-04-17T19:47:34.911 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.913+0000 7f501b400640 1 -- 172.21.15.29:0/1400459337 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5010004230 con 0x7f501c0c7420 2024-04-17T19:47:34.911 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.913+0000 7f501b400640 1 -- 172.21.15.29:0/1400459337 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5010003620 con 0x7f501c0c7420 2024-04-17T19:47:34.911 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.913+0000 7f5023800640 1 -- 172.21.15.29:0/1400459337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c0c7420 msgr2=0x7f501c1526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:34.911 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.913+0000 7f5023800640 1 --2- 172.21.15.29:0/1400459337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c0c7420 0x7f501c1526a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f501c06dfa0 tx=0x7f501001e8b0 comp rx=0 tx=0).stop 2024-04-17T19:47:34.911 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.913+0000 7f5023800640 1 -- 172.21.15.29:0/1400459337 shutdown_connections 2024-04-17T19:47:34.911 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.913+0000 7f5023800640 1 --2- 172.21.15.29:0/1400459337 >> v2:172.21.15.37:3300/0 conn(0x7f501c158480 0x7f501c158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:34.911 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.913+0000 7f5023800640 1 --2- 172.21.15.29:0/1400459337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c0c7420 0x7f501c1526a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:34.912 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.913+0000 7f5023800640 1 -- 172.21.15.29:0/1400459337 >> 172.21.15.29:0/1400459337 conn(0x7f501c06c1c0 msgr2=0x7f501c06c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:34.912 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.913+0000 7f5023800640 1 -- 172.21.15.29:0/1400459337 shutdown_connections 2024-04-17T19:47:34.912 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.913+0000 7f5023800640 1 -- 172.21.15.29:0/1400459337 wait complete. 2024-04-17T19:47:34.912 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5023800640 1 Processor -- start 2024-04-17T19:47:34.912 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5023800640 1 -- start start 2024-04-17T19:47:34.912 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5023800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c0c7420 0x7f501c1f8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:34.912 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5023800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f501c0cc790 0x7f501c1eb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:34.912 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5023800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f501c158480 0x7f501c148d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:34.912 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5023800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f501c0c7050 con 0x7f501c0c7420 2024-04-17T19:47:34.912 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5023800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f501c0c6ed0 con 0x7f501c158480 2024-04-17T19:47:34.912 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5023800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f501c0cc360 con 0x7f501c0cc790 2024-04-17T19:47:34.912 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5020e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c0c7420 0x7f501c1f8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:34.913 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5020e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c0c7420 0x7f501c1f8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50816/0 (socket says 172.21.15.29:50816) 2024-04-17T19:47:34.913 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5020e00640 1 -- 172.21.15.29:0/45732148 learned_addr learned my addr 172.21.15.29:0/45732148 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:34.913 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5021800640 1 --2- 172.21.15.29:0/45732148 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f501c158480 0x7f501c148d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:34.913 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f501be00640 1 --2- 172.21.15.29:0/45732148 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f501c0cc790 0x7f501c1eb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:34.913 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5020e00640 1 -- 172.21.15.29:0/45732148 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f501c0cc790 msgr2=0x7f501c1eb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:34.913 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5020e00640 1 --2- 172.21.15.29:0/45732148 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f501c0cc790 0x7f501c1eb2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:34.913 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5020e00640 1 -- 172.21.15.29:0/45732148 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f501c158480 msgr2=0x7f501c148d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:34.913 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5020e00640 1 --2- 172.21.15.29:0/45732148 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f501c158480 0x7f501c148d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:34.913 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5020e00640 1 -- 172.21.15.29:0/45732148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f501c1f8cc0 con 0x7f501c0c7420 2024-04-17T19:47:34.914 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5021800640 1 --2- 172.21.15.29:0/45732148 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f501c158480 0x7f501c148d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:34.914 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5020e00640 1 --2- 172.21.15.29:0/45732148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c0c7420 0x7f501c1f8780 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f501001c4e0 tx=0x7f501001b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:34.914 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5018c00640 1 -- 172.21.15.29:0/45732148 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50100b7070 con 0x7f501c0c7420 2024-04-17T19:47:34.914 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5018c00640 1 -- 172.21.15.29:0/45732148 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50100a7610 con 0x7f501c0c7420 2024-04-17T19:47:34.914 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5018c00640 1 -- 172.21.15.29:0/45732148 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50100ad050 con 0x7f501c0c7420 2024-04-17T19:47:34.914 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5023800640 1 -- 172.21.15.29:0/45732148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f501c153600 con 0x7f501c0c7420 2024-04-17T19:47:34.914 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.917+0000 7f5023800640 1 -- 172.21.15.29:0/45732148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f501c154450 con 0x7f501c0c7420 2024-04-17T19:47:34.914 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.921+0000 7f5023800640 1 -- 172.21.15.29:0/45732148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f501c069b90 con 0x7f501c0c7420 2024-04-17T19:47:34.914 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.921+0000 7f5018c00640 1 -- 172.21.15.29:0/45732148 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) ==== 87931+0+0 (secure 0 0 0) 0x7f50100c2060 con 0x7f501c0c7420 2024-04-17T19:47:34.915 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.921+0000 7f5018c00640 1 --2- 172.21.15.29:0/45732148 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4ffc0682f0 0x7f4ffc06a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:34.915 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.921+0000 7f5018c00640 1 -- 172.21.15.29:0/45732148 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7f501011fab0 con 0x7f501c0c7420 2024-04-17T19:47:34.915 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.921+0000 7f501be00640 1 --2- 172.21.15.29:0/45732148 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4ffc0682f0 0x7f4ffc06a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:34.916 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.921+0000 7f501be00640 1 --2- 172.21.15.29:0/45732148 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4ffc0682f0 0x7f4ffc06a800 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f501c0c5800 tx=0x7f5008008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:34.916 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:34.925+0000 7f5018c00640 1 -- 172.21.15.29:0/45732148 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f50100185c0 con 0x7f501c0c7420 2024-04-17T19:47:35.079 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:35.089+0000 7f5023800640 1 -- 172.21.15.29:0/45732148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f501c0c3fb0 con 0x7f501c0c7420 2024-04-17T19:47:35.080 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:35.093+0000 7f5018c00640 1 -- 172.21.15.29:0/45732148 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7f50100f0ff0 con 0x7f501c0c7420 2024-04-17T19:47:35.080 INFO:teuthology.misc.health.smithi029.stdout: 2024-04-17T19:47:35.080 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":12,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:47:32.834306+0000","last_up_change":"0.000000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:47:35.082 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:35.093+0000 7f5023800640 1 -- 172.21.15.29:0/45732148 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4ffc0682f0 msgr2=0x7f4ffc06a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:35.082 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:35.093+0000 7f5023800640 1 --2- 172.21.15.29:0/45732148 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4ffc0682f0 0x7f4ffc06a800 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f501c0c5800 tx=0x7f5008008040 comp rx=0 tx=0).stop 2024-04-17T19:47:35.082 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:35.093+0000 7f5023800640 1 -- 172.21.15.29:0/45732148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c0c7420 msgr2=0x7f501c1f8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:35.082 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:35.093+0000 7f5023800640 1 --2- 172.21.15.29:0/45732148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c0c7420 0x7f501c1f8780 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f501001c4e0 tx=0x7f501001b650 comp rx=0 tx=0).stop 2024-04-17T19:47:35.083 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:35.093+0000 7f5023800640 1 -- 172.21.15.29:0/45732148 shutdown_connections 2024-04-17T19:47:35.083 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:35.093+0000 7f5023800640 1 --2- 172.21.15.29:0/45732148 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4ffc0682f0 0x7f4ffc06a800 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:35.083 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:35.093+0000 7f5023800640 1 --2- 172.21.15.29:0/45732148 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f501c158480 0x7f501c148d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:35.083 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:35.093+0000 7f5023800640 1 --2- 172.21.15.29:0/45732148 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f501c0cc790 0x7f501c1eb2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:35.083 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:35.093+0000 7f5023800640 1 --2- 172.21.15.29:0/45732148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c0c7420 0x7f501c1f8780 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:35.083 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:35.093+0000 7f5023800640 1 -- 172.21.15.29:0/45732148 >> 172.21.15.29:0/45732148 conn(0x7f501c06c1c0 msgr2=0x7f501c0c80f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:35.083 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:35.093+0000 7f5023800640 1 -- 172.21.15.29:0/45732148 shutdown_connections 2024-04-17T19:47:35.083 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:35.093+0000 7f5023800640 1 -- 172.21.15.29:0/45732148 wait complete. 2024-04-17T19:47:35.100 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-17T19:47:35.206 INFO:tasks.ceph.osd.5.smithi037.stderr:2024-04-17T19:47:35.216+0000 7fd20b418640 -1 osd.5 0 waiting for initial osdmap 2024-04-17T19:47:35.221 INFO:tasks.ceph.osd.6.smithi037.stderr:2024-04-17T19:47:35.232+0000 7f91c6c12640 -1 osd.6 0 waiting for initial osdmap 2024-04-17T19:47:35.224 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-17T19:47:35.236+0000 7f7111295640 -1 osd.4 0 waiting for initial osdmap 2024-04-17T19:47:35.232 INFO:tasks.ceph.osd.5.smithi037.stderr:2024-04-17T19:47:35.244+0000 7fd203219640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T19:47:35.244 INFO:tasks.ceph.osd.6.smithi037.stderr:2024-04-17T19:47:35.256+0000 7f91bea13640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T19:47:35.252 INFO:tasks.ceph.osd.7.smithi037.stderr:2024-04-17T19:47:35.264+0000 7f41dc11b640 -1 osd.7 0 waiting for initial osdmap 2024-04-17T19:47:35.255 INFO:tasks.ceph.osd.4.smithi037.stderr:2024-04-17T19:47:35.268+0000 7f7109096640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T19:47:35.288 INFO:tasks.ceph.osd.7.smithi037.stderr:2024-04-17T19:47:35.300+0000 7f41d3708640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T19:47:41.906 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-17T19:47:42.061 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.073+0000 7f1320000640 1 Processor -- start 2024-04-17T19:47:42.061 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.073+0000 7f1320000640 1 -- start start 2024-04-17T19:47:42.063 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.073+0000 7f1320000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1318154050 0x7f13180c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.063 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.073+0000 7f1320000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f131815a5e0 0x7f131815a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.063 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.073+0000 7f1320000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13180c79d0 0x7f13180c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.063 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.073+0000 7f1320000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f1318071830 con 0x7f1318154050 2024-04-17T19:47:42.063 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.073+0000 7f1320000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f1318070260 con 0x7f131815a5e0 2024-04-17T19:47:42.064 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.073+0000 7f1320000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f131806c290 con 0x7f13180c79d0 2024-04-17T19:47:42.064 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.073+0000 7f131d600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1318154050 0x7f13180c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.064 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.073+0000 7f131d600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1318154050 0x7f13180c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45478/0 (socket says 172.21.15.29:45478) 2024-04-17T19:47:42.064 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.073+0000 7f131d600640 1 -- 172.21.15.29:0/1683406724 learned_addr learned my addr 172.21.15.29:0/1683406724 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:42.064 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.073+0000 7f131cc00640 1 --2- 172.21.15.29:0/1683406724 >> v2:172.21.15.37:3300/0 conn(0x7f131815a5e0 0x7f131815a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.064 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.073+0000 7f131e000640 1 --2- 172.21.15.29:0/1683406724 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13180c79d0 0x7f13180c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.064 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.073+0000 7f131d600640 1 -- 172.21.15.29:0/1683406724 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13180c79d0 msgr2=0x7f13180c7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:42.064 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.073+0000 7f131d600640 1 --2- 172.21.15.29:0/1683406724 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13180c79d0 0x7f13180c7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.064 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.073+0000 7f131d600640 1 -- 172.21.15.29:0/1683406724 >> v2:172.21.15.37:3300/0 conn(0x7f131815a5e0 msgr2=0x7f131815a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:42.064 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.073+0000 7f131d600640 1 --2- 172.21.15.29:0/1683406724 >> v2:172.21.15.37:3300/0 conn(0x7f131815a5e0 0x7f131815a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.064 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.073+0000 7f131d600640 1 -- 172.21.15.29:0/1683406724 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13180c9200 con 0x7f1318154050 2024-04-17T19:47:42.066 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.077+0000 7f131d600640 1 --2- 172.21.15.29:0/1683406724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1318154050 0x7f13180c8cc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1318074300 tx=0x7f130c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2985da0d7f7dc0c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:42.067 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.077+0000 7f1317e00640 1 -- 172.21.15.29:0/1683406724 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c0aa660 con 0x7f1318154050 2024-04-17T19:47:42.067 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.077+0000 7f1317e00640 1 -- 172.21.15.29:0/1683406724 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f130c0af070 con 0x7f1318154050 2024-04-17T19:47:42.067 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.077+0000 7f1317e00640 1 -- 172.21.15.29:0/1683406724 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f130c0aaa20 con 0x7f1318154050 2024-04-17T19:47:42.067 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.077+0000 7f1320000640 1 -- 172.21.15.29:0/1683406724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1318154050 msgr2=0x7f13180c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:42.068 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.077+0000 7f1320000640 1 --2- 172.21.15.29:0/1683406724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1318154050 0x7f13180c8cc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1318074300 tx=0x7f130c01c520 comp rx=0 tx=0).stop 2024-04-17T19:47:42.068 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.077+0000 7f1320000640 1 -- 172.21.15.29:0/1683406724 shutdown_connections 2024-04-17T19:47:42.068 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.077+0000 7f1320000640 1 --2- 172.21.15.29:0/1683406724 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13180c79d0 0x7f13180c7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.068 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.077+0000 7f1320000640 1 --2- 172.21.15.29:0/1683406724 >> v2:172.21.15.37:3300/0 conn(0x7f131815a5e0 0x7f131815a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.068 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.077+0000 7f1320000640 1 --2- 172.21.15.29:0/1683406724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1318154050 0x7f13180c8cc0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.068 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.077+0000 7f1320000640 1 -- 172.21.15.29:0/1683406724 >> 172.21.15.29:0/1683406724 conn(0x7f13180cadc0 msgr2=0x7f13180cb1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:42.068 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.077+0000 7f1320000640 1 -- 172.21.15.29:0/1683406724 shutdown_connections 2024-04-17T19:47:42.068 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.077+0000 7f1320000640 1 -- 172.21.15.29:0/1683406724 wait complete. 2024-04-17T19:47:42.069 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.081+0000 7f1320000640 1 Processor -- start 2024-04-17T19:47:42.069 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.081+0000 7f1320000640 1 -- start start 2024-04-17T19:47:42.072 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.081+0000 7f1320000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13180c79d0 0x7f13181f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.072 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.081+0000 7f1320000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1318154050 0x7f131815c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.072 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.081+0000 7f1320000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f131815a5e0 0x7f1318140920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.072 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.081+0000 7f1320000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f13181545e0 con 0x7f1318154050 2024-04-17T19:47:42.073 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.081+0000 7f1320000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1318154460 con 0x7f131815a5e0 2024-04-17T19:47:42.073 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.081+0000 7f1320000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f13180c9690 con 0x7f13180c79d0 2024-04-17T19:47:42.073 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.081+0000 7f131cc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1318154050 0x7f131815c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.073 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.081+0000 7f131cc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1318154050 0x7f131815c720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45484/0 (socket says 172.21.15.29:45484) 2024-04-17T19:47:42.073 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.081+0000 7f131cc00640 1 -- 172.21.15.29:0/1158939891 learned_addr learned my addr 172.21.15.29:0/1158939891 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:42.073 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.081+0000 7f131d600640 1 --2- 172.21.15.29:0/1158939891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13180c79d0 0x7f13181f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.073 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.081+0000 7f131e000640 1 --2- 172.21.15.29:0/1158939891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f131815a5e0 0x7f1318140920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.073 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.081+0000 7f131cc00640 1 -- 172.21.15.29:0/1158939891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13180c79d0 msgr2=0x7f13181f6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:42.073 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.081+0000 7f131cc00640 1 --2- 172.21.15.29:0/1158939891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13180c79d0 0x7f13181f6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.073 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.081+0000 7f131cc00640 1 -- 172.21.15.29:0/1158939891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f131815a5e0 msgr2=0x7f1318140920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:42.073 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.081+0000 7f131cc00640 1 --2- 172.21.15.29:0/1158939891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f131815a5e0 0x7f1318140920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.073 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.081+0000 7f131cc00640 1 -- 172.21.15.29:0/1158939891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f131814c890 con 0x7f1318154050 2024-04-17T19:47:42.073 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.085+0000 7f131cc00640 1 --2- 172.21.15.29:0/1158939891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1318154050 0x7f131815c720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f130400d5f0 tx=0x7f130400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:42.074 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.085+0000 7f1315600640 1 -- 172.21.15.29:0/1158939891 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1304004240 con 0x7f1318154050 2024-04-17T19:47:42.074 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.085+0000 7f1315600640 1 -- 172.21.15.29:0/1158939891 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13040043e0 con 0x7f1318154050 2024-04-17T19:47:42.074 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.085+0000 7f1315600640 1 -- 172.21.15.29:0/1158939891 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1304003840 con 0x7f1318154050 2024-04-17T19:47:42.074 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.085+0000 7f1320000640 1 -- 172.21.15.29:0/1158939891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f131814cb30 con 0x7f1318154050 2024-04-17T19:47:42.074 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.085+0000 7f1320000640 1 -- 172.21.15.29:0/1158939891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f131814e330 con 0x7f1318154050 2024-04-17T19:47:42.074 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.085+0000 7f1320000640 1 -- 172.21.15.29:0/1158939891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f131814efe0 con 0x7f1318154050 2024-04-17T19:47:42.074 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.085+0000 7f1315600640 1 -- 172.21.15.29:0/1158939891 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f13040039e0 con 0x7f1318154050 2024-04-17T19:47:42.076 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.085+0000 7f1315600640 1 --2- 172.21.15.29:0/1158939891 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f12f8068380 0x7f12f806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.076 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.089+0000 7f1315600640 1 -- 172.21.15.29:0/1158939891 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f1304086920 con 0x7f1318154050 2024-04-17T19:47:42.076 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.089+0000 7f131d600640 1 --2- 172.21.15.29:0/1158939891 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f12f8068380 0x7f12f806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.077 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.089+0000 7f131d600640 1 --2- 172.21.15.29:0/1158939891 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f12f8068380 0x7f12f806a890 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f130c0ae000 tx=0x7f130c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:42.081 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.093+0000 7f1315600640 1 -- 172.21.15.29:0/1158939891 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f130401c360 con 0x7f1318154050 2024-04-17T19:47:42.246 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.257+0000 7f1320000640 1 -- 172.21.15.29:0/1158939891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1318069b90 con 0x7f1318154050 2024-04-17T19:47:42.247 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.257+0000 7f1315600640 1 -- 172.21.15.29:0/1158939891 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12764 (secure 0 0 0) 0x7f1304057230 con 0x7f1318154050 2024-04-17T19:47:42.248 INFO:teuthology.misc.health.smithi029.stdout: 2024-04-17T19:47:42.248 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":16,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:47:38.916050+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:47:42.250 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.261+0000 7f1320000640 1 -- 172.21.15.29:0/1158939891 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f12f8068380 msgr2=0x7f12f806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:42.250 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.261+0000 7f1320000640 1 --2- 172.21.15.29:0/1158939891 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f12f8068380 0x7f12f806a890 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f130c0ae000 tx=0x7f130c0a7000 comp rx=0 tx=0).stop 2024-04-17T19:47:42.250 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.261+0000 7f1320000640 1 -- 172.21.15.29:0/1158939891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1318154050 msgr2=0x7f131815c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:42.250 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.261+0000 7f1320000640 1 --2- 172.21.15.29:0/1158939891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1318154050 0x7f131815c720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f130400d5f0 tx=0x7f130400dab0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.251 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.261+0000 7f1320000640 1 -- 172.21.15.29:0/1158939891 shutdown_connections 2024-04-17T19:47:42.251 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.261+0000 7f1320000640 1 --2- 172.21.15.29:0/1158939891 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f12f8068380 0x7f12f806a890 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.251 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.261+0000 7f1320000640 1 --2- 172.21.15.29:0/1158939891 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f131815a5e0 0x7f1318140920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.251 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.261+0000 7f1320000640 1 --2- 172.21.15.29:0/1158939891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1318154050 0x7f131815c720 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.251 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.261+0000 7f1320000640 1 --2- 172.21.15.29:0/1158939891 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f13180c79d0 0x7f13181f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.251 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.261+0000 7f1320000640 1 -- 172.21.15.29:0/1158939891 >> 172.21.15.29:0/1158939891 conn(0x7f13180cadc0 msgr2=0x7f13181481f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:42.251 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.261+0000 7f1320000640 1 -- 172.21.15.29:0/1158939891 shutdown_connections 2024-04-17T19:47:42.251 INFO:teuthology.misc.health.smithi029.stderr:2024-04-17T19:47:42.261+0000 7f1320000640 1 -- 172.21.15.29:0/1158939891 wait complete. 2024-04-17T19:47:42.268 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-17T19:47:42.268 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-17T19:47:42.268 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T19:47:42.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T19:47:42.271 INFO:tasks.ceph.mds.a:Started 2024-04-17T19:47:42.271 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T19:47:42.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T19:47:42.274 INFO:tasks.ceph.mds.c:Started 2024-04-17T19:47:42.275 INFO:tasks.ceph.mds.e:Restarting daemon 2024-04-17T19:47:42.275 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-04-17T19:47:42.278 INFO:tasks.ceph.mds.e:Started 2024-04-17T19:47:42.278 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T19:47:42.278 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T19:47:42.281 INFO:tasks.ceph.mds.b:Started 2024-04-17T19:47:42.281 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T19:47:42.281 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T19:47:42.284 INFO:tasks.ceph.mds.d:Started 2024-04-17T19:47:42.284 INFO:tasks.ceph.mds.f:Restarting daemon 2024-04-17T19:47:42.284 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-04-17T19:47:42.286 INFO:tasks.ceph.mds.f:Started 2024-04-17T19:47:42.286 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-17T19:47:42.287 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:47:42.399 INFO:tasks.ceph.mds.d.smithi037.stdout:starting mds.d at 2024-04-17T19:47:42.399 INFO:tasks.ceph.mds.b.smithi037.stdout:starting mds.b at 2024-04-17T19:47:42.400 INFO:tasks.ceph.mds.f.smithi037.stdout:starting mds.f at 2024-04-17T19:47:42.400 INFO:tasks.ceph.mds.d.smithi037.stderr:2024-04-17T19:47:42.411+0000 7f6bce839640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6836/0: (98) Address already in use 2024-04-17T19:47:42.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.453+0000 7f8b3a000640 1 Processor -- start 2024-04-17T19:47:42.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.453+0000 7f8b3a000640 1 -- start start 2024-04-17T19:47:42.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.453+0000 7f8b3a000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8b3415fdb0 0x7f8b341601c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.453+0000 7f8b3a000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b340cae50 0x7f8b340cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.453+0000 7f8b3a000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b340cbc20 con 0x7f8b340cc7f0 2024-04-17T19:47:42.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.453+0000 7f8b3a000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f8b3415d140 con 0x7f8b340cae50 2024-04-17T19:47:42.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.453+0000 7f8b3a000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8b3415c180 con 0x7f8b3415fdb0 2024-04-17T19:47:42.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.453+0000 7f8b33e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b340cae50 0x7f8b340cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.453+0000 7f8b2be00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8b3415fdb0 0x7f8b341601c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.453+0000 7f8b33400640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f8b340cc7f0 0x7f8b340c9ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:35074/0 (socket says 172.21.15.29:35074) 2024-04-17T19:47:42.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.453+0000 7f8b33400640 1 -- 172.21.15.29:0/249530918 learned_addr learned my addr 172.21.15.29:0/249530918 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:47:42.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.457+0000 7f8b33e00640 1 -- 172.21.15.29:0/249530918 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8b3415fdb0 msgr2=0x7f8b341601c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:42.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.457+0000 7f8b33e00640 1 --2- 172.21.15.29:0/249530918 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8b3415fdb0 0x7f8b341601c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.457+0000 7f8b33e00640 1 -- 172.21.15.29:0/249530918 >> v1:172.21.15.37:6789/0 conn(0x7f8b340cc7f0 legacy=0x7f8b340c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:42.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.457+0000 7f8b32a00640 1 -- 172.21.15.29:0/249530918 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 521539585 0 0) 0x7f8b24001120 con 0x7f8b340cc7f0 2024-04-17T19:47:42.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.457+0000 7f8b33e00640 1 -- 172.21.15.29:0/249530918 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b3406ca70 con 0x7f8b340cae50 2024-04-17T19:47:42.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.457+0000 7f8b33e00640 1 --2- 172.21.15.29:0/249530918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b340cae50 0x7f8b340cb260 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c009dc0 tx=0x7f8b2c01c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e151adbdef6a89 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:42.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.457+0000 7f8b32a00640 1 -- 172.21.15.29:0/249530918 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c0a6700 con 0x7f8b340cae50 2024-04-17T19:47:42.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.457+0000 7f8b32a00640 1 -- 172.21.15.29:0/249530918 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b2c0a68a0 con 0x7f8b340cae50 2024-04-17T19:47:42.447 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.457+0000 7f8b32a00640 1 -- 172.21.15.29:0/249530918 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c0a6c70 con 0x7f8b340cae50 2024-04-17T19:47:42.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.457+0000 7f8b3a000640 1 -- 172.21.15.29:0/249530918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b340cae50 msgr2=0x7f8b340cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:42.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.457+0000 7f8b3a000640 1 --2- 172.21.15.29:0/249530918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b340cae50 0x7f8b340cb260 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c009dc0 tx=0x7f8b2c01c830 comp rx=0 tx=0).stop 2024-04-17T19:47:42.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.461+0000 7f8b3a000640 1 -- 172.21.15.29:0/249530918 shutdown_connections 2024-04-17T19:47:42.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.461+0000 7f8b3a000640 1 --2- 172.21.15.29:0/249530918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b340cae50 0x7f8b340cb260 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.461+0000 7f8b3a000640 1 --2- 172.21.15.29:0/249530918 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8b3415fdb0 0x7f8b341601c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.461+0000 7f8b3a000640 1 -- 172.21.15.29:0/249530918 >> 172.21.15.29:0/249530918 conn(0x7f8b34067eb0 msgr2=0x7f8b34070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:42.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.461+0000 7f8b3a000640 1 -- 172.21.15.29:0/249530918 shutdown_connections 2024-04-17T19:47:42.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.461+0000 7f8b3a000640 1 -- 172.21.15.29:0/249530918 wait complete. 2024-04-17T19:47:42.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.461+0000 7f8b3a000640 1 Processor -- start 2024-04-17T19:47:42.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.461+0000 7f8b3a000640 1 -- start start 2024-04-17T19:47:42.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.461+0000 7f8b3a000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8b340cae50 0x7f8b340c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.461+0000 7f8b3a000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b340cc7f0 0x7f8b340c5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.461+0000 7f8b3a000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8b3415fdb0 0x7f8b340ce830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.461+0000 7f8b3a000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3406c790 con 0x7f8b340cc7f0 2024-04-17T19:47:42.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.461+0000 7f8b3a000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3406c4b0 con 0x7f8b340cae50 2024-04-17T19:47:42.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.461+0000 7f8b3a000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8b34169240 con 0x7f8b3415fdb0 2024-04-17T19:47:42.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.461+0000 7f8b33400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8b340cae50 0x7f8b340c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.461+0000 7f8b2be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b340cc7f0 0x7f8b340c5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.465+0000 7f8b33400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8b340cae50 0x7f8b340c6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:34376/0 (socket says 172.21.15.29:34376) 2024-04-17T19:47:42.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.465+0000 7f8b33400640 1 -- 172.21.15.29:0/390416334 learned_addr learned my addr 172.21.15.29:0/390416334 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:42.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.465+0000 7f8b33e00640 1 --2- 172.21.15.29:0/390416334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8b3415fdb0 0x7f8b340ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.465+0000 7f8b33400640 1 -- 172.21.15.29:0/390416334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8b3415fdb0 msgr2=0x7f8b340ce830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:42.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.465+0000 7f8b33400640 1 --2- 172.21.15.29:0/390416334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8b3415fdb0 0x7f8b340ce830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.465+0000 7f8b33400640 1 -- 172.21.15.29:0/390416334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b340cc7f0 msgr2=0x7f8b340c5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:42.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.465+0000 7f8b33400640 1 --2- 172.21.15.29:0/390416334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b340cc7f0 0x7f8b340c5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.465+0000 7f8b33400640 1 -- 172.21.15.29:0/390416334 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b340d0270 con 0x7f8b340cae50 2024-04-17T19:47:42.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.465+0000 7f8b2be00640 1 --2- 172.21.15.29:0/390416334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b340cc7f0 0x7f8b340c5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:47:42.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.465+0000 7f8b33400640 1 --2- 172.21.15.29:0/390416334 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8b340cae50 0x7f8b340c6f60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8b2400c4e0 tx=0x7f8b2400c9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:42.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.465+0000 7f8b2b400640 1 -- 172.21.15.29:0/390416334 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b24018070 con 0x7f8b340cae50 2024-04-17T19:47:42.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.465+0000 7f8b2b400640 1 -- 172.21.15.29:0/390416334 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b24007500 con 0x7f8b340cae50 2024-04-17T19:47:42.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.465+0000 7f8b3a000640 1 -- 172.21.15.29:0/390416334 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b340c78f0 con 0x7f8b340cae50 2024-04-17T19:47:42.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.465+0000 7f8b2b400640 1 -- 172.21.15.29:0/390416334 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b24004a30 con 0x7f8b340cae50 2024-04-17T19:47:42.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.465+0000 7f8b3a000640 1 -- 172.21.15.29:0/390416334 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b340cfa70 con 0x7f8b340cae50 2024-04-17T19:47:42.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.465+0000 7f8b3a000640 1 -- 172.21.15.29:0/390416334 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b340cf580 con 0x7f8b340cae50 2024-04-17T19:47:42.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.465+0000 7f8b2b400640 1 -- 172.21.15.29:0/390416334 <== mon.1 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8b2401b070 con 0x7f8b340cae50 2024-04-17T19:47:42.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.469+0000 7f8b2b400640 1 --2- 172.21.15.29:0/390416334 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8b10068380 0x7f8b1006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.469+0000 7f8b2b400640 1 -- 172.21.15.29:0/390416334 <== mon.1 v2:172.21.15.37:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f8b240859e0 con 0x7f8b340cae50 2024-04-17T19:47:42.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.469+0000 7f8b2be00640 1 --2- 172.21.15.29:0/390416334 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8b10068380 0x7f8b1006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.469+0000 7f8b2be00640 1 --2- 172.21.15.29:0/390416334 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8b10068380 0x7f8b1006a890 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8b340c6ef0 tx=0x7f8b1c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:42.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.473+0000 7f8b2b400640 1 -- 172.21.15.29:0/390416334 <== mon.1 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8b340cbc20 con 0x7f8b340cae50 2024-04-17T19:47:42.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.645+0000 7f8b3a000640 1 -- 172.21.15.29:0/390416334 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b34069b90 con 0x7f8b340cae50 2024-04-17T19:47:42.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.645+0000 7f8b2b400640 1 -- 172.21.15.29:0/390416334 <== mon.1 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12764 (secure 0 0 0) 0x7f8b24055b70 con 0x7f8b340cae50 2024-04-17T19:47:42.636 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:47:42.636 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":16,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:47:38.916050+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:47:42.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.649+0000 7f8b3a000640 1 -- 172.21.15.29:0/390416334 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8b10068380 msgr2=0x7f8b1006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:42.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.649+0000 7f8b3a000640 1 --2- 172.21.15.29:0/390416334 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8b10068380 0x7f8b1006a890 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8b340c6ef0 tx=0x7f8b1c009190 comp rx=0 tx=0).stop 2024-04-17T19:47:42.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.649+0000 7f8b3a000640 1 -- 172.21.15.29:0/390416334 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8b340cae50 msgr2=0x7f8b340c6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:42.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.649+0000 7f8b3a000640 1 --2- 172.21.15.29:0/390416334 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8b340cae50 0x7f8b340c6f60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8b2400c4e0 tx=0x7f8b2400c9a0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.649+0000 7f8b3a000640 1 -- 172.21.15.29:0/390416334 shutdown_connections 2024-04-17T19:47:42.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.649+0000 7f8b3a000640 1 --2- 172.21.15.29:0/390416334 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8b10068380 0x7f8b1006a890 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.649+0000 7f8b3a000640 1 --2- 172.21.15.29:0/390416334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8b3415fdb0 0x7f8b340ce830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.649+0000 7f8b3a000640 1 --2- 172.21.15.29:0/390416334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b340cc7f0 0x7f8b340c5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.649+0000 7f8b3a000640 1 --2- 172.21.15.29:0/390416334 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8b340cae50 0x7f8b340c6f60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.649+0000 7f8b3a000640 1 -- 172.21.15.29:0/390416334 >> 172.21.15.29:0/390416334 conn(0x7f8b34067eb0 msgr2=0x7f8b3406dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:42.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.649+0000 7f8b3a000640 1 -- 172.21.15.29:0/390416334 shutdown_connections 2024-04-17T19:47:42.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.649+0000 7f8b3a000640 1 -- 172.21.15.29:0/390416334 wait complete. 2024-04-17T19:47:42.657 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T19:47:36.674676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T19:47:42.658 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T19:47:42.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.825+0000 7f623e400640 1 Processor -- start 2024-04-17T19:47:42.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.825+0000 7f623e400640 1 -- start start 2024-04-17T19:47:42.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.825+0000 7f623e400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6238071ce0 0x7f6238141de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.825+0000 7f623e400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62381593d0 0x7f62381597e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.825+0000 7f623e400640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6238142320 con 0x7f62380c9f00 2024-04-17T19:47:42.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.825+0000 7f623e400640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f62380c7bd0 con 0x7f62381593d0 2024-04-17T19:47:42.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.825+0000 7f623e400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f623815c6d0 con 0x7f6238071ce0 2024-04-17T19:47:42.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.825+0000 7f622fe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62381593d0 0x7f62381597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.825+0000 7f622fe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62381593d0 0x7f62381597e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45526/0 (socket says 172.21.15.29:45526) 2024-04-17T19:47:42.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.825+0000 7f622fe00640 1 -- 172.21.15.29:0/1554494206 learned_addr learned my addr 172.21.15.29:0/1554494206 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:42.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.825+0000 7f6237400640 1 --2- 172.21.15.29:0/1554494206 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6238071ce0 0x7f6238141de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.825+0000 7f622fe00640 1 -- 172.21.15.29:0/1554494206 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6238071ce0 msgr2=0x7f6238141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:42.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.825+0000 7f622fe00640 1 --2- 172.21.15.29:0/1554494206 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6238071ce0 0x7f6238141de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.825+0000 7f622fe00640 1 -- 172.21.15.29:0/1554494206 >> v1:172.21.15.37:6789/0 conn(0x7f62380c9f00 legacy=0x7f62380c6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:42.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.825+0000 7f622fe00640 1 -- 172.21.15.29:0/1554494206 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f623815c9b0 con 0x7f62381593d0 2024-04-17T19:47:42.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.829+0000 7f622fe00640 1 --2- 172.21.15.29:0/1554494206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62381593d0 0x7f62381597e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6220009870 tx=0x7f622001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b3a9d0d7d4ffe0b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:42.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.829+0000 7f6236a00640 1 -- 172.21.15.29:0/1554494206 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62200b2070 con 0x7f62381593d0 2024-04-17T19:47:42.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.829+0000 7f6236a00640 1 -- 172.21.15.29:0/1554494206 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62200a9070 con 0x7f62381593d0 2024-04-17T19:47:42.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.829+0000 7f6236a00640 1 -- 172.21.15.29:0/1554494206 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62200ad880 con 0x7f62381593d0 2024-04-17T19:47:42.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.829+0000 7f623e400640 1 -- 172.21.15.29:0/1554494206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62381593d0 msgr2=0x7f62381597e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:42.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.829+0000 7f623e400640 1 --2- 172.21.15.29:0/1554494206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62381593d0 0x7f62381597e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6220009870 tx=0x7f622001c560 comp rx=0 tx=0).stop 2024-04-17T19:47:42.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.829+0000 7f623e400640 1 -- 172.21.15.29:0/1554494206 shutdown_connections 2024-04-17T19:47:42.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.829+0000 7f623e400640 1 --2- 172.21.15.29:0/1554494206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62381593d0 0x7f62381597e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.829+0000 7f623e400640 1 --2- 172.21.15.29:0/1554494206 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6238071ce0 0x7f6238141de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.829+0000 7f623e400640 1 -- 172.21.15.29:0/1554494206 >> 172.21.15.29:0/1554494206 conn(0x7f623806bf90 msgr2=0x7f623806c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:42.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.829+0000 7f623e400640 1 -- 172.21.15.29:0/1554494206 shutdown_connections 2024-04-17T19:47:42.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.829+0000 7f623e400640 1 -- 172.21.15.29:0/1554494206 wait complete. 2024-04-17T19:47:42.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f623e400640 1 Processor -- start 2024-04-17T19:47:42.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f623e400640 1 -- start start 2024-04-17T19:47:42.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f623e400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6238071ce0 0x7f62381f5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f623e400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62380c9f00 0x7f6238153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f623e400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62381593d0 0x7f6238153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f623e400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f62380720f0 con 0x7f62380c9f00 2024-04-17T19:47:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f623e400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f62380ca720 con 0x7f6238071ce0 2024-04-17T19:47:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f623e400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6238072270 con 0x7f62381593d0 2024-04-17T19:47:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f622fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62380c9f00 0x7f6238153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f622fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62380c9f00 0x7f6238153760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45538/0 (socket says 172.21.15.29:45538) 2024-04-17T19:47:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f622fe00640 1 -- 172.21.15.29:0/85915029 learned_addr learned my addr 172.21.15.29:0/85915029 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f6237400640 1 --2- 172.21.15.29:0/85915029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6238071ce0 0x7f62381f5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f6237e00640 1 --2- 172.21.15.29:0/85915029 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62381593d0 0x7f6238153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f622fe00640 1 -- 172.21.15.29:0/85915029 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62381593d0 msgr2=0x7f6238153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f622fe00640 1 --2- 172.21.15.29:0/85915029 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62381593d0 0x7f6238153ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f622fe00640 1 -- 172.21.15.29:0/85915029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6238071ce0 msgr2=0x7f62381f5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f622fe00640 1 --2- 172.21.15.29:0/85915029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6238071ce0 0x7f62381f5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f622fe00640 1 -- 172.21.15.29:0/85915029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62381f61d0 con 0x7f62380c9f00 2024-04-17T19:47:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f622fe00640 1 --2- 172.21.15.29:0/85915029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62380c9f00 0x7f6238153760 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f62200b0040 tx=0x7f6220019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:42.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f622f400640 1 -- 172.21.15.29:0/85915029 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62200b2040 con 0x7f62380c9f00 2024-04-17T19:47:42.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f623e400640 1 -- 172.21.15.29:0/85915029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6238163680 con 0x7f62380c9f00 2024-04-17T19:47:42.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f622f400640 1 -- 172.21.15.29:0/85915029 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62200aec60 con 0x7f62380c9f00 2024-04-17T19:47:42.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f622f400640 1 -- 172.21.15.29:0/85915029 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62200b8640 con 0x7f62380c9f00 2024-04-17T19:47:42.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.833+0000 7f623e400640 1 -- 172.21.15.29:0/85915029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6238156e10 con 0x7f62380c9f00 2024-04-17T19:47:42.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.837+0000 7f623e400640 1 -- 172.21.15.29:0/85915029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6238069b90 con 0x7f62380c9f00 2024-04-17T19:47:42.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.837+0000 7f622f400640 1 -- 172.21.15.29:0/85915029 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f62200c0020 con 0x7f62380c9f00 2024-04-17T19:47:42.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.837+0000 7f622f400640 1 --2- 172.21.15.29:0/85915029 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6214068380 0x7f621406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:42.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.837+0000 7f622f400640 1 -- 172.21.15.29:0/85915029 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f6220120340 con 0x7f62380c9f00 2024-04-17T19:47:42.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.837+0000 7f6237400640 1 --2- 172.21.15.29:0/85915029 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6214068380 0x7f621406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:42.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.841+0000 7f6237400640 1 --2- 172.21.15.29:0/85915029 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6214068380 0x7f621406a890 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f622800b930 tx=0x7f6228008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:42.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:42.845+0000 7f622f400640 1 -- 172.21.15.29:0/85915029 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6220016150 con 0x7f62380c9f00 2024-04-17T19:47:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.009+0000 7f623e400640 1 -- 172.21.15.29:0/85915029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f623806c770 con 0x7f62380c9f00 2024-04-17T19:47:42.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.009+0000 7f622f400640 1 -- 172.21.15.29:0/85915029 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f62200f0c60 con 0x7f62380c9f00 2024-04-17T19:47:42.997 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-04-17T19:47:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.009+0000 7f623e400640 1 -- 172.21.15.29:0/85915029 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6214068380 msgr2=0x7f621406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.009+0000 7f623e400640 1 --2- 172.21.15.29:0/85915029 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6214068380 0x7f621406a890 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f622800b930 tx=0x7f6228008040 comp rx=0 tx=0).stop 2024-04-17T19:47:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.009+0000 7f623e400640 1 -- 172.21.15.29:0/85915029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62380c9f00 msgr2=0x7f6238153760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.009+0000 7f623e400640 1 --2- 172.21.15.29:0/85915029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62380c9f00 0x7f6238153760 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f62200b0040 tx=0x7f6220019300 comp rx=0 tx=0).stop 2024-04-17T19:47:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.009+0000 7f623e400640 1 -- 172.21.15.29:0/85915029 shutdown_connections 2024-04-17T19:47:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.009+0000 7f623e400640 1 --2- 172.21.15.29:0/85915029 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6214068380 0x7f621406a890 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.009+0000 7f623e400640 1 --2- 172.21.15.29:0/85915029 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f62381593d0 0x7f6238153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.009+0000 7f623e400640 1 --2- 172.21.15.29:0/85915029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62380c9f00 0x7f6238153760 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.009+0000 7f623e400640 1 --2- 172.21.15.29:0/85915029 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6238071ce0 0x7f62381f5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.009+0000 7f623e400640 1 -- 172.21.15.29:0/85915029 >> 172.21.15.29:0/85915029 conn(0x7f623806bf90 msgr2=0x7f623815b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.009+0000 7f623e400640 1 -- 172.21.15.29:0/85915029 shutdown_connections 2024-04-17T19:47:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.009+0000 7f623e400640 1 -- 172.21.15.29:0/85915029 wait complete. 2024-04-17T19:47:43.018 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:47:43.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.185+0000 7f4a87200640 1 Processor -- start 2024-04-17T19:47:43.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.185+0000 7f4a87200640 1 -- start start 2024-04-17T19:47:43.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.185+0000 7f4a87200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f4a880740f0 0x7f4a8815b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:43.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.185+0000 7f4a87200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4a8815bd40 0x7f4a88162b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:43.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.185+0000 7f4a87200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a88074500 con 0x7f4a88073b30 2024-04-17T19:47:43.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.185+0000 7f4a87200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f4a8806edb0 con 0x7f4a8815bd40 2024-04-17T19:47:43.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.185+0000 7f4a87200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f4a880c9590 con 0x7f4a880740f0 2024-04-17T19:47:43.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.185+0000 7f4a85e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4a88073b30 0x7f4a880c30f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56554/0 (socket says 172.21.15.29:56554) 2024-04-17T19:47:43.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.185+0000 7f4a85e00640 1 -- 172.21.15.29:0/2984188357 learned_addr learned my addr 172.21.15.29:0/2984188357 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:47:43.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.185+0000 7f4a86800640 1 --2- 172.21.15.29:0/2984188357 >> v2:172.21.15.29:3300/0 conn(0x7f4a8815bd40 0x7f4a88162b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:43.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.185+0000 7f4a85400640 1 --2- 172.21.15.29:0/2984188357 >> v2:172.21.15.37:3300/0 conn(0x7f4a880740f0 0x7f4a8815b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:43.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.185+0000 7f4a84a00640 1 -- 172.21.15.29:0/2984188357 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 643304530 0 0) 0x7f4a88074500 con 0x7f4a88073b30 2024-04-17T19:47:43.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.185+0000 7f4a84a00640 1 -- 172.21.15.29:0/2984188357 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a6c003530 con 0x7f4a88073b30 2024-04-17T19:47:43.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.185+0000 7f4a86800640 1 -- 172.21.15.29:0/2984188357 >> v2:172.21.15.37:3300/0 conn(0x7f4a880740f0 msgr2=0x7f4a8815b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:43.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.185+0000 7f4a86800640 1 --2- 172.21.15.29:0/2984188357 >> v2:172.21.15.37:3300/0 conn(0x7f4a880740f0 0x7f4a8815b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:43.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.185+0000 7f4a86800640 1 -- 172.21.15.29:0/2984188357 >> v1:172.21.15.29:6789/0 conn(0x7f4a88073b30 legacy=0x7f4a880c30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:43.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.185+0000 7f4a86800640 1 -- 172.21.15.29:0/2984188357 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a880c92d0 con 0x7f4a8815bd40 2024-04-17T19:47:43.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.189+0000 7f4a86800640 1 --2- 172.21.15.29:0/2984188357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a8815bd40 0x7f4a88162b20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4a880cab40 tx=0x7f4a8001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c859fe4ef7294dd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:43.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.189+0000 7f4a84a00640 1 -- 172.21.15.29:0/2984188357 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a800b2070 con 0x7f4a8815bd40 2024-04-17T19:47:43.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.189+0000 7f4a84a00640 1 -- 172.21.15.29:0/2984188357 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a800a5860 con 0x7f4a8815bd40 2024-04-17T19:47:43.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.189+0000 7f4a84a00640 1 -- 172.21.15.29:0/2984188357 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a800a5c10 con 0x7f4a8815bd40 2024-04-17T19:47:43.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.189+0000 7f4a87200640 1 -- 172.21.15.29:0/2984188357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a8815bd40 msgr2=0x7f4a88162b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:43.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.189+0000 7f4a87200640 1 --2- 172.21.15.29:0/2984188357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a8815bd40 0x7f4a88162b20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4a880cab40 tx=0x7f4a8001c930 comp rx=0 tx=0).stop 2024-04-17T19:47:43.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.189+0000 7f4a87200640 1 -- 172.21.15.29:0/2984188357 shutdown_connections 2024-04-17T19:47:43.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.189+0000 7f4a87200640 1 --2- 172.21.15.29:0/2984188357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a8815bd40 0x7f4a88162b20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:43.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.189+0000 7f4a87200640 1 --2- 172.21.15.29:0/2984188357 >> v2:172.21.15.37:3300/0 conn(0x7f4a880740f0 0x7f4a8815b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:43.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.189+0000 7f4a87200640 1 -- 172.21.15.29:0/2984188357 >> 172.21.15.29:0/2984188357 conn(0x7f4a880ccf80 msgr2=0x7f4a880cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:43.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.189+0000 7f4a87200640 1 -- 172.21.15.29:0/2984188357 shutdown_connections 2024-04-17T19:47:43.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.189+0000 7f4a87200640 1 -- 172.21.15.29:0/2984188357 wait complete. 2024-04-17T19:47:43.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.193+0000 7f4a87200640 1 Processor -- start 2024-04-17T19:47:43.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.193+0000 7f4a87200640 1 -- start start 2024-04-17T19:47:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.193+0000 7f4a87200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a88073b30 0x7f4a88159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.193+0000 7f4a87200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a880740f0 0x7f4a880cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.193+0000 7f4a87200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a8815bd40 0x7f4a8814e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.193+0000 7f4a87200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4a880c3800 con 0x7f4a88073b30 2024-04-17T19:47:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.193+0000 7f4a87200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4a8806ea40 con 0x7f4a880740f0 2024-04-17T19:47:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.193+0000 7f4a87200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4a880c3980 con 0x7f4a8815bd40 2024-04-17T19:47:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.193+0000 7f4a85e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a88073b30 0x7f4a88159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.193+0000 7f4a85e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a88073b30 0x7f4a88159c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45548/0 (socket says 172.21.15.29:45548) 2024-04-17T19:47:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.193+0000 7f4a85e00640 1 -- 172.21.15.29:0/1390650301 learned_addr learned my addr 172.21.15.29:0/1390650301 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.197+0000 7f4a85400640 1 --2- 172.21.15.29:0/1390650301 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a880740f0 0x7f4a880cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.197+0000 7f4a86800640 1 --2- 172.21.15.29:0/1390650301 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a8815bd40 0x7f4a8814e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.197+0000 7f4a85e00640 1 -- 172.21.15.29:0/1390650301 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a8815bd40 msgr2=0x7f4a8814e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.197+0000 7f4a85e00640 1 --2- 172.21.15.29:0/1390650301 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a8815bd40 0x7f4a8814e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.197+0000 7f4a85e00640 1 -- 172.21.15.29:0/1390650301 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a880740f0 msgr2=0x7f4a880cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.197+0000 7f4a85e00640 1 --2- 172.21.15.29:0/1390650301 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a880740f0 0x7f4a880cbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.197+0000 7f4a85e00640 1 -- 172.21.15.29:0/1390650301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a881f60b0 con 0x7f4a88073b30 2024-04-17T19:47:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.197+0000 7f4a86800640 1 --2- 172.21.15.29:0/1390650301 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a8815bd40 0x7f4a8814e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.197+0000 7f4a85e00640 1 --2- 172.21.15.29:0/1390650301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a88073b30 0x7f4a88159c50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800c3f0 tx=0x7f4a7800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.197+0000 7f4a7e000640 1 -- 172.21.15.29:0/1390650301 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a78017070 con 0x7f4a88073b30 2024-04-17T19:47:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.197+0000 7f4a7e000640 1 -- 172.21.15.29:0/1390650301 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a78005cf0 con 0x7f4a88073b30 2024-04-17T19:47:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.197+0000 7f4a7e000640 1 -- 172.21.15.29:0/1390650301 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a780072f0 con 0x7f4a88073b30 2024-04-17T19:47:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.197+0000 7f4a87200640 1 -- 172.21.15.29:0/1390650301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a8815a190 con 0x7f4a88073b30 2024-04-17T19:47:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.197+0000 7f4a87200640 1 -- 172.21.15.29:0/1390650301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a8815b4f0 con 0x7f4a88073b30 2024-04-17T19:47:43.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.197+0000 7f4a87200640 1 -- 172.21.15.29:0/1390650301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a8804ff10 con 0x7f4a88073b30 2024-04-17T19:47:43.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.197+0000 7f4a7e000640 1 -- 172.21.15.29:0/1390650301 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4a7801a070 con 0x7f4a88073b30 2024-04-17T19:47:43.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.201+0000 7f4a7e000640 1 --2- 172.21.15.29:0/1390650301 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4a68068380 0x7f4a6806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:43.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.201+0000 7f4a7e000640 1 -- 172.21.15.29:0/1390650301 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f4a78085dc0 con 0x7f4a88073b30 2024-04-17T19:47:43.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.201+0000 7f4a85400640 1 --2- 172.21.15.29:0/1390650301 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4a68068380 0x7f4a6806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:43.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.201+0000 7f4a85400640 1 --2- 172.21.15.29:0/1390650301 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4a68068380 0x7f4a6806a890 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4a880c8530 tx=0x7f4a74009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:43.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.201+0000 7f4a7e000640 1 -- 172.21.15.29:0/1390650301 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4a78021020 con 0x7f4a88073b30 2024-04-17T19:47:43.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.401+0000 7f4a87200640 1 -- 172.21.15.29:0/1390650301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a881518f0 con 0x7f4a88073b30 2024-04-17T19:47:43.390 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:47:43.391 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4229,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-17T19:47:43.391 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 2 2024-04-17T19:47:43.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.401+0000 7f4a7e000640 1 -- 172.21.15.29:0/1390650301 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1392 (secure 0 0 0) 0x7f4a78056160 con 0x7f4a88073b30 2024-04-17T19:47:43.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.405+0000 7f4a87200640 1 -- 172.21.15.29:0/1390650301 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4a68068380 msgr2=0x7f4a6806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:43.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.405+0000 7f4a87200640 1 --2- 172.21.15.29:0/1390650301 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4a68068380 0x7f4a6806a890 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4a880c8530 tx=0x7f4a74009190 comp rx=0 tx=0).stop 2024-04-17T19:47:43.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.405+0000 7f4a87200640 1 -- 172.21.15.29:0/1390650301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a88073b30 msgr2=0x7f4a88159c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:43.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.405+0000 7f4a87200640 1 --2- 172.21.15.29:0/1390650301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a88073b30 0x7f4a88159c50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800c3f0 tx=0x7f4a7800c8b0 comp rx=0 tx=0).stop 2024-04-17T19:47:43.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.405+0000 7f4a87200640 1 -- 172.21.15.29:0/1390650301 shutdown_connections 2024-04-17T19:47:43.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.405+0000 7f4a87200640 1 --2- 172.21.15.29:0/1390650301 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4a68068380 0x7f4a6806a890 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:43.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.405+0000 7f4a87200640 1 --2- 172.21.15.29:0/1390650301 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a8815bd40 0x7f4a8814e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:43.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.405+0000 7f4a87200640 1 --2- 172.21.15.29:0/1390650301 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a880740f0 0x7f4a880cbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:43.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.405+0000 7f4a87200640 1 --2- 172.21.15.29:0/1390650301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a88073b30 0x7f4a88159c50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:43.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.405+0000 7f4a87200640 1 -- 172.21.15.29:0/1390650301 >> 172.21.15.29:0/1390650301 conn(0x7f4a880ccf80 msgr2=0x7f4a8814f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:43.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.405+0000 7f4a87200640 1 -- 172.21.15.29:0/1390650301 shutdown_connections 2024-04-17T19:47:43.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:43.405+0000 7f4a87200640 1 -- 172.21.15.29:0/1390650301 wait complete. 2024-04-17T19:47:44.424 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2024-04-17T19:47:44.425 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2024-04-17T19:47:44.425 INFO:tasks.ceph.mds.e.smithi029.stdout:starting mds.e at 2024-04-17T19:47:44.427 INFO:tasks.ceph.mds.c.smithi029.stderr:2024-04-17T19:47:44.437+0000 7ff3b7a00640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-04-17T19:47:45.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:47:45.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.981+0000 7fe861c00640 1 Processor -- start 2024-04-17T19:47:45.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.981+0000 7fe861c00640 1 -- start start 2024-04-17T19:47:45.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.985+0000 7fe861c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe85c1560d0 0x7fe85c15be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:45.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.985+0000 7fe861c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe85c15c370 0x7fe85c1667b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:45.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.985+0000 7fe861c00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe85c0c55e0 con 0x7fe85c0c9b20 2024-04-17T19:47:45.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.985+0000 7fe861c00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fe85c0c8fa0 con 0x7fe85c1560d0 2024-04-17T19:47:45.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.985+0000 7fe861c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe85c0c9f30 con 0x7fe85c15c370 2024-04-17T19:47:45.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.985+0000 7fe853e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe85c1560d0 0x7fe85c15be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:45.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.985+0000 7fe85aa00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe85c0c9b20 0x7fe85c06e300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56580/0 (socket says 172.21.15.29:56580) 2024-04-17T19:47:45.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.985+0000 7fe85aa00640 1 -- 172.21.15.29:0/2879719007 learned_addr learned my addr 172.21.15.29:0/2879719007 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:47:45.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.985+0000 7fe85b400640 1 --2- 172.21.15.29:0/2879719007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe85c15c370 0x7fe85c1667b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:45.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.985+0000 7fe853e00640 1 -- 172.21.15.29:0/2879719007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe85c15c370 msgr2=0x7fe85c1667b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:45.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.985+0000 7fe853e00640 1 --2- 172.21.15.29:0/2879719007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe85c15c370 0x7fe85c1667b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:45.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.985+0000 7fe853e00640 1 -- 172.21.15.29:0/2879719007 >> v1:172.21.15.29:6789/0 conn(0x7fe85c0c9b20 legacy=0x7fe85c06e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:45.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.985+0000 7fe853e00640 1 -- 172.21.15.29:0/2879719007 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe85c0ca1d0 con 0x7fe85c1560d0 2024-04-17T19:47:45.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.985+0000 7fe85a000640 1 -- 172.21.15.29:0/2879719007 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1996481076 0 0) 0x7fe84c001120 con 0x7fe85c0c9b20 2024-04-17T19:47:45.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.985+0000 7fe853e00640 1 --2- 172.21.15.29:0/2879719007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe85c1560d0 0x7fe85c15be30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe8440097c0 tx=0x7fe84401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b4f60c803935af server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:45.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.989+0000 7fe85a000640 1 -- 172.21.15.29:0/2879719007 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8440b5070 con 0x7fe85c1560d0 2024-04-17T19:47:45.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.989+0000 7fe85a000640 1 -- 172.21.15.29:0/2879719007 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8440a8990 con 0x7fe85c1560d0 2024-04-17T19:47:45.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.989+0000 7fe85a000640 1 -- 172.21.15.29:0/2879719007 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8440a8d60 con 0x7fe85c1560d0 2024-04-17T19:47:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.989+0000 7fe861c00640 1 -- 172.21.15.29:0/2879719007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe85c1560d0 msgr2=0x7fe85c15be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.989+0000 7fe861c00640 1 --2- 172.21.15.29:0/2879719007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe85c1560d0 0x7fe85c15be30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe8440097c0 tx=0x7fe84401ec90 comp rx=0 tx=0).stop 2024-04-17T19:47:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.989+0000 7fe861c00640 1 -- 172.21.15.29:0/2879719007 shutdown_connections 2024-04-17T19:47:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.989+0000 7fe861c00640 1 --2- 172.21.15.29:0/2879719007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe85c15c370 0x7fe85c1667b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.989+0000 7fe861c00640 1 --2- 172.21.15.29:0/2879719007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe85c1560d0 0x7fe85c15be30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.989+0000 7fe861c00640 1 -- 172.21.15.29:0/2879719007 >> 172.21.15.29:0/2879719007 conn(0x7fe85c06c3c0 msgr2=0x7fe85c0ca3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.989+0000 7fe861c00640 1 -- 172.21.15.29:0/2879719007 shutdown_connections 2024-04-17T19:47:45.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.989+0000 7fe861c00640 1 -- 172.21.15.29:0/2879719007 wait complete. 2024-04-17T19:47:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.989+0000 7fe861c00640 1 Processor -- start 2024-04-17T19:47:45.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.989+0000 7fe861c00640 1 -- start start 2024-04-17T19:47:45.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe861c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe85c0c9b20 0x7fe85c1f66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:45.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe861c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe85c1560d0 0x7fe85c154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:45.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe861c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe85c15c370 0x7fe85c155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:45.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe861c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe85c0c8670 con 0x7fe85c0c9b20 2024-04-17T19:47:45.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe861c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe85c0c8390 con 0x7fe85c1560d0 2024-04-17T19:47:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe861c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe85c0c8950 con 0x7fe85c15c370 2024-04-17T19:47:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe85aa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe85c0c9b20 0x7fe85c1f66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe85aa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe85c0c9b20 0x7fe85c1f66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45582/0 (socket says 172.21.15.29:45582) 2024-04-17T19:47:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe85aa00640 1 -- 172.21.15.29:0/3348663439 learned_addr learned my addr 172.21.15.29:0/3348663439 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe853e00640 1 --2- 172.21.15.29:0/3348663439 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe85c1560d0 0x7fe85c154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe85b400640 1 --2- 172.21.15.29:0/3348663439 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe85c15c370 0x7fe85c155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe85aa00640 1 -- 172.21.15.29:0/3348663439 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe85c15c370 msgr2=0x7fe85c155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe85aa00640 1 --2- 172.21.15.29:0/3348663439 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe85c15c370 0x7fe85c155450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe85aa00640 1 -- 172.21.15.29:0/3348663439 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe85c1560d0 msgr2=0x7fe85c154f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe85aa00640 1 --2- 172.21.15.29:0/3348663439 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe85c1560d0 0x7fe85c154f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe85aa00640 1 -- 172.21.15.29:0/3348663439 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe85c1f6c30 con 0x7fe85c0c9b20 2024-04-17T19:47:45.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe85aa00640 1 --2- 172.21.15.29:0/3348663439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe85c0c9b20 0x7fe85c1f66f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe84c00c480 tx=0x7fe84c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:45.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe852a00640 1 -- 172.21.15.29:0/3348663439 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84c018070 con 0x7fe85c0c9b20 2024-04-17T19:47:45.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe852a00640 1 -- 172.21.15.29:0/3348663439 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe84c007500 con 0x7fe85c0c9b20 2024-04-17T19:47:45.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe852a00640 1 -- 172.21.15.29:0/3348663439 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84c004a30 con 0x7fe85c0c9b20 2024-04-17T19:47:45.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe861c00640 1 -- 172.21.15.29:0/3348663439 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe85c0cd020 con 0x7fe85c0c9b20 2024-04-17T19:47:45.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe861c00640 1 -- 172.21.15.29:0/3348663439 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe85c0cd5e0 con 0x7fe85c0c9b20 2024-04-17T19:47:45.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.993+0000 7fe861c00640 1 -- 172.21.15.29:0/3348663439 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe85c069b90 con 0x7fe85c0c9b20 2024-04-17T19:47:45.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.997+0000 7fe852a00640 1 -- 172.21.15.29:0/3348663439 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe84c01b070 con 0x7fe85c0c9b20 2024-04-17T19:47:45.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.997+0000 7fe852a00640 1 --2- 172.21.15.29:0/3348663439 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe8380683b0 0x7fe83806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:45.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.997+0000 7fe852a00640 1 -- 172.21.15.29:0/3348663439 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fe84c085900 con 0x7fe85c0c9b20 2024-04-17T19:47:45.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.997+0000 7fe853e00640 1 --2- 172.21.15.29:0/3348663439 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe8380683b0 0x7fe83806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:45.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:45.997+0000 7fe853e00640 1 --2- 172.21.15.29:0/3348663439 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe8380683b0 0x7fe83806a8c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe8440b2000 tx=0x7fe844008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:45.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.001+0000 7fe852a00640 1 -- 172.21.15.29:0/3348663439 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe85c0c55e0 con 0x7fe85c0c9b20 2024-04-17T19:47:46.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.205+0000 7fe861c00640 1 -- 172.21.15.29:0/3348663439 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe85c069580 con 0x7fe85c0c9b20 2024-04-17T19:47:46.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.205+0000 7fe852a00640 1 -- 172.21.15.29:0/3348663439 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+5503 (secure 0 0 0) 0x7fe84c056200 con 0x7fe85c0c9b20 2024-04-17T19:47:46.193 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:47:46.193 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6838/332907250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":332907250},{"type":"v1","addr":"172.21.15.37:6838","nonce":332907250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":4302,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":4305,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[]} 2024-04-17T19:47:46.193 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 4 2024-04-17T19:47:46.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.205+0000 7fe861c00640 1 -- 172.21.15.29:0/3348663439 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe8380683b0 msgr2=0x7fe83806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:46.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.205+0000 7fe861c00640 1 --2- 172.21.15.29:0/3348663439 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe8380683b0 0x7fe83806a8c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe8440b2000 tx=0x7fe844008040 comp rx=0 tx=0).stop 2024-04-17T19:47:46.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.205+0000 7fe861c00640 1 -- 172.21.15.29:0/3348663439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe85c0c9b20 msgr2=0x7fe85c1f66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:46.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.205+0000 7fe861c00640 1 --2- 172.21.15.29:0/3348663439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe85c0c9b20 0x7fe85c1f66f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe84c00c480 tx=0x7fe84c00c940 comp rx=0 tx=0).stop 2024-04-17T19:47:46.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.205+0000 7fe861c00640 1 -- 172.21.15.29:0/3348663439 shutdown_connections 2024-04-17T19:47:46.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.205+0000 7fe861c00640 1 --2- 172.21.15.29:0/3348663439 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe8380683b0 0x7fe83806a8c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.205+0000 7fe861c00640 1 --2- 172.21.15.29:0/3348663439 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe85c15c370 0x7fe85c155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.205+0000 7fe861c00640 1 --2- 172.21.15.29:0/3348663439 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe85c1560d0 0x7fe85c154f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.205+0000 7fe861c00640 1 --2- 172.21.15.29:0/3348663439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe85c0c9b20 0x7fe85c1f66f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.205+0000 7fe861c00640 1 -- 172.21.15.29:0/3348663439 >> 172.21.15.29:0/3348663439 conn(0x7fe85c06c3c0 msgr2=0x7fe85c072b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:46.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.205+0000 7fe861c00640 1 -- 172.21.15.29:0/3348663439 shutdown_connections 2024-04-17T19:47:46.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.205+0000 7fe861c00640 1 -- 172.21.15.29:0/3348663439 wait complete. 2024-04-17T19:47:46.215 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:47:46.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.377+0000 7fd875600640 1 Processor -- start 2024-04-17T19:47:46.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.377+0000 7fd875600640 1 -- start start 2024-04-17T19:47:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.381+0000 7fd875600640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd8700cc8c0 0x7fd8700cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.381+0000 7fd875600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd8701597a0 0x7fd8701635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.381+0000 7fd875600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8700c88b0 con 0x7fd870159390 2024-04-17T19:47:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.381+0000 7fd875600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fd8700c6ce0 con 0x7fd8701597a0 2024-04-17T19:47:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.381+0000 7fd875600640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd8700c6fc0 con 0x7fd8700cc8c0 2024-04-17T19:47:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.381+0000 7fd86f400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd8701597a0 0x7fd8701635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:46.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.381+0000 7fd86f400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd8701597a0 0x7fd8701635e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45592/0 (socket says 172.21.15.29:45592) 2024-04-17T19:47:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.381+0000 7fd867e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd870159390 0x7fd8700ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56590/0 (socket says 172.21.15.29:56590) 2024-04-17T19:47:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.381+0000 7fd86f400640 1 -- 172.21.15.29:0/887847482 learned_addr learned my addr 172.21.15.29:0/887847482 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.381+0000 7fd86ea00640 1 --2- 172.21.15.29:0/887847482 >> v2:172.21.15.37:3300/0 conn(0x7fd8700cc8c0 0x7fd8700cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.381+0000 7fd86f400640 1 -- 172.21.15.29:0/887847482 >> v2:172.21.15.37:3300/0 conn(0x7fd8700cc8c0 msgr2=0x7fd8700cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.381+0000 7fd86f400640 1 --2- 172.21.15.29:0/887847482 >> v2:172.21.15.37:3300/0 conn(0x7fd8700cc8c0 0x7fd8700cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.381+0000 7fd86f400640 1 -- 172.21.15.29:0/887847482 >> v1:172.21.15.29:6789/0 conn(0x7fd870159390 legacy=0x7fd8700ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:46.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.381+0000 7fd86f400640 1 -- 172.21.15.29:0/887847482 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8700c72a0 con 0x7fd8701597a0 2024-04-17T19:47:46.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.385+0000 7fd86f400640 1 --2- 172.21.15.29:0/887847482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8701597a0 0x7fd8701635e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd8700721b0 tx=0x7fd86801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d96de016137a2ff server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.385+0000 7fd86e000640 1 -- 172.21.15.29:0/887847482 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8680a56c0 con 0x7fd8701597a0 2024-04-17T19:47:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.385+0000 7fd86e000640 1 -- 172.21.15.29:0/887847482 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8680a5860 con 0x7fd8701597a0 2024-04-17T19:47:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.385+0000 7fd86e000640 1 -- 172.21.15.29:0/887847482 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8680a5c30 con 0x7fd8701597a0 2024-04-17T19:47:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.385+0000 7fd875600640 1 -- 172.21.15.29:0/887847482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8701597a0 msgr2=0x7fd8701635e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.385+0000 7fd875600640 1 --2- 172.21.15.29:0/887847482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8701597a0 0x7fd8701635e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd8700721b0 tx=0x7fd86801c9f0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.385+0000 7fd875600640 1 -- 172.21.15.29:0/887847482 shutdown_connections 2024-04-17T19:47:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.385+0000 7fd875600640 1 --2- 172.21.15.29:0/887847482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8701597a0 0x7fd8701635e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.385+0000 7fd875600640 1 --2- 172.21.15.29:0/887847482 >> v2:172.21.15.37:3300/0 conn(0x7fd8700cc8c0 0x7fd8700cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.385+0000 7fd875600640 1 -- 172.21.15.29:0/887847482 >> 172.21.15.29:0/887847482 conn(0x7fd87006fde0 msgr2=0x7fd8700701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.385+0000 7fd875600640 1 -- 172.21.15.29:0/887847482 shutdown_connections 2024-04-17T19:47:46.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.385+0000 7fd875600640 1 -- 172.21.15.29:0/887847482 wait complete. 2024-04-17T19:47:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.385+0000 7fd875600640 1 Processor -- start 2024-04-17T19:47:46.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.385+0000 7fd875600640 1 -- start start 2024-04-17T19:47:46.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd875600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd8700cc8c0 0x7fd870162030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:46.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd875600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd870159390 0x7fd870140aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:46.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd875600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8701597a0 0x7fd870150bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:46.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd875600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd8700cabc0 con 0x7fd8701597a0 2024-04-17T19:47:46.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd875600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd8700caa40 con 0x7fd870159390 2024-04-17T19:47:46.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd875600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd8700c6880 con 0x7fd8700cc8c0 2024-04-17T19:47:46.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd86f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8701597a0 0x7fd870150bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:46.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd86f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8701597a0 0x7fd870150bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45602/0 (socket says 172.21.15.29:45602) 2024-04-17T19:47:46.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd86f400640 1 -- 172.21.15.29:0/2497581536 learned_addr learned my addr 172.21.15.29:0/2497581536 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:46.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd867e00640 1 --2- 172.21.15.29:0/2497581536 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd870159390 0x7fd870140aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:46.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd86ea00640 1 --2- 172.21.15.29:0/2497581536 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd8700cc8c0 0x7fd870162030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:46.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd86f400640 1 -- 172.21.15.29:0/2497581536 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd8700cc8c0 msgr2=0x7fd870162030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:46.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd86f400640 1 --2- 172.21.15.29:0/2497581536 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd8700cc8c0 0x7fd870162030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd86f400640 1 -- 172.21.15.29:0/2497581536 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd870159390 msgr2=0x7fd870140aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:46.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd86f400640 1 --2- 172.21.15.29:0/2497581536 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd870159390 0x7fd870140aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd86f400640 1 -- 172.21.15.29:0/2497581536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd870162570 con 0x7fd8701597a0 2024-04-17T19:47:46.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd867e00640 1 --2- 172.21.15.29:0/2497581536 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd870159390 0x7fd870140aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:46.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd86f400640 1 --2- 172.21.15.29:0/2497581536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8701597a0 0x7fd870150bc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd8680af040 tx=0x7fd868019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:46.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd866a00640 1 -- 172.21.15.29:0/2497581536 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8680bc020 con 0x7fd8701597a0 2024-04-17T19:47:46.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd866a00640 1 -- 172.21.15.29:0/2497581536 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8680b8670 con 0x7fd8701597a0 2024-04-17T19:47:46.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd875600640 1 -- 172.21.15.29:0/2497581536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd870163290 con 0x7fd8701597a0 2024-04-17T19:47:46.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd866a00640 1 -- 172.21.15.29:0/2497581536 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd868009510 con 0x7fd8701597a0 2024-04-17T19:47:46.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd875600640 1 -- 172.21.15.29:0/2497581536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd870161830 con 0x7fd8701597a0 2024-04-17T19:47:46.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.389+0000 7fd875600640 1 -- 172.21.15.29:0/2497581536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd870162740 con 0x7fd8701597a0 2024-04-17T19:47:46.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.393+0000 7fd866a00640 1 -- 172.21.15.29:0/2497581536 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd8680bf020 con 0x7fd8701597a0 2024-04-17T19:47:46.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.393+0000 7fd866a00640 1 --2- 172.21.15.29:0/2497581536 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd84c0683b0 0x7fd84c06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:46.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.393+0000 7fd866a00640 1 -- 172.21.15.29:0/2497581536 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fd868121740 con 0x7fd8701597a0 2024-04-17T19:47:46.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.393+0000 7fd86ea00640 1 --2- 172.21.15.29:0/2497581536 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd84c0683b0 0x7fd84c06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:46.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.393+0000 7fd86ea00640 1 --2- 172.21.15.29:0/2497581536 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd84c0683b0 0x7fd84c06a8c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd860009a50 tx=0x7fd860008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:46.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.397+0000 7fd866a00640 1 -- 172.21.15.29:0/2497581536 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd8680c6030 con 0x7fd8701597a0 2024-04-17T19:47:46.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.561+0000 7fd875600640 1 -- 172.21.15.29:0/2497581536 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd870069b90 con 0x7fd8701597a0 2024-04-17T19:47:46.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.565+0000 7fd866a00640 1 -- 172.21.15.29:0/2497581536 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12764 (secure 0 0 0) 0x7fd8680165e0 con 0x7fd8701597a0 2024-04-17T19:47:46.553 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:47:46.553 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":16,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:47:38.916050+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:47:46.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.565+0000 7fd875600640 1 -- 172.21.15.29:0/2497581536 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd84c0683b0 msgr2=0x7fd84c06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:46.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.565+0000 7fd875600640 1 --2- 172.21.15.29:0/2497581536 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd84c0683b0 0x7fd84c06a8c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd860009a50 tx=0x7fd860008040 comp rx=0 tx=0).stop 2024-04-17T19:47:46.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.565+0000 7fd875600640 1 -- 172.21.15.29:0/2497581536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8701597a0 msgr2=0x7fd870150bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:46.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.565+0000 7fd875600640 1 --2- 172.21.15.29:0/2497581536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8701597a0 0x7fd870150bc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd8680af040 tx=0x7fd868019790 comp rx=0 tx=0).stop 2024-04-17T19:47:46.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.565+0000 7fd875600640 1 -- 172.21.15.29:0/2497581536 shutdown_connections 2024-04-17T19:47:46.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.565+0000 7fd875600640 1 --2- 172.21.15.29:0/2497581536 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd84c0683b0 0x7fd84c06a8c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.565+0000 7fd875600640 1 --2- 172.21.15.29:0/2497581536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8701597a0 0x7fd870150bc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.565+0000 7fd875600640 1 --2- 172.21.15.29:0/2497581536 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd870159390 0x7fd870140aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.565+0000 7fd875600640 1 --2- 172.21.15.29:0/2497581536 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd8700cc8c0 0x7fd870162030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.565+0000 7fd875600640 1 -- 172.21.15.29:0/2497581536 >> 172.21.15.29:0/2497581536 conn(0x7fd87006fde0 msgr2=0x7fd8701528a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:46.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.565+0000 7fd875600640 1 -- 172.21.15.29:0/2497581536 shutdown_connections 2024-04-17T19:47:46.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.565+0000 7fd875600640 1 -- 172.21.15.29:0/2497581536 wait complete. 2024-04-17T19:47:46.575 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T19:47:36.674676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T19:47:46.575 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T19:47:46.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.741+0000 7f36e2000640 1 Processor -- start 2024-04-17T19:47:46.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.741+0000 7f36e2000640 1 -- start start 2024-04-17T19:47:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.741+0000 7f36e2000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f36dc071ce0 0x7f36dc141de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.741+0000 7f36e2000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f36dc1597e0 0x7f36dc163620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.741+0000 7f36e2000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36dc142320 con 0x7f36dc1593d0 2024-04-17T19:47:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.741+0000 7f36e2000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f36dc0c7bd0 con 0x7f36dc071ce0 2024-04-17T19:47:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.741+0000 7f36e2000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f36dc15c6d0 con 0x7f36dc1597e0 2024-04-17T19:47:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.741+0000 7f36db400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f36dc071ce0 0x7f36dc141de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.741+0000 7f36db400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f36dc071ce0 0x7f36dc141de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45610/0 (socket says 172.21.15.29:45610) 2024-04-17T19:47:46.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.741+0000 7f36db400640 1 -- 172.21.15.29:0/4072345762 learned_addr learned my addr 172.21.15.29:0/4072345762 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:46.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.745+0000 7f36dbe00640 1 --2- 172.21.15.29:0/4072345762 >> v2:172.21.15.37:3300/0 conn(0x7f36dc1597e0 0x7f36dc163620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:46.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.745+0000 7f36db400640 1 -- 172.21.15.29:0/4072345762 >> v2:172.21.15.37:3300/0 conn(0x7f36dc1597e0 msgr2=0x7f36dc163620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:46.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.745+0000 7f36db400640 1 --2- 172.21.15.29:0/4072345762 >> v2:172.21.15.37:3300/0 conn(0x7f36dc1597e0 0x7f36dc163620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.745+0000 7f36db400640 1 -- 172.21.15.29:0/4072345762 >> v1:172.21.15.37:6789/0 conn(0x7f36dc1593d0 legacy=0x7f36dc0c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:46.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.745+0000 7f36db400640 1 -- 172.21.15.29:0/4072345762 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36dc15c9b0 con 0x7f36dc071ce0 2024-04-17T19:47:46.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.745+0000 7f36db400640 1 --2- 172.21.15.29:0/4072345762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36dc071ce0 0x7f36dc141de0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f36dc0caaa0 tx=0x7f36cc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce319fbfff1ad17e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:46.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.745+0000 7f36da000640 1 -- 172.21.15.29:0/4072345762 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc0b0070 con 0x7f36dc071ce0 2024-04-17T19:47:46.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.745+0000 7f36da000640 1 -- 172.21.15.29:0/4072345762 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36cc0ab600 con 0x7f36dc071ce0 2024-04-17T19:47:46.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.745+0000 7f36da000640 1 -- 172.21.15.29:0/4072345762 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc0ab9d0 con 0x7f36dc071ce0 2024-04-17T19:47:46.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.749+0000 7f36e2000640 1 -- 172.21.15.29:0/4072345762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36dc071ce0 msgr2=0x7f36dc141de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:46.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.749+0000 7f36e2000640 1 --2- 172.21.15.29:0/4072345762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36dc071ce0 0x7f36dc141de0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f36dc0caaa0 tx=0x7f36cc01c520 comp rx=0 tx=0).stop 2024-04-17T19:47:46.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.749+0000 7f36e2000640 1 -- 172.21.15.29:0/4072345762 shutdown_connections 2024-04-17T19:47:46.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.749+0000 7f36e2000640 1 --2- 172.21.15.29:0/4072345762 >> v2:172.21.15.37:3300/0 conn(0x7f36dc1597e0 0x7f36dc163620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.749+0000 7f36e2000640 1 --2- 172.21.15.29:0/4072345762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36dc071ce0 0x7f36dc141de0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.749+0000 7f36e2000640 1 -- 172.21.15.29:0/4072345762 >> 172.21.15.29:0/4072345762 conn(0x7f36dc06bf90 msgr2=0x7f36dc06c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:46.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.749+0000 7f36e2000640 1 -- 172.21.15.29:0/4072345762 shutdown_connections 2024-04-17T19:47:46.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.749+0000 7f36e2000640 1 -- 172.21.15.29:0/4072345762 wait complete. 2024-04-17T19:47:46.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.749+0000 7f36e2000640 1 Processor -- start 2024-04-17T19:47:46.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.749+0000 7f36e2000640 1 -- start start 2024-04-17T19:47:46.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.749+0000 7f36e2000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36dc071ce0 0x7f36dc1f8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:46.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.749+0000 7f36e2000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36dc1593d0 0x7f36dc139990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:46.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.749+0000 7f36e2000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36dc1597e0 0x7f36dc13bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:46.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.749+0000 7f36e2000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f36dc0720f0 con 0x7f36dc1597e0 2024-04-17T19:47:46.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.749+0000 7f36e2000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f36dc159bf0 con 0x7f36dc1593d0 2024-04-17T19:47:46.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.749+0000 7f36e2000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f36dc072270 con 0x7f36dc071ce0 2024-04-17T19:47:46.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.753+0000 7f36dbe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36dc1597e0 0x7f36dc13bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:46.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.753+0000 7f36dbe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36dc1597e0 0x7f36dc13bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45618/0 (socket says 172.21.15.29:45618) 2024-04-17T19:47:46.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.753+0000 7f36dbe00640 1 -- 172.21.15.29:0/3308725821 learned_addr learned my addr 172.21.15.29:0/3308725821 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:46.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.753+0000 7f36daa00640 1 --2- 172.21.15.29:0/3308725821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36dc1593d0 0x7f36dc139990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:46.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.753+0000 7f36db400640 1 --2- 172.21.15.29:0/3308725821 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36dc071ce0 0x7f36dc1f8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:46.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.753+0000 7f36dbe00640 1 -- 172.21.15.29:0/3308725821 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36dc071ce0 msgr2=0x7f36dc1f8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:46.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.753+0000 7f36dbe00640 1 --2- 172.21.15.29:0/3308725821 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36dc071ce0 0x7f36dc1f8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.753+0000 7f36dbe00640 1 -- 172.21.15.29:0/3308725821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36dc1593d0 msgr2=0x7f36dc139990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:46.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.753+0000 7f36dbe00640 1 --2- 172.21.15.29:0/3308725821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36dc1593d0 0x7f36dc139990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.753+0000 7f36dbe00640 1 -- 172.21.15.29:0/3308725821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36dc0c7900 con 0x7f36dc1597e0 2024-04-17T19:47:46.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.753+0000 7f36dbe00640 1 --2- 172.21.15.29:0/3308725821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36dc1597e0 0x7f36dc13bad0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f36d4009fd0 tx=0x7f36d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:46.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.753+0000 7f36d3400640 1 -- 172.21.15.29:0/3308725821 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d4014420 con 0x7f36dc1597e0 2024-04-17T19:47:46.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.753+0000 7f36d3400640 1 -- 172.21.15.29:0/3308725821 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36d4019070 con 0x7f36dc1597e0 2024-04-17T19:47:46.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.753+0000 7f36e2000640 1 -- 172.21.15.29:0/3308725821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36dc0c5da0 con 0x7f36dc1597e0 2024-04-17T19:47:46.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.753+0000 7f36d3400640 1 -- 172.21.15.29:0/3308725821 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36d4014800 con 0x7f36dc1597e0 2024-04-17T19:47:46.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.753+0000 7f36e2000640 1 -- 172.21.15.29:0/3308725821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36dc15bc20 con 0x7f36dc1597e0 2024-04-17T19:47:46.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.753+0000 7f36e2000640 1 -- 172.21.15.29:0/3308725821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36dc069b90 con 0x7f36dc1597e0 2024-04-17T19:47:46.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.753+0000 7f36d3400640 1 -- 172.21.15.29:0/3308725821 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f36d40149a0 con 0x7f36dc1597e0 2024-04-17T19:47:46.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.757+0000 7f36d3400640 1 --2- 172.21.15.29:0/3308725821 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f36bc068380 0x7f36bc06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:46.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.757+0000 7f36d3400640 1 -- 172.21.15.29:0/3308725821 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f36d4086e50 con 0x7f36dc1597e0 2024-04-17T19:47:46.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.757+0000 7f36db400640 1 --2- 172.21.15.29:0/3308725821 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f36bc068380 0x7f36bc06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:46.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.757+0000 7f36db400640 1 --2- 172.21.15.29:0/3308725821 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f36bc068380 0x7f36bc06a890 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0ad000 tx=0x7f36cc0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:46.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.761+0000 7f36d3400640 1 -- 172.21.15.29:0/3308725821 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f36d4056fe0 con 0x7f36dc1597e0 2024-04-17T19:47:46.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.925+0000 7f36e2000640 1 -- 172.21.15.29:0/3308725821 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f36dc13af90 con 0x7f36dc1597e0 2024-04-17T19:47:46.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.925+0000 7f36d3400640 1 -- 172.21.15.29:0/3308725821 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f36d405b2e0 con 0x7f36dc1597e0 2024-04-17T19:47:46.914 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-04-17T19:47:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.925+0000 7f36e2000640 1 -- 172.21.15.29:0/3308725821 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f36bc068380 msgr2=0x7f36bc06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.925+0000 7f36e2000640 1 --2- 172.21.15.29:0/3308725821 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f36bc068380 0x7f36bc06a890 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0ad000 tx=0x7f36cc0a6000 comp rx=0 tx=0).stop 2024-04-17T19:47:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.925+0000 7f36e2000640 1 -- 172.21.15.29:0/3308725821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36dc1597e0 msgr2=0x7f36dc13bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.925+0000 7f36e2000640 1 --2- 172.21.15.29:0/3308725821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36dc1597e0 0x7f36dc13bad0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f36d4009fd0 tx=0x7f36d400e4b0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.925+0000 7f36e2000640 1 -- 172.21.15.29:0/3308725821 shutdown_connections 2024-04-17T19:47:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.925+0000 7f36e2000640 1 --2- 172.21.15.29:0/3308725821 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f36bc068380 0x7f36bc06a890 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.925+0000 7f36e2000640 1 --2- 172.21.15.29:0/3308725821 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36dc1597e0 0x7f36dc13bad0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.925+0000 7f36e2000640 1 --2- 172.21.15.29:0/3308725821 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36dc1593d0 0x7f36dc139990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.925+0000 7f36e2000640 1 --2- 172.21.15.29:0/3308725821 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36dc071ce0 0x7f36dc1f8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:46.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.925+0000 7f36e2000640 1 -- 172.21.15.29:0/3308725821 >> 172.21.15.29:0/3308725821 conn(0x7f36dc06bf90 msgr2=0x7f36dc06c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:46.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.929+0000 7f36e2000640 1 -- 172.21.15.29:0/3308725821 shutdown_connections 2024-04-17T19:47:46.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:46.929+0000 7f36e2000640 1 -- 172.21.15.29:0/3308725821 wait complete. 2024-04-17T19:47:46.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T19:47:47.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.097+0000 7f03db600640 1 Processor -- start 2024-04-17T19:47:47.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.097+0000 7f03db600640 1 -- start start 2024-04-17T19:47:47.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03db600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03d40cc210 0x7f03d40cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:47.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03db600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03d406e7b0 con 0x7f03d406e3a0 2024-04-17T19:47:47.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03db600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03d40cb610 con 0x7f03d415f7e0 2024-04-17T19:47:47.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03db600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f03d40cb8d0 con 0x7f03d40cc210 2024-04-17T19:47:47.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d8c00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f03d406e3a0 0x7f03d406ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56600/0 (socket says 172.21.15.29:56600) 2024-04-17T19:47:47.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d8c00640 1 -- 172.21.15.29:0/4288433216 learned_addr learned my addr 172.21.15.29:0/4288433216 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:47:47.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3e00640 1 --2- 172.21.15.29:0/4288433216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03d40cc210 0x7f03d40cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:47.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3400640 1 -- 172.21.15.29:0/4288433216 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2677684379 0 0) 0x7f03d406e7b0 con 0x7f03d406e3a0 2024-04-17T19:47:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3400640 1 -- 172.21.15.29:0/4288433216 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03bc003530 con 0x7f03d406e3a0 2024-04-17T19:47:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3400640 1 -- 172.21.15.29:0/4288433216 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2987171757 0 0) 0x7f03d40cb610 con 0x7f03d415f7e0 2024-04-17T19:47:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3400640 1 -- 172.21.15.29:0/4288433216 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f03d406e7b0 con 0x7f03d415f7e0 2024-04-17T19:47:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3400640 1 -- 172.21.15.29:0/4288433216 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2126300977 0 0) 0x7f03bc003530 con 0x7f03d406e3a0 2024-04-17T19:47:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3400640 1 -- 172.21.15.29:0/4288433216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03d40cc210 msgr2=0x7f03d40cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3400640 1 --2- 172.21.15.29:0/4288433216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03d40cc210 0x7f03d40cc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3400640 1 -- 172.21.15.29:0/4288433216 >> v1:172.21.15.37:6789/0 conn(0x7f03d415f7e0 legacy=0x7f03d415fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3400640 1 -- 172.21.15.29:0/4288433216 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d40cbbb0 con 0x7f03d406e3a0 2024-04-17T19:47:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3400640 1 -- 172.21.15.29:0/4288433216 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f03c8001680 con 0x7f03d406e3a0 2024-04-17T19:47:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3e00640 1 --2- 172.21.15.29:0/4288433216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03d40cc210 0x7f03d40cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:47:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3400640 1 -- 172.21.15.29:0/4288433216 >> v1:172.21.15.29:6789/0 conn(0x7f03d406e3a0 legacy=0x7f03d406ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3400640 1 --2- 172.21.15.29:0/4288433216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03d40cc210 0x7f03bc003bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3400640 1 --2- 172.21.15.29:0/4288433216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc004150 0x7f03bc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3400640 1 --2- 172.21.15.29:0/4288433216 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03bc004d50 0x7f03bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:47.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3400640 1 -- 172.21.15.29:0/4288433216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f03d40cb8d0 con 0x7f03bc004150 2024-04-17T19:47:47.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3400640 1 -- 172.21.15.29:0/4288433216 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f03bc00bbc0 con 0x7f03bc004d50 2024-04-17T19:47:47.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3400640 1 -- 172.21.15.29:0/4288433216 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f03bc00bdc0 con 0x7f03d40cc210 2024-04-17T19:47:47.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3e00640 1 --2- 172.21.15.29:0/4288433216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc004150 0x7f03bc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:47.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d8c00640 1 --2- 172.21.15.29:0/4288433216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03d40cc210 0x7f03bc003bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:47.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d9600640 1 --2- 172.21.15.29:0/4288433216 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03bc004d50 0x7f03bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:47.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3e00640 1 -- 172.21.15.29:0/4288433216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03d40cc210 msgr2=0x7f03bc003bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:47.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3e00640 1 --2- 172.21.15.29:0/4288433216 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03d40cc210 0x7f03bc003bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3e00640 1 -- 172.21.15.29:0/4288433216 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03bc004d50 msgr2=0x7f03bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:47.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3e00640 1 --2- 172.21.15.29:0/4288433216 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03bc004d50 0x7f03bc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d3e00640 1 -- 172.21.15.29:0/4288433216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03bc003530 con 0x7f03bc004150 2024-04-17T19:47:47.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.101+0000 7f03d9600640 1 --2- 172.21.15.29:0/4288433216 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03bc004d50 0x7f03bc0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:47.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.105+0000 7f03d3e00640 1 --2- 172.21.15.29:0/4288433216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc004150 0x7f03bc004810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f03c0009c60 tx=0x7f03c001cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c2e104f97b53dd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:47.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.105+0000 7f03d3400640 1 -- 172.21.15.29:0/4288433216 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c0009860 con 0x7f03bc004150 2024-04-17T19:47:47.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.105+0000 7f03d3400640 1 -- 172.21.15.29:0/4288433216 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03c0004030 con 0x7f03bc004150 2024-04-17T19:47:47.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.105+0000 7f03d3400640 1 -- 172.21.15.29:0/4288433216 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c0004460 con 0x7f03bc004150 2024-04-17T19:47:47.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.105+0000 7f03db600640 1 -- 172.21.15.29:0/4288433216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc004150 msgr2=0x7f03bc004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:47.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.105+0000 7f03db600640 1 --2- 172.21.15.29:0/4288433216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc004150 0x7f03bc004810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f03c0009c60 tx=0x7f03c001cc10 comp rx=0 tx=0).stop 2024-04-17T19:47:47.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.105+0000 7f03d9600640 1 -- 172.21.15.29:0/4288433216 reap_dead start 2024-04-17T19:47:47.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.105+0000 7f03db600640 1 -- 172.21.15.29:0/4288433216 shutdown_connections 2024-04-17T19:47:47.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.105+0000 7f03db600640 1 -- 172.21.15.29:0/4288433216 >> 172.21.15.29:0/4288433216 conn(0x7f03d4071ca0 msgr2=0x7f03d40720e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:47.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.105+0000 7f03db600640 1 -- 172.21.15.29:0/4288433216 shutdown_connections 2024-04-17T19:47:47.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.105+0000 7f03db600640 1 -- 172.21.15.29:0/4288433216 wait complete. 2024-04-17T19:47:47.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03db600640 1 Processor -- start 2024-04-17T19:47:47.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03db600640 1 -- start start 2024-04-17T19:47:47.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03db600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc004150 0x7f03d41e53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03db600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03d41f33b0 0x7f03d41e7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03db600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03d40cd1e0 0x7f03d4148f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03db600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f03d40ca9b0 con 0x7f03bc004150 2024-04-17T19:47:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03db600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f03d40ca6d0 con 0x7f03d41f33b0 2024-04-17T19:47:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03db600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f03d40b1f80 con 0x7f03d40cd1e0 2024-04-17T19:47:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03d8c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc004150 0x7f03d41e53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03d8c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc004150 0x7f03d41e53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45640/0 (socket says 172.21.15.29:45640) 2024-04-17T19:47:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03d8c00640 1 -- 172.21.15.29:0/3384228024 learned_addr learned my addr 172.21.15.29:0/3384228024 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03d3e00640 1 --2- 172.21.15.29:0/3384228024 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03d41f33b0 0x7f03d41e7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03d9600640 1 --2- 172.21.15.29:0/3384228024 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03d40cd1e0 0x7f03d4148f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03d8c00640 1 -- 172.21.15.29:0/3384228024 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03d40cd1e0 msgr2=0x7f03d4148f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03d8c00640 1 --2- 172.21.15.29:0/3384228024 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03d40cd1e0 0x7f03d4148f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03d8c00640 1 -- 172.21.15.29:0/3384228024 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03d41f33b0 msgr2=0x7f03d41e7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03d8c00640 1 --2- 172.21.15.29:0/3384228024 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03d41f33b0 0x7f03d41e7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03d8c00640 1 -- 172.21.15.29:0/3384228024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03d41e7fd0 con 0x7f03bc004150 2024-04-17T19:47:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03d9600640 1 --2- 172.21.15.29:0/3384228024 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03d40cd1e0 0x7f03d4148f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:47.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03d8c00640 1 --2- 172.21.15.29:0/3384228024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc004150 0x7f03d41e53e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f03c8012290 tx=0x7f03c8012750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:47.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03d0c00640 1 -- 172.21.15.29:0/3384228024 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c801c070 con 0x7f03bc004150 2024-04-17T19:47:47.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03d0c00640 1 -- 172.21.15.29:0/3384228024 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03c8014750 con 0x7f03bc004150 2024-04-17T19:47:47.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03db600640 1 -- 172.21.15.29:0/3384228024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03d41f28d0 con 0x7f03bc004150 2024-04-17T19:47:47.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03d0c00640 1 -- 172.21.15.29:0/3384228024 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03c8014b40 con 0x7f03bc004150 2024-04-17T19:47:47.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.109+0000 7f03db600640 1 -- 172.21.15.29:0/3384228024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03d415ea90 con 0x7f03bc004150 2024-04-17T19:47:47.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.113+0000 7f03db600640 1 -- 172.21.15.29:0/3384228024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03d41f26c0 con 0x7f03bc004150 2024-04-17T19:47:47.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.113+0000 7f03d0c00640 1 -- 172.21.15.29:0/3384228024 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f03c801f070 con 0x7f03bc004150 2024-04-17T19:47:47.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.113+0000 7f03d0c00640 1 --2- 172.21.15.29:0/3384228024 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f03b40683b0 0x7f03b406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:47.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.113+0000 7f03d0c00640 1 -- 172.21.15.29:0/3384228024 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f03c8089090 con 0x7f03bc004150 2024-04-17T19:47:47.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.113+0000 7f03d3e00640 1 --2- 172.21.15.29:0/3384228024 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f03b40683b0 0x7f03b406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:47.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.113+0000 7f03d3e00640 1 --2- 172.21.15.29:0/3384228024 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f03b40683b0 0x7f03b406a8c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f03c00ae000 tx=0x7f03c00046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:47.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.117+0000 7f03d0c00640 1 -- 172.21.15.29:0/3384228024 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f03c8059220 con 0x7f03bc004150 2024-04-17T19:47:47.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.285+0000 7f03db600640 1 -- 172.21.15.29:0/3384228024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f03d4069b90 con 0x7f03bc004150 2024-04-17T19:47:47.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.285+0000 7f03d0c00640 1 -- 172.21.15.29:0/3384228024 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7f03c805d520 con 0x7f03bc004150 2024-04-17T19:47:47.273 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:47:47.273 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2024-04-17T19:47:47.273 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-04-17T19:47:47.273 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2024-04-17T19:47:47.273 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2024-04-17T19:47:47.274 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-17T19:47:47.274 INFO:teuthology.orchestra.run.smithi029.stdout:] 2024-04-17T19:47:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.285+0000 7f03db600640 1 -- 172.21.15.29:0/3384228024 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f03b40683b0 msgr2=0x7f03b406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.285+0000 7f03db600640 1 --2- 172.21.15.29:0/3384228024 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f03b40683b0 0x7f03b406a8c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f03c00ae000 tx=0x7f03c00046d0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.285+0000 7f03db600640 1 -- 172.21.15.29:0/3384228024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc004150 msgr2=0x7f03d41e53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.285+0000 7f03db600640 1 --2- 172.21.15.29:0/3384228024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc004150 0x7f03d41e53e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f03c8012290 tx=0x7f03c8012750 comp rx=0 tx=0).stop 2024-04-17T19:47:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.285+0000 7f03db600640 1 -- 172.21.15.29:0/3384228024 shutdown_connections 2024-04-17T19:47:47.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.285+0000 7f03db600640 1 --2- 172.21.15.29:0/3384228024 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f03b40683b0 0x7f03b406a8c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.285+0000 7f03db600640 1 --2- 172.21.15.29:0/3384228024 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f03d40cd1e0 0x7f03d4148f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.285+0000 7f03db600640 1 --2- 172.21.15.29:0/3384228024 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f03d41f33b0 0x7f03d41e7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.285+0000 7f03db600640 1 --2- 172.21.15.29:0/3384228024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03bc004150 0x7f03d41e53e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.285+0000 7f03db600640 1 -- 172.21.15.29:0/3384228024 >> 172.21.15.29:0/3384228024 conn(0x7f03d4071ca0 msgr2=0x7f03d4153460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.285+0000 7f03db600640 1 -- 172.21.15.29:0/3384228024 shutdown_connections 2024-04-17T19:47:47.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.285+0000 7f03db600640 1 -- 172.21.15.29:0/3384228024 wait complete. 2024-04-17T19:47:47.296 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T19:47:47.296 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T19:47:47.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.461+0000 7f4133e00640 1 Processor -- start 2024-04-17T19:47:47.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.461+0000 7f4133e00640 1 -- start start 2024-04-17T19:47:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.465+0000 7f4133e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f412c0cb870 0x7f412c0ca400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.465+0000 7f4133e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f412c0b5100 0x7f412c0cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.465+0000 7f4133e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f412c0cc340 0x7f412c0c52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.465+0000 7f4133e00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f412c1605e0 con 0x7f412c0cc340 2024-04-17T19:47:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.465+0000 7f4133e00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f412c15c310 con 0x7f412c0b5100 2024-04-17T19:47:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.465+0000 7f4133e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f412c0b5510 con 0x7f412c0cb870 2024-04-17T19:47:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.465+0000 7f4131e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f412c0cc340 0x7f412c0c52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.465+0000 7f4131e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f412c0cc340 0x7f412c0c52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45654/0 (socket says 172.21.15.29:45654) 2024-04-17T19:47:47.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.465+0000 7f4131e00640 1 -- 172.21.15.29:0/3108623269 learned_addr learned my addr 172.21.15.29:0/3108623269 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.465+0000 7f4130a00640 1 --2- 172.21.15.29:0/3108623269 >> v2:172.21.15.37:3300/0 conn(0x7f412c0b5100 0x7f412c0cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.465+0000 7f4131400640 1 --2- 172.21.15.29:0/3108623269 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f412c0cb870 0x7f412c0ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.465+0000 7f4131e00640 1 -- 172.21.15.29:0/3108623269 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f412c0cb870 msgr2=0x7f412c0ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.465+0000 7f4131e00640 1 --2- 172.21.15.29:0/3108623269 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f412c0cb870 0x7f412c0ca400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.465+0000 7f4131e00640 1 -- 172.21.15.29:0/3108623269 >> v2:172.21.15.37:3300/0 conn(0x7f412c0b5100 msgr2=0x7f412c0cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.465+0000 7f4131e00640 1 --2- 172.21.15.29:0/3108623269 >> v2:172.21.15.37:3300/0 conn(0x7f412c0b5100 0x7f412c0cbe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.465+0000 7f4131e00640 1 -- 172.21.15.29:0/3108623269 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f412c15c5b0 con 0x7f412c0cc340 2024-04-17T19:47:47.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.465+0000 7f4130a00640 1 --2- 172.21.15.29:0/3108623269 >> v2:172.21.15.37:3300/0 conn(0x7f412c0b5100 0x7f412c0cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:47.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.469+0000 7f4131e00640 1 --2- 172.21.15.29:0/3108623269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f412c0cc340 0x7f412c0c52f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f412c0729a0 tx=0x7f412401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acba2ced32d07ea server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.469+0000 7f412be00640 1 -- 172.21.15.29:0/3108623269 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41240a5670 con 0x7f412c0cc340 2024-04-17T19:47:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.469+0000 7f412be00640 1 -- 172.21.15.29:0/3108623269 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41240a5810 con 0x7f412c0cc340 2024-04-17T19:47:47.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.469+0000 7f412be00640 1 -- 172.21.15.29:0/3108623269 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41240a5be0 con 0x7f412c0cc340 2024-04-17T19:47:47.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.469+0000 7f4133e00640 1 -- 172.21.15.29:0/3108623269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f412c0cc340 msgr2=0x7f412c0c52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:47.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.469+0000 7f4133e00640 1 --2- 172.21.15.29:0/3108623269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f412c0cc340 0x7f412c0c52f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f412c0729a0 tx=0x7f412401c9f0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.469+0000 7f4133e00640 1 -- 172.21.15.29:0/3108623269 shutdown_connections 2024-04-17T19:47:47.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.469+0000 7f4133e00640 1 --2- 172.21.15.29:0/3108623269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f412c0cc340 0x7f412c0c52f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.469+0000 7f4133e00640 1 --2- 172.21.15.29:0/3108623269 >> v2:172.21.15.37:3300/0 conn(0x7f412c0b5100 0x7f412c0cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.469+0000 7f4133e00640 1 --2- 172.21.15.29:0/3108623269 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f412c0cb870 0x7f412c0ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.469+0000 7f4133e00640 1 -- 172.21.15.29:0/3108623269 >> 172.21.15.29:0/3108623269 conn(0x7f412c071a00 msgr2=0x7f412c071e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:47.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.469+0000 7f4133e00640 1 -- 172.21.15.29:0/3108623269 shutdown_connections 2024-04-17T19:47:47.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.469+0000 7f4133e00640 1 -- 172.21.15.29:0/3108623269 wait complete. 2024-04-17T19:47:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.469+0000 7f4133e00640 1 Processor -- start 2024-04-17T19:47:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4133e00640 1 -- start start 2024-04-17T19:47:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4133e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f412c0b5100 0x7f412c15b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4133e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f412c0cb870 0x7f412c148230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4133e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f412c0cc340 0x7f412c145cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4133e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f412c0c5c40 con 0x7f412c0cb870 2024-04-17T19:47:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4133e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f412c156960 con 0x7f412c0b5100 2024-04-17T19:47:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4133e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f412c0707e0 con 0x7f412c0cc340 2024-04-17T19:47:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4130a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f412c0cb870 0x7f412c148230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4130a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f412c0cb870 0x7f412c148230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45668/0 (socket says 172.21.15.29:45668) 2024-04-17T19:47:47.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4130a00640 1 -- 172.21.15.29:0/3052954274 learned_addr learned my addr 172.21.15.29:0/3052954274 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4131400640 1 --2- 172.21.15.29:0/3052954274 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f412c0b5100 0x7f412c15b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4131e00640 1 --2- 172.21.15.29:0/3052954274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f412c0cc340 0x7f412c145cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4130a00640 1 -- 172.21.15.29:0/3052954274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f412c0cc340 msgr2=0x7f412c145cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4130a00640 1 --2- 172.21.15.29:0/3052954274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f412c0cc340 0x7f412c145cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4130a00640 1 -- 172.21.15.29:0/3052954274 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f412c0b5100 msgr2=0x7f412c15b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4130a00640 1 --2- 172.21.15.29:0/3052954274 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f412c0b5100 0x7f412c15b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4130a00640 1 -- 172.21.15.29:0/3052954274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f412c072290 con 0x7f412c0cb870 2024-04-17T19:47:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4130a00640 1 --2- 172.21.15.29:0/3052954274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f412c0cb870 0x7f412c148230 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f411800d800 tx=0x7f411800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4129600640 1 -- 172.21.15.29:0/3052954274 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4118018070 con 0x7f412c0cb870 2024-04-17T19:47:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4129600640 1 -- 172.21.15.29:0/3052954274 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4118004030 con 0x7f412c0cb870 2024-04-17T19:47:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4129600640 1 -- 172.21.15.29:0/3052954274 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4118004400 con 0x7f412c0cb870 2024-04-17T19:47:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4133e00640 1 -- 172.21.15.29:0/3052954274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f412c14ce70 con 0x7f412c0cb870 2024-04-17T19:47:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4133e00640 1 -- 172.21.15.29:0/3052954274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f412c15b960 con 0x7f412c0cb870 2024-04-17T19:47:47.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.473+0000 7f4133e00640 1 -- 172.21.15.29:0/3052954274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f412c069b90 con 0x7f412c0cb870 2024-04-17T19:47:47.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.477+0000 7f4129600640 1 -- 172.21.15.29:0/3052954274 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f41180096a0 con 0x7f412c0cb870 2024-04-17T19:47:47.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.477+0000 7f4129600640 1 --2- 172.21.15.29:0/3052954274 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f410c068380 0x7f410c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:47.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.477+0000 7f4129600640 1 -- 172.21.15.29:0/3052954274 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f41180868c0 con 0x7f412c0cb870 2024-04-17T19:47:47.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.477+0000 7f4131400640 1 --2- 172.21.15.29:0/3052954274 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f410c068380 0x7f410c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:47.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.477+0000 7f4131400640 1 --2- 172.21.15.29:0/3052954274 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f410c068380 0x7f410c06a890 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f41200079c0 tx=0x7f4120008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:47.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.481+0000 7f4129600640 1 -- 172.21.15.29:0/3052954274 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4118056b70 con 0x7f412c0cb870 2024-04-17T19:47:47.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.645+0000 7f4133e00640 1 -- 172.21.15.29:0/3052954274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f412c0d0260 con 0x7f412c0cb870 2024-04-17T19:47:47.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.821+0000 7f4129600640 1 -- 172.21.15.29:0/3052954274 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7f411805ae70 con 0x7f412c0cb870 2024-04-17T19:47:47.811 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2024-04-17T19:47:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.825+0000 7f4133e00640 1 -- 172.21.15.29:0/3052954274 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f410c068380 msgr2=0x7f410c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.825+0000 7f4133e00640 1 --2- 172.21.15.29:0/3052954274 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f410c068380 0x7f410c06a890 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f41200079c0 tx=0x7f4120008040 comp rx=0 tx=0).stop 2024-04-17T19:47:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.825+0000 7f4133e00640 1 -- 172.21.15.29:0/3052954274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f412c0cb870 msgr2=0x7f412c148230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.825+0000 7f4133e00640 1 --2- 172.21.15.29:0/3052954274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f412c0cb870 0x7f412c148230 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f411800d800 tx=0x7f411800dcc0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.825+0000 7f4133e00640 1 -- 172.21.15.29:0/3052954274 shutdown_connections 2024-04-17T19:47:47.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.825+0000 7f4133e00640 1 --2- 172.21.15.29:0/3052954274 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f410c068380 0x7f410c06a890 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.825+0000 7f4133e00640 1 --2- 172.21.15.29:0/3052954274 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f412c0cc340 0x7f412c145cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.825+0000 7f4133e00640 1 --2- 172.21.15.29:0/3052954274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f412c0cb870 0x7f412c148230 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.825+0000 7f4133e00640 1 --2- 172.21.15.29:0/3052954274 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f412c0b5100 0x7f412c15b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:47.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.825+0000 7f4133e00640 1 -- 172.21.15.29:0/3052954274 >> 172.21.15.29:0/3052954274 conn(0x7f412c071a00 msgr2=0x7f412c153580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:47.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.825+0000 7f4133e00640 1 -- 172.21.15.29:0/3052954274 shutdown_connections 2024-04-17T19:47:47.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:47.825+0000 7f4133e00640 1 -- 172.21.15.29:0/3052954274 wait complete. 2024-04-17T19:47:47.842 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-17T19:47:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.013+0000 7f5599c00640 1 Processor -- start 2024-04-17T19:47:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.013+0000 7f5599c00640 1 -- start start 2024-04-17T19:47:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.013+0000 7f5599c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55940c55e0 0x7f55940706e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.013+0000 7f5599c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5594164a20 0x7f5594166e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.013+0000 7f5599c00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55940b5520 con 0x7f55940c51d0 2024-04-17T19:47:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.013+0000 7f5599c00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f55940cb870 con 0x7f5594164a20 2024-04-17T19:47:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.013+0000 7f5599c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f55940ca4b0 con 0x7f55940c55e0 2024-04-17T19:47:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.013+0000 7f5593400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5594164a20 0x7f5594166e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.013+0000 7f5593400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5594164a20 0x7f5594166e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45670/0 (socket says 172.21.15.29:45670) 2024-04-17T19:47:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.013+0000 7f5593400640 1 -- 172.21.15.29:0/2293161465 learned_addr learned my addr 172.21.15.29:0/2293161465 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.013+0000 7f558be00640 1 --2- 172.21.15.29:0/2293161465 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55940c55e0 0x7f55940706e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.013+0000 7f5593400640 1 -- 172.21.15.29:0/2293161465 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55940c55e0 msgr2=0x7f55940706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.013+0000 7f5593400640 1 --2- 172.21.15.29:0/2293161465 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55940c55e0 0x7f55940706e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.013+0000 7f5593400640 1 -- 172.21.15.29:0/2293161465 >> v1:172.21.15.37:6789/0 conn(0x7f55940c51d0 legacy=0x7f559406ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:48.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.013+0000 7f5593400640 1 -- 172.21.15.29:0/2293161465 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f559415cfa0 con 0x7f5594164a20 2024-04-17T19:47:48.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.017+0000 7f5593400640 1 --2- 172.21.15.29:0/2293161465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5594164a20 0x7f5594166e40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f558c009e50 tx=0x7f558c01f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e704bee8352c861 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:48.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.017+0000 7f5592000640 1 -- 172.21.15.29:0/2293161465 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f558c0b4070 con 0x7f5594164a20 2024-04-17T19:47:48.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.017+0000 7f5592000640 1 -- 172.21.15.29:0/2293161465 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f558c0af400 con 0x7f5594164a20 2024-04-17T19:47:48.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.017+0000 7f5592000640 1 -- 172.21.15.29:0/2293161465 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f558c0af7d0 con 0x7f5594164a20 2024-04-17T19:47:48.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.017+0000 7f5599c00640 1 -- 172.21.15.29:0/2293161465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5594164a20 msgr2=0x7f5594166e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:48.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.017+0000 7f5599c00640 1 --2- 172.21.15.29:0/2293161465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5594164a20 0x7f5594166e40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f558c009e50 tx=0x7f558c01f210 comp rx=0 tx=0).stop 2024-04-17T19:47:48.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.017+0000 7f5599c00640 1 -- 172.21.15.29:0/2293161465 shutdown_connections 2024-04-17T19:47:48.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.017+0000 7f5599c00640 1 --2- 172.21.15.29:0/2293161465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5594164a20 0x7f5594166e40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:48.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.017+0000 7f5599c00640 1 --2- 172.21.15.29:0/2293161465 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55940c55e0 0x7f55940706e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:48.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.017+0000 7f5599c00640 1 -- 172.21.15.29:0/2293161465 >> 172.21.15.29:0/2293161465 conn(0x7f5594071a00 msgr2=0x7f5594071e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:48.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.017+0000 7f5599c00640 1 -- 172.21.15.29:0/2293161465 shutdown_connections 2024-04-17T19:47:48.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.017+0000 7f5599c00640 1 -- 172.21.15.29:0/2293161465 wait complete. 2024-04-17T19:47:48.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f5599c00640 1 Processor -- start 2024-04-17T19:47:48.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f5599c00640 1 -- start start 2024-04-17T19:47:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f5599c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55940c51d0 0x7f55941f2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f5599c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55940c55e0 0x7f5594144130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f5599c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5594164a20 0x7f5594159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f5599c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f55940cc080 con 0x7f55940c51d0 2024-04-17T19:47:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f5599c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f55940cbda0 con 0x7f5594164a20 2024-04-17T19:47:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f5599c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f55940cc360 con 0x7f55940c55e0 2024-04-17T19:47:48.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f5592a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55940c51d0 0x7f55941f2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f5592a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55940c51d0 0x7f55941f2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45676/0 (socket says 172.21.15.29:45676) 2024-04-17T19:47:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f5592a00640 1 -- 172.21.15.29:0/2352243567 learned_addr learned my addr 172.21.15.29:0/2352243567 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f558be00640 1 --2- 172.21.15.29:0/2352243567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55940c55e0 0x7f5594144130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f5593400640 1 --2- 172.21.15.29:0/2352243567 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5594164a20 0x7f5594159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f5592a00640 1 -- 172.21.15.29:0/2352243567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55940c55e0 msgr2=0x7f5594144130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f5592a00640 1 --2- 172.21.15.29:0/2352243567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55940c55e0 0x7f5594144130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f5592a00640 1 -- 172.21.15.29:0/2352243567 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5594164a20 msgr2=0x7f5594159fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f5592a00640 1 --2- 172.21.15.29:0/2352243567 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5594164a20 0x7f5594159fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f5592a00640 1 -- 172.21.15.29:0/2352243567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f559414bbe0 con 0x7f55940c51d0 2024-04-17T19:47:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.021+0000 7f5592a00640 1 --2- 172.21.15.29:0/2352243567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55940c51d0 0x7f55941f2a50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f558400c3a0 tx=0x7f558400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.025+0000 7f558aa00640 1 -- 172.21.15.29:0/2352243567 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55840052d0 con 0x7f55940c51d0 2024-04-17T19:47:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.025+0000 7f558aa00640 1 -- 172.21.15.29:0/2352243567 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5584005470 con 0x7f55940c51d0 2024-04-17T19:47:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.025+0000 7f5599c00640 1 -- 172.21.15.29:0/2352243567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55941f2f90 con 0x7f55940c51d0 2024-04-17T19:47:48.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.025+0000 7f558aa00640 1 -- 172.21.15.29:0/2352243567 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5584007260 con 0x7f55940c51d0 2024-04-17T19:47:48.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.025+0000 7f5599c00640 1 -- 172.21.15.29:0/2352243567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f559414b870 con 0x7f55940c51d0 2024-04-17T19:47:48.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.025+0000 7f5599c00640 1 -- 172.21.15.29:0/2352243567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5594069b90 con 0x7f55940c51d0 2024-04-17T19:47:48.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.025+0000 7f558aa00640 1 -- 172.21.15.29:0/2352243567 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5584007400 con 0x7f55940c51d0 2024-04-17T19:47:48.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.025+0000 7f558aa00640 1 --2- 172.21.15.29:0/2352243567 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f55700683b0 0x7f557006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:48.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.025+0000 7f558aa00640 1 -- 172.21.15.29:0/2352243567 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7f55840861b0 con 0x7f55940c51d0 2024-04-17T19:47:48.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.025+0000 7f558be00640 1 --2- 172.21.15.29:0/2352243567 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f55700683b0 0x7f557006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:48.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.029+0000 7f558be00640 1 --2- 172.21.15.29:0/2352243567 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f55700683b0 0x7f557006a8c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f55941f0330 tx=0x7f557c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:48.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.029+0000 7f558aa00640 1 -- 172.21.15.29:0/2352243567 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f559414bbe0 con 0x7f55940c51d0 2024-04-17T19:47:48.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:47:48.134 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:47:48.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.197+0000 7f5599c00640 1 -- 172.21.15.29:0/2352243567 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f55940cb440 con 0x7f55940c51d0 2024-04-17T19:47:48.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.829+0000 7f558aa00640 1 -- 172.21.15.29:0/2352243567 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) ==== 171+0+0 (secure 0 0 0) 0x7f5584056930 con 0x7f55940c51d0 2024-04-17T19:47:48.820 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2024-04-17T19:47:48.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.833+0000 7f5599c00640 1 -- 172.21.15.29:0/2352243567 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f55700683b0 msgr2=0x7f557006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:48.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.833+0000 7f5599c00640 1 --2- 172.21.15.29:0/2352243567 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f55700683b0 0x7f557006a8c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f55941f0330 tx=0x7f557c008040 comp rx=0 tx=0).stop 2024-04-17T19:47:48.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.833+0000 7f5599c00640 1 -- 172.21.15.29:0/2352243567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55940c51d0 msgr2=0x7f55941f2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:48.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.833+0000 7f5599c00640 1 --2- 172.21.15.29:0/2352243567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55940c51d0 0x7f55941f2a50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f558400c3a0 tx=0x7f558400c860 comp rx=0 tx=0).stop 2024-04-17T19:47:48.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.833+0000 7f5599c00640 1 -- 172.21.15.29:0/2352243567 shutdown_connections 2024-04-17T19:47:48.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.833+0000 7f5599c00640 1 --2- 172.21.15.29:0/2352243567 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f55700683b0 0x7f557006a8c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:48.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.833+0000 7f5599c00640 1 --2- 172.21.15.29:0/2352243567 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5594164a20 0x7f5594159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:48.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.833+0000 7f5599c00640 1 --2- 172.21.15.29:0/2352243567 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f55940c55e0 0x7f5594144130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:48.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.833+0000 7f5599c00640 1 --2- 172.21.15.29:0/2352243567 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55940c51d0 0x7f55941f2a50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:48.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.833+0000 7f5599c00640 1 -- 172.21.15.29:0/2352243567 >> 172.21.15.29:0/2352243567 conn(0x7f5594071a00 msgr2=0x7f5594156be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:48.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.837+0000 7f5599c00640 1 -- 172.21.15.29:0/2352243567 shutdown_connections 2024-04-17T19:47:48.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:48.837+0000 7f5599c00640 1 -- 172.21.15.29:0/2352243567 wait complete. 2024-04-17T19:47:48.871 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T19:47:49.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.089+0000 7f9050800640 1 Processor -- start 2024-04-17T19:47:49.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.089+0000 7f9050800640 1 -- start start 2024-04-17T19:47:49.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f9050800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f90480ca1a0 0x7f90480ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:49.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f9050800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90480ccaa0 con 0x7f904815c330 2024-04-17T19:47:49.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f9050800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f904806c9c0 con 0x7f90480cbf80 2024-04-17T19:47:49.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f9050800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f904806cca0 con 0x7f90480ca1a0 2024-04-17T19:47:49.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904d400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f904815c330 0x7f904815c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33782/0 (socket says 172.21.15.29:33782) 2024-04-17T19:47:49.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904d400640 1 -- 172.21.15.29:0/527322507 learned_addr learned my addr 172.21.15.29:0/527322507 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:47:49.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904de00640 1 --2- 172.21.15.29:0/527322507 >> v2:172.21.15.37:3300/0 conn(0x7f90480ca1a0 0x7f90480ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:49.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904ca00640 1 -- 172.21.15.29:0/527322507 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3889273587 0 0) 0x7f90480ccaa0 con 0x7f904815c330 2024-04-17T19:47:49.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904ca00640 1 -- 172.21.15.29:0/527322507 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9030003530 con 0x7f904815c330 2024-04-17T19:47:49.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904ca00640 1 -- 172.21.15.29:0/527322507 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4247006430 0 0) 0x7f9030003530 con 0x7f904815c330 2024-04-17T19:47:49.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904ca00640 1 -- 172.21.15.29:0/527322507 >> v2:172.21.15.37:3300/0 conn(0x7f90480ca1a0 msgr2=0x7f90480ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:49.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904ca00640 1 --2- 172.21.15.29:0/527322507 >> v2:172.21.15.37:3300/0 conn(0x7f90480ca1a0 0x7f90480ca5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:49.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904ca00640 1 -- 172.21.15.29:0/527322507 >> v1:172.21.15.37:6789/0 conn(0x7f90480cbf80 legacy=0x7f90480cc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:49.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904ca00640 1 -- 172.21.15.29:0/527322507 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f904806cf80 con 0x7f904815c330 2024-04-17T19:47:49.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904ca00640 1 -- 172.21.15.29:0/527322507 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f9038001680 con 0x7f904815c330 2024-04-17T19:47:49.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904ca00640 1 -- 172.21.15.29:0/527322507 >> v1:172.21.15.29:6789/0 conn(0x7f904815c330 legacy=0x7f904815c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:49.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904ca00640 1 --2- 172.21.15.29:0/527322507 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90300039c0 0x7f9030003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:49.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904ca00640 1 --2- 172.21.15.29:0/527322507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9030004810 0x7f9030004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:49.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904ca00640 1 --2- 172.21.15.29:0/527322507 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f90300051d0 0x7f903000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:49.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904ca00640 1 -- 172.21.15.29:0/527322507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f904806cca0 con 0x7f9030004810 2024-04-17T19:47:49.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904ca00640 1 -- 172.21.15.29:0/527322507 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f903000c160 con 0x7f90300051d0 2024-04-17T19:47:49.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904ca00640 1 -- 172.21.15.29:0/527322507 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f903000c360 con 0x7f90300039c0 2024-04-17T19:47:49.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904d400640 1 --2- 172.21.15.29:0/527322507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9030004810 0x7f9030004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:49.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904e800640 1 --2- 172.21.15.29:0/527322507 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f90300051d0 0x7f903000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:49.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904de00640 1 --2- 172.21.15.29:0/527322507 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90300039c0 0x7f9030003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:49.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904d400640 1 -- 172.21.15.29:0/527322507 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90300039c0 msgr2=0x7f9030003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:49.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904d400640 1 --2- 172.21.15.29:0/527322507 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90300039c0 0x7f9030003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:49.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904d400640 1 -- 172.21.15.29:0/527322507 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f90300051d0 msgr2=0x7f903000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:49.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904d400640 1 --2- 172.21.15.29:0/527322507 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f90300051d0 0x7f903000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:49.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904d400640 1 -- 172.21.15.29:0/527322507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9030003530 con 0x7f9030004810 2024-04-17T19:47:49.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904e800640 1 --2- 172.21.15.29:0/527322507 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f90300051d0 0x7f903000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:49.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.093+0000 7f904e800640 1 -- 172.21.15.29:0/527322507 reap_dead start 2024-04-17T19:47:49.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.097+0000 7f904d400640 1 --2- 172.21.15.29:0/527322507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9030004810 0x7f9030004c20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f903800c240 tx=0x7f903801cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2226c69536d44c66 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:49.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.097+0000 7f904ca00640 1 -- 172.21.15.29:0/527322507 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9038007d20 con 0x7f9030004810 2024-04-17T19:47:49.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.097+0000 7f904ca00640 1 -- 172.21.15.29:0/527322507 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90380a9070 con 0x7f9030004810 2024-04-17T19:47:49.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.097+0000 7f904ca00640 1 -- 172.21.15.29:0/527322507 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90380a5e20 con 0x7f9030004810 2024-04-17T19:47:49.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.097+0000 7f9050800640 1 -- 172.21.15.29:0/527322507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9030004810 msgr2=0x7f9030004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:49.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.097+0000 7f9050800640 1 --2- 172.21.15.29:0/527322507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9030004810 0x7f9030004c20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f903800c240 tx=0x7f903801cc80 comp rx=0 tx=0).stop 2024-04-17T19:47:49.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.097+0000 7f9050800640 1 -- 172.21.15.29:0/527322507 shutdown_connections 2024-04-17T19:47:49.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.097+0000 7f9050800640 1 --2- 172.21.15.29:0/527322507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9030004810 0x7f9030004c20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:49.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.097+0000 7f9050800640 1 -- 172.21.15.29:0/527322507 >> 172.21.15.29:0/527322507 conn(0x7f9048071a60 msgr2=0x7f9048071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:49.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.097+0000 7f9050800640 1 -- 172.21.15.29:0/527322507 shutdown_connections 2024-04-17T19:47:49.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.097+0000 7f9050800640 1 -- 172.21.15.29:0/527322507 wait complete. 2024-04-17T19:47:49.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f9050800640 1 Processor -- start 2024-04-17T19:47:49.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f9050800640 1 -- start start 2024-04-17T19:47:49.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f9050800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9030004810 0x7f90481590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:49.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f9050800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90481589b0 0x7f904814cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:49.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f9050800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9048159bc0 0x7f904814d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:49.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f9050800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f904806bf80 con 0x7f9030004810 2024-04-17T19:47:49.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f9050800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f904806d5d0 con 0x7f9048159bc0 2024-04-17T19:47:49.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f9050800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f904806c100 con 0x7f90481589b0 2024-04-17T19:47:49.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f904de00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9030004810 0x7f90481590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:49.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f904de00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9030004810 0x7f90481590d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51640/0 (socket says 172.21.15.29:51640) 2024-04-17T19:47:49.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f904de00640 1 -- 172.21.15.29:0/178339591 learned_addr learned my addr 172.21.15.29:0/178339591 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:49.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f904e800640 1 --2- 172.21.15.29:0/178339591 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9048159bc0 0x7f904814d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:49.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f904d400640 1 --2- 172.21.15.29:0/178339591 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90481589b0 0x7f904814cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:49.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f904de00640 1 -- 172.21.15.29:0/178339591 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90481589b0 msgr2=0x7f904814cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:49.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f904de00640 1 --2- 172.21.15.29:0/178339591 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90481589b0 0x7f904814cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:49.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f904de00640 1 -- 172.21.15.29:0/178339591 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9048159bc0 msgr2=0x7f904814d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:49.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f904de00640 1 --2- 172.21.15.29:0/178339591 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9048159bc0 0x7f904814d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:49.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f904de00640 1 -- 172.21.15.29:0/178339591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90481f3d90 con 0x7f9030004810 2024-04-17T19:47:49.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f904de00640 1 --2- 172.21.15.29:0/178339591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9030004810 0x7f90481590d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f903c011500 tx=0x7f903c0119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:49.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f9042000640 1 -- 172.21.15.29:0/178339591 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c015870 con 0x7f9030004810 2024-04-17T19:47:49.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f9042000640 1 -- 172.21.15.29:0/178339591 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f903c015a10 con 0x7f9030004810 2024-04-17T19:47:49.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f9042000640 1 -- 172.21.15.29:0/178339591 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f903c015dc0 con 0x7f9030004810 2024-04-17T19:47:49.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f9050800640 1 -- 172.21.15.29:0/178339591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90481f2fd0 con 0x7f9030004810 2024-04-17T19:47:49.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.101+0000 7f9050800640 1 -- 172.21.15.29:0/178339591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90481f3590 con 0x7f9030004810 2024-04-17T19:47:49.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.105+0000 7f9050800640 1 -- 172.21.15.29:0/178339591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9048069b90 con 0x7f9030004810 2024-04-17T19:47:49.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.105+0000 7f9042000640 1 -- 172.21.15.29:0/178339591 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f903c013400 con 0x7f9030004810 2024-04-17T19:47:49.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.105+0000 7f9042000640 1 --2- 172.21.15.29:0/178339591 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f902c068380 0x7f902c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:49.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.105+0000 7f9042000640 1 -- 172.21.15.29:0/178339591 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7f903c087390 con 0x7f9030004810 2024-04-17T19:47:49.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.105+0000 7f904d400640 1 --2- 172.21.15.29:0/178339591 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f902c068380 0x7f902c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:49.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.109+0000 7f904d400640 1 --2- 172.21.15.29:0/178339591 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f902c068380 0x7f902c06a890 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f90380ab000 tx=0x7f90380b0040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:49.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.109+0000 7f9042000640 1 -- 172.21.15.29:0/178339591 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f903c019030 con 0x7f9030004810 2024-04-17T19:47:49.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.313+0000 7f9050800640 1 -- 172.21.15.29:0/178339591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f90480c5ea0 con 0x7f9030004810 2024-04-17T19:47:49.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.833+0000 7f9042000640 1 -- 172.21.15.29:0/178339591 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v5) ==== 172+0+0 (secure 0 0 0) 0x7f903c057890 con 0x7f9030004810 2024-04-17T19:47:49.822 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 2 and data pool 3 2024-04-17T19:47:49.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.837+0000 7f9050800640 1 -- 172.21.15.29:0/178339591 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f902c068380 msgr2=0x7f902c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:49.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.837+0000 7f9050800640 1 --2- 172.21.15.29:0/178339591 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f902c068380 0x7f902c06a890 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f90380ab000 tx=0x7f90380b0040 comp rx=0 tx=0).stop 2024-04-17T19:47:49.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.837+0000 7f9050800640 1 -- 172.21.15.29:0/178339591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9030004810 msgr2=0x7f90481590d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:49.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.837+0000 7f9050800640 1 --2- 172.21.15.29:0/178339591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9030004810 0x7f90481590d0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f903c011500 tx=0x7f903c0119c0 comp rx=0 tx=0).stop 2024-04-17T19:47:49.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.837+0000 7f9050800640 1 -- 172.21.15.29:0/178339591 shutdown_connections 2024-04-17T19:47:49.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.837+0000 7f9050800640 1 --2- 172.21.15.29:0/178339591 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f902c068380 0x7f902c06a890 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:49.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.837+0000 7f9050800640 1 --2- 172.21.15.29:0/178339591 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9048159bc0 0x7f904814d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:49.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.837+0000 7f9050800640 1 --2- 172.21.15.29:0/178339591 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f90481589b0 0x7f904814cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:49.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.837+0000 7f9050800640 1 --2- 172.21.15.29:0/178339591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9030004810 0x7f90481590d0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:49.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.837+0000 7f9050800640 1 -- 172.21.15.29:0/178339591 >> 172.21.15.29:0/178339591 conn(0x7f9048071a60 msgr2=0x7f90480cdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:49.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.837+0000 7f9050800640 1 -- 172.21.15.29:0/178339591 shutdown_connections 2024-04-17T19:47:49.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:49.837+0000 7f9050800640 1 -- 172.21.15.29:0/178339591 wait complete. 2024-04-17T19:47:49.876 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-17T19:47:49.876 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-17T19:47:50.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.049+0000 7fd695400640 1 Processor -- start 2024-04-17T19:47:50.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.049+0000 7fd695400640 1 -- start start 2024-04-17T19:47:50.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.049+0000 7fd695400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd690154220 0x7fd6900cba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:50.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.049+0000 7fd695400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd6900cbf70 0x7fd6900cc380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:50.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.049+0000 7fd695400640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6900caf00 con 0x7fd690153e10 2024-04-17T19:47:50.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.049+0000 7fd695400640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fd6900ca8c0 con 0x7fd6900cbf70 2024-04-17T19:47:50.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.049+0000 7fd695400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd6900ca5e0 con 0x7fd690154220 2024-04-17T19:47:50.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.049+0000 7fd68f200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd6900cbf70 0x7fd6900cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:50.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.049+0000 7fd68f200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd6900cbf70 0x7fd6900cc380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51646/0 (socket says 172.21.15.29:51646) 2024-04-17T19:47:50.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.049+0000 7fd68f200640 1 -- 172.21.15.29:0/3172165863 learned_addr learned my addr 172.21.15.29:0/3172165863 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:50.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.049+0000 7fd687e00640 1 --2- 172.21.15.29:0/3172165863 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd690154220 0x7fd6900cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:50.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.049+0000 7fd68f200640 1 -- 172.21.15.29:0/3172165863 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd690154220 msgr2=0x7fd6900cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:50.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.049+0000 7fd68f200640 1 --2- 172.21.15.29:0/3172165863 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd690154220 0x7fd6900cba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:50.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.049+0000 7fd68f200640 1 -- 172.21.15.29:0/3172165863 >> v1:172.21.15.37:6789/0 conn(0x7fd690153e10 legacy=0x7fd6900c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:50.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.049+0000 7fd68f200640 1 -- 172.21.15.29:0/3172165863 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd69015d430 con 0x7fd6900cbf70 2024-04-17T19:47:50.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.053+0000 7fd68f200640 1 --2- 172.21.15.29:0/3172165863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6900cbf70 0x7fd6900cc380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd688009e50 tx=0x7fd68801ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e3d1d7bc3ced11 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:50.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.053+0000 7fd68de00640 1 -- 172.21.15.29:0/3172165863 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6880a58d0 con 0x7fd6900cbf70 2024-04-17T19:47:50.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.053+0000 7fd68de00640 1 -- 172.21.15.29:0/3172165863 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6880a5a70 con 0x7fd6900cbf70 2024-04-17T19:47:50.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.053+0000 7fd68de00640 1 -- 172.21.15.29:0/3172165863 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6880a5e40 con 0x7fd6900cbf70 2024-04-17T19:47:50.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.053+0000 7fd695400640 1 -- 172.21.15.29:0/3172165863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6900cbf70 msgr2=0x7fd6900cc380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:50.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.053+0000 7fd695400640 1 --2- 172.21.15.29:0/3172165863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6900cbf70 0x7fd6900cc380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd688009e50 tx=0x7fd68801ca00 comp rx=0 tx=0).stop 2024-04-17T19:47:50.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.053+0000 7fd695400640 1 -- 172.21.15.29:0/3172165863 shutdown_connections 2024-04-17T19:47:50.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.053+0000 7fd695400640 1 --2- 172.21.15.29:0/3172165863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6900cbf70 0x7fd6900cc380 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:50.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.053+0000 7fd695400640 1 --2- 172.21.15.29:0/3172165863 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd690154220 0x7fd6900cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:50.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.053+0000 7fd695400640 1 -- 172.21.15.29:0/3172165863 >> 172.21.15.29:0/3172165863 conn(0x7fd690071a70 msgr2=0x7fd690071e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:50.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.053+0000 7fd695400640 1 -- 172.21.15.29:0/3172165863 shutdown_connections 2024-04-17T19:47:50.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.053+0000 7fd695400640 1 -- 172.21.15.29:0/3172165863 wait complete. 2024-04-17T19:47:50.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd695400640 1 Processor -- start 2024-04-17T19:47:50.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd695400640 1 -- start start 2024-04-17T19:47:50.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd695400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6900cbf70 0x7fd6900b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:50.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd695400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd690153e10 0x7fd690144de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:50.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd695400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd690154220 0x7fd690145320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:50.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd695400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd6900c4b10 con 0x7fd6900cbf70 2024-04-17T19:47:50.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd695400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd69015e1f0 con 0x7fd690153e10 2024-04-17T19:47:50.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd695400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd6900ca2c0 con 0x7fd690154220 2024-04-17T19:47:50.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd68e800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6900cbf70 0x7fd6900b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd68e800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6900cbf70 0x7fd6900b45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51654/0 (socket says 172.21.15.29:51654) 2024-04-17T19:47:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd68e800640 1 -- 172.21.15.29:0/4130379971 learned_addr learned my addr 172.21.15.29:0/4130379971 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd687e00640 1 --2- 172.21.15.29:0/4130379971 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd690153e10 0x7fd690144de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd68f200640 1 --2- 172.21.15.29:0/4130379971 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd690154220 0x7fd690145320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd68e800640 1 -- 172.21.15.29:0/4130379971 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd690154220 msgr2=0x7fd690145320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd68e800640 1 --2- 172.21.15.29:0/4130379971 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd690154220 0x7fd690145320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd68e800640 1 -- 172.21.15.29:0/4130379971 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd690153e10 msgr2=0x7fd690144de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd68e800640 1 --2- 172.21.15.29:0/4130379971 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd690153e10 0x7fd690144de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd68e800640 1 -- 172.21.15.29:0/4130379971 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd690152710 con 0x7fd6900cbf70 2024-04-17T19:47:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.057+0000 7fd68e800640 1 --2- 172.21.15.29:0/4130379971 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6900cbf70 0x7fd6900b45c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd68000c3a0 tx=0x7fd68000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.061+0000 7fd686a00640 1 -- 172.21.15.29:0/4130379971 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6800052d0 con 0x7fd6900cbf70 2024-04-17T19:47:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.061+0000 7fd686a00640 1 -- 172.21.15.29:0/4130379971 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd680005470 con 0x7fd6900cbf70 2024-04-17T19:47:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.061+0000 7fd695400640 1 -- 172.21.15.29:0/4130379971 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6901529b0 con 0x7fd6900cbf70 2024-04-17T19:47:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.061+0000 7fd686a00640 1 -- 172.21.15.29:0/4130379971 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd680007260 con 0x7fd6900cbf70 2024-04-17T19:47:50.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.061+0000 7fd695400640 1 -- 172.21.15.29:0/4130379971 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd690152c50 con 0x7fd6900cbf70 2024-04-17T19:47:50.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.061+0000 7fd695400640 1 -- 172.21.15.29:0/4130379971 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd690069b90 con 0x7fd6900cbf70 2024-04-17T19:47:50.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.061+0000 7fd686a00640 1 -- 172.21.15.29:0/4130379971 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd680007400 con 0x7fd6900cbf70 2024-04-17T19:47:50.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.061+0000 7fd686a00640 1 --2- 172.21.15.29:0/4130379971 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd66c068380 0x7fd66c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:50.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.061+0000 7fd686a00640 1 -- 172.21.15.29:0/4130379971 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7fd680085f50 con 0x7fd6900cbf70 2024-04-17T19:47:50.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.061+0000 7fd687e00640 1 --2- 172.21.15.29:0/4130379971 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd66c068380 0x7fd66c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:50.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.065+0000 7fd687e00640 1 --2- 172.21.15.29:0/4130379971 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd66c068380 0x7fd66c06a890 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd6900b5190 tx=0x7fd678008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:50.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.065+0000 7fd686a00640 1 -- 172.21.15.29:0/4130379971 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd6800563e0 con 0x7fd6900cbf70 2024-04-17T19:47:50.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.241+0000 7fd695400640 1 -- 172.21.15.29:0/4130379971 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fd6900cb320 con 0x7fd6900cbf70 2024-04-17T19:47:50.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.837+0000 7fd686a00640 1 -- 172.21.15.29:0/4130379971 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) ==== 157+0+0 (secure 0 0 0) 0x7fd680055ce0 con 0x7fd6900cbf70 2024-04-17T19:47:50.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.837+0000 7fd695400640 1 -- 172.21.15.29:0/4130379971 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd66c068380 msgr2=0x7fd66c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:50.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.837+0000 7fd695400640 1 --2- 172.21.15.29:0/4130379971 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd66c068380 0x7fd66c06a890 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd6900b5190 tx=0x7fd678008040 comp rx=0 tx=0).stop 2024-04-17T19:47:50.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.837+0000 7fd695400640 1 -- 172.21.15.29:0/4130379971 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6900cbf70 msgr2=0x7fd6900b45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:50.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.837+0000 7fd695400640 1 --2- 172.21.15.29:0/4130379971 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6900cbf70 0x7fd6900b45c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd68000c3a0 tx=0x7fd68000c860 comp rx=0 tx=0).stop 2024-04-17T19:47:50.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.837+0000 7fd695400640 1 -- 172.21.15.29:0/4130379971 shutdown_connections 2024-04-17T19:47:50.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.837+0000 7fd695400640 1 --2- 172.21.15.29:0/4130379971 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd66c068380 0x7fd66c06a890 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:50.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.837+0000 7fd695400640 1 --2- 172.21.15.29:0/4130379971 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd690154220 0x7fd690145320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:50.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.837+0000 7fd695400640 1 --2- 172.21.15.29:0/4130379971 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd690153e10 0x7fd690144de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:50.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.837+0000 7fd695400640 1 --2- 172.21.15.29:0/4130379971 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6900cbf70 0x7fd6900b45c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:50.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.837+0000 7fd695400640 1 -- 172.21.15.29:0/4130379971 >> 172.21.15.29:0/4130379971 conn(0x7fd690071a70 msgr2=0x7fd690071e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:50.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.841+0000 7fd695400640 1 -- 172.21.15.29:0/4130379971 shutdown_connections 2024-04-17T19:47:50.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:50.841+0000 7fd695400640 1 -- 172.21.15.29:0/4130379971 wait complete. 2024-04-17T19:47:50.853 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T19:47:51.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.029+0000 7f3474a00640 1 Processor -- start 2024-04-17T19:47:51.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.029+0000 7f3474a00640 1 -- start start 2024-04-17T19:47:51.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.033+0000 7f3474a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34700c9d80 0x7f347015bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:51.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.033+0000 7f3474a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f347015c4a0 0x7f34701666b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:51.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.033+0000 7f3474a00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34700ca4c0 con 0x7f3470159510 2024-04-17T19:47:51.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.033+0000 7f3474a00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f34700cc3b0 con 0x7f347015c4a0 2024-04-17T19:47:51.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.033+0000 7f3474a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f34700cb7e0 con 0x7f34700c9d80 2024-04-17T19:47:51.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.033+0000 7f346fe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f347015c4a0 0x7f34701666b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:51.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.033+0000 7f346fe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f347015c4a0 0x7f34701666b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51658/0 (socket says 172.21.15.29:51658) 2024-04-17T19:47:51.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.033+0000 7f346fe00640 1 -- 172.21.15.29:0/1140524607 learned_addr learned my addr 172.21.15.29:0/1140524607 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:51.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.033+0000 7f346f400640 1 --1- 172.21.15.29:0/1140524607 >> v1:172.21.15.29:6789/0 conn(0x7f3470159510 0x7f34700cab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33794/0 (socket says 172.21.15.29:33794) 2024-04-17T19:47:51.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.033+0000 7f3466a00640 1 --2- 172.21.15.29:0/1140524607 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34700c9d80 0x7f347015bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:51.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.033+0000 7f346fe00640 1 -- 172.21.15.29:0/1140524607 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34700c9d80 msgr2=0x7f347015bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:51.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.033+0000 7f346fe00640 1 --2- 172.21.15.29:0/1140524607 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34700c9d80 0x7f347015bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:51.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.033+0000 7f346fe00640 1 -- 172.21.15.29:0/1140524607 >> v1:172.21.15.29:6789/0 conn(0x7f3470159510 legacy=0x7f34700cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:51.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.033+0000 7f346fe00640 1 -- 172.21.15.29:0/1140524607 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34700cba80 con 0x7f347015c4a0 2024-04-17T19:47:51.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.037+0000 7f346fe00640 1 --2- 172.21.15.29:0/1140524607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f347015c4a0 0x7f34701666b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3468009e50 tx=0x7f346801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d59d284dab10fc27 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:51.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.037+0000 7f346ea00640 1 -- 172.21.15.29:0/1140524607 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34680a57f0 con 0x7f347015c4a0 2024-04-17T19:47:51.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.037+0000 7f346ea00640 1 -- 172.21.15.29:0/1140524607 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34680a5990 con 0x7f347015c4a0 2024-04-17T19:47:51.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.037+0000 7f346ea00640 1 -- 172.21.15.29:0/1140524607 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34680a5d60 con 0x7f347015c4a0 2024-04-17T19:47:51.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.037+0000 7f3474a00640 1 -- 172.21.15.29:0/1140524607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f347015c4a0 msgr2=0x7f34701666b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:51.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.037+0000 7f3474a00640 1 --2- 172.21.15.29:0/1140524607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f347015c4a0 0x7f34701666b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3468009e50 tx=0x7f346801ca30 comp rx=0 tx=0).stop 2024-04-17T19:47:51.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.037+0000 7f3474a00640 1 -- 172.21.15.29:0/1140524607 shutdown_connections 2024-04-17T19:47:51.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.037+0000 7f3474a00640 1 --2- 172.21.15.29:0/1140524607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f347015c4a0 0x7f34701666b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:51.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.037+0000 7f3474a00640 1 --2- 172.21.15.29:0/1140524607 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34700c9d80 0x7f347015bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:51.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.037+0000 7f3474a00640 1 -- 172.21.15.29:0/1140524607 >> 172.21.15.29:0/1140524607 conn(0x7f3470072070 msgr2=0x7f347006dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:51.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.037+0000 7f3474a00640 1 -- 172.21.15.29:0/1140524607 shutdown_connections 2024-04-17T19:47:51.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.037+0000 7f3474a00640 1 -- 172.21.15.29:0/1140524607 wait complete. 2024-04-17T19:47:51.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.037+0000 7f3474a00640 1 Processor -- start 2024-04-17T19:47:51.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f3474a00640 1 -- start start 2024-04-17T19:47:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f3474a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34700c9d80 0x7f34700cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f3474a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3470159510 0x7f34700cf070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f3474a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f347015c4a0 0x7f347014eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f3474a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f34700ca190 con 0x7f34700c9d80 2024-04-17T19:47:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f3474a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f347006d1d0 con 0x7f347015c4a0 2024-04-17T19:47:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f3474a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f347006e750 con 0x7f3470159510 2024-04-17T19:47:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f346f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34700c9d80 0x7f34700cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:51.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f346f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34700c9d80 0x7f34700cd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51672/0 (socket says 172.21.15.29:51672) 2024-04-17T19:47:51.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f346f400640 1 -- 172.21.15.29:0/1633951576 learned_addr learned my addr 172.21.15.29:0/1633951576 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:51.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f346fe00640 1 --2- 172.21.15.29:0/1633951576 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f347015c4a0 0x7f347014eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:51.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f3466a00640 1 --2- 172.21.15.29:0/1633951576 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3470159510 0x7f34700cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:51.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f346f400640 1 -- 172.21.15.29:0/1633951576 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3470159510 msgr2=0x7f34700cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:51.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f346f400640 1 --2- 172.21.15.29:0/1633951576 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3470159510 0x7f34700cf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:51.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f346f400640 1 -- 172.21.15.29:0/1633951576 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f347015c4a0 msgr2=0x7f347014eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:51.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f346f400640 1 --2- 172.21.15.29:0/1633951576 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f347015c4a0 0x7f347014eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:51.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f346f400640 1 -- 172.21.15.29:0/1633951576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34700cd8c0 con 0x7f34700c9d80 2024-04-17T19:47:51.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f346f400640 1 --2- 172.21.15.29:0/1633951576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34700c9d80 0x7f34700cd380 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f346000c3a0 tx=0x7f346000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:51.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f3467e00640 1 -- 172.21.15.29:0/1633951576 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34600074e0 con 0x7f34700c9d80 2024-04-17T19:47:51.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f3467e00640 1 -- 172.21.15.29:0/1633951576 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3460007680 con 0x7f34700c9d80 2024-04-17T19:47:51.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f3467e00640 1 -- 172.21.15.29:0/1633951576 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3460004a30 con 0x7f34700c9d80 2024-04-17T19:47:51.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f3474a00640 1 -- 172.21.15.29:0/1633951576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3470149b10 con 0x7f34700c9d80 2024-04-17T19:47:51.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f3474a00640 1 -- 172.21.15.29:0/1633951576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3470147910 con 0x7f34700c9d80 2024-04-17T19:47:51.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.041+0000 7f3474a00640 1 -- 172.21.15.29:0/1633951576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34700c44f0 con 0x7f34700c9d80 2024-04-17T19:47:51.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.045+0000 7f3467e00640 1 -- 172.21.15.29:0/1633951576 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f346001a020 con 0x7f34700c9d80 2024-04-17T19:47:51.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.045+0000 7f3467e00640 1 --2- 172.21.15.29:0/1633951576 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f344c068380 0x7f344c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:51.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.045+0000 7f3467e00640 1 -- 172.21.15.29:0/1633951576 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 6007+0+0 (secure 0 0 0) 0x7f3460086310 con 0x7f34700c9d80 2024-04-17T19:47:51.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.045+0000 7f3466a00640 1 --2- 172.21.15.29:0/1633951576 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f344c068380 0x7f344c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:51.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.045+0000 7f3466a00640 1 --2- 172.21.15.29:0/1633951576 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f344c068380 0x7f344c06a890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f347014a4b0 tx=0x7f3458008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:51.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.049+0000 7f3467e00640 1 -- 172.21.15.29:0/1633951576 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3460056710 con 0x7f34700c9d80 2024-04-17T19:47:51.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:51.217+0000 7f3474a00640 1 -- 172.21.15.29:0/1633951576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f34700ca950 con 0x7f34700c9d80 2024-04-17T19:47:52.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:52.873+0000 7f3467e00640 1 -- 172.21.15.29:0/1633951576 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) ==== 228+0+0 (secure 0 0 0) 0x7f3460056010 con 0x7f34700c9d80 2024-04-17T19:47:52.863 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data_ec' created 2024-04-17T19:47:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:52.877+0000 7f3474a00640 1 -- 172.21.15.29:0/1633951576 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f344c068380 msgr2=0x7f344c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:52.877+0000 7f3474a00640 1 --2- 172.21.15.29:0/1633951576 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f344c068380 0x7f344c06a890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f347014a4b0 tx=0x7f3458008040 comp rx=0 tx=0).stop 2024-04-17T19:47:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:52.877+0000 7f3474a00640 1 -- 172.21.15.29:0/1633951576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34700c9d80 msgr2=0x7f34700cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:52.877+0000 7f3474a00640 1 --2- 172.21.15.29:0/1633951576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34700c9d80 0x7f34700cd380 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f346000c3a0 tx=0x7f346000c860 comp rx=0 tx=0).stop 2024-04-17T19:47:52.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:52.877+0000 7f3474a00640 1 -- 172.21.15.29:0/1633951576 shutdown_connections 2024-04-17T19:47:52.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:52.877+0000 7f3474a00640 1 --2- 172.21.15.29:0/1633951576 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f344c068380 0x7f344c06a890 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:52.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:52.877+0000 7f3474a00640 1 --2- 172.21.15.29:0/1633951576 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f347015c4a0 0x7f347014eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:52.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:52.877+0000 7f3474a00640 1 --2- 172.21.15.29:0/1633951576 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3470159510 0x7f34700cf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:52.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:52.877+0000 7f3474a00640 1 --2- 172.21.15.29:0/1633951576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34700c9d80 0x7f34700cd380 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:52.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:52.877+0000 7f3474a00640 1 -- 172.21.15.29:0/1633951576 >> 172.21.15.29:0/1633951576 conn(0x7f3470072070 msgr2=0x7f3470075420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:52.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:52.881+0000 7f3474a00640 1 -- 172.21.15.29:0/1633951576 shutdown_connections 2024-04-17T19:47:52.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:52.881+0000 7f3474a00640 1 -- 172.21.15.29:0/1633951576 wait complete. 2024-04-17T19:47:52.905 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-17T19:47:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.077+0000 7f98e8400640 1 Processor -- start 2024-04-17T19:47:53.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.077+0000 7f98e8400640 1 -- start start 2024-04-17T19:47:53.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.077+0000 7f98e8400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f98e00c97c0 0x7f98e01527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:53.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.077+0000 7f98e8400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f98e006f6d0 0x7f98e006fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:53.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.077+0000 7f98e8400640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98e0072c30 con 0x7f98e0072e70 2024-04-17T19:47:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.077+0000 7f98e8400640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f98e00c7250 con 0x7f98e00c97c0 2024-04-17T19:47:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.077+0000 7f98e8400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f98e00c7530 con 0x7f98e006f6d0 2024-04-17T19:47:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.077+0000 7f98e5000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f98e00c97c0 0x7f98e01527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.077+0000 7f98e5000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f98e00c97c0 0x7f98e01527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51680/0 (socket says 172.21.15.29:51680) 2024-04-17T19:47:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.077+0000 7f98e5000640 1 -- 172.21.15.29:0/1673655589 learned_addr learned my addr 172.21.15.29:0/1673655589 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.077+0000 7f98e6400640 1 --2- 172.21.15.29:0/1673655589 >> v2:172.21.15.37:3300/0 conn(0x7f98e006f6d0 0x7f98e006fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.077+0000 7f98e5000640 1 -- 172.21.15.29:0/1673655589 >> v2:172.21.15.37:3300/0 conn(0x7f98e006f6d0 msgr2=0x7f98e006fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.077+0000 7f98e5000640 1 --2- 172.21.15.29:0/1673655589 >> v2:172.21.15.37:3300/0 conn(0x7f98e006f6d0 0x7f98e006fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.077+0000 7f98e5000640 1 -- 172.21.15.29:0/1673655589 >> v1:172.21.15.37:6789/0 conn(0x7f98e0072e70 legacy=0x7f98e00733c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.077+0000 7f98e5000640 1 -- 172.21.15.29:0/1673655589 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e00c8f70 con 0x7f98e00c97c0 2024-04-17T19:47:53.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.077+0000 7f98e6400640 1 --2- 172.21.15.29:0/1673655589 >> v2:172.21.15.37:3300/0 conn(0x7f98e006f6d0 0x7f98e006fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:53.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.081+0000 7f98e5000640 1 --2- 172.21.15.29:0/1673655589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e00c97c0 0x7f98e01527c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f98e00c7840 tx=0x7f98cc01e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34ef9300e57abae3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.081+0000 7f98dfe00640 1 -- 172.21.15.29:0/1673655589 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98cc0ad070 con 0x7f98e00c97c0 2024-04-17T19:47:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.081+0000 7f98dfe00640 1 -- 172.21.15.29:0/1673655589 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98cc004230 con 0x7f98e00c97c0 2024-04-17T19:47:53.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.081+0000 7f98dfe00640 1 -- 172.21.15.29:0/1673655589 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98cc003620 con 0x7f98e00c97c0 2024-04-17T19:47:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.081+0000 7f98e8400640 1 -- 172.21.15.29:0/1673655589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e00c97c0 msgr2=0x7f98e01527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.081+0000 7f98e8400640 1 --2- 172.21.15.29:0/1673655589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e00c97c0 0x7f98e01527c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f98e00c7840 tx=0x7f98cc01e9d0 comp rx=0 tx=0).stop 2024-04-17T19:47:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.081+0000 7f98e8400640 1 -- 172.21.15.29:0/1673655589 shutdown_connections 2024-04-17T19:47:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.081+0000 7f98e8400640 1 --2- 172.21.15.29:0/1673655589 >> v2:172.21.15.37:3300/0 conn(0x7f98e006f6d0 0x7f98e006fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.081+0000 7f98e8400640 1 --2- 172.21.15.29:0/1673655589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e00c97c0 0x7f98e01527c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.081+0000 7f98e8400640 1 -- 172.21.15.29:0/1673655589 >> 172.21.15.29:0/1673655589 conn(0x7f98e006bf90 msgr2=0x7f98e006c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.081+0000 7f98e8400640 1 -- 172.21.15.29:0/1673655589 shutdown_connections 2024-04-17T19:47:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.081+0000 7f98e8400640 1 -- 172.21.15.29:0/1673655589 wait complete. 2024-04-17T19:47:53.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e8400640 1 Processor -- start 2024-04-17T19:47:53.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e8400640 1 -- start start 2024-04-17T19:47:53.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e8400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98e006f6d0 0x7f98e00cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:53.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e8400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e0072e70 0x7f98e014b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:53.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e8400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98e00c97c0 0x7f98e014bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:53.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e8400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f98e0164f80 con 0x7f98e0072e70 2024-04-17T19:47:53.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e8400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f98e0164e00 con 0x7f98e006f6d0 2024-04-17T19:47:53.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e8400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f98e0165100 con 0x7f98e00c97c0 2024-04-17T19:47:53.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e5000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e0072e70 0x7f98e014b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:53.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e5000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e0072e70 0x7f98e014b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51696/0 (socket says 172.21.15.29:51696) 2024-04-17T19:47:53.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e5000640 1 -- 172.21.15.29:0/1439946424 learned_addr learned my addr 172.21.15.29:0/1439946424 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:53.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e5a00640 1 --2- 172.21.15.29:0/1439946424 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98e006f6d0 0x7f98e00cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:53.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e6400640 1 --2- 172.21.15.29:0/1439946424 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98e00c97c0 0x7f98e014bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:53.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e5000640 1 -- 172.21.15.29:0/1439946424 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98e00c97c0 msgr2=0x7f98e014bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:53.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e5000640 1 --2- 172.21.15.29:0/1439946424 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98e00c97c0 0x7f98e014bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:53.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e5000640 1 -- 172.21.15.29:0/1439946424 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98e006f6d0 msgr2=0x7f98e00cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e5000640 1 --2- 172.21.15.29:0/1439946424 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98e006f6d0 0x7f98e00cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e5000640 1 -- 172.21.15.29:0/1439946424 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98e0153fc0 con 0x7f98e0072e70 2024-04-17T19:47:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e5000640 1 --2- 172.21.15.29:0/1439946424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e0072e70 0x7f98e014b9d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f98cc024fe0 tx=0x7f98cc01b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98dd600640 1 -- 172.21.15.29:0/1439946424 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98cc0b7070 con 0x7f98e0072e70 2024-04-17T19:47:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98dd600640 1 -- 172.21.15.29:0/1439946424 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98cc003620 con 0x7f98e0072e70 2024-04-17T19:47:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e8400640 1 -- 172.21.15.29:0/1439946424 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98e0154820 con 0x7f98e0072e70 2024-04-17T19:47:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98dd600640 1 -- 172.21.15.29:0/1439946424 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98cc0ad050 con 0x7f98e0072e70 2024-04-17T19:47:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.085+0000 7f98e8400640 1 -- 172.21.15.29:0/1439946424 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98e0154540 con 0x7f98e0072e70 2024-04-17T19:47:53.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.089+0000 7f98e8400640 1 -- 172.21.15.29:0/1439946424 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98e0069b90 con 0x7f98e0072e70 2024-04-17T19:47:53.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.089+0000 7f98dd600640 1 -- 172.21.15.29:0/1439946424 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f98cc0c2060 con 0x7f98e0072e70 2024-04-17T19:47:53.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.089+0000 7f98dd600640 1 --2- 172.21.15.29:0/1439946424 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f98c0068380 0x7f98c006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:53.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.089+0000 7f98dd600640 1 -- 172.21.15.29:0/1439946424 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 6500+0+0 (secure 0 0 0) 0x7f98cc120e60 con 0x7f98e0072e70 2024-04-17T19:47:53.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.089+0000 7f98e5a00640 1 --2- 172.21.15.29:0/1439946424 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f98c0068380 0x7f98c006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:53.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.093+0000 7f98e5a00640 1 --2- 172.21.15.29:0/1439946424 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f98c0068380 0x7f98c006a890 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f98d4000f70 tx=0x7f98d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:53.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.093+0000 7f98dd600640 1 -- 172.21.15.29:0/1439946424 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f98cc0185c0 con 0x7f98e0072e70 2024-04-17T19:47:53.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.261+0000 7f98e8400640 1 -- 172.21.15.29:0/1439946424 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f98e00cb810 con 0x7f98e0072e70 2024-04-17T19:47:53.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.877+0000 7f98dd600640 1 -- 172.21.15.29:0/1439946424 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) ==== 169+0+0 (secure 0 0 0) 0x7f98cc0f1060 con 0x7f98e0072e70 2024-04-17T19:47:53.866 INFO:teuthology.orchestra.run.smithi029.stderr:set pool 4 allow_ec_overwrites to true 2024-04-17T19:47:53.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.881+0000 7f98e8400640 1 -- 172.21.15.29:0/1439946424 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f98c0068380 msgr2=0x7f98c006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:53.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.881+0000 7f98e8400640 1 --2- 172.21.15.29:0/1439946424 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f98c0068380 0x7f98c006a890 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f98d4000f70 tx=0x7f98d4009040 comp rx=0 tx=0).stop 2024-04-17T19:47:53.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.881+0000 7f98e8400640 1 -- 172.21.15.29:0/1439946424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e0072e70 msgr2=0x7f98e014b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:53.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.881+0000 7f98e8400640 1 --2- 172.21.15.29:0/1439946424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e0072e70 0x7f98e014b9d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f98cc024fe0 tx=0x7f98cc01b770 comp rx=0 tx=0).stop 2024-04-17T19:47:53.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.881+0000 7f98e8400640 1 -- 172.21.15.29:0/1439946424 shutdown_connections 2024-04-17T19:47:53.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.881+0000 7f98e8400640 1 --2- 172.21.15.29:0/1439946424 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f98c0068380 0x7f98c006a890 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:53.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.881+0000 7f98e8400640 1 --2- 172.21.15.29:0/1439946424 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98e00c97c0 0x7f98e014bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:53.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.881+0000 7f98e8400640 1 --2- 172.21.15.29:0/1439946424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98e0072e70 0x7f98e014b9d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:53.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.881+0000 7f98e8400640 1 --2- 172.21.15.29:0/1439946424 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98e006f6d0 0x7f98e00cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:53.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.881+0000 7f98e8400640 1 -- 172.21.15.29:0/1439946424 >> 172.21.15.29:0/1439946424 conn(0x7f98e006bf90 msgr2=0x7f98e006fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:53.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.881+0000 7f98e8400640 1 -- 172.21.15.29:0/1439946424 shutdown_connections 2024-04-17T19:47:53.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:53.881+0000 7f98e8400640 1 -- 172.21.15.29:0/1439946424 wait complete. 2024-04-17T19:47:53.913 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-17T19:47:54.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.089+0000 7f64b4e00640 1 Processor -- start 2024-04-17T19:47:54.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.089+0000 7f64b4e00640 1 -- start start 2024-04-17T19:47:54.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.089+0000 7f64b4e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f64b01593d0 0x7f64b01597e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:54.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.089+0000 7f64b4e00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64b015c6d0 con 0x7f64b0071ce0 2024-04-17T19:47:54.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.089+0000 7f64b4e00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64b015c9b0 con 0x7f64b00c9f00 2024-04-17T19:47:54.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.089+0000 7f64b4e00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f64b00c7bd0 con 0x7f64b01593d0 2024-04-17T19:47:54.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.089+0000 7f64ad400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f64b01593d0 0x7f64b01597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:54.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.089+0000 7f64ade00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f64b0071ce0 0x7f64b0141de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33806/0 (socket says 172.21.15.29:33806) 2024-04-17T19:47:54.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.089+0000 7f64ade00640 1 -- 172.21.15.29:0/785501350 learned_addr learned my addr 172.21.15.29:0/785501350 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:47:54.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.089+0000 7f64ad400640 1 -- 172.21.15.29:0/785501350 >> v1:172.21.15.37:6789/0 conn(0x7f64b00c9f00 legacy=0x7f64b0163620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:54.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.089+0000 7f64ad400640 1 -- 172.21.15.29:0/785501350 >> v1:172.21.15.29:6789/0 conn(0x7f64b0071ce0 legacy=0x7f64b0141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:54.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.089+0000 7f64aca00640 1 -- 172.21.15.29:0/785501350 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2040009671 0 0) 0x7f64b015c6d0 con 0x7f64b0071ce0 2024-04-17T19:47:54.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.089+0000 7f64ad400640 1 -- 172.21.15.29:0/785501350 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b015cc90 con 0x7f64b01593d0 2024-04-17T19:47:54.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.093+0000 7f64ad400640 1 --2- 172.21.15.29:0/785501350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64b01593d0 0x7f64b01597e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f649c0097c0 tx=0x7f649c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c1a0d4e148932f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:54.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.093+0000 7f64aca00640 1 -- 172.21.15.29:0/785501350 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c0b1070 con 0x7f64b01593d0 2024-04-17T19:47:54.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.093+0000 7f64aca00640 1 -- 172.21.15.29:0/785501350 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f649c0ac410 con 0x7f64b01593d0 2024-04-17T19:47:54.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.093+0000 7f64aca00640 1 -- 172.21.15.29:0/785501350 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c0ac7e0 con 0x7f64b01593d0 2024-04-17T19:47:54.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.093+0000 7f64b4e00640 1 -- 172.21.15.29:0/785501350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64b01593d0 msgr2=0x7f64b01597e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:54.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.093+0000 7f64b4e00640 1 --2- 172.21.15.29:0/785501350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64b01593d0 0x7f64b01597e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f649c0097c0 tx=0x7f649c01c4a0 comp rx=0 tx=0).stop 2024-04-17T19:47:54.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.093+0000 7f64b4e00640 1 -- 172.21.15.29:0/785501350 shutdown_connections 2024-04-17T19:47:54.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.093+0000 7f64b4e00640 1 --2- 172.21.15.29:0/785501350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64b01593d0 0x7f64b01597e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:54.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.093+0000 7f64b4e00640 1 -- 172.21.15.29:0/785501350 >> 172.21.15.29:0/785501350 conn(0x7f64b006bf90 msgr2=0x7f64b006c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:54.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.093+0000 7f64b4e00640 1 -- 172.21.15.29:0/785501350 shutdown_connections 2024-04-17T19:47:54.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.093+0000 7f64b4e00640 1 -- 172.21.15.29:0/785501350 wait complete. 2024-04-17T19:47:54.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64b4e00640 1 Processor -- start 2024-04-17T19:47:54.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64b4e00640 1 -- start start 2024-04-17T19:47:54.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64b4e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64b0071ce0 0x7f64b0159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:54.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64b4e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64b00c9f00 0x7f64b0151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:54.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64b4e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64b01593d0 0x7f64b0139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:54.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64b4e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f64b0072270 con 0x7f64b00c9f00 2024-04-17T19:47:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64b4e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f64b00720f0 con 0x7f64b0071ce0 2024-04-17T19:47:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64b4e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f64b015c240 con 0x7f64b01593d0 2024-04-17T19:47:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64ad400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64b00c9f00 0x7f64b0151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64ad400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64b00c9f00 0x7f64b0151cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51708/0 (socket says 172.21.15.29:51708) 2024-04-17T19:47:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64ad400640 1 -- 172.21.15.29:0/156571007 learned_addr learned my addr 172.21.15.29:0/156571007 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64ade00640 1 --2- 172.21.15.29:0/156571007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64b0071ce0 0x7f64b0159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64ae800640 1 --2- 172.21.15.29:0/156571007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64b01593d0 0x7f64b0139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64ad400640 1 -- 172.21.15.29:0/156571007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64b01593d0 msgr2=0x7f64b0139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64ad400640 1 --2- 172.21.15.29:0/156571007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64b01593d0 0x7f64b0139e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64ad400640 1 -- 172.21.15.29:0/156571007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64b0071ce0 msgr2=0x7f64b0159e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64ad400640 1 --2- 172.21.15.29:0/156571007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64b0071ce0 0x7f64b0159e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64ad400640 1 -- 172.21.15.29:0/156571007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b01f6370 con 0x7f64b00c9f00 2024-04-17T19:47:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.097+0000 7f64ad400640 1 --2- 172.21.15.29:0/156571007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64b00c9f00 0x7f64b0151cb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f649c0af040 tx=0x7f649c019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:54.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.101+0000 7f64a6000640 1 -- 172.21.15.29:0/156571007 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c0bc070 con 0x7f64b00c9f00 2024-04-17T19:47:54.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.101+0000 7f64a6000640 1 -- 172.21.15.29:0/156571007 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f649c0acda0 con 0x7f64b00c9f00 2024-04-17T19:47:54.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.101+0000 7f64b4e00640 1 -- 172.21.15.29:0/156571007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64b015bc20 con 0x7f64b00c9f00 2024-04-17T19:47:54.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.101+0000 7f64a6000640 1 -- 172.21.15.29:0/156571007 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f649c0b1040 con 0x7f64b00c9f00 2024-04-17T19:47:54.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.101+0000 7f64b4e00640 1 -- 172.21.15.29:0/156571007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64b01f6090 con 0x7f64b00c9f00 2024-04-17T19:47:54.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.101+0000 7f64b4e00640 1 -- 172.21.15.29:0/156571007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64b01f5e80 con 0x7f64b00c9f00 2024-04-17T19:47:54.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.101+0000 7f64a6000640 1 -- 172.21.15.29:0/156571007 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f649c0bf070 con 0x7f64b00c9f00 2024-04-17T19:47:54.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.101+0000 7f64a6000640 1 --2- 172.21.15.29:0/156571007 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f64900683b0 0x7f649006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:54.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.101+0000 7f64a6000640 1 -- 172.21.15.29:0/156571007 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7f649c11faf0 con 0x7f64b00c9f00 2024-04-17T19:47:54.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.101+0000 7f64ade00640 1 --2- 172.21.15.29:0/156571007 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f64900683b0 0x7f649006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:54.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.105+0000 7f64ade00640 1 --2- 172.21.15.29:0/156571007 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f64900683b0 0x7f649006a8c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f64a0000f70 tx=0x7f64a0007240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:54.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.105+0000 7f64a6000640 1 -- 172.21.15.29:0/156571007 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f649c016090 con 0x7f64b00c9f00 2024-04-17T19:47:54.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.317+0000 7f64b4e00640 1 -- 172.21.15.29:0/156571007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f64b0069b90 con 0x7f64b00c9f00 2024-04-17T19:47:54.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.929+0000 7f64a6000640 1 -- 172.21.15.29:0/156571007 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v6) ==== 137+0+0 (secure 0 0 0) 0x7f649c0ef6a0 con 0x7f64b00c9f00 2024-04-17T19:47:54.917 INFO:teuthology.orchestra.run.smithi029.stderr:added data pool 4 to fsmap 2024-04-17T19:47:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.929+0000 7f64b4e00640 1 -- 172.21.15.29:0/156571007 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f64900683b0 msgr2=0x7f649006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.929+0000 7f64b4e00640 1 --2- 172.21.15.29:0/156571007 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f64900683b0 0x7f649006a8c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f64a0000f70 tx=0x7f64a0007240 comp rx=0 tx=0).stop 2024-04-17T19:47:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.929+0000 7f64b4e00640 1 -- 172.21.15.29:0/156571007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64b00c9f00 msgr2=0x7f64b0151cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.929+0000 7f64b4e00640 1 --2- 172.21.15.29:0/156571007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64b00c9f00 0x7f64b0151cb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f649c0af040 tx=0x7f649c019240 comp rx=0 tx=0).stop 2024-04-17T19:47:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.929+0000 7f64b4e00640 1 -- 172.21.15.29:0/156571007 shutdown_connections 2024-04-17T19:47:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.929+0000 7f64b4e00640 1 --2- 172.21.15.29:0/156571007 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f64900683b0 0x7f649006a8c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.929+0000 7f64b4e00640 1 --2- 172.21.15.29:0/156571007 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f64b01593d0 0x7f64b0139e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:54.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.929+0000 7f64b4e00640 1 --2- 172.21.15.29:0/156571007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f64b00c9f00 0x7f64b0151cb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.929+0000 7f64b4e00640 1 --2- 172.21.15.29:0/156571007 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f64b0071ce0 0x7f64b0159e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.929+0000 7f64b4e00640 1 -- 172.21.15.29:0/156571007 >> 172.21.15.29:0/156571007 conn(0x7f64b006bf90 msgr2=0x7f64b0140720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.929+0000 7f64b4e00640 1 -- 172.21.15.29:0/156571007 shutdown_connections 2024-04-17T19:47:54.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:54.933+0000 7f64b4e00640 1 -- 172.21.15.29:0/156571007 wait complete. 2024-04-17T19:47:54.943 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:47:55.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.109+0000 7f267f000640 1 Processor -- start 2024-04-17T19:47:55.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.109+0000 7f267f000640 1 -- start start 2024-04-17T19:47:55.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.109+0000 7f267f000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26780c6f70 0x7f26780c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:55.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.109+0000 7f267f000640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f267815a0a0 con 0x7f267806c660 2024-04-17T19:47:55.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.109+0000 7f267f000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f267815abe0 con 0x7f2678159c90 2024-04-17T19:47:55.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.109+0000 7f267f000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f267815a340 con 0x7f26780c6f70 2024-04-17T19:47:55.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.113+0000 7f267d000640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f267806c660 0x7f267806ca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33816/0 (socket says 172.21.15.29:33816) 2024-04-17T19:47:55.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.113+0000 7f267d000640 1 -- 172.21.15.29:0/2691833620 learned_addr learned my addr 172.21.15.29:0/2691833620 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:47:55.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.113+0000 7f2677400640 1 --2- 172.21.15.29:0/2691833620 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26780c6f70 0x7f26780c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:55.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.113+0000 7f2676a00640 1 -- 172.21.15.29:0/2691833620 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1850539970 0 0) 0x7f267815a0a0 con 0x7f267806c660 2024-04-17T19:47:55.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.113+0000 7f2676a00640 1 -- 172.21.15.29:0/2691833620 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f265c003530 con 0x7f267806c660 2024-04-17T19:47:55.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.113+0000 7f2676a00640 1 -- 172.21.15.29:0/2691833620 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3179719322 0 0) 0x7f267815abe0 con 0x7f2678159c90 2024-04-17T19:47:55.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.113+0000 7f2676a00640 1 -- 172.21.15.29:0/2691833620 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f267815a0a0 con 0x7f2678159c90 2024-04-17T19:47:55.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.113+0000 7f2677400640 1 -- 172.21.15.29:0/2691833620 >> v1:172.21.15.37:6789/0 conn(0x7f2678159c90 legacy=0x7f26780ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:55.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.113+0000 7f2677400640 1 -- 172.21.15.29:0/2691833620 >> v1:172.21.15.29:6789/0 conn(0x7f267806c660 legacy=0x7f267806ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:55.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.113+0000 7f2676a00640 1 -- 172.21.15.29:0/2691833620 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 117724529 0 0) 0x7f265c003530 con 0x7f267806c660 2024-04-17T19:47:55.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.113+0000 7f2677400640 1 -- 172.21.15.29:0/2691833620 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f267815a620 con 0x7f26780c6f70 2024-04-17T19:47:55.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.113+0000 7f2677400640 1 --2- 172.21.15.29:0/2691833620 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26780c6f70 0x7f26780c7380 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f26640097f0 tx=0x7f266401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17c009a6a273994a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:55.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.117+0000 7f2676a00640 1 -- 172.21.15.29:0/2691833620 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26640b0070 con 0x7f26780c6f70 2024-04-17T19:47:55.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.117+0000 7f2676a00640 1 -- 172.21.15.29:0/2691833620 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26640ab400 con 0x7f26780c6f70 2024-04-17T19:47:55.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.117+0000 7f2676a00640 1 -- 172.21.15.29:0/2691833620 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2670001a20 con 0x7f26780c6f70 2024-04-17T19:47:55.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.117+0000 7f267f000640 1 -- 172.21.15.29:0/2691833620 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26780c6f70 msgr2=0x7f26780c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:55.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.117+0000 7f267f000640 1 --2- 172.21.15.29:0/2691833620 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26780c6f70 0x7f26780c7380 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f26640097f0 tx=0x7f266401c400 comp rx=0 tx=0).stop 2024-04-17T19:47:55.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.117+0000 7f267f000640 1 -- 172.21.15.29:0/2691833620 shutdown_connections 2024-04-17T19:47:55.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.117+0000 7f267f000640 1 --2- 172.21.15.29:0/2691833620 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26780c6f70 0x7f26780c7380 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.117+0000 7f267f000640 1 -- 172.21.15.29:0/2691833620 >> 172.21.15.29:0/2691833620 conn(0x7f267806fde0 msgr2=0x7f26780701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:55.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.117+0000 7f267f000640 1 -- 172.21.15.29:0/2691833620 shutdown_connections 2024-04-17T19:47:55.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.117+0000 7f267f000640 1 -- 172.21.15.29:0/2691833620 wait complete. 2024-04-17T19:47:55.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.117+0000 7f267f000640 1 Processor -- start 2024-04-17T19:47:55.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.117+0000 7f267f000640 1 -- start start 2024-04-17T19:47:55.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f267f000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267806c660 0x7f26780c5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:55.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f267f000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26780c6f70 0x7f267814f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:55.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f267f000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2678159c90 0x7f2678148c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f267f000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f26780cab40 con 0x7f26780c6f70 2024-04-17T19:47:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f267f000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f26780ca9c0 con 0x7f2678159c90 2024-04-17T19:47:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f267f000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2678167e40 con 0x7f267806c660 2024-04-17T19:47:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f2677400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26780c6f70 0x7f267814f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f2677400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26780c6f70 0x7f267814f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51720/0 (socket says 172.21.15.29:51720) 2024-04-17T19:47:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f2677400640 1 -- 172.21.15.29:0/433557000 learned_addr learned my addr 172.21.15.29:0/433557000 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f267d000640 1 --2- 172.21.15.29:0/433557000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2678159c90 0x7f2678148c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f2677e00640 1 --2- 172.21.15.29:0/433557000 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267806c660 0x7f26780c5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f2677400640 1 -- 172.21.15.29:0/433557000 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267806c660 msgr2=0x7f26780c5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f2677400640 1 --2- 172.21.15.29:0/433557000 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267806c660 0x7f26780c5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f2677400640 1 -- 172.21.15.29:0/433557000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2678159c90 msgr2=0x7f2678148c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f2677400640 1 --2- 172.21.15.29:0/433557000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2678159c90 0x7f2678148c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f2677400640 1 -- 172.21.15.29:0/433557000 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26781f8cf0 con 0x7f26780c6f70 2024-04-17T19:47:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f267d000640 1 --2- 172.21.15.29:0/433557000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2678159c90 0x7f2678148c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:55.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f2677400640 1 --2- 172.21.15.29:0/433557000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26780c6f70 0x7f267814f3d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f26640097f0 tx=0x7f26640191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:55.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f266fe00640 1 -- 172.21.15.29:0/433557000 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26640b0040 con 0x7f26780c6f70 2024-04-17T19:47:55.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f266fe00640 1 -- 172.21.15.29:0/433557000 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26640bb070 con 0x7f26780c6f70 2024-04-17T19:47:55.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f267f000640 1 -- 172.21.15.29:0/433557000 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26781f8fd0 con 0x7f26780c6f70 2024-04-17T19:47:55.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f266fe00640 1 -- 172.21.15.29:0/433557000 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2664003a20 con 0x7f26780c6f70 2024-04-17T19:47:55.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f267f000640 1 -- 172.21.15.29:0/433557000 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26780c4910 con 0x7f26780c6f70 2024-04-17T19:47:55.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.121+0000 7f267f000640 1 -- 172.21.15.29:0/433557000 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26781f62f0 con 0x7f26780c6f70 2024-04-17T19:47:55.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.125+0000 7f266fe00640 1 -- 172.21.15.29:0/433557000 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f26640b3070 con 0x7f26780c6f70 2024-04-17T19:47:55.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.125+0000 7f266fe00640 1 --2- 172.21.15.29:0/433557000 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f26580683b0 0x7f265806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:55.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.125+0000 7f266fe00640 1 -- 172.21.15.29:0/433557000 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2664120a40 con 0x7f26780c6f70 2024-04-17T19:47:55.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.125+0000 7f2677e00640 1 --2- 172.21.15.29:0/433557000 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f26580683b0 0x7f265806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:55.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.125+0000 7f2677e00640 1 --2- 172.21.15.29:0/433557000 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f26580683b0 0x7f265806a8c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f26680011f0 tx=0x7f2668009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:55.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.129+0000 7f266fe00640 1 -- 172.21.15.29:0/433557000 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2664015ff0 con 0x7f26780c6f70 2024-04-17T19:47:55.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.329+0000 7f267f000640 1 -- 172.21.15.29:0/433557000 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2678069b90 con 0x7f26780c6f70 2024-04-17T19:47:55.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.329+0000 7f266fe00640 1 -- 172.21.15.29:0/433557000 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+6816 (secure 0 0 0) 0x7f26640f0cb0 con 0x7f26780c6f70 2024-04-17T19:47:55.319 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:47:55.319 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6838/332907250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":332907250},{"type":"v1","addr":"172.21.15.37:6838","nonce":332907250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":4302,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:47:54.919907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4305,"qdb_cluster":[4305]},"id":1}]} 2024-04-17T19:47:55.320 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 7 2024-04-17T19:47:55.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.333+0000 7f267f000640 1 -- 172.21.15.29:0/433557000 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f26580683b0 msgr2=0x7f265806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:55.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.333+0000 7f267f000640 1 --2- 172.21.15.29:0/433557000 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f26580683b0 0x7f265806a8c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f26680011f0 tx=0x7f2668009040 comp rx=0 tx=0).stop 2024-04-17T19:47:55.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.333+0000 7f267f000640 1 -- 172.21.15.29:0/433557000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26780c6f70 msgr2=0x7f267814f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:55.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.333+0000 7f267f000640 1 --2- 172.21.15.29:0/433557000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26780c6f70 0x7f267814f3d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f26640097f0 tx=0x7f26640191a0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.333+0000 7f267f000640 1 -- 172.21.15.29:0/433557000 shutdown_connections 2024-04-17T19:47:55.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.333+0000 7f267f000640 1 --2- 172.21.15.29:0/433557000 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f26580683b0 0x7f265806a8c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.333+0000 7f267f000640 1 --2- 172.21.15.29:0/433557000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2678159c90 0x7f2678148c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.333+0000 7f267f000640 1 --2- 172.21.15.29:0/433557000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26780c6f70 0x7f267814f3d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.333+0000 7f267f000640 1 --2- 172.21.15.29:0/433557000 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f267806c660 0x7f26780c5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.333+0000 7f267f000640 1 -- 172.21.15.29:0/433557000 >> 172.21.15.29:0/433557000 conn(0x7f267806fde0 msgr2=0x7f26780720e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:55.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.333+0000 7f267f000640 1 -- 172.21.15.29:0/433557000 shutdown_connections 2024-04-17T19:47:55.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.333+0000 7f267f000640 1 -- 172.21.15.29:0/433557000 wait complete. 2024-04-17T19:47:55.342 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:47:55.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.505+0000 7fa173c00640 1 Processor -- start 2024-04-17T19:47:55.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.505+0000 7fa173c00640 1 -- start start 2024-04-17T19:47:55.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.505+0000 7fa173c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa16c0c7f50 0x7fa16c0ca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:55.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.505+0000 7fa173c00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa16c155da0 0x7fa16c1561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:55.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.505+0000 7fa173c00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa16c1566f0 con 0x7fa16c0cbf90 2024-04-17T19:47:55.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.505+0000 7fa173c00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fa16c06bdc0 con 0x7fa16c0c7f50 2024-04-17T19:47:55.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.505+0000 7fa173c00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa16c0cbba0 con 0x7fa16c155da0 2024-04-17T19:47:55.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.505+0000 7fa16be00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa16c0c7f50 0x7fa16c0ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:55.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.505+0000 7fa16be00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa16c0c7f50 0x7fa16c0ca410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51736/0 (socket says 172.21.15.29:51736) 2024-04-17T19:47:55.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.505+0000 7fa16be00640 1 -- 172.21.15.29:0/3600975099 learned_addr learned my addr 172.21.15.29:0/3600975099 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:55.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.505+0000 7fa171c00640 1 --2- 172.21.15.29:0/3600975099 >> v2:172.21.15.37:3300/0 conn(0x7fa16c155da0 0x7fa16c1561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:55.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.505+0000 7fa16be00640 1 -- 172.21.15.29:0/3600975099 >> v2:172.21.15.37:3300/0 conn(0x7fa16c155da0 msgr2=0x7fa16c1561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:55.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.505+0000 7fa16be00640 1 --2- 172.21.15.29:0/3600975099 >> v2:172.21.15.37:3300/0 conn(0x7fa16c155da0 0x7fa16c1561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.505+0000 7fa16be00640 1 -- 172.21.15.29:0/3600975099 >> v1:172.21.15.37:6789/0 conn(0x7fa16c0cbf90 legacy=0x7fa16c15ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:55.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.505+0000 7fa16be00640 1 -- 172.21.15.29:0/3600975099 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16c06c010 con 0x7fa16c0c7f50 2024-04-17T19:47:55.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.509+0000 7fa16be00640 1 --2- 172.21.15.29:0/3600975099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa16c0c7f50 0x7fa16c0ca410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0c56a0 tx=0x7fa15801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e49bfbbde6ef0a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:55.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.509+0000 7fa16b400640 1 -- 172.21.15.29:0/3600975099 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1580ab610 con 0x7fa16c0c7f50 2024-04-17T19:47:55.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.509+0000 7fa16b400640 1 -- 172.21.15.29:0/3600975099 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1580b0070 con 0x7fa16c0c7f50 2024-04-17T19:47:55.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.509+0000 7fa16b400640 1 -- 172.21.15.29:0/3600975099 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1580ab9d0 con 0x7fa16c0c7f50 2024-04-17T19:47:55.499 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.509+0000 7fa173c00640 1 -- 172.21.15.29:0/3600975099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa16c0c7f50 msgr2=0x7fa16c0ca410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:55.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.509+0000 7fa173c00640 1 --2- 172.21.15.29:0/3600975099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa16c0c7f50 0x7fa16c0ca410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0c56a0 tx=0x7fa15801c520 comp rx=0 tx=0).stop 2024-04-17T19:47:55.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.509+0000 7fa173c00640 1 -- 172.21.15.29:0/3600975099 shutdown_connections 2024-04-17T19:47:55.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.509+0000 7fa173c00640 1 --2- 172.21.15.29:0/3600975099 >> v2:172.21.15.37:3300/0 conn(0x7fa16c155da0 0x7fa16c1561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.509+0000 7fa173c00640 1 --2- 172.21.15.29:0/3600975099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa16c0c7f50 0x7fa16c0ca410 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.509+0000 7fa173c00640 1 -- 172.21.15.29:0/3600975099 >> 172.21.15.29:0/3600975099 conn(0x7fa16c06fde0 msgr2=0x7fa16c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:55.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.509+0000 7fa173c00640 1 -- 172.21.15.29:0/3600975099 shutdown_connections 2024-04-17T19:47:55.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.509+0000 7fa173c00640 1 -- 172.21.15.29:0/3600975099 wait complete. 2024-04-17T19:47:55.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.513+0000 7fa173c00640 1 Processor -- start 2024-04-17T19:47:55.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.513+0000 7fa173c00640 1 -- start start 2024-04-17T19:47:55.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.513+0000 7fa173c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa16c0c7f50 0x7fa16c14b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:55.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.513+0000 7fa173c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa16c0cbf90 0x7fa16c156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:55.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.513+0000 7fa173c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa16c155da0 0x7fa16c156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:55.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.513+0000 7fa173c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c0caad0 con 0x7fa16c155da0 2024-04-17T19:47:55.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.513+0000 7fa173c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c0ca950 con 0x7fa16c0cbf90 2024-04-17T19:47:55.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.513+0000 7fa173c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa16c0cb760 con 0x7fa16c0c7f50 2024-04-17T19:47:55.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.513+0000 7fa171c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa16c155da0 0x7fa16c156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:55.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.513+0000 7fa171c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa16c155da0 0x7fa16c156e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51748/0 (socket says 172.21.15.29:51748) 2024-04-17T19:47:55.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.513+0000 7fa171c00640 1 -- 172.21.15.29:0/3499046962 learned_addr learned my addr 172.21.15.29:0/3499046962 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:55.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.513+0000 7fa171200640 1 --2- 172.21.15.29:0/3499046962 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa16c0c7f50 0x7fa16c14b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:55.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.513+0000 7fa16be00640 1 --2- 172.21.15.29:0/3499046962 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa16c0cbf90 0x7fa16c156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.517+0000 7fa171c00640 1 -- 172.21.15.29:0/3499046962 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa16c0c7f50 msgr2=0x7fa16c14b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.517+0000 7fa171c00640 1 --2- 172.21.15.29:0/3499046962 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa16c0c7f50 0x7fa16c14b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.517+0000 7fa171c00640 1 -- 172.21.15.29:0/3499046962 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa16c0cbf90 msgr2=0x7fa16c156910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.517+0000 7fa171c00640 1 --2- 172.21.15.29:0/3499046962 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa16c0cbf90 0x7fa16c156910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.517+0000 7fa171c00640 1 -- 172.21.15.29:0/3499046962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16c15de20 con 0x7fa16c155da0 2024-04-17T19:47:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.517+0000 7fa171c00640 1 --2- 172.21.15.29:0/3499046962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa16c155da0 0x7fa16c156e50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa164009cb0 tx=0x7fa16400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.517+0000 7fa168c00640 1 -- 172.21.15.29:0/3499046962 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164018020 con 0x7fa16c155da0 2024-04-17T19:47:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.517+0000 7fa168c00640 1 -- 172.21.15.29:0/3499046962 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa164014480 con 0x7fa16c155da0 2024-04-17T19:47:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.517+0000 7fa168c00640 1 -- 172.21.15.29:0/3499046962 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164014870 con 0x7fa16c155da0 2024-04-17T19:47:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.517+0000 7fa173c00640 1 -- 172.21.15.29:0/3499046962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa16c1499c0 con 0x7fa16c155da0 2024-04-17T19:47:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.517+0000 7fa173c00640 1 -- 172.21.15.29:0/3499046962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa16c15d340 con 0x7fa16c155da0 2024-04-17T19:47:55.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.517+0000 7fa173c00640 1 -- 172.21.15.29:0/3499046962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa16c069b90 con 0x7fa16c155da0 2024-04-17T19:47:55.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.517+0000 7fa168c00640 1 -- 172.21.15.29:0/3499046962 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa16401b020 con 0x7fa16c155da0 2024-04-17T19:47:55.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.521+0000 7fa168c00640 1 --2- 172.21.15.29:0/3499046962 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa14c068380 0x7fa14c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:55.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.521+0000 7fa168c00640 1 -- 172.21.15.29:0/3499046962 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa164087a00 con 0x7fa16c155da0 2024-04-17T19:47:55.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.521+0000 7fa171200640 1 --2- 172.21.15.29:0/3499046962 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa14c068380 0x7fa14c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:55.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.521+0000 7fa171200640 1 --2- 172.21.15.29:0/3499046962 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa14c068380 0x7fa14c06a890 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa160001770 tx=0x7fa160009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:55.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.525+0000 7fa168c00640 1 -- 172.21.15.29:0/3499046962 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa164022030 con 0x7fa16c155da0 2024-04-17T19:47:55.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.689+0000 7fa173c00640 1 -- 172.21.15.29:0/3499046962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa16c150090 con 0x7fa16c155da0 2024-04-17T19:47:55.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.689+0000 7fa168c00640 1 -- 172.21.15.29:0/3499046962 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18394 (secure 0 0 0) 0x7fa164010040 con 0x7fa16c155da0 2024-04-17T19:47:55.678 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:47:55.678 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:47:54.321794+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T19:47:47.658449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T19:47:48.209047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T19:47:51.887909+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:47:55.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.693+0000 7fa173c00640 1 -- 172.21.15.29:0/3499046962 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa14c068380 msgr2=0x7fa14c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.693+0000 7fa173c00640 1 --2- 172.21.15.29:0/3499046962 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa14c068380 0x7fa14c06a890 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa160001770 tx=0x7fa160009040 comp rx=0 tx=0).stop 2024-04-17T19:47:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.693+0000 7fa173c00640 1 -- 172.21.15.29:0/3499046962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa16c155da0 msgr2=0x7fa16c156e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.693+0000 7fa173c00640 1 --2- 172.21.15.29:0/3499046962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa16c155da0 0x7fa16c156e50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa164009cb0 tx=0x7fa16400c490 comp rx=0 tx=0).stop 2024-04-17T19:47:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.693+0000 7fa173c00640 1 -- 172.21.15.29:0/3499046962 shutdown_connections 2024-04-17T19:47:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.693+0000 7fa173c00640 1 --2- 172.21.15.29:0/3499046962 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa14c068380 0x7fa14c06a890 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.693+0000 7fa173c00640 1 --2- 172.21.15.29:0/3499046962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa16c155da0 0x7fa16c156e50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.693+0000 7fa173c00640 1 --2- 172.21.15.29:0/3499046962 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa16c0cbf90 0x7fa16c156910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.693+0000 7fa173c00640 1 --2- 172.21.15.29:0/3499046962 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa16c0c7f50 0x7fa16c14b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.693+0000 7fa173c00640 1 -- 172.21.15.29:0/3499046962 >> 172.21.15.29:0/3499046962 conn(0x7fa16c06fde0 msgr2=0x7fa16c0cda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.693+0000 7fa173c00640 1 -- 172.21.15.29:0/3499046962 shutdown_connections 2024-04-17T19:47:55.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.693+0000 7fa173c00640 1 -- 172.21.15.29:0/3499046962 wait complete. 2024-04-17T19:47:55.701 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:47:55.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.865+0000 7ff0d3200640 1 Processor -- start 2024-04-17T19:47:55.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.865+0000 7ff0d3200640 1 -- start start 2024-04-17T19:47:55.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.865+0000 7ff0d3200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0d4153c00 0x7ff0d4072200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:55.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.865+0000 7ff0d3200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff0d406d210 0x7ff0d40c7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:55.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.869+0000 7ff0d3200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0d4154010 con 0x7ff0d406d620 2024-04-17T19:47:55.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.869+0000 7ff0d3200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff0d4159c30 con 0x7ff0d406d210 2024-04-17T19:47:55.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.869+0000 7ff0d3200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff0d4071000 con 0x7ff0d4153c00 2024-04-17T19:47:55.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.869+0000 7ff0d1e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0d4153c00 0x7ff0d4072200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:55.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.869+0000 7ff0cbe00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff0d406d210 0x7ff0d40c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:55.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.869+0000 7ff0d2800640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ff0d406d620 0x7ff0d415d110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:37028/0 (socket says 172.21.15.29:37028) 2024-04-17T19:47:55.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.869+0000 7ff0d2800640 1 -- 172.21.15.29:0/3495111579 learned_addr learned my addr 172.21.15.29:0/3495111579 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:47:55.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.869+0000 7ff0cbe00640 1 -- 172.21.15.29:0/3495111579 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0d4153c00 msgr2=0x7ff0d4072200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:55.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.869+0000 7ff0cbe00640 1 --2- 172.21.15.29:0/3495111579 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0d4153c00 0x7ff0d4072200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.869+0000 7ff0cbe00640 1 -- 172.21.15.29:0/3495111579 >> v1:172.21.15.37:6789/0 conn(0x7ff0d406d620 legacy=0x7ff0d415d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:55.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.869+0000 7ff0cbe00640 1 -- 172.21.15.29:0/3495111579 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d4072b20 con 0x7ff0d406d210 2024-04-17T19:47:55.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.869+0000 7ff0d1e00640 1 --2- 172.21.15.29:0/3495111579 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0d4153c00 0x7ff0d4072200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:47:55.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.873+0000 7ff0cbe00640 1 --2- 172.21.15.29:0/3495111579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0d406d210 0x7ff0d40c7980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc0097c0 tx=0x7ff0bc01c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c362e13593d6d2df server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:55.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.873+0000 7ff0d1400640 1 -- 172.21.15.29:0/3495111579 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc0b2070 con 0x7ff0d406d210 2024-04-17T19:47:55.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.873+0000 7ff0d1400640 1 -- 172.21.15.29:0/3495111579 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0bc0ad680 con 0x7ff0d406d210 2024-04-17T19:47:55.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.873+0000 7ff0d1400640 1 -- 172.21.15.29:0/3495111579 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0bc0ada50 con 0x7ff0d406d210 2024-04-17T19:47:55.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.873+0000 7ff0d3200640 1 -- 172.21.15.29:0/3495111579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0d406d210 msgr2=0x7ff0d40c7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:55.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.873+0000 7ff0d3200640 1 --2- 172.21.15.29:0/3495111579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0d406d210 0x7ff0d40c7980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc0097c0 tx=0x7ff0bc01c470 comp rx=0 tx=0).stop 2024-04-17T19:47:55.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.873+0000 7ff0d3200640 1 -- 172.21.15.29:0/3495111579 shutdown_connections 2024-04-17T19:47:55.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.873+0000 7ff0d3200640 1 --2- 172.21.15.29:0/3495111579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0d406d210 0x7ff0d40c7980 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.873+0000 7ff0d3200640 1 --2- 172.21.15.29:0/3495111579 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0d4153c00 0x7ff0d4072200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.873+0000 7ff0d3200640 1 -- 172.21.15.29:0/3495111579 >> 172.21.15.29:0/3495111579 conn(0x7ff0d40ccf80 msgr2=0x7ff0d40cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:55.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.873+0000 7ff0d3200640 1 -- 172.21.15.29:0/3495111579 shutdown_connections 2024-04-17T19:47:55.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.873+0000 7ff0d3200640 1 -- 172.21.15.29:0/3495111579 wait complete. 2024-04-17T19:47:55.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.873+0000 7ff0d3200640 1 Processor -- start 2024-04-17T19:47:55.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.873+0000 7ff0d3200640 1 -- start start 2024-04-17T19:47:55.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d3200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d406d210 0x7ff0d4150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:55.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d3200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0d406d620 0x7ff0d4158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:55.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d3200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0d4153c00 0x7ff0d4159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:55.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d3200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d4072f80 con 0x7ff0d406d210 2024-04-17T19:47:55.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d3200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff0d4072e00 con 0x7ff0d4153c00 2024-04-17T19:47:55.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d3200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff0d415a9b0 con 0x7ff0d406d620 2024-04-17T19:47:55.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d1e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d406d210 0x7ff0d4150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:55.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d1e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d406d210 0x7ff0d4150810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51764/0 (socket says 172.21.15.29:51764) 2024-04-17T19:47:55.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d1e00640 1 -- 172.21.15.29:0/3105750520 learned_addr learned my addr 172.21.15.29:0/3105750520 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:55.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d2800640 1 --2- 172.21.15.29:0/3105750520 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0d4153c00 0x7ff0d4159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:55.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0cbe00640 1 --2- 172.21.15.29:0/3105750520 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0d406d620 0x7ff0d4158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:55.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d1e00640 1 -- 172.21.15.29:0/3105750520 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0d406d620 msgr2=0x7ff0d4158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:55.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d1e00640 1 --2- 172.21.15.29:0/3105750520 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0d406d620 0x7ff0d4158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d1e00640 1 -- 172.21.15.29:0/3105750520 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0d4153c00 msgr2=0x7ff0d4159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:55.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d1e00640 1 --2- 172.21.15.29:0/3105750520 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0d4153c00 0x7ff0d4159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:55.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d1e00640 1 -- 172.21.15.29:0/3105750520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0d4158b50 con 0x7ff0d406d210 2024-04-17T19:47:55.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d1e00640 1 --2- 172.21.15.29:0/3105750520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d406d210 0x7ff0d4150810 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400e9b0 tx=0x7ff0c400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:55.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0ca000640 1 -- 172.21.15.29:0/3105750520 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4017070 con 0x7ff0d406d210 2024-04-17T19:47:55.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0ca000640 1 -- 172.21.15.29:0/3105750520 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0c400ca00 con 0x7ff0d406d210 2024-04-17T19:47:55.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d3200640 1 -- 172.21.15.29:0/3105750520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0d4158450 con 0x7ff0d406d210 2024-04-17T19:47:55.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0ca000640 1 -- 172.21.15.29:0/3105750520 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0c400cdd0 con 0x7ff0d406d210 2024-04-17T19:47:55.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d3200640 1 -- 172.21.15.29:0/3105750520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0d40c2d30 con 0x7ff0d406d210 2024-04-17T19:47:55.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.877+0000 7ff0d3200640 1 -- 172.21.15.29:0/3105750520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0d404ff10 con 0x7ff0d406d210 2024-04-17T19:47:55.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.881+0000 7ff0ca000640 1 -- 172.21.15.29:0/3105750520 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff0c401a070 con 0x7ff0d406d210 2024-04-17T19:47:55.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.881+0000 7ff0ca000640 1 --2- 172.21.15.29:0/3105750520 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff0b0068380 0x7ff0b006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:55.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.881+0000 7ff0ca000640 1 -- 172.21.15.29:0/3105750520 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff0c4086ff0 con 0x7ff0d406d210 2024-04-17T19:47:55.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.881+0000 7ff0cbe00640 1 --2- 172.21.15.29:0/3105750520 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff0b0068380 0x7ff0b006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:55.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.881+0000 7ff0cbe00640 1 --2- 172.21.15.29:0/3105750520 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff0b0068380 0x7ff0b006a890 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc0af000 tx=0x7ff0bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:55.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:55.885+0000 7ff0ca000640 1 -- 172.21.15.29:0/3105750520 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff0c40571d0 con 0x7ff0d406d210 2024-04-17T19:47:56.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.053+0000 7ff0d3200640 1 -- 172.21.15.29:0/3105750520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff0d4069460 con 0x7ff0d406d210 2024-04-17T19:47:56.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.053+0000 7ff0ca000640 1 -- 172.21.15.29:0/3105750520 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18394 (secure 0 0 0) 0x7ff0c4056ad0 con 0x7ff0d406d210 2024-04-17T19:47:56.043 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:47:56.044 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:47:54.321794+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T19:47:47.658449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T19:47:48.209047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T19:47:51.887909+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:47:56.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.057+0000 7ff0d3200640 1 -- 172.21.15.29:0/3105750520 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff0b0068380 msgr2=0x7ff0b006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.057+0000 7ff0d3200640 1 --2- 172.21.15.29:0/3105750520 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff0b0068380 0x7ff0b006a890 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff0bc0af000 tx=0x7ff0bc008040 comp rx=0 tx=0).stop 2024-04-17T19:47:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.057+0000 7ff0d3200640 1 -- 172.21.15.29:0/3105750520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d406d210 msgr2=0x7ff0d4150810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.057+0000 7ff0d3200640 1 --2- 172.21.15.29:0/3105750520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d406d210 0x7ff0d4150810 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff0c400e9b0 tx=0x7ff0c400ee70 comp rx=0 tx=0).stop 2024-04-17T19:47:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.057+0000 7ff0d3200640 1 -- 172.21.15.29:0/3105750520 shutdown_connections 2024-04-17T19:47:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.057+0000 7ff0d3200640 1 --2- 172.21.15.29:0/3105750520 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff0b0068380 0x7ff0b006a890 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.057+0000 7ff0d3200640 1 --2- 172.21.15.29:0/3105750520 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff0d4153c00 0x7ff0d4159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.057+0000 7ff0d3200640 1 --2- 172.21.15.29:0/3105750520 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff0d406d620 0x7ff0d4158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.057+0000 7ff0d3200640 1 --2- 172.21.15.29:0/3105750520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d406d210 0x7ff0d4150810 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.057+0000 7ff0d3200640 1 -- 172.21.15.29:0/3105750520 >> 172.21.15.29:0/3105750520 conn(0x7ff0d40ccf80 msgr2=0x7ff0d4143ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.057+0000 7ff0d3200640 1 -- 172.21.15.29:0/3105750520 shutdown_connections 2024-04-17T19:47:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.057+0000 7ff0d3200640 1 -- 172.21.15.29:0/3105750520 wait complete. 2024-04-17T19:47:56.079 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:47:56.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.241+0000 7f41f7600640 1 Processor -- start 2024-04-17T19:47:56.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.241+0000 7f41f7600640 1 -- start start 2024-04-17T19:47:56.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.241+0000 7f41f7600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41f006ced0 0x7f41f006e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:56.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.241+0000 7f41f7600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41f01563a0 0x7f41f015fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:56.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.241+0000 7f41f7600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41f00cba40 con 0x7f41f0160170 2024-04-17T19:47:56.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.241+0000 7f41f7600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f41f00715a0 con 0x7f41f01563a0 2024-04-17T19:47:56.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.241+0000 7f41f7600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f41f00cab20 con 0x7f41f006ced0 2024-04-17T19:47:56.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.241+0000 7f41efe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41f01563a0 0x7f41f015fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:56.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.241+0000 7f41f5600640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f41f0160170 0x7f41f00cc1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33832/0 (socket says 172.21.15.29:33832) 2024-04-17T19:47:56.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.245+0000 7f41f5600640 1 -- 172.21.15.29:0/2748628734 learned_addr learned my addr 172.21.15.29:0/2748628734 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:47:56.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.245+0000 7f41f4c00640 1 --2- 172.21.15.29:0/2748628734 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41f006ced0 0x7f41f006e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:56.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.245+0000 7f41efe00640 1 -- 172.21.15.29:0/2748628734 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41f006ced0 msgr2=0x7f41f006e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:56.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.245+0000 7f41efe00640 1 --2- 172.21.15.29:0/2748628734 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41f006ced0 0x7f41f006e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.245+0000 7f41efe00640 1 -- 172.21.15.29:0/2748628734 >> v1:172.21.15.29:6789/0 conn(0x7f41f0160170 legacy=0x7f41f00cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:56.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.245+0000 7f41efe00640 1 -- 172.21.15.29:0/2748628734 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f00ca560 con 0x7f41f01563a0 2024-04-17T19:47:56.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.245+0000 7f41ef400640 1 -- 172.21.15.29:0/2748628734 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2189929345 0 0) 0x7f41f00cba40 con 0x7f41f0160170 2024-04-17T19:47:56.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.245+0000 7f41f4c00640 1 --2- 172.21.15.29:0/2748628734 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41f006ced0 0x7f41f006e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:56.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.245+0000 7f41efe00640 1 --2- 172.21.15.29:0/2748628734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41f01563a0 0x7f41f015fc30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0097c0 tx=0x7f41dc01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d6020228e443a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:56.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.245+0000 7f41ef400640 1 -- 172.21.15.29:0/2748628734 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc0b0070 con 0x7f41f01563a0 2024-04-17T19:47:56.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.245+0000 7f41ef400640 1 -- 172.21.15.29:0/2748628734 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41dc0ab600 con 0x7f41f01563a0 2024-04-17T19:47:56.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.245+0000 7f41ef400640 1 -- 172.21.15.29:0/2748628734 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41dc0ab9d0 con 0x7f41f01563a0 2024-04-17T19:47:56.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.249+0000 7f41f7600640 1 -- 172.21.15.29:0/2748628734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41f01563a0 msgr2=0x7f41f015fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:56.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.249+0000 7f41f7600640 1 --2- 172.21.15.29:0/2748628734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41f01563a0 0x7f41f015fc30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0097c0 tx=0x7f41dc01c4a0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.249+0000 7f41f7600640 1 -- 172.21.15.29:0/2748628734 shutdown_connections 2024-04-17T19:47:56.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.249+0000 7f41f7600640 1 --2- 172.21.15.29:0/2748628734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41f01563a0 0x7f41f015fc30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.249+0000 7f41f7600640 1 --2- 172.21.15.29:0/2748628734 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41f006ced0 0x7f41f006e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.249+0000 7f41f7600640 1 -- 172.21.15.29:0/2748628734 >> 172.21.15.29:0/2748628734 conn(0x7f41f0071ca0 msgr2=0x7f41f00720e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:56.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.249+0000 7f41f7600640 1 -- 172.21.15.29:0/2748628734 shutdown_connections 2024-04-17T19:47:56.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.249+0000 7f41f7600640 1 -- 172.21.15.29:0/2748628734 wait complete. 2024-04-17T19:47:56.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.249+0000 7f41f7600640 1 Processor -- start 2024-04-17T19:47:56.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.249+0000 7f41f7600640 1 -- start start 2024-04-17T19:47:56.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.249+0000 7f41f7600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41f006ced0 0x7f41f00ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:56.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.249+0000 7f41f7600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41f01563a0 0x7f41f00c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:56.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.249+0000 7f41f7600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41f0160170 0x7f41f00c8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:56.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41f7600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f41f00cca80 con 0x7f41f0160170 2024-04-17T19:47:56.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41f7600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f41f00cc900 con 0x7f41f006ced0 2024-04-17T19:47:56.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41f7600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f41f0169130 con 0x7f41f01563a0 2024-04-17T19:47:56.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41f5600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41f0160170 0x7f41f00c8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:56.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41f5600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41f0160170 0x7f41f00c8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51772/0 (socket says 172.21.15.29:51772) 2024-04-17T19:47:56.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41f5600640 1 -- 172.21.15.29:0/3780886186 learned_addr learned my addr 172.21.15.29:0/3780886186 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:56.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41efe00640 1 --2- 172.21.15.29:0/3780886186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41f01563a0 0x7f41f00c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:56.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41f4c00640 1 --2- 172.21.15.29:0/3780886186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41f006ced0 0x7f41f00ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:56.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41f5600640 1 -- 172.21.15.29:0/3780886186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41f01563a0 msgr2=0x7f41f00c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:56.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41f5600640 1 --2- 172.21.15.29:0/3780886186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41f01563a0 0x7f41f00c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41f5600640 1 -- 172.21.15.29:0/3780886186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41f006ced0 msgr2=0x7f41f00ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:56.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41f5600640 1 --2- 172.21.15.29:0/3780886186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41f006ced0 0x7f41f00ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41f5600640 1 -- 172.21.15.29:0/3780886186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f01f6400 con 0x7f41f0160170 2024-04-17T19:47:56.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41f5600640 1 --2- 172.21.15.29:0/3780886186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41f0160170 0x7f41f00c8d30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f41e800c700 tx=0x7f41e800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:56.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41ecc00640 1 -- 172.21.15.29:0/3780886186 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e8018070 con 0x7f41f0160170 2024-04-17T19:47:56.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41ecc00640 1 -- 172.21.15.29:0/3780886186 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41e800ce10 con 0x7f41f0160170 2024-04-17T19:47:56.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41ecc00640 1 -- 172.21.15.29:0/3780886186 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e800a880 con 0x7f41f0160170 2024-04-17T19:47:56.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41f7600640 1 -- 172.21.15.29:0/3780886186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41f00ce580 con 0x7f41f0160170 2024-04-17T19:47:56.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41f7600640 1 -- 172.21.15.29:0/3780886186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41f01f6120 con 0x7f41f0160170 2024-04-17T19:47:56.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41f7600640 1 -- 172.21.15.29:0/3780886186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41f0069b90 con 0x7f41f0160170 2024-04-17T19:47:56.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.253+0000 7f41ecc00640 1 -- 172.21.15.29:0/3780886186 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f41e800aa20 con 0x7f41f0160170 2024-04-17T19:47:56.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.257+0000 7f41ecc00640 1 --2- 172.21.15.29:0/3780886186 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f41d00683b0 0x7f41d006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:56.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.257+0000 7f41ecc00640 1 -- 172.21.15.29:0/3780886186 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f41e8086210 con 0x7f41f0160170 2024-04-17T19:47:56.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.257+0000 7f41f4c00640 1 --2- 172.21.15.29:0/3780886186 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f41d00683b0 0x7f41d006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:56.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.257+0000 7f41f4c00640 1 --2- 172.21.15.29:0/3780886186 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f41d00683b0 0x7f41d006a8c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f41e400ba10 tx=0x7f41e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:56.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.261+0000 7f41ecc00640 1 -- 172.21.15.29:0/3780886186 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f41e80563f0 con 0x7f41f0160170 2024-04-17T19:47:56.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.461+0000 7f41f7600640 1 -- 172.21.15.29:0/3780886186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41f0155c60 con 0x7f41f0160170 2024-04-17T19:47:56.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.461+0000 7f41ecc00640 1 -- 172.21.15.29:0/3780886186 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+6816 (secure 0 0 0) 0x7f41e8055cf0 con 0x7f41f0160170 2024-04-17T19:47:56.449 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:47:56.449 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6838/332907250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":332907250},{"type":"v1","addr":"172.21.15.37:6838","nonce":332907250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":4302,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:47:54.919907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4305,"qdb_cluster":[4305]},"id":1}]} 2024-04-17T19:47:56.449 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 7 2024-04-17T19:47:56.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.461+0000 7f41f7600640 1 -- 172.21.15.29:0/3780886186 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f41d00683b0 msgr2=0x7f41d006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:56.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.461+0000 7f41f7600640 1 --2- 172.21.15.29:0/3780886186 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f41d00683b0 0x7f41d006a8c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f41e400ba10 tx=0x7f41e4008040 comp rx=0 tx=0).stop 2024-04-17T19:47:56.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.461+0000 7f41f7600640 1 -- 172.21.15.29:0/3780886186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41f0160170 msgr2=0x7f41f00c8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:56.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.461+0000 7f41f7600640 1 --2- 172.21.15.29:0/3780886186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41f0160170 0x7f41f00c8d30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f41e800c700 tx=0x7f41e800cbc0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.461+0000 7f41f7600640 1 -- 172.21.15.29:0/3780886186 shutdown_connections 2024-04-17T19:47:56.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.461+0000 7f41f7600640 1 --2- 172.21.15.29:0/3780886186 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f41d00683b0 0x7f41d006a8c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.461+0000 7f41f7600640 1 --2- 172.21.15.29:0/3780886186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41f0160170 0x7f41f00c8d30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.461+0000 7f41f7600640 1 --2- 172.21.15.29:0/3780886186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f41f01563a0 0x7f41f00c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.461+0000 7f41f7600640 1 --2- 172.21.15.29:0/3780886186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f41f006ced0 0x7f41f00ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.461+0000 7f41f7600640 1 -- 172.21.15.29:0/3780886186 >> 172.21.15.29:0/3780886186 conn(0x7f41f0071ca0 msgr2=0x7f41f00720b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:56.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.461+0000 7f41f7600640 1 -- 172.21.15.29:0/3780886186 shutdown_connections 2024-04-17T19:47:56.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.461+0000 7f41f7600640 1 -- 172.21.15.29:0/3780886186 wait complete. 2024-04-17T19:47:56.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:47:56.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.633+0000 7f7623800640 1 Processor -- start 2024-04-17T19:47:56.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.633+0000 7f7623800640 1 -- start start 2024-04-17T19:47:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.637+0000 7f7623800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f761c0cc8c0 0x7f761c0cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.637+0000 7f7623800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f761c159390 0x7f761c1597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.637+0000 7f7623800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f761c0ca330 0x7f761c0ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.637+0000 7f7623800640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f761c0c88b0 con 0x7f761c0cc8c0 2024-04-17T19:47:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.637+0000 7f7623800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f761c0c6ce0 con 0x7f761c0ca330 2024-04-17T19:47:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.637+0000 7f7623800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f761c0c6fc0 con 0x7f761c159390 2024-04-17T19:47:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.637+0000 7f7620e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f761c0cc8c0 0x7f761c0cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:56.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.637+0000 7f7620e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f761c0cc8c0 0x7f761c0cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51786/0 (socket says 172.21.15.29:51786) 2024-04-17T19:47:56.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.637+0000 7f7620e00640 1 -- 172.21.15.29:0/3576272495 learned_addr learned my addr 172.21.15.29:0/3576272495 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:56.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.637+0000 7f761be00640 1 --2- 172.21.15.29:0/3576272495 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f761c159390 0x7f761c1597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:56.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.637+0000 7f7621800640 1 --2- 172.21.15.29:0/3576272495 >> v2:172.21.15.37:3300/0 conn(0x7f761c0ca330 0x7f761c0ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:56.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.637+0000 7f7620e00640 1 -- 172.21.15.29:0/3576272495 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f761c159390 msgr2=0x7f761c1597a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:56.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.637+0000 7f7620e00640 1 --2- 172.21.15.29:0/3576272495 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f761c159390 0x7f761c1597a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.637+0000 7f7620e00640 1 -- 172.21.15.29:0/3576272495 >> v2:172.21.15.37:3300/0 conn(0x7f761c0ca330 msgr2=0x7f761c0ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:56.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.637+0000 7f7620e00640 1 --2- 172.21.15.29:0/3576272495 >> v2:172.21.15.37:3300/0 conn(0x7f761c0ca330 0x7f761c0ca740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.637+0000 7f7620e00640 1 -- 172.21.15.29:0/3576272495 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f761c0c72a0 con 0x7f761c0cc8c0 2024-04-17T19:47:56.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.641+0000 7f7620e00640 1 --2- 172.21.15.29:0/3576272495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0cc8c0 0x7f761c0cccd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f761c071e60 tx=0x7f761001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=936b6d1795e2f20d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:56.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.641+0000 7f761b400640 1 -- 172.21.15.29:0/3576272495 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76100b0070 con 0x7f761c0cc8c0 2024-04-17T19:47:56.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.641+0000 7f761b400640 1 -- 172.21.15.29:0/3576272495 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76100ab600 con 0x7f761c0cc8c0 2024-04-17T19:47:56.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.641+0000 7f761b400640 1 -- 172.21.15.29:0/3576272495 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76100ab9d0 con 0x7f761c0cc8c0 2024-04-17T19:47:56.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.641+0000 7f7623800640 1 -- 172.21.15.29:0/3576272495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0cc8c0 msgr2=0x7f761c0cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:56.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.641+0000 7f7623800640 1 --2- 172.21.15.29:0/3576272495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0cc8c0 0x7f761c0cccd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f761c071e60 tx=0x7f761001c520 comp rx=0 tx=0).stop 2024-04-17T19:47:56.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.641+0000 7f7623800640 1 -- 172.21.15.29:0/3576272495 shutdown_connections 2024-04-17T19:47:56.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.641+0000 7f7623800640 1 --2- 172.21.15.29:0/3576272495 >> v2:172.21.15.37:3300/0 conn(0x7f761c0ca330 0x7f761c0ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.641+0000 7f7623800640 1 --2- 172.21.15.29:0/3576272495 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f761c159390 0x7f761c1597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.641+0000 7f7623800640 1 --2- 172.21.15.29:0/3576272495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0cc8c0 0x7f761c0cccd0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.641+0000 7f7623800640 1 -- 172.21.15.29:0/3576272495 >> 172.21.15.29:0/3576272495 conn(0x7f761c06fde0 msgr2=0x7f761c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:56.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.641+0000 7f7623800640 1 -- 172.21.15.29:0/3576272495 shutdown_connections 2024-04-17T19:47:56.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.641+0000 7f7623800640 1 -- 172.21.15.29:0/3576272495 wait complete. 2024-04-17T19:47:56.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f7623800640 1 Processor -- start 2024-04-17T19:47:56.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f7623800640 1 -- start start 2024-04-17T19:47:56.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f7623800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f761c0ca330 0x7f761c156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:56.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f7623800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0cc8c0 0x7f761c1f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:56.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f7623800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f761c159390 0x7f761c140b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f7623800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f761c15ca60 con 0x7f761c0cc8c0 2024-04-17T19:47:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f7623800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f761c0c6a90 con 0x7f761c159390 2024-04-17T19:47:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f7623800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f761c15cbe0 con 0x7f761c0ca330 2024-04-17T19:47:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f761be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0cc8c0 0x7f761c1f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f761be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0cc8c0 0x7f761c1f7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51794/0 (socket says 172.21.15.29:51794) 2024-04-17T19:47:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f761be00640 1 -- 172.21.15.29:0/613444038 learned_addr learned my addr 172.21.15.29:0/613444038 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f7620e00640 1 --2- 172.21.15.29:0/613444038 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f761c0ca330 0x7f761c156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f7621800640 1 --2- 172.21.15.29:0/613444038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f761c159390 0x7f761c140b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f761be00640 1 -- 172.21.15.29:0/613444038 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f761c0ca330 msgr2=0x7f761c156ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f761be00640 1 --2- 172.21.15.29:0/613444038 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f761c0ca330 0x7f761c156ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f761be00640 1 -- 172.21.15.29:0/613444038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f761c159390 msgr2=0x7f761c140b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f761be00640 1 --2- 172.21.15.29:0/613444038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f761c159390 0x7f761c140b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f761be00640 1 -- 172.21.15.29:0/613444038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f761c071fa0 con 0x7f761c0cc8c0 2024-04-17T19:47:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f7621800640 1 --2- 172.21.15.29:0/613444038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f761c159390 0x7f761c140b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f7620e00640 1 --2- 172.21.15.29:0/613444038 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f761c0ca330 0x7f761c156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:56.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f761be00640 1 --2- 172.21.15.29:0/613444038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0cc8c0 0x7f761c1f7330 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f760800d7d0 tx=0x7f760800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:56.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f7618c00640 1 -- 172.21.15.29:0/613444038 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760800b6c0 con 0x7f761c0cc8c0 2024-04-17T19:47:56.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f7618c00640 1 -- 172.21.15.29:0/613444038 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7608019070 con 0x7f761c0cc8c0 2024-04-17T19:47:56.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f7623800640 1 -- 172.21.15.29:0/613444038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f761c1f7870 con 0x7f761c0cc8c0 2024-04-17T19:47:56.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f7618c00640 1 -- 172.21.15.29:0/613444038 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760800bac0 con 0x7f761c0cc8c0 2024-04-17T19:47:56.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.645+0000 7f7623800640 1 -- 172.21.15.29:0/613444038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f761c13c150 con 0x7f761c0cc8c0 2024-04-17T19:47:56.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.649+0000 7f7623800640 1 -- 172.21.15.29:0/613444038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f761c069b90 con 0x7f761c0cc8c0 2024-04-17T19:47:56.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.649+0000 7f7618c00640 1 -- 172.21.15.29:0/613444038 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f760800bc60 con 0x7f761c0cc8c0 2024-04-17T19:47:56.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.649+0000 7f7618c00640 1 --2- 172.21.15.29:0/613444038 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f75fc0683b0 0x7f75fc06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:56.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.649+0000 7f7618c00640 1 -- 172.21.15.29:0/613444038 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7608087150 con 0x7f761c0cc8c0 2024-04-17T19:47:56.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.649+0000 7f7620e00640 1 --2- 172.21.15.29:0/613444038 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f75fc0683b0 0x7f75fc06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:56.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.649+0000 7f7620e00640 1 --2- 172.21.15.29:0/613444038 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f75fc0683b0 0x7f75fc06a8c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f76100ad000 tx=0x7f7610008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:56.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.653+0000 7f7618c00640 1 -- 172.21.15.29:0/613444038 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f76080573c0 con 0x7f761c0cc8c0 2024-04-17T19:47:56.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.817+0000 7f7623800640 1 -- 172.21.15.29:0/613444038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f761c0cb790 con 0x7f761c0cc8c0 2024-04-17T19:47:56.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.821+0000 7f7618c00640 1 -- 172.21.15.29:0/613444038 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18394 (secure 0 0 0) 0x7f7608056cc0 con 0x7f761c0cc8c0 2024-04-17T19:47:56.809 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:47:56.809 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:47:54.321794+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T19:47:47.658449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T19:47:48.209047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T19:47:51.887909+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:47:56.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.821+0000 7f7623800640 1 -- 172.21.15.29:0/613444038 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f75fc0683b0 msgr2=0x7f75fc06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:56.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.821+0000 7f7623800640 1 --2- 172.21.15.29:0/613444038 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f75fc0683b0 0x7f75fc06a8c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f76100ad000 tx=0x7f7610008040 comp rx=0 tx=0).stop 2024-04-17T19:47:56.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.821+0000 7f7623800640 1 -- 172.21.15.29:0/613444038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0cc8c0 msgr2=0x7f761c1f7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:56.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.821+0000 7f7623800640 1 --2- 172.21.15.29:0/613444038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0cc8c0 0x7f761c1f7330 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f760800d7d0 tx=0x7f760800dc90 comp rx=0 tx=0).stop 2024-04-17T19:47:56.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.821+0000 7f7623800640 1 -- 172.21.15.29:0/613444038 shutdown_connections 2024-04-17T19:47:56.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.821+0000 7f7623800640 1 --2- 172.21.15.29:0/613444038 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f75fc0683b0 0x7f75fc06a8c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.821+0000 7f7623800640 1 --2- 172.21.15.29:0/613444038 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f761c159390 0x7f761c140b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.821+0000 7f7623800640 1 --2- 172.21.15.29:0/613444038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f761c0cc8c0 0x7f761c1f7330 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.821+0000 7f7623800640 1 --2- 172.21.15.29:0/613444038 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f761c0ca330 0x7f761c156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:56.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.821+0000 7f7623800640 1 -- 172.21.15.29:0/613444038 >> 172.21.15.29:0/613444038 conn(0x7f761c06fde0 msgr2=0x7f761c06bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:56.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.821+0000 7f7623800640 1 -- 172.21.15.29:0/613444038 shutdown_connections 2024-04-17T19:47:56.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:56.821+0000 7f7623800640 1 -- 172.21.15.29:0/613444038 wait complete. 2024-04-17T19:47:57.435 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T19:47:49.315515+0000', 'modified': '2024-04-17T19:47:54.919907+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4305}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4305': {'gid': 4305, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.29:6837/2691590158', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6833', 'nonce': 2691590158}, {'type': 'v1', 'addr': '172.21.15.29:6837', 'nonce': 2691590158}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4305, 'qdb_cluster': [4305]} 2024-04-17T19:47:57.435 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T19:47:57.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-17T19:47:57.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.605+0000 7fc872200640 1 Processor -- start 2024-04-17T19:47:57.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.605+0000 7fc872200640 1 -- start start 2024-04-17T19:47:57.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.609+0000 7fc872200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc86c1541c0 0x7fc86c0c5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:57.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.609+0000 7fc872200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc86c0c4b70 con 0x7fc86c0cbf80 2024-04-17T19:47:57.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.609+0000 7fc872200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc86c0ca210 con 0x7fc86c0cc390 2024-04-17T19:47:57.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.609+0000 7fc872200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fc86c0ca790 con 0x7fc86c1541c0 2024-04-17T19:47:57.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.609+0000 7fc86b400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc86c1541c0 0x7fc86c0c5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:57.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.609+0000 7fc863e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc86c0cbf80 0x7fc86c15a4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33842/0 (socket says 172.21.15.29:33842) 2024-04-17T19:47:57.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.609+0000 7fc863e00640 1 -- 172.21.15.29:0/3275777996 learned_addr learned my addr 172.21.15.29:0/3275777996 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:47:57.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.609+0000 7fc86b400640 1 -- 172.21.15.29:0/3275777996 >> v1:172.21.15.37:6789/0 conn(0x7fc86c0cc390 legacy=0x7fc86c164790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:57.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.609+0000 7fc86b400640 1 -- 172.21.15.29:0/3275777996 >> v1:172.21.15.29:6789/0 conn(0x7fc86c0cbf80 legacy=0x7fc86c15a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:57.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.609+0000 7fc86aa00640 1 -- 172.21.15.29:0/3275777996 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2800205230 0 0) 0x7fc86c0c4b70 con 0x7fc86c0cbf80 2024-04-17T19:47:57.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.609+0000 7fc86b400640 1 -- 172.21.15.29:0/3275777996 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86c0ca4b0 con 0x7fc86c1541c0 2024-04-17T19:47:57.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.609+0000 7fc86b400640 1 --2- 172.21.15.29:0/3275777996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc86c1541c0 0x7fc86c0c5870 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc86c0ca3e0 tx=0x7fc85c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bdcf3e27eaec8a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:57.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.609+0000 7fc86aa00640 1 -- 172.21.15.29:0/3275777996 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c0b0070 con 0x7fc86c1541c0 2024-04-17T19:47:57.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.613+0000 7fc86aa00640 1 -- 172.21.15.29:0/3275777996 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc85c0ab600 con 0x7fc86c1541c0 2024-04-17T19:47:57.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.613+0000 7fc86aa00640 1 -- 172.21.15.29:0/3275777996 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c0ab9d0 con 0x7fc86c1541c0 2024-04-17T19:47:57.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.613+0000 7fc872200640 1 -- 172.21.15.29:0/3275777996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc86c1541c0 msgr2=0x7fc86c0c5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:57.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.613+0000 7fc872200640 1 --2- 172.21.15.29:0/3275777996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc86c1541c0 0x7fc86c0c5870 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc86c0ca3e0 tx=0x7fc85c01c410 comp rx=0 tx=0).stop 2024-04-17T19:47:57.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.613+0000 7fc872200640 1 -- 172.21.15.29:0/3275777996 shutdown_connections 2024-04-17T19:47:57.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.613+0000 7fc872200640 1 --2- 172.21.15.29:0/3275777996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc86c1541c0 0x7fc86c0c5870 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:57.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.613+0000 7fc872200640 1 -- 172.21.15.29:0/3275777996 >> 172.21.15.29:0/3275777996 conn(0x7fc86c071a60 msgr2=0x7fc86c071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:57.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.613+0000 7fc872200640 1 -- 172.21.15.29:0/3275777996 shutdown_connections 2024-04-17T19:47:57.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.613+0000 7fc872200640 1 -- 172.21.15.29:0/3275777996 wait complete. 2024-04-17T19:47:57.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.613+0000 7fc872200640 1 Processor -- start 2024-04-17T19:47:57.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.613+0000 7fc872200640 1 -- start start 2024-04-17T19:47:57.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc872200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc86c0cbf80 0x7fc86c06be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:57.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc872200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc86c0cc390 0x7fc86c15b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:57.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc872200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc86c1541c0 0x7fc86c146690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:57.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc872200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc86c0cc7a0 con 0x7fc86c0cc390 2024-04-17T19:47:57.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc872200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc86c0c5db0 con 0x7fc86c1541c0 2024-04-17T19:47:57.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc872200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc86c0caad0 con 0x7fc86c0cbf80 2024-04-17T19:47:57.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc863e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc86c0cc390 0x7fc86c15b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:57.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc863e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc86c0cc390 0x7fc86c15b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51808/0 (socket says 172.21.15.29:51808) 2024-04-17T19:47:57.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc863e00640 1 -- 172.21.15.29:0/3807274695 learned_addr learned my addr 172.21.15.29:0/3807274695 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:57.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc86b400640 1 --2- 172.21.15.29:0/3807274695 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc86c0cbf80 0x7fc86c06be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:57.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc86be00640 1 --2- 172.21.15.29:0/3807274695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc86c1541c0 0x7fc86c146690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:57.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc863e00640 1 -- 172.21.15.29:0/3807274695 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc86c0cbf80 msgr2=0x7fc86c06be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:57.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc863e00640 1 --2- 172.21.15.29:0/3807274695 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc86c0cbf80 0x7fc86c06be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:57.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc863e00640 1 -- 172.21.15.29:0/3807274695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc86c1541c0 msgr2=0x7fc86c146690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:57.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc863e00640 1 --2- 172.21.15.29:0/3807274695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc86c1541c0 0x7fc86c146690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:57.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc863e00640 1 -- 172.21.15.29:0/3807274695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86c1f58f0 con 0x7fc86c0cc390 2024-04-17T19:47:57.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc863e00640 1 --2- 172.21.15.29:0/3807274695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc86c0cc390 0x7fc86c15b9e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc85400c440 tx=0x7fc85400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:57.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc863400640 1 -- 172.21.15.29:0/3807274695 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc854018070 con 0x7fc86c0cc390 2024-04-17T19:47:57.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc863400640 1 -- 172.21.15.29:0/3807274695 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8540074e0 con 0x7fc86c0cc390 2024-04-17T19:47:57.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc863400640 1 -- 172.21.15.29:0/3807274695 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc854004a10 con 0x7fc86c0cc390 2024-04-17T19:47:57.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc872200640 1 -- 172.21.15.29:0/3807274695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc86c1f5bd0 con 0x7fc86c0cc390 2024-04-17T19:47:57.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.617+0000 7fc872200640 1 -- 172.21.15.29:0/3807274695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc86c14cc40 con 0x7fc86c0cc390 2024-04-17T19:47:57.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc863400640 1 -- 172.21.15.29:0/3807274695 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc854004bb0 con 0x7fc86c0cc390 2024-04-17T19:47:57.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc863400640 1 --2- 172.21.15.29:0/3807274695 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc848068380 0x7fc84806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:57.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc863400640 1 -- 172.21.15.29:0/3807274695 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc854085ef0 con 0x7fc86c0cc390 2024-04-17T19:47:57.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc86b400640 1 --2- 172.21.15.29:0/3807274695 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc848068380 0x7fc84806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:57.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc879964000 1 Processor -- start 2024-04-17T19:47:57.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc879964000 1 -- start start 2024-04-17T19:47:57.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc86b400640 1 --2- 172.21.15.29:0/3807274695 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc848068380 0x7fc84806a890 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc85c0ad000 tx=0x7fc85c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:57.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc879964000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a979c10 0x55ed2aae31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:57.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc879964000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a972930 0x55ed2aae36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:57.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc879964000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a572280 0x55ed2aae3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:57.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc879964000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ed2a8f04f0 con 0x55ed2a979c10 2024-04-17T19:47:57.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc879964000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x55ed2a88ac10 con 0x55ed2a572280 2024-04-17T19:47:57.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc879964000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x55ed2a8ec4b0 con 0x55ed2a972930 2024-04-17T19:47:57.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc86b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a979c10 0x55ed2aae31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:57.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc86b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a979c10 0x55ed2aae31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51810/0 (socket says 172.21.15.29:51810) 2024-04-17T19:47:57.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc86b400640 1 -- 172.21.15.29:0/396337545 learned_addr learned my addr 172.21.15.29:0/396337545 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:57.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc863e00640 1 --2- 172.21.15.29:0/396337545 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a972930 0x55ed2aae36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:57.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc86be00640 1 --2- 172.21.15.29:0/396337545 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a572280 0x55ed2aae3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:57.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc86b400640 1 -- 172.21.15.29:0/396337545 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a972930 msgr2=0x55ed2aae36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:57.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc86b400640 1 --2- 172.21.15.29:0/396337545 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a972930 0x55ed2aae36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:57.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc86b400640 1 -- 172.21.15.29:0/396337545 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a572280 msgr2=0x55ed2aae3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:57.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc86b400640 1 --2- 172.21.15.29:0/396337545 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a572280 0x55ed2aae3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:57.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc86b400640 1 -- 172.21.15.29:0/396337545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed2a88ae60 con 0x55ed2a979c10 2024-04-17T19:47:57.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.621+0000 7fc863e00640 1 --2- 172.21.15.29:0/396337545 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a972930 0x55ed2aae36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:57.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc86b400640 1 --2- 172.21.15.29:0/396337545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a979c10 0x55ed2aae31b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc85c01a040 tx=0x7fc85c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ffd59aa641e8d40 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:57.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc85b400640 1 -- 172.21.15.29:0/396337545 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c0c6070 con 0x55ed2a979c10 2024-04-17T19:47:57.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc85b400640 1 -- 172.21.15.29:0/396337545 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc85c0b6470 con 0x55ed2a979c10 2024-04-17T19:47:57.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc85b400640 1 -- 172.21.15.29:0/396337545 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c0b0040 con 0x55ed2a979c10 2024-04-17T19:47:57.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc879964000 1 -- 172.21.15.29:0/396337545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a979c10 msgr2=0x55ed2aae31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:57.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc879964000 1 --2- 172.21.15.29:0/396337545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a979c10 0x55ed2aae31b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc85c01a040 tx=0x7fc85c004740 comp rx=0 tx=0).stop 2024-04-17T19:47:57.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc879964000 1 -- 172.21.15.29:0/396337545 shutdown_connections 2024-04-17T19:47:57.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc879964000 1 --2- 172.21.15.29:0/396337545 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a572280 0x55ed2aae3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:57.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc879964000 1 --2- 172.21.15.29:0/396337545 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a972930 0x55ed2aae36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:57.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc879964000 1 --2- 172.21.15.29:0/396337545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a979c10 0x55ed2aae31b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:57.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc879964000 1 -- 172.21.15.29:0/396337545 >> 172.21.15.29:0/396337545 conn(0x55ed2a9bb670 msgr2=0x55ed2aae0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:57.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc879964000 1 -- 172.21.15.29:0/396337545 shutdown_connections 2024-04-17T19:47:57.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc879964000 1 -- 172.21.15.29:0/396337545 wait complete. 2024-04-17T19:47:57.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc879964000 1 -- start start 2024-04-17T19:47:57.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc879964000 1 Processor -- start 2024-04-17T19:47:57.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc879964000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a572280 0x55ed2aadcf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:57.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc879964000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a972930 0x55ed2aadd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:57.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc879964000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a979c10 0x55ed2aaa4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:57.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc879964000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ed2a9319e0 con 0x55ed2a972930 2024-04-17T19:47:57.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc879964000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x55ed2a8ada80 con 0x55ed2a979c10 2024-04-17T19:47:57.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc879964000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x55ed2a8abd10 con 0x55ed2a572280 2024-04-17T19:47:57.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.625+0000 7fc879964000 10 client.0 _ll_register_callbacks cb 0x55ed2aa8db20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T19:47:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc863e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a972930 0x55ed2aadd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc863e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a972930 0x55ed2aadd450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51826/0 (socket says 172.21.15.29:51826) 2024-04-17T19:47:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc863e00640 1 -- 172.21.15.29:0/1261143242 learned_addr learned my addr 172.21.15.29:0/1261143242 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc86be00640 1 --2- 172.21.15.29:0/1261143242 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a979c10 0x55ed2aaa4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc86b400640 1 --2- 172.21.15.29:0/1261143242 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a572280 0x55ed2aadcf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc863e00640 1 -- 172.21.15.29:0/1261143242 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a572280 msgr2=0x55ed2aadcf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc863e00640 1 --2- 172.21.15.29:0/1261143242 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a572280 0x55ed2aadcf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc863e00640 1 -- 172.21.15.29:0/1261143242 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a979c10 msgr2=0x55ed2aaa4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc863e00640 1 --2- 172.21.15.29:0/1261143242 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a979c10 0x55ed2aaa4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc863e00640 1 -- 172.21.15.29:0/1261143242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed2a8bbe20 con 0x55ed2a972930 2024-04-17T19:47:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc86be00640 1 --2- 172.21.15.29:0/1261143242 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a979c10 0x55ed2aaa4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc86b400640 1 --2- 172.21.15.29:0/1261143242 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a572280 0x55ed2aadcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc863e00640 1 --2- 172.21.15.29:0/1261143242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a972930 0x55ed2aadd450 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc854000fc0 tx=0x7fc854055530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a04aed5cad42354 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:57.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc84fe00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-17T19:47:57.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc84fe00640 1 -- 172.21.15.29:0/1261143242 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85405c270 con 0x55ed2a972930 2024-04-17T19:47:57.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc84fe00640 1 -- 172.21.15.29:0/1261143242 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc854054f60 con 0x55ed2a972930 2024-04-17T19:47:57.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc879964000 1 -- 172.21.15.29:0/1261143242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc85c0b6470 con 0x55ed2a972930 2024-04-17T19:47:57.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc84fe00640 1 -- 172.21.15.29:0/1261143242 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fc854055160 con 0x55ed2a972930 2024-04-17T19:47:57.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc879964000 10 client.4431 fetch_fsmap learned FSMap version 7 2024-04-17T19:47:57.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc879964000 1 -- 172.21.15.29:0/1261143242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55ed2a9bba80 con 0x55ed2a972930 2024-04-17T19:47:57.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc84fe00640 1 -- 172.21.15.29:0/1261143242 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc85c0b6470 con 0x55ed2a972930 2024-04-17T19:47:57.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc84fe00640 1 -- 172.21.15.29:0/1261143242 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 7) ==== 4712+0+0 (secure 0 0 0) 0x7fc854074350 con 0x55ed2a972930 2024-04-17T19:47:57.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc879964000 10 client.4431 fetch_fsmap finished waiting for FSMap version 7 2024-04-17T19:47:57.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc879964000 10 client.4431 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T19:47:57.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc879964000 20 client.4431 populate_metadata read hostname 'smithi029' 2024-04-17T19:47:57.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc879964000 1 --2- 172.21.15.29:0/1261143242 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x55ed2a5967d0 0x55ed2aaa58d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:57.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc879964000 4 client.4431 mds_command: new command op to 4305 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T19:47:57.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc879964000 1 -- 172.21.15.29:0/1261143242 --> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed2a889be0 con 0x55ed2a5967d0 2024-04-17T19:47:57.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc86b400640 1 --2- 172.21.15.29:0/1261143242 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x55ed2a5967d0 0x55ed2aaa58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:57.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc86b400640 1 --2- 172.21.15.29:0/1261143242 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x55ed2a5967d0 0x55ed2aaa58d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e768d095070c4993 server_cookie=5af0f49b96d3ecfa in_seq=0 out_seq=0 2024-04-17T19:47:57.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:57.629+0000 7fc84fe00640 10 client.4431 ms_handle_connect on v2:172.21.15.29:6833/2691590158 2024-04-17T19:47:59.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.933+0000 7fc84fe00640 1 -- 172.21.15.29:0/1261143242 <== mds.0 v2:172.21.15.29:6833/2691590158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ed2a889be0 con 0x55ed2a5967d0 2024-04-17T19:47:59.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.933+0000 7fc84fe00640 10 client.4431 handle_command_reply: tid=0 2024-04-17T19:47:59.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.933+0000 7fc879964000 1 client.4431 shutdown 2024-04-17T19:47:59.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.933+0000 7fc879964000 1 -- 172.21.15.29:0/1261143242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a972930 msgr2=0x55ed2aadd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:59.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.933+0000 7fc879964000 1 --2- 172.21.15.29:0/1261143242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a972930 0x55ed2aadd450 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc854000fc0 tx=0x7fc854055530 comp rx=0 tx=0).stop 2024-04-17T19:47:59.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.933+0000 7fc879964000 1 -- 172.21.15.29:0/1261143242 shutdown_connections 2024-04-17T19:47:59.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.933+0000 7fc879964000 1 --2- 172.21.15.29:0/1261143242 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x55ed2a5967d0 0x55ed2aaa58d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:59.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.933+0000 7fc879964000 1 --2- 172.21.15.29:0/1261143242 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a979c10 0x55ed2aaa4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:59.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.933+0000 7fc879964000 1 --2- 172.21.15.29:0/1261143242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a972930 0x55ed2aadd450 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:59.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.933+0000 7fc879964000 1 --2- 172.21.15.29:0/1261143242 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a572280 0x55ed2aadcf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:59.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.933+0000 7fc84fe00640 0 client.4431 ms_handle_reset on v2:172.21.15.29:6833/2691590158 2024-04-17T19:47:59.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.933+0000 7fc879964000 1 -- 172.21.15.29:0/1261143242 >> 172.21.15.29:0/1261143242 conn(0x55ed2a9bb670 msgr2=0x55ed2aade720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:59.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.933+0000 7fc879964000 1 -- 172.21.15.29:0/1261143242 shutdown_connections 2024-04-17T19:47:59.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.933+0000 7fc879964000 1 -- 172.21.15.29:0/1261143242 wait complete. 2024-04-17T19:47:59.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.933+0000 7fc879964000 20 client.4431 trim_cache size 0 max 16384 2024-04-17T19:47:59.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.953+0000 7fc879964000 1 Processor -- start 2024-04-17T19:47:59.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.953+0000 7fc879964000 1 -- start start 2024-04-17T19:47:59.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.953+0000 7fc879964000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a572280 0x55ed2aadb160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:59.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.953+0000 7fc879964000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a972930 0x55ed2aadb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:59.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.953+0000 7fc879964000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a979c10 0x55ed2aadc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:59.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.953+0000 7fc879964000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc86c2026c0 con 0x55ed2a979c10 2024-04-17T19:47:59.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.953+0000 7fc879964000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc86c201aa0 con 0x55ed2a972930 2024-04-17T19:47:59.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.953+0000 7fc879964000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc86c202070 con 0x55ed2a572280 2024-04-17T19:47:59.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.953+0000 7fc86be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a979c10 0x55ed2aadc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:59.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.953+0000 7fc86be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a979c10 0x55ed2aadc010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51876/0 (socket says 172.21.15.29:51876) 2024-04-17T19:47:59.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.953+0000 7fc86be00640 1 -- 172.21.15.29:0/2993249073 learned_addr learned my addr 172.21.15.29:0/2993249073 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:59.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc863e00640 1 --2- 172.21.15.29:0/2993249073 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a972930 0x55ed2aadb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:59.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc86b400640 1 --2- 172.21.15.29:0/2993249073 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a572280 0x55ed2aadb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc86be00640 1 -- 172.21.15.29:0/2993249073 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a572280 msgr2=0x55ed2aadb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc86be00640 1 --2- 172.21.15.29:0/2993249073 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a572280 0x55ed2aadb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc86be00640 1 -- 172.21.15.29:0/2993249073 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a972930 msgr2=0x55ed2aadb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc86be00640 1 --2- 172.21.15.29:0/2993249073 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a972930 0x55ed2aadb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc86be00640 1 -- 172.21.15.29:0/2993249073 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed2a949e30 con 0x55ed2a979c10 2024-04-17T19:47:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc86b400640 1 --2- 172.21.15.29:0/2993249073 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a572280 0x55ed2aadb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:47:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc86be00640 1 --2- 172.21.15.29:0/2993249073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a979c10 0x55ed2aadc010 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc864005080 tx=0x7fc864001610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e518d665eee7e89e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc85aa00640 1 -- 172.21.15.29:0/2993249073 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc864001de0 con 0x55ed2a979c10 2024-04-17T19:47:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc85aa00640 1 -- 172.21.15.29:0/2993249073 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc864001f80 con 0x55ed2a979c10 2024-04-17T19:47:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc85aa00640 1 -- 172.21.15.29:0/2993249073 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc864016650 con 0x55ed2a979c10 2024-04-17T19:47:59.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc879964000 1 -- 172.21.15.29:0/2993249073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a979c10 msgr2=0x55ed2aadc010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:47:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc879964000 1 --2- 172.21.15.29:0/2993249073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a979c10 0x55ed2aadc010 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc864005080 tx=0x7fc864001610 comp rx=0 tx=0).stop 2024-04-17T19:47:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc879964000 1 -- 172.21.15.29:0/2993249073 shutdown_connections 2024-04-17T19:47:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc879964000 1 --2- 172.21.15.29:0/2993249073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a979c10 0x55ed2aadc010 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc879964000 1 --2- 172.21.15.29:0/2993249073 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a972930 0x55ed2aadb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc879964000 1 --2- 172.21.15.29:0/2993249073 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a572280 0x55ed2aadb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc879964000 1 -- 172.21.15.29:0/2993249073 >> 172.21.15.29:0/2993249073 conn(0x55ed2a9bb670 msgr2=0x55ed2aafea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:47:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc879964000 1 -- 172.21.15.29:0/2993249073 shutdown_connections 2024-04-17T19:47:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc879964000 1 -- 172.21.15.29:0/2993249073 wait complete. 2024-04-17T19:47:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc879964000 1 -- start start 2024-04-17T19:47:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.957+0000 7fc879964000 1 Processor -- start 2024-04-17T19:47:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc879964000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a572280 0x55ed2ab54910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc879964000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a972930 0x55ed2ab54e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc879964000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a979c10 0x55ed2aafc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc879964000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55ed2a8f4b20 con 0x55ed2a572280 2024-04-17T19:47:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc879964000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x55ed2a8ef800 con 0x55ed2a979c10 2024-04-17T19:47:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc879964000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x55ed2a8fd700 con 0x55ed2a972930 2024-04-17T19:47:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc879964000 10 client.0 _ll_register_callbacks cb 0x7fc85c0ae070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T19:47:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc86b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a572280 0x55ed2ab54910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc86b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a572280 0x55ed2ab54910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51882/0 (socket says 172.21.15.29:51882) 2024-04-17T19:47:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc86b400640 1 -- 172.21.15.29:0/1509549774 learned_addr learned my addr 172.21.15.29:0/1509549774 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:47:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc86be00640 1 --2- 172.21.15.29:0/1509549774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a979c10 0x55ed2aafc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc863e00640 1 --2- 172.21.15.29:0/1509549774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a972930 0x55ed2ab54e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc86b400640 1 -- 172.21.15.29:0/1509549774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a972930 msgr2=0x55ed2ab54e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc86b400640 1 --2- 172.21.15.29:0/1509549774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a972930 0x55ed2ab54e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc86b400640 1 -- 172.21.15.29:0/1509549774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a979c10 msgr2=0x55ed2aafc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:47:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc86b400640 1 --2- 172.21.15.29:0/1509549774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a979c10 0x55ed2aafc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:47:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc86b400640 1 -- 172.21.15.29:0/1509549774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed2a8827e0 con 0x55ed2a572280 2024-04-17T19:47:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc86b400640 1 --2- 172.21.15.29:0/1509549774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a572280 0x55ed2ab54910 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc85c01c8f0 tx=0x7fc85c01ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d0dbc3a2d6e7de server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:47:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc84fe00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-17T19:47:59.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc84fe00640 1 -- 172.21.15.29:0/1509549774 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c0c6040 con 0x55ed2a572280 2024-04-17T19:47:59.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc84fe00640 1 -- 172.21.15.29:0/1509549774 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc85c0b9a60 con 0x55ed2a572280 2024-04-17T19:47:59.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc879964000 1 -- 172.21.15.29:0/1509549774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc864001f80 con 0x55ed2a572280 2024-04-17T19:47:59.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc84fe00640 1 -- 172.21.15.29:0/1509549774 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fc85c0b0040 con 0x55ed2a572280 2024-04-17T19:47:59.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc879964000 10 client.4437 fetch_fsmap learned FSMap version 7 2024-04-17T19:47:59.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc879964000 1 -- 172.21.15.29:0/1509549774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55ed2a895670 con 0x55ed2a572280 2024-04-17T19:47:59.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc84fe00640 1 -- 172.21.15.29:0/1509549774 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc864001f80 con 0x55ed2a572280 2024-04-17T19:47:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc84fe00640 1 -- 172.21.15.29:0/1509549774 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 7) ==== 4712+0+0 (secure 0 0 0) 0x7fc85c0b9c60 con 0x55ed2a572280 2024-04-17T19:47:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc879964000 10 client.4437 fetch_fsmap finished waiting for FSMap version 7 2024-04-17T19:47:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc879964000 10 client.4437 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T19:47:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc879964000 20 client.4437 populate_metadata read hostname 'smithi029' 2024-04-17T19:47:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc879964000 1 --2- 172.21.15.29:0/1509549774 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x55ed2a584830 0x55ed2aafcce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:47:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc879964000 4 client.4437 mds_command: new command op to 4305 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T19:47:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc879964000 1 -- 172.21.15.29:0/1509549774 --> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] -- command(tid 0: {"prefix": "status"}) -- 0x55ed2a8fdf10 con 0x55ed2a584830 2024-04-17T19:47:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.961+0000 7fc863e00640 1 --2- 172.21.15.29:0/1509549774 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x55ed2a584830 0x55ed2aafcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:47:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.965+0000 7fc863e00640 1 -- 172.21.15.29:0/1509549774 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x55ed2a584830 msgr2=0x55ed2aafcce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-04-17T19:47:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.965+0000 7fc863e00640 1 -- 172.21.15.29:0/1509549774 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x55ed2a584830 msgr2=0x55ed2aafcce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-17T19:47:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.965+0000 7fc863e00640 1 --2- 172.21.15.29:0/1509549774 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x55ed2a584830 0x55ed2aafcce0 crc :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-17T19:47:59.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:47:59.965+0000 7fc863e00640 1 --2- 172.21.15.29:0/1509549774 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x55ed2a584830 0x55ed2aafcce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T19:48:00.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc863e00640 1 --2- 172.21.15.29:0/1509549774 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x55ed2a584830 0x55ed2aafcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:00.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc863e00640 1 --2- 172.21.15.29:0/1509549774 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x55ed2a584830 0x55ed2aafcce0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43dbcde517e6ff66 server_cookie=9c42b8abcfcbd18c in_seq=0 out_seq=0 2024-04-17T19:48:00.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc84fe00640 10 client.4437 ms_handle_connect on v2:172.21.15.29:6833/2691590158 2024-04-17T19:48:00.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc84fe00640 1 -- 172.21.15.29:0/1509549774 <== mds.0 v2:172.21.15.29:6833/2691590158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+318 (crc 0 0 0) 0x55ed2a8fdf10 con 0x55ed2a584830 2024-04-17T19:48:00.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc84fe00640 10 client.4437 handle_command_reply: tid=0 2024-04-17T19:48:00.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc879964000 1 client.4437 shutdown 2024-04-17T19:48:00.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc879964000 1 -- 172.21.15.29:0/1509549774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a572280 msgr2=0x55ed2ab54910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:00.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc879964000 1 --2- 172.21.15.29:0/1509549774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a572280 0x55ed2ab54910 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc85c01c8f0 tx=0x7fc85c01ac50 comp rx=0 tx=0).stop 2024-04-17T19:48:00.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc879964000 1 -- 172.21.15.29:0/1509549774 shutdown_connections 2024-04-17T19:48:00.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc879964000 1 --2- 172.21.15.29:0/1509549774 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x55ed2a584830 0x55ed2aafcce0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:00.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc879964000 1 --2- 172.21.15.29:0/1509549774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55ed2a979c10 0x55ed2aafc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:00.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc879964000 1 --2- 172.21.15.29:0/1509549774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55ed2a972930 0x55ed2ab54e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:00.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc879964000 1 --2- 172.21.15.29:0/1509549774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55ed2a572280 0x55ed2ab54910 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:00.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc84fe00640 0 client.4437 ms_handle_reset on v2:172.21.15.29:6833/2691590158 2024-04-17T19:48:00.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc879964000 1 -- 172.21.15.29:0/1509549774 >> 172.21.15.29:0/1509549774 conn(0x55ed2a9bb670 msgr2=0x55ed2aadc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:00.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc879964000 1 -- 172.21.15.29:0/1509549774 shutdown_connections 2024-04-17T19:48:00.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc879964000 1 -- 172.21.15.29:0/1509549774 wait complete. 2024-04-17T19:48:00.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc879964000 20 client.4437 trim_cache size 0 max 16384 2024-04-17T19:48:00.159 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-17T19:48:00.159 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "a680ee90-875c-4283-a901-86d7638393c3", 2024-04-17T19:48:00.160 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2024-04-17T19:48:00.160 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 4305, 2024-04-17T19:48:00.160 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-04-17T19:48:00.160 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-04-17T19:48:00.160 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-04-17T19:48:00.160 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 10.334242176, 2024-04-17T19:48:00.160 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 6, 2024-04-17T19:48:00.160 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 24, 2024-04-17T19:48:00.160 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 24, 2024-04-17T19:48:00.160 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 15.727180037 2024-04-17T19:48:00.160 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-17T19:48:00.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc872200640 1 -- 172.21.15.29:0/3807274695 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc848068380 msgr2=0x7fc84806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:00.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc872200640 1 --2- 172.21.15.29:0/3807274695 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc848068380 0x7fc84806a890 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc85c0ad000 tx=0x7fc85c008040 comp rx=0 tx=0).stop 2024-04-17T19:48:00.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc872200640 1 -- 172.21.15.29:0/3807274695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc86c0cc390 msgr2=0x7fc86c15b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:00.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc872200640 1 --2- 172.21.15.29:0/3807274695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc86c0cc390 0x7fc86c15b9e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc85400c440 tx=0x7fc85400c900 comp rx=0 tx=0).stop 2024-04-17T19:48:00.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc872200640 1 -- 172.21.15.29:0/3807274695 shutdown_connections 2024-04-17T19:48:00.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc872200640 1 --2- 172.21.15.29:0/3807274695 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc848068380 0x7fc84806a890 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:00.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc872200640 1 --2- 172.21.15.29:0/3807274695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc86c1541c0 0x7fc86c146690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:00.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc872200640 1 --2- 172.21.15.29:0/3807274695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc86c0cc390 0x7fc86c15b9e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:00.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc872200640 1 --2- 172.21.15.29:0/3807274695 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc86c0cbf80 0x7fc86c06be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:00.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc872200640 1 -- 172.21.15.29:0/3807274695 >> 172.21.15.29:0/3807274695 conn(0x7fc86c071a60 msgr2=0x7fc86c14ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:00.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc872200640 1 -- 172.21.15.29:0/3807274695 shutdown_connections 2024-04-17T19:48:00.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.169+0000 7fc872200640 1 -- 172.21.15.29:0/3807274695 wait complete. 2024-04-17T19:48:00.179 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-17T19:48:00.179 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-17T19:48:00.179 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T19:48:00.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.341+0000 7fb3a2e00640 1 Processor -- start 2024-04-17T19:48:00.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.341+0000 7fb3a2e00640 1 -- start start 2024-04-17T19:48:00.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.345+0000 7fb3a2e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb39c0ce470 0x7fb39c0ce880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:00.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.345+0000 7fb3a2e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb39c15f920 0x7fb39c15fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:00.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.345+0000 7fb3a2e00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb39c06c980 con 0x7fb39c155f10 2024-04-17T19:48:00.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.345+0000 7fb3a2e00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fb39c06cc60 con 0x7fb39c0ce470 2024-04-17T19:48:00.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.345+0000 7fb3a2e00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb39c06cf40 con 0x7fb39c15f920 2024-04-17T19:48:00.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.345+0000 7fb3a1000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb39c0ce470 0x7fb39c0ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:00.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.345+0000 7fb3a1a00640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fb39c155f10 0x7fb39c15c830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:51464/0 (socket says 172.21.15.29:51464) 2024-04-17T19:48:00.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.345+0000 7fb3a1a00640 1 -- 172.21.15.29:0/3440122261 learned_addr learned my addr 172.21.15.29:0/3440122261 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:00.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.345+0000 7fb3a2400640 1 --2- 172.21.15.29:0/3440122261 >> v2:172.21.15.37:3300/0 conn(0x7fb39c15f920 0x7fb39c15fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:00.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.345+0000 7fb3a1000640 1 -- 172.21.15.29:0/3440122261 >> v2:172.21.15.37:3300/0 conn(0x7fb39c15f920 msgr2=0x7fb39c15fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:00.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.345+0000 7fb3a1000640 1 --2- 172.21.15.29:0/3440122261 >> v2:172.21.15.37:3300/0 conn(0x7fb39c15f920 0x7fb39c15fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:00.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.345+0000 7fb3a1000640 1 -- 172.21.15.29:0/3440122261 >> v1:172.21.15.37:6789/0 conn(0x7fb39c155f10 legacy=0x7fb39c15c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:00.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.345+0000 7fb3a1000640 1 -- 172.21.15.29:0/3440122261 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39c06d220 con 0x7fb39c0ce470 2024-04-17T19:48:00.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.349+0000 7fb3a1000640 1 --2- 172.21.15.29:0/3440122261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb39c0ce470 0x7fb39c0ce880 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb39c072b70 tx=0x7fb38801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9196261b6ae97754 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:00.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.349+0000 7fb39be00640 1 -- 172.21.15.29:0/3440122261 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3880b0070 con 0x7fb39c0ce470 2024-04-17T19:48:00.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.349+0000 7fb39be00640 1 -- 172.21.15.29:0/3440122261 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3880ab600 con 0x7fb39c0ce470 2024-04-17T19:48:00.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.349+0000 7fb39be00640 1 -- 172.21.15.29:0/3440122261 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3880ab9d0 con 0x7fb39c0ce470 2024-04-17T19:48:00.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.349+0000 7fb3a2e00640 1 -- 172.21.15.29:0/3440122261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb39c0ce470 msgr2=0x7fb39c0ce880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:00.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.349+0000 7fb3a2e00640 1 --2- 172.21.15.29:0/3440122261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb39c0ce470 0x7fb39c0ce880 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb39c072b70 tx=0x7fb38801c480 comp rx=0 tx=0).stop 2024-04-17T19:48:00.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.349+0000 7fb3a2e00640 1 -- 172.21.15.29:0/3440122261 shutdown_connections 2024-04-17T19:48:00.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.349+0000 7fb3a2e00640 1 --2- 172.21.15.29:0/3440122261 >> v2:172.21.15.37:3300/0 conn(0x7fb39c15f920 0x7fb39c15fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:00.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.349+0000 7fb3a2e00640 1 --2- 172.21.15.29:0/3440122261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb39c0ce470 0x7fb39c0ce880 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:00.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.349+0000 7fb3a2e00640 1 -- 172.21.15.29:0/3440122261 >> 172.21.15.29:0/3440122261 conn(0x7fb39c072db0 msgr2=0x7fb39c0731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:00.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.349+0000 7fb3a2e00640 1 -- 172.21.15.29:0/3440122261 shutdown_connections 2024-04-17T19:48:00.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.349+0000 7fb3a2e00640 1 -- 172.21.15.29:0/3440122261 wait complete. 2024-04-17T19:48:00.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.349+0000 7fb3a2e00640 1 Processor -- start 2024-04-17T19:48:00.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.349+0000 7fb3a2e00640 1 -- start start 2024-04-17T19:48:00.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a2e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb39c0ce470 0x7fb39c138290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:00.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a2e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb39c155f10 0x7fb39c0c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:00.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a2e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb39c15f920 0x7fb39c0c7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:00.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a2e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c1690d0 con 0x7fb39c0ce470 2024-04-17T19:48:00.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a2e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb39c168f50 con 0x7fb39c15f920 2024-04-17T19:48:00.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a2e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb39c169250 con 0x7fb39c155f10 2024-04-17T19:48:00.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a1a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb39c0ce470 0x7fb39c138290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:00.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a1a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb39c0ce470 0x7fb39c138290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51894/0 (socket says 172.21.15.29:51894) 2024-04-17T19:48:00.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a1a00640 1 -- 172.21.15.29:0/2923942454 learned_addr learned my addr 172.21.15.29:0/2923942454 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:00.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a1000640 1 --2- 172.21.15.29:0/2923942454 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb39c155f10 0x7fb39c0c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:00.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a2400640 1 --2- 172.21.15.29:0/2923942454 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb39c15f920 0x7fb39c0c7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:00.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a1a00640 1 -- 172.21.15.29:0/2923942454 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb39c155f10 msgr2=0x7fb39c0c7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:00.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a1a00640 1 --2- 172.21.15.29:0/2923942454 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb39c155f10 0x7fb39c0c7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:00.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a1a00640 1 -- 172.21.15.29:0/2923942454 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb39c15f920 msgr2=0x7fb39c0c7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:00.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a1a00640 1 --2- 172.21.15.29:0/2923942454 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb39c15f920 0x7fb39c0c7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:00.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a1a00640 1 -- 172.21.15.29:0/2923942454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39c1387d0 con 0x7fb39c0ce470 2024-04-17T19:48:00.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a2400640 1 --2- 172.21.15.29:0/2923942454 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb39c15f920 0x7fb39c0c7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:00.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a1a00640 1 --2- 172.21.15.29:0/2923942454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb39c0ce470 0x7fb39c138290 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb39000c3d0 tx=0x7fb39000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:00.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb399600640 1 -- 172.21.15.29:0/2923942454 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb390017070 con 0x7fb39c0ce470 2024-04-17T19:48:00.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb399600640 1 -- 172.21.15.29:0/2923942454 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb390005cf0 con 0x7fb39c0ce470 2024-04-17T19:48:00.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb399600640 1 -- 172.21.15.29:0/2923942454 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3900072f0 con 0x7fb39c0ce470 2024-04-17T19:48:00.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a2e00640 1 -- 172.21.15.29:0/2923942454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb39c13a970 con 0x7fb39c0ce470 2024-04-17T19:48:00.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a2e00640 1 -- 172.21.15.29:0/2923942454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb39c137a90 con 0x7fb39c0ce470 2024-04-17T19:48:00.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.353+0000 7fb3a2e00640 1 -- 172.21.15.29:0/2923942454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb39c04ff10 con 0x7fb39c0ce470 2024-04-17T19:48:00.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.357+0000 7fb399600640 1 -- 172.21.15.29:0/2923942454 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb39001a070 con 0x7fb39c0ce470 2024-04-17T19:48:00.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.357+0000 7fb399600640 1 --2- 172.21.15.29:0/2923942454 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb37c0683b0 0x7fb37c06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:00.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.357+0000 7fb399600640 1 -- 172.21.15.29:0/2923942454 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb390085f40 con 0x7fb39c0ce470 2024-04-17T19:48:00.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.357+0000 7fb3a1000640 1 --2- 172.21.15.29:0/2923942454 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb37c0683b0 0x7fb37c06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:00.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.357+0000 7fb3a1000640 1 --2- 172.21.15.29:0/2923942454 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb37c0683b0 0x7fb37c06a8c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb3880ad000 tx=0x7fb388008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:00.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.361+0000 7fb399600640 1 -- 172.21.15.29:0/2923942454 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb390055ad0 con 0x7fb39c0ce470 2024-04-17T19:48:00.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.561+0000 7fb3a2e00640 1 -- 172.21.15.29:0/2923942454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb39c159a00 con 0x7fb39c0ce470 2024-04-17T19:48:00.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.561+0000 7fb399600640 1 -- 172.21.15.29:0/2923942454 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+88 (secure 0 0 0) 0x7fb390059dd0 con 0x7fb39c0ce470 2024-04-17T19:48:00.551 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-17T19:48:00.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.565+0000 7fb3a2e00640 1 -- 172.21.15.29:0/2923942454 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb37c0683b0 msgr2=0x7fb37c06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:00.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.565+0000 7fb3a2e00640 1 --2- 172.21.15.29:0/2923942454 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb37c0683b0 0x7fb37c06a8c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb3880ad000 tx=0x7fb388008040 comp rx=0 tx=0).stop 2024-04-17T19:48:00.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.565+0000 7fb3a2e00640 1 -- 172.21.15.29:0/2923942454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb39c0ce470 msgr2=0x7fb39c138290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:00.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.565+0000 7fb3a2e00640 1 --2- 172.21.15.29:0/2923942454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb39c0ce470 0x7fb39c138290 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb39000c3d0 tx=0x7fb39000c890 comp rx=0 tx=0).stop 2024-04-17T19:48:00.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.565+0000 7fb3a2e00640 1 -- 172.21.15.29:0/2923942454 shutdown_connections 2024-04-17T19:48:00.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.565+0000 7fb3a2e00640 1 --2- 172.21.15.29:0/2923942454 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb37c0683b0 0x7fb37c06a8c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:00.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.565+0000 7fb3a2e00640 1 --2- 172.21.15.29:0/2923942454 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb39c15f920 0x7fb39c0c7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:00.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.565+0000 7fb3a2e00640 1 --2- 172.21.15.29:0/2923942454 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb39c155f10 0x7fb39c0c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:00.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.565+0000 7fb3a2e00640 1 --2- 172.21.15.29:0/2923942454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb39c0ce470 0x7fb39c138290 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:00.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.565+0000 7fb3a2e00640 1 -- 172.21.15.29:0/2923942454 >> 172.21.15.29:0/2923942454 conn(0x7fb39c072db0 msgr2=0x7fb39c0c98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:00.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.565+0000 7fb3a2e00640 1 -- 172.21.15.29:0/2923942454 shutdown_connections 2024-04-17T19:48:00.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00.565+0000 7fb3a2e00640 1 -- 172.21.15.29:0/2923942454 wait complete. 2024-04-17T19:48:00.573 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T19:48:00.573 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T19:48:00.573 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2024-04-17T19:48:00.573 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-17T19:48:00.573 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-17T19:48:00.573 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T19:48:00.573 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-17T19:48:00.573 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T19:48:00.573 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-17T19:48:00.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:00.574 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2024-04-17T19:48:00.583 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T19:48:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T19:48:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T19:48:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-04-17T19:48:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2024-04-17T19:48:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-04-17T19:48:00.583 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T19:48:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2024-04-17T19:48:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2024-04-17T19:48:00.584 INFO:teuthology.orchestra.run.smithi029.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T19:48:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2024-04-17T19:48:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2024-04-17T19:48:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-17T19:48:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 42360sec preferred_lft 42360sec 2024-04-17T19:48:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2024-04-17T19:48:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-04-17T19:48:00.584 INFO:teuthology.orchestra.run.smithi029.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T19:48:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2024-04-17T19:48:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2024-04-17T19:48:00.585 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T19:48:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2024-04-17T19:48:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2024-04-17T19:48:00.585 INFO:teuthology.orchestra.run.smithi029.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-17T19:48:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 02:42:9c:5a:03:85 brd ff:ff:ff:ff:ff:ff 2024-04-17T19:48:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-17T19:48:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-04-17T19:48:00.586 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-17T19:48:00.586 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T19:48:00.586 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-04-17T19:48:00.586 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add name ceph-brx type bridge 2024-04-17T19:48:00.586 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr flush dev ceph-brx 2024-04-17T19:48:00.586 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set ceph-brx up 2024-04-17T19:48:00.586 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-17T19:48:00.586 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-04-17T19:48:00.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T19:48:00.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T19:48:00.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:00.744 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T19:48:00.762 INFO:teuthology.orchestra.run.smithi029.stdout:1 2024-04-17T19:48:00.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:00.765 DEBUG:teuthology.orchestra.run.smithi029:> route 2024-04-17T19:48:00.833 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2024-04-17T19:48:00.833 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T19:48:00.833 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-04-17T19:48:00.833 INFO:teuthology.orchestra.run.smithi029.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-17T19:48:00.833 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-04-17T19:48:00.833 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T19:48:00.834 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T19:48:00.834 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-04-17T19:48:00.834 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-17T19:48:00.834 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-17T19:48:00.834 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-17T19:48:00.834 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-04-17T19:48:00.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T19:48:00.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T19:48:00.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:00.972 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-04-17T19:48:01.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:01.029 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list-id 2024-04-17T19:48:01.080 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T19:48:01.080 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-04-17T19:48:01.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-17T19:48:01.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-17T19:48:01.081 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-04-17T19:48:01.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T19:48:01.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T19:48:01.210 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-17T19:48:01.210 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T19:48:01.210 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-04-17T19:48:01.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-17T19:48:01.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-17T19:48:01.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-17T19:48:01.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-17T19:48:01.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-17T19:48:01.211 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-04-17T19:48:01.259 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T19:48:01.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T19:48:01.388 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T19:48:01.388 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-04-17T19:48:01.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2024-04-17T19:48:01.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set dev brx.0 master ceph-brx 2024-04-17T19:48:01.388 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-04-17T19:48:01.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T19:48:01.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T19:48:01.475 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-17T19:48:01.475 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T19:48:01.476 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-17T19:48:01.484 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-17T19:48:01.485 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T19:48:01.485 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-17T19:48:01.535 DEBUG:teuthology.orchestra.run.smithi029:> sudo modprobe fuse 2024-04-17T19:48:01.595 DEBUG:teuthology.orchestra.run.smithi029:> cat /proc/self/mounts | awk '{print $2}' 2024-04-17T19:48:01.649 INFO:teuthology.orchestra.run.smithi029.stdout:/sys 2024-04-17T19:48:01.649 INFO:teuthology.orchestra.run.smithi029.stdout:/proc 2024-04-17T19:48:01.649 INFO:teuthology.orchestra.run.smithi029.stdout:/dev 2024-04-17T19:48:01.649 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/pts 2024-04-17T19:48:01.649 INFO:teuthology.orchestra.run.smithi029.stdout:/run 2024-04-17T19:48:01.649 INFO:teuthology.orchestra.run.smithi029.stdout:/ 2024-04-17T19:48:01.649 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/security 2024-04-17T19:48:01.649 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/shm 2024-04-17T19:48:01.649 INFO:teuthology.orchestra.run.smithi029.stdout:/run/lock 2024-04-17T19:48:01.649 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/cgroup 2024-04-17T19:48:01.649 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/pstore 2024-04-17T19:48:01.649 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/bpf 2024-04-17T19:48:01.649 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-04-17T19:48:01.650 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/hugepages 2024-04-17T19:48:01.650 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/mqueue 2024-04-17T19:48:01.650 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/debug 2024-04-17T19:48:01.650 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/tracing 2024-04-17T19:48:01.650 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/fs/nfsd 2024-04-17T19:48:01.650 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/fuse/connections 2024-04-17T19:48:01.650 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/config 2024-04-17T19:48:01.650 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-sysusers.service 2024-04-17T19:48:01.650 INFO:teuthology.orchestra.run.smithi029.stdout:/run/rpc_pipefs 2024-04-17T19:48:01.650 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/1000 2024-04-17T19:48:01.650 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph 2024-04-17T19:48:01.650 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph/osd/ceph-0 2024-04-17T19:48:01.650 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph/osd/ceph-1 2024-04-17T19:48:01.650 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph/osd/ceph-2 2024-04-17T19:48:01.651 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph/osd/ceph-3 2024-04-17T19:48:01.651 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns 2024-04-17T19:48:01.651 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-17T19:48:01.651 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-17T19:48:01.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:01.652 DEBUG:teuthology.orchestra.run.smithi029:> ls /sys/fs/fuse/connections 2024-04-17T19:48:01.703 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-17T19:48:01.703 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-17T19:48:01.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo modprobe fuse 2024-04-17T19:48:01.770 DEBUG:teuthology.orchestra.run.smithi029:> cat /proc/self/mounts | awk '{print $2}' 2024-04-17T19:48:01.820 INFO:teuthology.orchestra.run.smithi029.stdout:/sys 2024-04-17T19:48:01.821 INFO:teuthology.orchestra.run.smithi029.stdout:/proc 2024-04-17T19:48:01.821 INFO:teuthology.orchestra.run.smithi029.stdout:/dev 2024-04-17T19:48:01.821 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/pts 2024-04-17T19:48:01.821 INFO:teuthology.orchestra.run.smithi029.stdout:/run 2024-04-17T19:48:01.821 INFO:teuthology.orchestra.run.smithi029.stdout:/ 2024-04-17T19:48:01.821 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/security 2024-04-17T19:48:01.821 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/shm 2024-04-17T19:48:01.821 INFO:teuthology.orchestra.run.smithi029.stdout:/run/lock 2024-04-17T19:48:01.821 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/cgroup 2024-04-17T19:48:01.821 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/pstore 2024-04-17T19:48:01.821 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/bpf 2024-04-17T19:48:01.821 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-04-17T19:48:01.821 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/hugepages 2024-04-17T19:48:01.822 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/mqueue 2024-04-17T19:48:01.822 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/debug 2024-04-17T19:48:01.822 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/tracing 2024-04-17T19:48:01.822 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/fs/nfsd 2024-04-17T19:48:01.822 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/fuse/connections 2024-04-17T19:48:01.822 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/config 2024-04-17T19:48:01.822 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-sysusers.service 2024-04-17T19:48:01.822 INFO:teuthology.orchestra.run.smithi029.stdout:/run/rpc_pipefs 2024-04-17T19:48:01.822 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/1000 2024-04-17T19:48:01.822 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph 2024-04-17T19:48:01.822 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph/osd/ceph-0 2024-04-17T19:48:01.822 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph/osd/ceph-1 2024-04-17T19:48:01.822 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph/osd/ceph-2 2024-04-17T19:48:01.822 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph/osd/ceph-3 2024-04-17T19:48:01.823 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns 2024-04-17T19:48:01.823 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-17T19:48:01.823 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-17T19:48:01.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:01.824 DEBUG:teuthology.orchestra.run.smithi029:> ls /sys/fs/fuse/connections 2024-04-17T19:48:01.860 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi029.stderr:2024-04-17T19:48:01.869+0000 7f3f0fe9b580 -1 init, newargv = 0x55da3b62a3d0 newargc=15 2024-04-17T19:48:01.860 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi029.stderr:ceph-fuse[32615]: starting ceph client 2024-04-17T19:48:01.872 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi029.stderr:ceph-fuse[32615]: starting fuse 2024-04-17T19:48:02.969 DEBUG:teuthology.orchestra.run.smithi029:> sudo modprobe fuse 2024-04-17T19:48:02.994 DEBUG:teuthology.orchestra.run.smithi029:> cat /proc/self/mounts | awk '{print $2}' 2024-04-17T19:48:03.050 INFO:teuthology.orchestra.run.smithi029.stdout:/sys 2024-04-17T19:48:03.050 INFO:teuthology.orchestra.run.smithi029.stdout:/proc 2024-04-17T19:48:03.050 INFO:teuthology.orchestra.run.smithi029.stdout:/dev 2024-04-17T19:48:03.051 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/pts 2024-04-17T19:48:03.051 INFO:teuthology.orchestra.run.smithi029.stdout:/run 2024-04-17T19:48:03.051 INFO:teuthology.orchestra.run.smithi029.stdout:/ 2024-04-17T19:48:03.051 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/security 2024-04-17T19:48:03.051 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/shm 2024-04-17T19:48:03.051 INFO:teuthology.orchestra.run.smithi029.stdout:/run/lock 2024-04-17T19:48:03.051 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/cgroup 2024-04-17T19:48:03.052 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/pstore 2024-04-17T19:48:03.052 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/bpf 2024-04-17T19:48:03.052 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/sys/fs/binfmt_misc 2024-04-17T19:48:03.052 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/hugepages 2024-04-17T19:48:03.052 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/mqueue 2024-04-17T19:48:03.052 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/debug 2024-04-17T19:48:03.052 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/tracing 2024-04-17T19:48:03.052 INFO:teuthology.orchestra.run.smithi029.stdout:/proc/fs/nfsd 2024-04-17T19:48:03.052 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/fs/fuse/connections 2024-04-17T19:48:03.052 INFO:teuthology.orchestra.run.smithi029.stdout:/sys/kernel/config 2024-04-17T19:48:03.052 INFO:teuthology.orchestra.run.smithi029.stdout:/run/credentials/systemd-sysusers.service 2024-04-17T19:48:03.052 INFO:teuthology.orchestra.run.smithi029.stdout:/run/rpc_pipefs 2024-04-17T19:48:03.052 INFO:teuthology.orchestra.run.smithi029.stdout:/run/user/1000 2024-04-17T19:48:03.053 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph 2024-04-17T19:48:03.053 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph/osd/ceph-0 2024-04-17T19:48:03.053 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph/osd/ceph-1 2024-04-17T19:48:03.053 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph/osd/ceph-2 2024-04-17T19:48:03.053 INFO:teuthology.orchestra.run.smithi029.stdout:/var/lib/ceph/osd/ceph-3 2024-04-17T19:48:03.053 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns 2024-04-17T19:48:03.053 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-17T19:48:03.053 INFO:teuthology.orchestra.run.smithi029.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-17T19:48:03.053 INFO:teuthology.orchestra.run.smithi029.stdout:/home/ubuntu/cephtest/mnt.admin 2024-04-17T19:48:03.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:03.054 DEBUG:teuthology.orchestra.run.smithi029:> ls /sys/fs/fuse/connections 2024-04-17T19:48:03.104 INFO:teuthology.orchestra.run.smithi029.stdout:44 2024-04-17T19:48:03.104 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [44] 2024-04-17T19:48:03.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo stdin-killer -- python3 -c ' 2024-04-17T19:48:03.104 DEBUG:teuthology.orchestra.run.smithi029:> import glob 2024-04-17T19:48:03.105 DEBUG:teuthology.orchestra.run.smithi029:> import re 2024-04-17T19:48:03.105 DEBUG:teuthology.orchestra.run.smithi029:> import os 2024-04-17T19:48:03.105 DEBUG:teuthology.orchestra.run.smithi029:> import subprocess 2024-04-17T19:48:03.105 DEBUG:teuthology.orchestra.run.smithi029:> 2024-04-17T19:48:03.105 DEBUG:teuthology.orchestra.run.smithi029:> def _find_admin_socket(client_name): 2024-04-17T19:48:03.105 DEBUG:teuthology.orchestra.run.smithi029:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-17T19:48:03.105 DEBUG:teuthology.orchestra.run.smithi029:> files = glob.glob(asok_path) 2024-04-17T19:48:03.105 DEBUG:teuthology.orchestra.run.smithi029:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-17T19:48:03.105 DEBUG:teuthology.orchestra.run.smithi029:> 2024-04-17T19:48:03.105 DEBUG:teuthology.orchestra.run.smithi029:> # Given a non-glob path, it better be there 2024-04-17T19:48:03.105 DEBUG:teuthology.orchestra.run.smithi029:> if "*" not in asok_path: 2024-04-17T19:48:03.105 DEBUG:teuthology.orchestra.run.smithi029:> assert(len(files) == 1) 2024-04-17T19:48:03.105 DEBUG:teuthology.orchestra.run.smithi029:> return files[0] 2024-04-17T19:48:03.106 DEBUG:teuthology.orchestra.run.smithi029:> 2024-04-17T19:48:03.106 DEBUG:teuthology.orchestra.run.smithi029:> for f in files: 2024-04-17T19:48:03.106 DEBUG:teuthology.orchestra.run.smithi029:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-17T19:48:03.106 DEBUG:teuthology.orchestra.run.smithi029:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-17T19:48:03.106 DEBUG:teuthology.orchestra.run.smithi029:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-17T19:48:03.106 DEBUG:teuthology.orchestra.run.smithi029:> contents = proc_f.read() 2024-04-17T19:48:03.106 DEBUG:teuthology.orchestra.run.smithi029:> if mountpoint in contents: 2024-04-17T19:48:03.106 DEBUG:teuthology.orchestra.run.smithi029:> return f 2024-04-17T19:48:03.106 DEBUG:teuthology.orchestra.run.smithi029:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-17T19:48:03.106 DEBUG:teuthology.orchestra.run.smithi029:> 2024-04-17T19:48:03.106 DEBUG:teuthology.orchestra.run.smithi029:> print(_find_admin_socket("client.admin")) 2024-04-17T19:48:03.106 DEBUG:teuthology.orchestra.run.smithi029:> ' 2024-04-17T19:48:03.239 INFO:teuthology.orchestra.run.smithi029.stdout:/var/run/ceph/ceph-client.admin.32615.asok 2024-04-17T19:48:03.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T19:48:03.252 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.32615.asok 2024-04-17T19:48:03.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:03.252 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.32615.asok status 2024-04-17T19:48:03.439 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-17T19:48:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: "metadata": { 2024-04-17T19:48:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_sha1": "f3a98d3f110319c567ec86bf70955c141495ccf6", 2024-04-17T19:48:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_version": "ceph version 19.0.0-3146-gf3a98d3f (f3a98d3f110319c567ec86bf70955c141495ccf6) squid (dev)", 2024-04-17T19:48:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: "entity_id": "admin", 2024-04-17T19:48:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: "hostname": "smithi029", 2024-04-17T19:48:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-17T19:48:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: "pid": "32615", 2024-04-17T19:48:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: "root": "/" 2024-04-17T19:48:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-17T19:48:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "dentry_count": 0, 2024-04-17T19:48:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "dentry_pinned_count": 0, 2024-04-17T19:48:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 4452, 2024-04-17T19:48:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "inst": { 2024-04-17T19:48:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "name": { 2024-04-17T19:48:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "client", 2024-04-17T19:48:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "num": 4452 2024-04-17T19:48:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-17T19:48:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": { 2024-04-17T19:48:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v1", 2024-04-17T19:48:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:0", 2024-04-17T19:48:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3186209574 2024-04-17T19:48:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-04-17T19:48:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-17T19:48:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": { 2024-04-17T19:48:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v1", 2024-04-17T19:48:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:0", 2024-04-17T19:48:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3186209574 2024-04-17T19:48:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-04-17T19:48:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "inst_str": "client.4452 172.21.15.29:0/3186209574", 2024-04-17T19:48:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "addr_str": "172.21.15.29:0/3186209574", 2024-04-17T19:48:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "inode_count": 1, 2024-04-17T19:48:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_epoch": 6, 2024-04-17T19:48:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_epoch": 24, 2024-04-17T19:48:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_epoch_barrier": 0, 2024-04-17T19:48:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "blocklisted": false, 2024-04-17T19:48:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs" 2024-04-17T19:48:03.441 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-17T19:48:03.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:03.455 DEBUG:teuthology.orchestra.run.smithi029:> stat --file-system '--printf=%T 2024-04-17T19:48:03.455 DEBUG:teuthology.orchestra.run.smithi029:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-17T19:48:03.464 INFO:teuthology.orchestra.run.smithi029.stdout:fuseblk 2024-04-17T19:48:03.464 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-17T19:48:03.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:03.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-17T19:48:03.532 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-17T19:48:03.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T19:48:03.651 INFO:teuthology.orchestra.run.smithi029.stdout:# file: . 2024-04-17T19:48:03.651 INFO:teuthology.orchestra.run.smithi029.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-17T19:48:03.651 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:03.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T19:48:03.658 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:48:03.658 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T19:48:03.666 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:48:03.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.873+0000 7f5ecba00640 1 Processor -- start 2024-04-17T19:48:03.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.873+0000 7f5ecba00640 1 -- start start 2024-04-17T19:48:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.873+0000 7f5ecba00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ec40c8d50 0x7f5ec406bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.873+0000 7f5ecba00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5ec415d7d0 0x7f5ec4164750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.873+0000 7f5ecba00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ec415dbe0 con 0x7f5ec415d3c0 2024-04-17T19:48:03.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.873+0000 7f5ecba00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f5ec415ad50 con 0x7f5ec415d7d0 2024-04-17T19:48:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.873+0000 7f5ecba00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5ec40c9590 con 0x7f5ec40c8d50 2024-04-17T19:48:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.873+0000 7f5ec9a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5ec415d7d0 0x7f5ec4164750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.873+0000 7f5ec9a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5ec415d7d0 0x7f5ec4164750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51900/0 (socket says 172.21.15.29:51900) 2024-04-17T19:48:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.873+0000 7f5ec9a00640 1 -- 172.21.15.29:0/1739670745 learned_addr learned my addr 172.21.15.29:0/1739670745 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.873+0000 7f5ec9000640 1 --2- 172.21.15.29:0/1739670745 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ec40c8d50 0x7f5ec406bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.873+0000 7f5ec9a00640 1 -- 172.21.15.29:0/1739670745 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ec40c8d50 msgr2=0x7f5ec406bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.873+0000 7f5ec9a00640 1 --2- 172.21.15.29:0/1739670745 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ec40c8d50 0x7f5ec406bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.873+0000 7f5ec9a00640 1 -- 172.21.15.29:0/1739670745 >> v1:172.21.15.37:6789/0 conn(0x7f5ec415d3c0 legacy=0x7f5ec40c7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:03.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.873+0000 7f5ec9a00640 1 -- 172.21.15.29:0/1739670745 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ec4159f90 con 0x7f5ec415d7d0 2024-04-17T19:48:03.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.877+0000 7f5ec9a00640 1 --2- 172.21.15.29:0/1739670745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec415d7d0 0x7f5ec4164750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc009e50 tx=0x7f5ebc01f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c8e4f7d1402d4e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.877+0000 7f5ec3400640 1 -- 172.21.15.29:0/1739670745 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc0b4070 con 0x7f5ec415d7d0 2024-04-17T19:48:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.877+0000 7f5ec3400640 1 -- 172.21.15.29:0/1739670745 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ebc0af400 con 0x7f5ec415d7d0 2024-04-17T19:48:03.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.877+0000 7f5ec3400640 1 -- 172.21.15.29:0/1739670745 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc0af7d0 con 0x7f5ec415d7d0 2024-04-17T19:48:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.877+0000 7f5ecba00640 1 -- 172.21.15.29:0/1739670745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec415d7d0 msgr2=0x7f5ec4164750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.877+0000 7f5ecba00640 1 --2- 172.21.15.29:0/1739670745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec415d7d0 0x7f5ec4164750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc009e50 tx=0x7f5ebc01f210 comp rx=0 tx=0).stop 2024-04-17T19:48:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.877+0000 7f5ecba00640 1 -- 172.21.15.29:0/1739670745 shutdown_connections 2024-04-17T19:48:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.877+0000 7f5ecba00640 1 --2- 172.21.15.29:0/1739670745 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec415d7d0 0x7f5ec4164750 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.877+0000 7f5ecba00640 1 --2- 172.21.15.29:0/1739670745 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ec40c8d50 0x7f5ec406bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.877+0000 7f5ecba00640 1 -- 172.21.15.29:0/1739670745 >> 172.21.15.29:0/1739670745 conn(0x7f5ec40713d0 msgr2=0x7f5ec40ce350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.877+0000 7f5ecba00640 1 -- 172.21.15.29:0/1739670745 shutdown_connections 2024-04-17T19:48:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.877+0000 7f5ecba00640 1 -- 172.21.15.29:0/1739670745 wait complete. 2024-04-17T19:48:03.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ecba00640 1 Processor -- start 2024-04-17T19:48:03.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ecba00640 1 -- start start 2024-04-17T19:48:03.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ecba00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ec40c8d50 0x7f5ec4142760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:03.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ecba00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec415d3c0 0x7f5ec4156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:03.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ecba00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ec415d7d0 0x7f5ec41593e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:03.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ecba00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec415a830 con 0x7f5ec415d3c0 2024-04-17T19:48:03.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ecba00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5ec415a550 con 0x7f5ec415d7d0 2024-04-17T19:48:03.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ecba00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5ec40c9870 con 0x7f5ec40c8d50 2024-04-17T19:48:03.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ec3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec415d3c0 0x7f5ec4156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:03.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ec3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec415d3c0 0x7f5ec4156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51912/0 (socket says 172.21.15.29:51912) 2024-04-17T19:48:03.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ec3e00640 1 -- 172.21.15.29:0/548395510 learned_addr learned my addr 172.21.15.29:0/548395510 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:03.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ec9a00640 1 --2- 172.21.15.29:0/548395510 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ec415d7d0 0x7f5ec41593e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:03.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ec9000640 1 --2- 172.21.15.29:0/548395510 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ec40c8d50 0x7f5ec4142760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:03.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ec3e00640 1 -- 172.21.15.29:0/548395510 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ec40c8d50 msgr2=0x7f5ec4142760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:03.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ec3e00640 1 --2- 172.21.15.29:0/548395510 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ec40c8d50 0x7f5ec4142760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:03.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ec3e00640 1 -- 172.21.15.29:0/548395510 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ec415d7d0 msgr2=0x7f5ec41593e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:03.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ec3e00640 1 --2- 172.21.15.29:0/548395510 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ec415d7d0 0x7f5ec41593e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:03.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ec3e00640 1 -- 172.21.15.29:0/548395510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ec4157410 con 0x7f5ec415d3c0 2024-04-17T19:48:03.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ec9000640 1 --2- 172.21.15.29:0/548395510 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ec40c8d50 0x7f5ec4142760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:03.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ec3e00640 1 --2- 172.21.15.29:0/548395510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec415d3c0 0x7f5ec4156ed0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000c310 tx=0x7f5eb000c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:03.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ec0c00640 1 -- 172.21.15.29:0/548395510 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb00052d0 con 0x7f5ec415d3c0 2024-04-17T19:48:03.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ec0c00640 1 -- 172.21.15.29:0/548395510 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5eb0005470 con 0x7f5ec415d3c0 2024-04-17T19:48:03.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ec0c00640 1 -- 172.21.15.29:0/548395510 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eb0007260 con 0x7f5ec415d3c0 2024-04-17T19:48:03.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ecba00640 1 -- 172.21.15.29:0/548395510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ec4154ae0 con 0x7f5ec415d3c0 2024-04-17T19:48:03.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.881+0000 7f5ecba00640 1 -- 172.21.15.29:0/548395510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ec4154dc0 con 0x7f5ec415d3c0 2024-04-17T19:48:03.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.885+0000 7f5ecba00640 1 -- 172.21.15.29:0/548395510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ec4069b90 con 0x7f5ec415d3c0 2024-04-17T19:48:03.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.885+0000 7f5ec0c00640 1 -- 172.21.15.29:0/548395510 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5eb0007400 con 0x7f5ec415d3c0 2024-04-17T19:48:03.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.885+0000 7f5ec0c00640 1 --2- 172.21.15.29:0/548395510 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5ea4068380 0x7f5ea406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:03.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.885+0000 7f5ec0c00640 1 -- 172.21.15.29:0/548395510 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5eb0086180 con 0x7f5ec415d3c0 2024-04-17T19:48:03.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.885+0000 7f5ec9000640 1 --2- 172.21.15.29:0/548395510 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5ea4068380 0x7f5ea406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:03.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.889+0000 7f5ec9000640 1 --2- 172.21.15.29:0/548395510 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5ea4068380 0x7f5ea406a890 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4157a20 tx=0x7f5eb8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:03.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:03.893+0000 7f5ec0c00640 1 -- 172.21.15.29:0/548395510 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5eb0055c80 con 0x7f5ec415d3c0 2024-04-17T19:48:04.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.061+0000 7f5ecba00640 1 -- 172.21.15.29:0/548395510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5ec4156810 con 0x7f5ec415d3c0 2024-04-17T19:48:04.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.061+0000 7f5ec0c00640 1 -- 172.21.15.29:0/548395510 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18394 (secure 0 0 0) 0x7f5eb0059f80 con 0x7f5ec415d3c0 2024-04-17T19:48:04.050 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:04.050 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:47:54.321794+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T19:47:47.658449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T19:47:48.209047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T19:47:51.887909+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:48:04.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.061+0000 7f5ecba00640 1 -- 172.21.15.29:0/548395510 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5ea4068380 msgr2=0x7f5ea406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.061+0000 7f5ecba00640 1 --2- 172.21.15.29:0/548395510 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5ea4068380 0x7f5ea406a890 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4157a20 tx=0x7f5eb8008040 comp rx=0 tx=0).stop 2024-04-17T19:48:04.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.061+0000 7f5ecba00640 1 -- 172.21.15.29:0/548395510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec415d3c0 msgr2=0x7f5ec4156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.061+0000 7f5ecba00640 1 --2- 172.21.15.29:0/548395510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec415d3c0 0x7f5ec4156ed0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000c310 tx=0x7f5eb000c7d0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.065+0000 7f5ecba00640 1 -- 172.21.15.29:0/548395510 shutdown_connections 2024-04-17T19:48:04.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.065+0000 7f5ecba00640 1 --2- 172.21.15.29:0/548395510 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5ea4068380 0x7f5ea406a890 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.065+0000 7f5ecba00640 1 --2- 172.21.15.29:0/548395510 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5ec415d7d0 0x7f5ec41593e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.065+0000 7f5ecba00640 1 --2- 172.21.15.29:0/548395510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec415d3c0 0x7f5ec4156ed0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.065+0000 7f5ecba00640 1 --2- 172.21.15.29:0/548395510 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5ec40c8d50 0x7f5ec4142760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.065+0000 7f5ecba00640 1 -- 172.21.15.29:0/548395510 >> 172.21.15.29:0/548395510 conn(0x7f5ec40713d0 msgr2=0x7f5ec40730c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:04.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.065+0000 7f5ecba00640 1 -- 172.21.15.29:0/548395510 shutdown_connections 2024-04-17T19:48:04.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.065+0000 7f5ecba00640 1 -- 172.21.15.29:0/548395510 wait complete. 2024-04-17T19:48:04.072 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T19:47:36.674676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T19:47:47.658449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T19:47:48.209047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T19:47:51.887909+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T19:48:04.072 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T19:48:04.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.237+0000 7f560a200640 1 Processor -- start 2024-04-17T19:48:04.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.237+0000 7f560a200640 1 -- start start 2024-04-17T19:48:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f560a200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56040c91e0 0x7f560415cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f560a200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f56040700d0 0x7f5604155b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f560a200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f560415d060 con 0x7f56040704e0 2024-04-17T19:48:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f560a200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f56040744a0 con 0x7f56040700d0 2024-04-17T19:48:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f560a200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f56040c95f0 con 0x7f56040c91e0 2024-04-17T19:48:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5603e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f56040704e0 0x7f560415f690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49660/0 (socket says 172.21.15.29:49660) 2024-04-17T19:48:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5603e00640 1 -- 172.21.15.29:0/777302181 learned_addr learned my addr 172.21.15.29:0/777302181 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602a00640 1 --2- 172.21.15.29:0/777302181 >> v2:172.21.15.37:3300/0 conn(0x7f56040700d0 0x7f5604155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5603400640 1 --2- 172.21.15.29:0/777302181 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56040c91e0 0x7f560415cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602000640 1 -- 172.21.15.29:0/777302181 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2138366260 0 0) 0x7f55fc0014a0 con 0x7f56040704e0 2024-04-17T19:48:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602000640 1 -- 172.21.15.29:0/777302181 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f55e8003530 con 0x7f56040704e0 2024-04-17T19:48:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602000640 1 -- 172.21.15.29:0/777302181 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3908013456 0 0) 0x7f55e8003530 con 0x7f56040704e0 2024-04-17T19:48:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602000640 1 -- 172.21.15.29:0/777302181 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56040c91e0 msgr2=0x7f560415cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602000640 1 --2- 172.21.15.29:0/777302181 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56040c91e0 0x7f560415cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602000640 1 -- 172.21.15.29:0/777302181 >> v2:172.21.15.37:3300/0 conn(0x7f56040700d0 msgr2=0x7f5604155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602000640 1 --2- 172.21.15.29:0/777302181 >> v2:172.21.15.37:3300/0 conn(0x7f56040700d0 0x7f5604155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602000640 1 -- 172.21.15.29:0/777302181 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56040cafd0 con 0x7f56040704e0 2024-04-17T19:48:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602000640 1 -- 172.21.15.29:0/777302181 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f55fc001bd0 con 0x7f56040704e0 2024-04-17T19:48:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602000640 1 -- 172.21.15.29:0/777302181 >> v1:172.21.15.29:6789/0 conn(0x7f56040704e0 legacy=0x7f560415f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602000640 1 --2- 172.21.15.29:0/777302181 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56040c91e0 0x7f55e80039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602000640 1 --2- 172.21.15.29:0/777302181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55e8003f40 0x7f55e8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602000640 1 --2- 172.21.15.29:0/777302181 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55e8004d50 0x7f55e8005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602000640 1 -- 172.21.15.29:0/777302181 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f56040744a0 con 0x7f55e8003f40 2024-04-17T19:48:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602000640 1 -- 172.21.15.29:0/777302181 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f56040c95f0 con 0x7f55e8004d50 2024-04-17T19:48:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602000640 1 -- 172.21.15.29:0/777302181 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f55e800bab0 con 0x7f56040c91e0 2024-04-17T19:48:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602a00640 1 --2- 172.21.15.29:0/777302181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55e8003f40 0x7f55e8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5603400640 1 --2- 172.21.15.29:0/777302181 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56040c91e0 0x7f55e80039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5603e00640 1 --2- 172.21.15.29:0/777302181 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55e8004d50 0x7f55e8005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602a00640 1 -- 172.21.15.29:0/777302181 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56040c91e0 msgr2=0x7f55e80039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602a00640 1 --2- 172.21.15.29:0/777302181 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56040c91e0 0x7f55e80039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602a00640 1 -- 172.21.15.29:0/777302181 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55e8004d50 msgr2=0x7f55e8005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:04.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602a00640 1 --2- 172.21.15.29:0/777302181 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55e8004d50 0x7f55e8005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.241+0000 7f5602a00640 1 -- 172.21.15.29:0/777302181 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55e8003530 con 0x7f55e8003f40 2024-04-17T19:48:04.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.245+0000 7f5602a00640 1 --2- 172.21.15.29:0/777302181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55e8003f40 0x7f55e8004810 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f56040ce9c0 tx=0x7f55f001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71a7719e898cb024 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.245+0000 7f5602000640 1 -- 172.21.15.29:0/777302181 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f00b2070 con 0x7f55e8003f40 2024-04-17T19:48:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.245+0000 7f5602000640 1 -- 172.21.15.29:0/777302181 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55f00a5640 con 0x7f55e8003f40 2024-04-17T19:48:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.245+0000 7f5602000640 1 -- 172.21.15.29:0/777302181 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55f00096b0 con 0x7f55e8003f40 2024-04-17T19:48:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.245+0000 7f560a200640 1 -- 172.21.15.29:0/777302181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55e8003f40 msgr2=0x7f55e8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.245+0000 7f560a200640 1 --2- 172.21.15.29:0/777302181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55e8003f40 0x7f55e8004810 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f56040ce9c0 tx=0x7f55f001ca50 comp rx=0 tx=0).stop 2024-04-17T19:48:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.245+0000 7f5603e00640 1 -- 172.21.15.29:0/777302181 reap_dead start 2024-04-17T19:48:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.245+0000 7f560a200640 1 -- 172.21.15.29:0/777302181 shutdown_connections 2024-04-17T19:48:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.245+0000 7f560a200640 1 -- 172.21.15.29:0/777302181 >> 172.21.15.29:0/777302181 conn(0x7f560406c420 msgr2=0x7f56040ca380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.245+0000 7f560a200640 1 -- 172.21.15.29:0/777302181 shutdown_connections 2024-04-17T19:48:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.245+0000 7f560a200640 1 -- 172.21.15.29:0/777302181 wait complete. 2024-04-17T19:48:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f560a200640 1 Processor -- start 2024-04-17T19:48:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f560a200640 1 -- start start 2024-04-17T19:48:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f560a200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55e8003f40 0x7f5604158f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f560a200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f560415dc70 0x7f5604159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f560a200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56040c44d0 0x7f560415ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f560a200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f56040c89e0 con 0x7f56040c44d0 2024-04-17T19:48:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f560a200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f56040c8700 con 0x7f55e8003f40 2024-04-17T19:48:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f560a200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f56040c8cc0 con 0x7f560415dc70 2024-04-17T19:48:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f5603e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56040c44d0 0x7f560415ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f5603e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56040c44d0 0x7f560415ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51936/0 (socket says 172.21.15.29:51936) 2024-04-17T19:48:04.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f5603e00640 1 -- 172.21.15.29:0/156888069 learned_addr learned my addr 172.21.15.29:0/156888069 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:04.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f5603400640 1 --2- 172.21.15.29:0/156888069 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55e8003f40 0x7f5604158f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f5602a00640 1 --2- 172.21.15.29:0/156888069 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f560415dc70 0x7f5604159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f5603e00640 1 -- 172.21.15.29:0/156888069 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f560415dc70 msgr2=0x7f5604159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f5603e00640 1 --2- 172.21.15.29:0/156888069 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f560415dc70 0x7f5604159490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f5603e00640 1 -- 172.21.15.29:0/156888069 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55e8003f40 msgr2=0x7f5604158f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f5603e00640 1 --2- 172.21.15.29:0/156888069 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55e8003f40 0x7f5604158f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f5603e00640 1 -- 172.21.15.29:0/156888069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56041f6540 con 0x7f56040c44d0 2024-04-17T19:48:04.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f5603e00640 1 --2- 172.21.15.29:0/156888069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56040c44d0 0x7f560415ef40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f56040c9770 tx=0x7f55fc00aec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:04.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f55fb400640 1 -- 172.21.15.29:0/156888069 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc017020 con 0x7f56040c44d0 2024-04-17T19:48:04.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f55fb400640 1 -- 172.21.15.29:0/156888069 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55fc0078c0 con 0x7f56040c44d0 2024-04-17T19:48:04.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f55fb400640 1 -- 172.21.15.29:0/156888069 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55fc0133f0 con 0x7f56040c44d0 2024-04-17T19:48:04.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f560a200640 1 -- 172.21.15.29:0/156888069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56041f2270 con 0x7f56040c44d0 2024-04-17T19:48:04.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.249+0000 7f560a200640 1 -- 172.21.15.29:0/156888069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56041f2510 con 0x7f56040c44d0 2024-04-17T19:48:04.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.253+0000 7f560a200640 1 -- 172.21.15.29:0/156888069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f560415e0e0 con 0x7f56040c44d0 2024-04-17T19:48:04.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.253+0000 7f55fb400640 1 -- 172.21.15.29:0/156888069 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f55fc01a020 con 0x7f56040c44d0 2024-04-17T19:48:04.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.253+0000 7f55fb400640 1 --2- 172.21.15.29:0/156888069 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f55e4068380 0x7f55e406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.253+0000 7f55fb400640 1 -- 172.21.15.29:0/156888069 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f55fc087580 con 0x7f56040c44d0 2024-04-17T19:48:04.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.253+0000 7f5603400640 1 --2- 172.21.15.29:0/156888069 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f55e4068380 0x7f55e406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.257+0000 7f5603400640 1 --2- 172.21.15.29:0/156888069 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f55e4068380 0x7f55e406a890 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f55f4009b40 tx=0x7f55f4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:04.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.257+0000 7f55fb400640 1 -- 172.21.15.29:0/156888069 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f55fc021020 con 0x7f56040c44d0 2024-04-17T19:48:04.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.425+0000 7f560a200640 1 -- 172.21.15.29:0/156888069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5604069b90 con 0x7f56040c44d0 2024-04-17T19:48:04.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.425+0000 7f55fb400640 1 -- 172.21.15.29:0/156888069 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f560415d060 con 0x7f56040c44d0 2024-04-17T19:48:04.414 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-04-17T19:48:04.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.429+0000 7f560a200640 1 -- 172.21.15.29:0/156888069 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f55e4068380 msgr2=0x7f55e406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.429+0000 7f560a200640 1 --2- 172.21.15.29:0/156888069 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f55e4068380 0x7f55e406a890 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f55f4009b40 tx=0x7f55f4009190 comp rx=0 tx=0).stop 2024-04-17T19:48:04.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.429+0000 7f560a200640 1 -- 172.21.15.29:0/156888069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56040c44d0 msgr2=0x7f560415ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.429+0000 7f560a200640 1 --2- 172.21.15.29:0/156888069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56040c44d0 0x7f560415ef40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f56040c9770 tx=0x7f55fc00aec0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.429+0000 7f560a200640 1 -- 172.21.15.29:0/156888069 shutdown_connections 2024-04-17T19:48:04.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.429+0000 7f560a200640 1 --2- 172.21.15.29:0/156888069 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f55e4068380 0x7f55e406a890 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.429+0000 7f560a200640 1 --2- 172.21.15.29:0/156888069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56040c44d0 0x7f560415ef40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.429+0000 7f560a200640 1 --2- 172.21.15.29:0/156888069 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f560415dc70 0x7f5604159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.429+0000 7f560a200640 1 --2- 172.21.15.29:0/156888069 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f55e8003f40 0x7f5604158f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.429+0000 7f560a200640 1 -- 172.21.15.29:0/156888069 >> 172.21.15.29:0/156888069 conn(0x7f560406c420 msgr2=0x7f560415abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:04.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.429+0000 7f560a200640 1 -- 172.21.15.29:0/156888069 shutdown_connections 2024-04-17T19:48:04.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.429+0000 7f560a200640 1 -- 172.21.15.29:0/156888069 wait complete. 2024-04-17T19:48:04.436 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T19:48:04.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.601+0000 7fb4b9e00640 1 Processor -- start 2024-04-17T19:48:04.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.601+0000 7fb4b9e00640 1 -- start start 2024-04-17T19:48:04.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b9e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb4b41561e0 0x7fb4b41678b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b9e00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b40c8210 con 0x7fb4b415bf20 2024-04-17T19:48:04.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b9e00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4b40c84f0 con 0x7fb4b40ce260 2024-04-17T19:48:04.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b9e00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb4b40c9590 con 0x7fb4b41561e0 2024-04-17T19:48:04.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4abe00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb4b415bf20 0x7fb4b406f8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49668/0 (socket says 172.21.15.29:49668) 2024-04-17T19:48:04.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4abe00640 1 -- 172.21.15.29:0/1768117819 learned_addr learned my addr 172.21.15.29:0/1768117819 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:04.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b3400640 1 --2- 172.21.15.29:0/1768117819 >> v2:172.21.15.37:3300/0 conn(0x7fb4b41561e0 0x7fb4b41678b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b2000640 1 -- 172.21.15.29:0/1768117819 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3913110223 0 0) 0x7fb4b40c8210 con 0x7fb4b415bf20 2024-04-17T19:48:04.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b2000640 1 -- 172.21.15.29:0/1768117819 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb498003530 con 0x7fb4b415bf20 2024-04-17T19:48:04.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b2000640 1 -- 172.21.15.29:0/1768117819 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 255532482 0 0) 0x7fb498003530 con 0x7fb4b415bf20 2024-04-17T19:48:04.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b2000640 1 -- 172.21.15.29:0/1768117819 >> v2:172.21.15.37:3300/0 conn(0x7fb4b41561e0 msgr2=0x7fb4b41678b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:04.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b2000640 1 --2- 172.21.15.29:0/1768117819 >> v2:172.21.15.37:3300/0 conn(0x7fb4b41561e0 0x7fb4b41678b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b2000640 1 -- 172.21.15.29:0/1768117819 >> v1:172.21.15.37:6789/0 conn(0x7fb4b40ce260 legacy=0x7fb4b415d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b2000640 1 -- 172.21.15.29:0/1768117819 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b40c87d0 con 0x7fb4b415bf20 2024-04-17T19:48:04.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b2000640 1 -- 172.21.15.29:0/1768117819 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fb49c001680 con 0x7fb4b415bf20 2024-04-17T19:48:04.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b2000640 1 -- 172.21.15.29:0/1768117819 >> v1:172.21.15.29:6789/0 conn(0x7fb4b415bf20 legacy=0x7fb4b406f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b2000640 1 --2- 172.21.15.29:0/1768117819 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4980039c0 0x7fb498003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b2000640 1 --2- 172.21.15.29:0/1768117819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb498004810 0x7fb498004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b2000640 1 --2- 172.21.15.29:0/1768117819 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4980051d0 0x7fb49800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b2000640 1 -- 172.21.15.29:0/1768117819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b40c9590 con 0x7fb498004810 2024-04-17T19:48:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b2000640 1 -- 172.21.15.29:0/1768117819 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb49800c160 con 0x7fb4980039c0 2024-04-17T19:48:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b2000640 1 -- 172.21.15.29:0/1768117819 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb49800c360 con 0x7fb4980051d0 2024-04-17T19:48:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4abe00640 1 --2- 172.21.15.29:0/1768117819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb498004810 0x7fb498004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b2a00640 1 --2- 172.21.15.29:0/1768117819 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4980039c0 0x7fb498003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b3400640 1 --2- 172.21.15.29:0/1768117819 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4980051d0 0x7fb49800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4abe00640 1 -- 172.21.15.29:0/1768117819 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4980051d0 msgr2=0x7fb49800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4abe00640 1 --2- 172.21.15.29:0/1768117819 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4980051d0 0x7fb49800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4abe00640 1 -- 172.21.15.29:0/1768117819 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4980039c0 msgr2=0x7fb498003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4abe00640 1 --2- 172.21.15.29:0/1768117819 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4980039c0 0x7fb498003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4abe00640 1 -- 172.21.15.29:0/1768117819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb498003530 con 0x7fb498004810 2024-04-17T19:48:04.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.605+0000 7fb4b3400640 1 -- 172.21.15.29:0/1768117819 reap_dead start 2024-04-17T19:48:04.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.609+0000 7fb4abe00640 1 --2- 172.21.15.29:0/1768117819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb498004810 0x7fb498004c20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb4b40ca030 tx=0x7fb49c01cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df2964e6309c9387 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:04.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.609+0000 7fb4b2000640 1 -- 172.21.15.29:0/1768117819 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49c007d20 con 0x7fb498004810 2024-04-17T19:48:04.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.609+0000 7fb4b2000640 1 -- 172.21.15.29:0/1768117819 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb49c0a9070 con 0x7fb498004810 2024-04-17T19:48:04.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.609+0000 7fb4b2000640 1 -- 172.21.15.29:0/1768117819 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb49c005d80 con 0x7fb498004810 2024-04-17T19:48:04.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.609+0000 7fb4b9e00640 1 -- 172.21.15.29:0/1768117819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb498004810 msgr2=0x7fb498004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.609+0000 7fb4b9e00640 1 --2- 172.21.15.29:0/1768117819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb498004810 0x7fb498004c20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb4b40ca030 tx=0x7fb49c01cba0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.609+0000 7fb4b9e00640 1 -- 172.21.15.29:0/1768117819 shutdown_connections 2024-04-17T19:48:04.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.609+0000 7fb4b9e00640 1 --2- 172.21.15.29:0/1768117819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb498004810 0x7fb498004c20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.609+0000 7fb4b9e00640 1 -- 172.21.15.29:0/1768117819 >> 172.21.15.29:0/1768117819 conn(0x7fb4b406c470 msgr2=0x7fb4b40cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:04.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.609+0000 7fb4b9e00640 1 -- 172.21.15.29:0/1768117819 shutdown_connections 2024-04-17T19:48:04.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.609+0000 7fb4b9e00640 1 -- 172.21.15.29:0/1768117819 wait complete. 2024-04-17T19:48:04.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b9e00640 1 Processor -- start 2024-04-17T19:48:04.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b9e00640 1 -- start start 2024-04-17T19:48:04.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b9e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb498004810 0x7fb4b4156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b9e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4b41544b0 0x7fb4b41f60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b9e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4b4073360 0x7fb4b4149a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b9e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b40c8d90 con 0x7fb4b4073360 2024-04-17T19:48:04.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b9e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb4b40c9100 con 0x7fb4b41544b0 2024-04-17T19:48:04.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b9e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb4b41565f0 con 0x7fb498004810 2024-04-17T19:48:04.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b3400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4b4073360 0x7fb4b4149a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b3400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4b4073360 0x7fb4b4149a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51946/0 (socket says 172.21.15.29:51946) 2024-04-17T19:48:04.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b3400640 1 -- 172.21.15.29:0/2083477065 learned_addr learned my addr 172.21.15.29:0/2083477065 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:04.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4abe00640 1 --2- 172.21.15.29:0/2083477065 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4b41544b0 0x7fb4b41f60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b2a00640 1 --2- 172.21.15.29:0/2083477065 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb498004810 0x7fb4b4156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b3400640 1 -- 172.21.15.29:0/2083477065 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb498004810 msgr2=0x7fb4b4156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b3400640 1 --2- 172.21.15.29:0/2083477065 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb498004810 0x7fb4b4156990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b3400640 1 -- 172.21.15.29:0/2083477065 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4b41544b0 msgr2=0x7fb4b41f60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b3400640 1 --2- 172.21.15.29:0/2083477065 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4b41544b0 0x7fb4b41f60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b3400640 1 -- 172.21.15.29:0/2083477065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4b41f6630 con 0x7fb4b4073360 2024-04-17T19:48:04.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b2a00640 1 --2- 172.21.15.29:0/2083477065 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb498004810 0x7fb4b4156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:04.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b3400640 1 --2- 172.21.15.29:0/2083477065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4b4073360 0x7fb4b4149a20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4071bc0 tx=0x7fb4ac0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:04.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4ab400640 1 -- 172.21.15.29:0/2083477065 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac011e40 con 0x7fb4b4073360 2024-04-17T19:48:04.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4ab400640 1 -- 172.21.15.29:0/2083477065 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4ac01b070 con 0x7fb4b4073360 2024-04-17T19:48:04.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4ab400640 1 -- 172.21.15.29:0/2083477065 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4ac0039a0 con 0x7fb4b4073360 2024-04-17T19:48:04.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b9e00640 1 -- 172.21.15.29:0/2083477065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4b41e5530 con 0x7fb4b4073360 2024-04-17T19:48:04.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.613+0000 7fb4b9e00640 1 -- 172.21.15.29:0/2083477065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4b41e5af0 con 0x7fb4b4073360 2024-04-17T19:48:04.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.617+0000 7fb4b9e00640 1 -- 172.21.15.29:0/2083477065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4b41e5d30 con 0x7fb4b4073360 2024-04-17T19:48:04.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.617+0000 7fb4ab400640 1 -- 172.21.15.29:0/2083477065 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb4ac003b40 con 0x7fb4b4073360 2024-04-17T19:48:04.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.617+0000 7fb4ab400640 1 --2- 172.21.15.29:0/2083477065 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb490068380 0x7fb49006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.617+0000 7fb4ab400640 1 -- 172.21.15.29:0/2083477065 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb4ac087690 con 0x7fb4b4073360 2024-04-17T19:48:04.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.617+0000 7fb4b2a00640 1 --2- 172.21.15.29:0/2083477065 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb490068380 0x7fb49006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.617+0000 7fb4b2a00640 1 --2- 172.21.15.29:0/2083477065 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb490068380 0x7fb49006a890 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4000c10 tx=0x7fb4a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:04.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.621+0000 7fb4ab400640 1 -- 172.21.15.29:0/2083477065 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb4b41e5d30 con 0x7fb4b4073360 2024-04-17T19:48:04.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.785+0000 7fb4b9e00640 1 -- 172.21.15.29:0/2083477065 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb4b41f5a50 con 0x7fb4b4073360 2024-04-17T19:48:04.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.789+0000 7fb4ab400640 1 -- 172.21.15.29:0/2083477065 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7fb4ac025020 con 0x7fb4b4073360 2024-04-17T19:48:04.776 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-04-17T19:48:04.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.789+0000 7fb4b9e00640 1 -- 172.21.15.29:0/2083477065 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb490068380 msgr2=0x7fb49006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.789+0000 7fb4b9e00640 1 --2- 172.21.15.29:0/2083477065 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb490068380 0x7fb49006a890 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4000c10 tx=0x7fb4a4009040 comp rx=0 tx=0).stop 2024-04-17T19:48:04.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.789+0000 7fb4b9e00640 1 -- 172.21.15.29:0/2083477065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4b4073360 msgr2=0x7fb4b4149a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.789+0000 7fb4b9e00640 1 --2- 172.21.15.29:0/2083477065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4b4073360 0x7fb4b4149a20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb4b4071bc0 tx=0x7fb4ac0115a0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.789+0000 7fb4b9e00640 1 -- 172.21.15.29:0/2083477065 shutdown_connections 2024-04-17T19:48:04.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.789+0000 7fb4b9e00640 1 --2- 172.21.15.29:0/2083477065 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb490068380 0x7fb49006a890 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.789+0000 7fb4b9e00640 1 --2- 172.21.15.29:0/2083477065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4b4073360 0x7fb4b4149a20 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.789+0000 7fb4b9e00640 1 --2- 172.21.15.29:0/2083477065 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb4b41544b0 0x7fb4b41f60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.789+0000 7fb4b9e00640 1 --2- 172.21.15.29:0/2083477065 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb498004810 0x7fb4b4156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.789+0000 7fb4b9e00640 1 -- 172.21.15.29:0/2083477065 >> 172.21.15.29:0/2083477065 conn(0x7fb4b406c470 msgr2=0x7fb4b414c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:04.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.789+0000 7fb4b9e00640 1 -- 172.21.15.29:0/2083477065 shutdown_connections 2024-04-17T19:48:04.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.789+0000 7fb4b9e00640 1 -- 172.21.15.29:0/2083477065 wait complete. 2024-04-17T19:48:04.798 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T19:48:04.802 INFO:tasks.ceph.mds.c.smithi029.stderr:2024-04-17T19:48:04.813+0000 7ff3b0200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T19:48:04.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.965+0000 7faaa1000640 1 Processor -- start 2024-04-17T19:48:04.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.965+0000 7faaa1000640 1 -- start start 2024-04-17T19:48:04.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.969+0000 7faaa1000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa9c071ce0 0x7faa9c141de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.969+0000 7faaa1000640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa9c142320 con 0x7faa9c1597e0 2024-04-17T19:48:04.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.969+0000 7faaa1000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa9c15c6d0 con 0x7faa9c1593d0 2024-04-17T19:48:04.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.969+0000 7faaa1000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7faa9c0c7bd0 con 0x7faa9c071ce0 2024-04-17T19:48:04.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.969+0000 7faa99e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa9c071ce0 0x7faa9c141de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.969+0000 7faa9a800640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faa9c1597e0 0x7faa9c163620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49682/0 (socket says 172.21.15.29:49682) 2024-04-17T19:48:04.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.969+0000 7faa9a800640 1 -- 172.21.15.29:0/120664521 learned_addr learned my addr 172.21.15.29:0/120664521 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:04.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.969+0000 7faa99e00640 1 -- 172.21.15.29:0/120664521 >> v1:172.21.15.37:6789/0 conn(0x7faa9c1593d0 legacy=0x7faa9c0c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.969+0000 7faa99e00640 1 -- 172.21.15.29:0/120664521 >> v1:172.21.15.29:6789/0 conn(0x7faa9c1597e0 legacy=0x7faa9c163620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.969+0000 7faa99e00640 1 -- 172.21.15.29:0/120664521 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa9c15c9b0 con 0x7faa9c071ce0 2024-04-17T19:48:04.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.973+0000 7faa99e00640 1 --2- 172.21.15.29:0/120664521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa9c071ce0 0x7faa9c141de0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0097c0 tx=0x7faa8c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5ebaf9cf101864d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:04.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.973+0000 7faa99400640 1 -- 172.21.15.29:0/120664521 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c0b0070 con 0x7faa9c071ce0 2024-04-17T19:48:04.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.973+0000 7faa99400640 1 -- 172.21.15.29:0/120664521 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa8c0ab600 con 0x7faa9c071ce0 2024-04-17T19:48:04.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.973+0000 7faa99400640 1 -- 172.21.15.29:0/120664521 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa8c0ab9d0 con 0x7faa9c071ce0 2024-04-17T19:48:04.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.973+0000 7faaa1000640 1 -- 172.21.15.29:0/120664521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa9c071ce0 msgr2=0x7faa9c141de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.973+0000 7faaa1000640 1 --2- 172.21.15.29:0/120664521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa9c071ce0 0x7faa9c141de0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0097c0 tx=0x7faa8c01c4a0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.973+0000 7faaa1000640 1 -- 172.21.15.29:0/120664521 shutdown_connections 2024-04-17T19:48:04.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.973+0000 7faaa1000640 1 --2- 172.21.15.29:0/120664521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa9c071ce0 0x7faa9c141de0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.973+0000 7faaa1000640 1 -- 172.21.15.29:0/120664521 >> 172.21.15.29:0/120664521 conn(0x7faa9c06bf90 msgr2=0x7faa9c06c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:04.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.973+0000 7faaa1000640 1 -- 172.21.15.29:0/120664521 shutdown_connections 2024-04-17T19:48:04.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.973+0000 7faaa1000640 1 -- 172.21.15.29:0/120664521 wait complete. 2024-04-17T19:48:04.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.973+0000 7faaa1000640 1 Processor -- start 2024-04-17T19:48:04.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.973+0000 7faaa1000640 1 -- start start 2024-04-17T19:48:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faaa1000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa9c071ce0 0x7faa9c1578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faaa1000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa9c1593d0 0x7faa9c163df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faaa1000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa9c1597e0 0x7faa9c1f4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faaa1000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faa9c0720f0 con 0x7faa9c1593d0 2024-04-17T19:48:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faaa1000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7faa9c159bf0 con 0x7faa9c1597e0 2024-04-17T19:48:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faaa1000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7faa9c072270 con 0x7faa9c071ce0 2024-04-17T19:48:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faa91400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa9c1593d0 0x7faa9c163df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faa91400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa9c1593d0 0x7faa9c163df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51966/0 (socket says 172.21.15.29:51966) 2024-04-17T19:48:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faa91400640 1 -- 172.21.15.29:0/2813695900 learned_addr learned my addr 172.21.15.29:0/2813695900 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faa9a800640 1 --2- 172.21.15.29:0/2813695900 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa9c1597e0 0x7faa9c1f4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faa99e00640 1 --2- 172.21.15.29:0/2813695900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa9c071ce0 0x7faa9c1578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faa91400640 1 -- 172.21.15.29:0/2813695900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa9c071ce0 msgr2=0x7faa9c1578a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faa91400640 1 --2- 172.21.15.29:0/2813695900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa9c071ce0 0x7faa9c1578a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faa91400640 1 -- 172.21.15.29:0/2813695900 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa9c1597e0 msgr2=0x7faa9c1f4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:04.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faa91400640 1 --2- 172.21.15.29:0/2813695900 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa9c1597e0 0x7faa9c1f4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:04.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faa91400640 1 -- 172.21.15.29:0/2813695900 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa9c068770 con 0x7faa9c1593d0 2024-04-17T19:48:04.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faa91400640 1 --2- 172.21.15.29:0/2813695900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa9c1593d0 0x7faa9c163df0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7faa8400c3f0 tx=0x7faa8400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:04.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faa92a00640 1 -- 172.21.15.29:0/2813695900 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa84017070 con 0x7faa9c1593d0 2024-04-17T19:48:04.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faa92a00640 1 -- 172.21.15.29:0/2813695900 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa840052d0 con 0x7faa9c1593d0 2024-04-17T19:48:04.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faa92a00640 1 -- 172.21.15.29:0/2813695900 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa84003840 con 0x7faa9c1593d0 2024-04-17T19:48:04.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faaa1000640 1 -- 172.21.15.29:0/2813695900 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa9c157de0 con 0x7faa9c1593d0 2024-04-17T19:48:04.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faaa1000640 1 -- 172.21.15.29:0/2813695900 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa9c1f5de0 con 0x7faa9c1593d0 2024-04-17T19:48:04.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.977+0000 7faaa1000640 1 -- 172.21.15.29:0/2813695900 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa9c069b90 con 0x7faa9c1593d0 2024-04-17T19:48:04.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.981+0000 7faa92a00640 1 -- 172.21.15.29:0/2813695900 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7faa8401a070 con 0x7faa9c1593d0 2024-04-17T19:48:04.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.981+0000 7faa92a00640 1 --2- 172.21.15.29:0/2813695900 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faa78068380 0x7faa7806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:04.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.981+0000 7faa92a00640 1 -- 172.21.15.29:0/2813695900 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7faa840860e0 con 0x7faa9c1593d0 2024-04-17T19:48:04.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.981+0000 7faa99e00640 1 --2- 172.21.15.29:0/2813695900 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faa78068380 0x7faa7806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:04.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.981+0000 7faa99e00640 1 --2- 172.21.15.29:0/2813695900 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faa78068380 0x7faa7806a890 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0ad000 tx=0x7faa8c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:04.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:04.985+0000 7faa92a00640 1 -- 172.21.15.29:0/2813695900 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7faa84005480 con 0x7faa9c1593d0 2024-04-17T19:48:05.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.149+0000 7faaa1000640 1 -- 172.21.15.29:0/2813695900 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7faa9c157500 con 0x7faa9c1593d0 2024-04-17T19:48:05.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.149+0000 7faa92a00640 1 -- 172.21.15.29:0/2813695900 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7faa84055be0 con 0x7faa9c1593d0 2024-04-17T19:48:05.136 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-04-17T19:48:05.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.149+0000 7faaa1000640 1 -- 172.21.15.29:0/2813695900 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faa78068380 msgr2=0x7faa7806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:05.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.149+0000 7faaa1000640 1 --2- 172.21.15.29:0/2813695900 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faa78068380 0x7faa7806a890 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0ad000 tx=0x7faa8c0a6000 comp rx=0 tx=0).stop 2024-04-17T19:48:05.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.149+0000 7faaa1000640 1 -- 172.21.15.29:0/2813695900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa9c1593d0 msgr2=0x7faa9c163df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:05.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.149+0000 7faaa1000640 1 --2- 172.21.15.29:0/2813695900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa9c1593d0 0x7faa9c163df0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7faa8400c3f0 tx=0x7faa8400c8b0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.149+0000 7faaa1000640 1 -- 172.21.15.29:0/2813695900 shutdown_connections 2024-04-17T19:48:05.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.149+0000 7faaa1000640 1 --2- 172.21.15.29:0/2813695900 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faa78068380 0x7faa7806a890 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.149+0000 7faaa1000640 1 --2- 172.21.15.29:0/2813695900 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa9c1597e0 0x7faa9c1f4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.149+0000 7faaa1000640 1 --2- 172.21.15.29:0/2813695900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa9c1593d0 0x7faa9c163df0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.149+0000 7faaa1000640 1 --2- 172.21.15.29:0/2813695900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa9c071ce0 0x7faa9c1578a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.149+0000 7faaa1000640 1 -- 172.21.15.29:0/2813695900 >> 172.21.15.29:0/2813695900 conn(0x7faa9c06bf90 msgr2=0x7faa9c06f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:05.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.149+0000 7faaa1000640 1 -- 172.21.15.29:0/2813695900 shutdown_connections 2024-04-17T19:48:05.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.149+0000 7faaa1000640 1 -- 172.21.15.29:0/2813695900 wait complete. 2024-04-17T19:48:05.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T19:48:05.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.321+0000 7fc4a7400640 1 Processor -- start 2024-04-17T19:48:05.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.321+0000 7fc4a7400640 1 -- start start 2024-04-17T19:48:05.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.325+0000 7fc4a7400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc4a00c91e0 0x7fc4a015cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:05.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.325+0000 7fc4a7400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc4a00700d0 0x7fc4a0155b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:05.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.325+0000 7fc4a7400640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4a015d060 con 0x7fc4a00704e0 2024-04-17T19:48:05.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.325+0000 7fc4a7400640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fc4a00744a0 con 0x7fc4a00c91e0 2024-04-17T19:48:05.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.325+0000 7fc4a7400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc4a00c95f0 con 0x7fc4a00700d0 2024-04-17T19:48:05.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.325+0000 7fc4a4a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc4a00c91e0 0x7fc4a015cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:05.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.325+0000 7fc4a5400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc4a00704e0 0x7fc4a015f690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49698/0 (socket says 172.21.15.29:49698) 2024-04-17T19:48:05.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.325+0000 7fc4a5400640 1 -- 172.21.15.29:0/1925729330 learned_addr learned my addr 172.21.15.29:0/1925729330 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:05.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.325+0000 7fc49fe00640 1 --2- 172.21.15.29:0/1925729330 >> v2:172.21.15.37:3300/0 conn(0x7fc4a00700d0 0x7fc4a0155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:05.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.325+0000 7fc4a4a00640 1 -- 172.21.15.29:0/1925729330 >> v2:172.21.15.37:3300/0 conn(0x7fc4a00700d0 msgr2=0x7fc4a0155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:05.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.325+0000 7fc4a4a00640 1 --2- 172.21.15.29:0/1925729330 >> v2:172.21.15.37:3300/0 conn(0x7fc4a00700d0 0x7fc4a0155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.325+0000 7fc4a4a00640 1 -- 172.21.15.29:0/1925729330 >> v1:172.21.15.29:6789/0 conn(0x7fc4a00704e0 legacy=0x7fc4a015f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:05.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.325+0000 7fc4a4a00640 1 -- 172.21.15.29:0/1925729330 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a00cafd0 con 0x7fc4a00c91e0 2024-04-17T19:48:05.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.325+0000 7fc49f400640 1 -- 172.21.15.29:0/1925729330 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3736299871 0 0) 0x7fc4980014a0 con 0x7fc4a00704e0 2024-04-17T19:48:05.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.329+0000 7fc4a4a00640 1 --2- 172.21.15.29:0/1925729330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4a00c91e0 0x7fc4a015cb20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00c4a30 tx=0x7fc49401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2b79359f97a5a1d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:05.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.329+0000 7fc49f400640 1 -- 172.21.15.29:0/1925729330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4940b0070 con 0x7fc4a00c91e0 2024-04-17T19:48:05.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.329+0000 7fc49f400640 1 -- 172.21.15.29:0/1925729330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4940ab600 con 0x7fc4a00c91e0 2024-04-17T19:48:05.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.329+0000 7fc49f400640 1 -- 172.21.15.29:0/1925729330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4940ab9d0 con 0x7fc4a00c91e0 2024-04-17T19:48:05.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.329+0000 7fc4a7400640 1 -- 172.21.15.29:0/1925729330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4a00c91e0 msgr2=0x7fc4a015cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:05.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.329+0000 7fc4a7400640 1 --2- 172.21.15.29:0/1925729330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4a00c91e0 0x7fc4a015cb20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc4a00c4a30 tx=0x7fc49401c480 comp rx=0 tx=0).stop 2024-04-17T19:48:05.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.329+0000 7fc4a7400640 1 -- 172.21.15.29:0/1925729330 shutdown_connections 2024-04-17T19:48:05.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.329+0000 7fc4a7400640 1 --2- 172.21.15.29:0/1925729330 >> v2:172.21.15.37:3300/0 conn(0x7fc4a00700d0 0x7fc4a0155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.329+0000 7fc4a7400640 1 --2- 172.21.15.29:0/1925729330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4a00c91e0 0x7fc4a015cb20 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.329+0000 7fc4a7400640 1 -- 172.21.15.29:0/1925729330 >> 172.21.15.29:0/1925729330 conn(0x7fc4a006c420 msgr2=0x7fc4a00ca380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:05.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.329+0000 7fc4a7400640 1 -- 172.21.15.29:0/1925729330 shutdown_connections 2024-04-17T19:48:05.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.329+0000 7fc4a7400640 1 -- 172.21.15.29:0/1925729330 wait complete. 2024-04-17T19:48:05.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a7400640 1 Processor -- start 2024-04-17T19:48:05.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a7400640 1 -- start start 2024-04-17T19:48:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a7400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4a00700d0 0x7fc4a0148840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a7400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4a00704e0 0x7fc4a014a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a7400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4a00c91e0 0x7fc4a014bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a7400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a01564c0 con 0x7fc4a00c91e0 2024-04-17T19:48:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a7400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc4a00c8cc0 con 0x7fc4a00704e0 2024-04-17T19:48:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a7400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc4a00c9f60 con 0x7fc4a00700d0 2024-04-17T19:48:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a5400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4a00c91e0 0x7fc4a014bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a5400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4a00c91e0 0x7fc4a014bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51986/0 (socket says 172.21.15.29:51986) 2024-04-17T19:48:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a5400640 1 -- 172.21.15.29:0/738444083 learned_addr learned my addr 172.21.15.29:0/738444083 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc49fe00640 1 --2- 172.21.15.29:0/738444083 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4a00704e0 0x7fc4a014a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a4a00640 1 --2- 172.21.15.29:0/738444083 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4a00700d0 0x7fc4a0148840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a5400640 1 -- 172.21.15.29:0/738444083 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4a00700d0 msgr2=0x7fc4a0148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a5400640 1 --2- 172.21.15.29:0/738444083 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4a00700d0 0x7fc4a0148840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a5400640 1 -- 172.21.15.29:0/738444083 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4a00704e0 msgr2=0x7fc4a014a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a5400640 1 --2- 172.21.15.29:0/738444083 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4a00704e0 0x7fc4a014a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a5400640 1 -- 172.21.15.29:0/738444083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4a015ecc0 con 0x7fc4a00c91e0 2024-04-17T19:48:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a4a00640 1 --2- 172.21.15.29:0/738444083 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4a00700d0 0x7fc4a0148840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a5400640 1 --2- 172.21.15.29:0/738444083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4a00c91e0 0x7fc4a014bdc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc49800c700 tx=0x7fc49800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc49cc00640 1 -- 172.21.15.29:0/738444083 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498018070 con 0x7fc4a00c91e0 2024-04-17T19:48:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc49cc00640 1 -- 172.21.15.29:0/738444083 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc49800ce10 con 0x7fc4a00c91e0 2024-04-17T19:48:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc49cc00640 1 -- 172.21.15.29:0/738444083 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49800a820 con 0x7fc4a00c91e0 2024-04-17T19:48:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a7400640 1 -- 172.21.15.29:0/738444083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4a014d1d0 con 0x7fc4a00c91e0 2024-04-17T19:48:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.333+0000 7fc4a7400640 1 -- 172.21.15.29:0/738444083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4a01f5f40 con 0x7fc4a00c91e0 2024-04-17T19:48:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.337+0000 7fc4a7400640 1 -- 172.21.15.29:0/738444083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4a0069b90 con 0x7fc4a00c91e0 2024-04-17T19:48:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.337+0000 7fc49cc00640 1 -- 172.21.15.29:0/738444083 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc49801b070 con 0x7fc4a00c91e0 2024-04-17T19:48:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.337+0000 7fc49cc00640 1 --2- 172.21.15.29:0/738444083 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc480068380 0x7fc48006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.337+0000 7fc49cc00640 1 -- 172.21.15.29:0/738444083 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc498086160 con 0x7fc4a00c91e0 2024-04-17T19:48:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.337+0000 7fc4a4a00640 1 --2- 172.21.15.29:0/738444083 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc480068380 0x7fc48006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.337+0000 7fc4a4a00640 1 --2- 172.21.15.29:0/738444083 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc480068380 0x7fc48006a890 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc4940ad000 tx=0x7fc4940a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.341+0000 7fc49cc00640 1 -- 172.21.15.29:0/738444083 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc4a015d060 con 0x7fc4a00c91e0 2024-04-17T19:48:05.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.505+0000 7fc4a7400640 1 -- 172.21.15.29:0/738444083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc4a0067ce0 con 0x7fc4a00c91e0 2024-04-17T19:48:05.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.505+0000 7fc49cc00640 1 -- 172.21.15.29:0/738444083 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7fc498055d80 con 0x7fc4a00c91e0 2024-04-17T19:48:05.494 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-04-17T19:48:05.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.509+0000 7fc4a7400640 1 -- 172.21.15.29:0/738444083 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc480068380 msgr2=0x7fc48006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:05.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.509+0000 7fc4a7400640 1 --2- 172.21.15.29:0/738444083 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc480068380 0x7fc48006a890 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc4940ad000 tx=0x7fc4940a6000 comp rx=0 tx=0).stop 2024-04-17T19:48:05.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.509+0000 7fc4a7400640 1 -- 172.21.15.29:0/738444083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4a00c91e0 msgr2=0x7fc4a014bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:05.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.509+0000 7fc4a7400640 1 --2- 172.21.15.29:0/738444083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4a00c91e0 0x7fc4a014bdc0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc49800c700 tx=0x7fc49800cbc0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.509+0000 7fc4a7400640 1 -- 172.21.15.29:0/738444083 shutdown_connections 2024-04-17T19:48:05.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.509+0000 7fc4a7400640 1 --2- 172.21.15.29:0/738444083 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc480068380 0x7fc48006a890 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.509+0000 7fc4a7400640 1 --2- 172.21.15.29:0/738444083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc4a00c91e0 0x7fc4a014bdc0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.509+0000 7fc4a7400640 1 --2- 172.21.15.29:0/738444083 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc4a00704e0 0x7fc4a014a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.509+0000 7fc4a7400640 1 --2- 172.21.15.29:0/738444083 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc4a00700d0 0x7fc4a0148840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.509+0000 7fc4a7400640 1 -- 172.21.15.29:0/738444083 >> 172.21.15.29:0/738444083 conn(0x7fc4a006c420 msgr2=0x7fc4a0073be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:05.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.509+0000 7fc4a7400640 1 -- 172.21.15.29:0/738444083 shutdown_connections 2024-04-17T19:48:05.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.509+0000 7fc4a7400640 1 -- 172.21.15.29:0/738444083 wait complete. 2024-04-17T19:48:05.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T19:48:05.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.681+0000 7f937a200640 1 Processor -- start 2024-04-17T19:48:05.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.681+0000 7f937a200640 1 -- start start 2024-04-17T19:48:05.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.681+0000 7f937a200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f93740c6f90 0x7f93740c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:05.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.681+0000 7f937a200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f937415d800 0x7f937415dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:05.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.681+0000 7f937a200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93740c8320 con 0x7f93740cacb0 2024-04-17T19:48:05.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.681+0000 7f937a200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f93740c9370 con 0x7f93740c6f90 2024-04-17T19:48:05.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.681+0000 7f937a200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f937415d370 con 0x7f937415d800 2024-04-17T19:48:05.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.681+0000 7f936be00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f93740c6f90 0x7f93740c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:05.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.681+0000 7f9373400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f93740cacb0 0x7f9374153e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49700/0 (socket says 172.21.15.29:49700) 2024-04-17T19:48:05.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.681+0000 7f9373400640 1 -- 172.21.15.29:0/3261336512 learned_addr learned my addr 172.21.15.29:0/3261336512 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:05.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.681+0000 7f9373e00640 1 --2- 172.21.15.29:0/3261336512 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f937415d800 0x7f937415dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:05.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.681+0000 7f9372a00640 1 -- 172.21.15.29:0/3261336512 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1660581932 0 0) 0x7f93740c8320 con 0x7f93740cacb0 2024-04-17T19:48:05.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.681+0000 7f936be00640 1 -- 172.21.15.29:0/3261336512 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f937415d800 msgr2=0x7f937415dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:05.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.681+0000 7f936be00640 1 --2- 172.21.15.29:0/3261336512 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f937415d800 0x7f937415dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.681+0000 7f936be00640 1 -- 172.21.15.29:0/3261336512 >> v1:172.21.15.29:6789/0 conn(0x7f93740cacb0 legacy=0x7f9374153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:05.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.681+0000 7f936be00640 1 -- 172.21.15.29:0/3261336512 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93740c8600 con 0x7f93740c6f90 2024-04-17T19:48:05.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.685+0000 7f936be00640 1 --2- 172.21.15.29:0/3261336512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93740c6f90 0x7f93740c73a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f935c0097c0 tx=0x7f935c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5f78532c2f5ab9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:05.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.685+0000 7f9372a00640 1 -- 172.21.15.29:0/3261336512 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935c0b2070 con 0x7f93740c6f90 2024-04-17T19:48:05.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.685+0000 7f9372a00640 1 -- 172.21.15.29:0/3261336512 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f935c0ad640 con 0x7f93740c6f90 2024-04-17T19:48:05.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.685+0000 7f9372a00640 1 -- 172.21.15.29:0/3261336512 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935c0ada10 con 0x7f93740c6f90 2024-04-17T19:48:05.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.685+0000 7f937a200640 1 -- 172.21.15.29:0/3261336512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93740c6f90 msgr2=0x7f93740c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:05.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.685+0000 7f937a200640 1 --2- 172.21.15.29:0/3261336512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93740c6f90 0x7f93740c73a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f935c0097c0 tx=0x7f935c01c4a0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.685+0000 7f937a200640 1 -- 172.21.15.29:0/3261336512 shutdown_connections 2024-04-17T19:48:05.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.685+0000 7f937a200640 1 --2- 172.21.15.29:0/3261336512 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f937415d800 0x7f937415dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.685+0000 7f937a200640 1 --2- 172.21.15.29:0/3261336512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93740c6f90 0x7f93740c73a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.685+0000 7f937a200640 1 -- 172.21.15.29:0/3261336512 >> 172.21.15.29:0/3261336512 conn(0x7f937406c420 msgr2=0x7f93740ca380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:05.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.685+0000 7f937a200640 1 -- 172.21.15.29:0/3261336512 shutdown_connections 2024-04-17T19:48:05.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.685+0000 7f937a200640 1 -- 172.21.15.29:0/3261336512 wait complete. 2024-04-17T19:48:05.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.689+0000 7f937a200640 1 Processor -- start 2024-04-17T19:48:05.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.689+0000 7f937a200640 1 -- start start 2024-04-17T19:48:05.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.689+0000 7f937a200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93740c6f90 0x7f93741f50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:05.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.689+0000 7f937a200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93740cacb0 0x7f93740cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:05.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.689+0000 7f937a200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f937415d800 0x7f9374150a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:05.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.689+0000 7f937a200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f93740c9eb0 con 0x7f93740cacb0 2024-04-17T19:48:05.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.689+0000 7f937a200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f937415e150 con 0x7f93740c6f90 2024-04-17T19:48:05.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.689+0000 7f937a200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f93740ca030 con 0x7f937415d800 2024-04-17T19:48:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.689+0000 7f936be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93740cacb0 0x7f93740cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.689+0000 7f936be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93740cacb0 0x7f93740cd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52002/0 (socket says 172.21.15.29:52002) 2024-04-17T19:48:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.689+0000 7f936be00640 1 -- 172.21.15.29:0/1389311539 learned_addr learned my addr 172.21.15.29:0/1389311539 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.689+0000 7f9373e00640 1 --2- 172.21.15.29:0/1389311539 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f937415d800 0x7f9374150a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.689+0000 7f936be00640 1 -- 172.21.15.29:0/1389311539 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f937415d800 msgr2=0x7f9374150a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.689+0000 7f936be00640 1 --2- 172.21.15.29:0/1389311539 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f937415d800 0x7f9374150a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.689+0000 7f936be00640 1 -- 172.21.15.29:0/1389311539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93740c6f90 msgr2=0x7f93741f50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.689+0000 7f936be00640 1 --2- 172.21.15.29:0/1389311539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93740c6f90 0x7f93741f50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.689+0000 7f936be00640 1 -- 172.21.15.29:0/1389311539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9374070480 con 0x7f93740cacb0 2024-04-17T19:48:05.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.689+0000 7f936be00640 1 --2- 172.21.15.29:0/1389311539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93740cacb0 0x7f93740cd170 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f935c01a0d0 tx=0x7f935c019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:05.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.693+0000 7f936b400640 1 -- 172.21.15.29:0/1389311539 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935c0b2040 con 0x7f93740cacb0 2024-04-17T19:48:05.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.693+0000 7f936b400640 1 -- 172.21.15.29:0/1389311539 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f935c0bc070 con 0x7f93740cacb0 2024-04-17T19:48:05.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.693+0000 7f937a200640 1 -- 172.21.15.29:0/1389311539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93741f9480 con 0x7f93740cacb0 2024-04-17T19:48:05.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.693+0000 7f936b400640 1 -- 172.21.15.29:0/1389311539 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935c0b6650 con 0x7f93740cacb0 2024-04-17T19:48:05.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.693+0000 7f937a200640 1 -- 172.21.15.29:0/1389311539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93741f89a0 con 0x7f93740cacb0 2024-04-17T19:48:05.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.693+0000 7f937a200640 1 -- 172.21.15.29:0/1389311539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9374069b90 con 0x7f93740cacb0 2024-04-17T19:48:05.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.697+0000 7f936b400640 1 -- 172.21.15.29:0/1389311539 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f935c0b67f0 con 0x7f93740cacb0 2024-04-17T19:48:05.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.697+0000 7f936b400640 1 --2- 172.21.15.29:0/1389311539 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9350068380 0x7f935006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:05.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.697+0000 7f936b400640 1 -- 172.21.15.29:0/1389311539 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f935c121400 con 0x7f93740cacb0 2024-04-17T19:48:05.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.697+0000 7f9373400640 1 --2- 172.21.15.29:0/1389311539 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9350068380 0x7f935006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:05.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.697+0000 7f9373400640 1 --2- 172.21.15.29:0/1389311539 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9350068380 0x7f935006a890 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f93640015a0 tx=0x7f9364007970 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:05.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.701+0000 7f936b400640 1 -- 172.21.15.29:0/1389311539 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f935c0c6020 con 0x7f93740cacb0 2024-04-17T19:48:05.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.881+0000 7f937a200640 1 -- 172.21.15.29:0/1389311539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9374073f10 con 0x7f93740cacb0 2024-04-17T19:48:05.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.885+0000 7f936b400640 1 -- 172.21.15.29:0/1389311539 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f935c016090 con 0x7f93740cacb0 2024-04-17T19:48:05.872 INFO:teuthology.orchestra.run.smithi029.stderr:listed 0 entries 2024-04-17T19:48:05.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.885+0000 7f937a200640 1 -- 172.21.15.29:0/1389311539 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9350068380 msgr2=0x7f935006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:05.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.885+0000 7f937a200640 1 --2- 172.21.15.29:0/1389311539 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9350068380 0x7f935006a890 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f93640015a0 tx=0x7f9364007970 comp rx=0 tx=0).stop 2024-04-17T19:48:05.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.885+0000 7f937a200640 1 -- 172.21.15.29:0/1389311539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93740cacb0 msgr2=0x7f93740cd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:05.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.885+0000 7f937a200640 1 --2- 172.21.15.29:0/1389311539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93740cacb0 0x7f93740cd170 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f935c01a0d0 tx=0x7f935c019240 comp rx=0 tx=0).stop 2024-04-17T19:48:05.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.885+0000 7f937a200640 1 -- 172.21.15.29:0/1389311539 shutdown_connections 2024-04-17T19:48:05.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.885+0000 7f937a200640 1 --2- 172.21.15.29:0/1389311539 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9350068380 0x7f935006a890 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.885+0000 7f937a200640 1 --2- 172.21.15.29:0/1389311539 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f937415d800 0x7f9374150a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.885+0000 7f937a200640 1 --2- 172.21.15.29:0/1389311539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93740cacb0 0x7f93740cd170 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.885+0000 7f937a200640 1 --2- 172.21.15.29:0/1389311539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f93740c6f90 0x7f93741f50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:05.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.885+0000 7f937a200640 1 -- 172.21.15.29:0/1389311539 >> 172.21.15.29:0/1389311539 conn(0x7f937406c420 msgr2=0x7f93740c9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:05.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.885+0000 7f937a200640 1 -- 172.21.15.29:0/1389311539 shutdown_connections 2024-04-17T19:48:05.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:05.885+0000 7f937a200640 1 -- 172.21.15.29:0/1389311539 wait complete. 2024-04-17T19:48:05.894 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:48:05.894 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T19:48:05.903 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:48:06.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.109+0000 7fc73a600640 1 Processor -- start 2024-04-17T19:48:06.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.109+0000 7fc73a600640 1 -- start start 2024-04-17T19:48:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.109+0000 7fc73a600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7341560d0 0x7fc73415be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.109+0000 7fc73a600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7340c55e0 con 0x7fc73415c370 2024-04-17T19:48:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.109+0000 7fc73a600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7340c9f30 con 0x7fc7340c9b20 2024-04-17T19:48:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.109+0000 7fc73a600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fc7340c8fa0 con 0x7fc7341560d0 2024-04-17T19:48:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.113+0000 7fc72aa00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7341560d0 0x7fc73415be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.113+0000 7fc733e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc73415c370 0x7fc7341667b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49714/0 (socket says 172.21.15.29:49714) 2024-04-17T19:48:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.113+0000 7fc733e00640 1 -- 172.21.15.29:0/2273251268 learned_addr learned my addr 172.21.15.29:0/2273251268 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.113+0000 7fc72aa00640 1 -- 172.21.15.29:0/2273251268 >> v1:172.21.15.37:6789/0 conn(0x7fc7340c9b20 legacy=0x7fc73406e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.113+0000 7fc72aa00640 1 -- 172.21.15.29:0/2273251268 >> v1:172.21.15.29:6789/0 conn(0x7fc73415c370 legacy=0x7fc7341667b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:06.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.113+0000 7fc72aa00640 1 -- 172.21.15.29:0/2273251268 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7340ca1d0 con 0x7fc7341560d0 2024-04-17T19:48:06.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.113+0000 7fc72aa00640 1 --2- 172.21.15.29:0/2273251268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7341560d0 0x7fc73415be30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc7240097c0 tx=0x7fc72401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7425f86981198a61 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:06.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.113+0000 7fc732a00640 1 -- 172.21.15.29:0/2273251268 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7240b5070 con 0x7fc7341560d0 2024-04-17T19:48:06.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.113+0000 7fc732a00640 1 -- 172.21.15.29:0/2273251268 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7240a8990 con 0x7fc7341560d0 2024-04-17T19:48:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.113+0000 7fc732a00640 1 -- 172.21.15.29:0/2273251268 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7240a8d60 con 0x7fc7341560d0 2024-04-17T19:48:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.117+0000 7fc73a600640 1 -- 172.21.15.29:0/2273251268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7341560d0 msgr2=0x7fc73415be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.117+0000 7fc73a600640 1 --2- 172.21.15.29:0/2273251268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7341560d0 0x7fc73415be30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc7240097c0 tx=0x7fc72401ec90 comp rx=0 tx=0).stop 2024-04-17T19:48:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.117+0000 7fc73a600640 1 -- 172.21.15.29:0/2273251268 shutdown_connections 2024-04-17T19:48:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.117+0000 7fc73a600640 1 --2- 172.21.15.29:0/2273251268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7341560d0 0x7fc73415be30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.117+0000 7fc73a600640 1 -- 172.21.15.29:0/2273251268 >> 172.21.15.29:0/2273251268 conn(0x7fc73406c3c0 msgr2=0x7fc7340ca3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:06.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.117+0000 7fc73a600640 1 -- 172.21.15.29:0/2273251268 shutdown_connections 2024-04-17T19:48:06.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.117+0000 7fc73a600640 1 -- 172.21.15.29:0/2273251268 wait complete. 2024-04-17T19:48:06.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.117+0000 7fc73a600640 1 Processor -- start 2024-04-17T19:48:06.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.117+0000 7fc73a600640 1 -- start start 2024-04-17T19:48:06.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc73a600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc7340c9b20 0x7fc7341f6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc73a600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc7341560d0 0x7fc73415aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc73a600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc73415c370 0x7fc734145240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc73a600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc7340c8670 con 0x7fc73415c370 2024-04-17T19:48:06.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc73a600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc7340c8390 con 0x7fc7341560d0 2024-04-17T19:48:06.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc73a600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc7340c8950 con 0x7fc7340c9b20 2024-04-17T19:48:06.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc733e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc73415c370 0x7fc734145240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc733e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc73415c370 0x7fc734145240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52026/0 (socket says 172.21.15.29:52026) 2024-04-17T19:48:06.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc733e00640 1 -- 172.21.15.29:0/1117962509 learned_addr learned my addr 172.21.15.29:0/1117962509 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:06.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc72aa00640 1 --2- 172.21.15.29:0/1117962509 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc7341560d0 0x7fc73415aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc733400640 1 --2- 172.21.15.29:0/1117962509 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc7340c9b20 0x7fc7341f6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc733e00640 1 -- 172.21.15.29:0/1117962509 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc7340c9b20 msgr2=0x7fc7341f6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:06.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc733e00640 1 --2- 172.21.15.29:0/1117962509 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc7340c9b20 0x7fc7341f6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc733e00640 1 -- 172.21.15.29:0/1117962509 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc7341560d0 msgr2=0x7fc73415aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:06.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc733e00640 1 --2- 172.21.15.29:0/1117962509 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc7341560d0 0x7fc73415aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc733e00640 1 -- 172.21.15.29:0/1117962509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7341f6fd0 con 0x7fc73415c370 2024-04-17T19:48:06.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc733e00640 1 --2- 172.21.15.29:0/1117962509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc73415c370 0x7fc734145240 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00caa0 tx=0x7fc72c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:06.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc72be00640 1 -- 172.21.15.29:0/1117962509 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c017070 con 0x7fc73415c370 2024-04-17T19:48:06.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc72be00640 1 -- 172.21.15.29:0/1117962509 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc72c00ac00 con 0x7fc73415c370 2024-04-17T19:48:06.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc72be00640 1 -- 172.21.15.29:0/1117962509 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c012400 con 0x7fc73415c370 2024-04-17T19:48:06.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc73a600640 1 -- 172.21.15.29:0/1117962509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc73415afb0 con 0x7fc73415c370 2024-04-17T19:48:06.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc73a600640 1 -- 172.21.15.29:0/1117962509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7340cd2b0 con 0x7fc73415c370 2024-04-17T19:48:06.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc73a600640 1 -- 172.21.15.29:0/1117962509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc734069b90 con 0x7fc73415c370 2024-04-17T19:48:06.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.121+0000 7fc72be00640 1 -- 172.21.15.29:0/1117962509 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc72c01a070 con 0x7fc73415c370 2024-04-17T19:48:06.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.125+0000 7fc72be00640 1 --2- 172.21.15.29:0/1117962509 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc710068380 0x7fc71006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.125+0000 7fc72be00640 1 -- 172.21.15.29:0/1117962509 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc72c0868d0 con 0x7fc73415c370 2024-04-17T19:48:06.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.125+0000 7fc733400640 1 --2- 172.21.15.29:0/1117962509 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc710068380 0x7fc71006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.125+0000 7fc733400640 1 --2- 172.21.15.29:0/1117962509 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc710068380 0x7fc71006a890 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc71c007fd0 tx=0x7fc71c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:06.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.129+0000 7fc72be00640 1 -- 172.21.15.29:0/1117962509 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc7340c55e0 con 0x7fc73415c370 2024-04-17T19:48:06.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.293+0000 7fc73a600640 1 -- 172.21.15.29:0/1117962509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc7340cd010 con 0x7fc73415c370 2024-04-17T19:48:06.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.293+0000 7fc72be00640 1 -- 172.21.15.29:0/1117962509 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18394 (secure 0 0 0) 0x7fc72c056530 con 0x7fc73415c370 2024-04-17T19:48:06.282 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:06.282 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:47:54.321794+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T19:47:47.658449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T19:47:48.209047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T19:47:51.887909+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:48:06.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.297+0000 7fc73a600640 1 -- 172.21.15.29:0/1117962509 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc710068380 msgr2=0x7fc71006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:06.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.297+0000 7fc73a600640 1 --2- 172.21.15.29:0/1117962509 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc710068380 0x7fc71006a890 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc71c007fd0 tx=0x7fc71c009040 comp rx=0 tx=0).stop 2024-04-17T19:48:06.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.297+0000 7fc73a600640 1 -- 172.21.15.29:0/1117962509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc73415c370 msgr2=0x7fc734145240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:06.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.297+0000 7fc73a600640 1 --2- 172.21.15.29:0/1117962509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc73415c370 0x7fc734145240 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00caa0 tx=0x7fc72c00cf60 comp rx=0 tx=0).stop 2024-04-17T19:48:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.297+0000 7fc73a600640 1 -- 172.21.15.29:0/1117962509 shutdown_connections 2024-04-17T19:48:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.297+0000 7fc73a600640 1 --2- 172.21.15.29:0/1117962509 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc710068380 0x7fc71006a890 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.297+0000 7fc73a600640 1 --2- 172.21.15.29:0/1117962509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc73415c370 0x7fc734145240 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.297+0000 7fc73a600640 1 --2- 172.21.15.29:0/1117962509 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc7341560d0 0x7fc73415aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.297+0000 7fc73a600640 1 --2- 172.21.15.29:0/1117962509 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc7340c9b20 0x7fc7341f6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.297+0000 7fc73a600640 1 -- 172.21.15.29:0/1117962509 >> 172.21.15.29:0/1117962509 conn(0x7fc73406c3c0 msgr2=0x7fc734155100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.297+0000 7fc73a600640 1 -- 172.21.15.29:0/1117962509 shutdown_connections 2024-04-17T19:48:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.297+0000 7fc73a600640 1 -- 172.21.15.29:0/1117962509 wait complete. 2024-04-17T19:48:06.304 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T19:47:36.674676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T19:47:47.658449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T19:47:48.209047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T19:47:51.887909+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T19:48:06.305 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T19:48:06.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.469+0000 7f73f6a00640 1 Processor -- start 2024-04-17T19:48:06.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.469+0000 7f73f6a00640 1 -- start start 2024-04-17T19:48:06.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.473+0000 7f73f6a00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f73f0153fe0 0x7f73f00c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.473+0000 7f73f6a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f73f00c4fb0 0x7f73f00c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.473+0000 7f73f6a00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73f015abf0 con 0x7f73f00ca520 2024-04-17T19:48:06.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.473+0000 7f73f6a00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f73f00c72e0 con 0x7f73f00c4fb0 2024-04-17T19:48:06.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.473+0000 7f73f6a00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f73f015d3b0 con 0x7f73f0153fe0 2024-04-17T19:48:06.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.473+0000 7f73efe00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f73f0153fe0 0x7f73f00c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.473+0000 7f73e7e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f73f00c4fb0 0x7f73f00c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.473+0000 7f73f4a00640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f73f00ca520 0x7f73f006f6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:51504/0 (socket says 172.21.15.29:51504) 2024-04-17T19:48:06.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.473+0000 7f73efe00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f73f0153fe0 0x7f73f00c4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:39162/0 (socket says 172.21.15.29:39162) 2024-04-17T19:48:06.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.473+0000 7f73f4a00640 1 -- 172.21.15.29:0/1782998349 learned_addr learned my addr 172.21.15.29:0/1782998349 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:06.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.473+0000 7f73e7e00640 1 -- 172.21.15.29:0/1782998349 >> v2:172.21.15.37:3300/0 conn(0x7f73f0153fe0 msgr2=0x7f73f00c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:06.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.473+0000 7f73e7e00640 1 --2- 172.21.15.29:0/1782998349 >> v2:172.21.15.37:3300/0 conn(0x7f73f0153fe0 0x7f73f00c4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.473+0000 7f73e7e00640 1 -- 172.21.15.29:0/1782998349 >> v1:172.21.15.37:6789/0 conn(0x7f73f00ca520 legacy=0x7f73f006f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:06.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.473+0000 7f73e7e00640 1 -- 172.21.15.29:0/1782998349 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f015d600 con 0x7f73f00c4fb0 2024-04-17T19:48:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.477+0000 7f73e7e00640 1 --2- 172.21.15.29:0/1782998349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f00c4fb0 0x7f73f00c9fe0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f73f006ccd0 tx=0x7f73d801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9973c75e9aa9742 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.477+0000 7f73ef400640 1 -- 172.21.15.29:0/1782998349 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d80b0070 con 0x7f73f00c4fb0 2024-04-17T19:48:06.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.477+0000 7f73ef400640 1 -- 172.21.15.29:0/1782998349 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d80ab600 con 0x7f73f00c4fb0 2024-04-17T19:48:06.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.477+0000 7f73ef400640 1 -- 172.21.15.29:0/1782998349 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d80ab9d0 con 0x7f73f00c4fb0 2024-04-17T19:48:06.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.477+0000 7f73f6a00640 1 -- 172.21.15.29:0/1782998349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f00c4fb0 msgr2=0x7f73f00c9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:06.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.477+0000 7f73f6a00640 1 --2- 172.21.15.29:0/1782998349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f00c4fb0 0x7f73f00c9fe0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f73f006ccd0 tx=0x7f73d801c480 comp rx=0 tx=0).stop 2024-04-17T19:48:06.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.477+0000 7f73f6a00640 1 -- 172.21.15.29:0/1782998349 shutdown_connections 2024-04-17T19:48:06.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.477+0000 7f73f6a00640 1 --2- 172.21.15.29:0/1782998349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f00c4fb0 0x7f73f00c9fe0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.477+0000 7f73f6a00640 1 --2- 172.21.15.29:0/1782998349 >> v2:172.21.15.37:3300/0 conn(0x7f73f0153fe0 0x7f73f00c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.477+0000 7f73f6a00640 1 -- 172.21.15.29:0/1782998349 >> 172.21.15.29:0/1782998349 conn(0x7f73f006bf90 msgr2=0x7f73f006c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:06.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.477+0000 7f73f6a00640 1 -- 172.21.15.29:0/1782998349 shutdown_connections 2024-04-17T19:48:06.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.477+0000 7f73f6a00640 1 -- 172.21.15.29:0/1782998349 wait complete. 2024-04-17T19:48:06.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.477+0000 7f73f6a00640 1 Processor -- start 2024-04-17T19:48:06.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.477+0000 7f73f6a00640 1 -- start start 2024-04-17T19:48:06.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73f6a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73f00c4fb0 0x7f73f0152b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73f6a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73f00ca520 0x7f73f0144c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73f6a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f0153fe0 0x7f73f0145180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73f6a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f73f01543f0 con 0x7f73f0153fe0 2024-04-17T19:48:06.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73f6a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f73f015df30 con 0x7f73f00ca520 2024-04-17T19:48:06.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73f6a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f73f0154570 con 0x7f73f00c4fb0 2024-04-17T19:48:06.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73f4a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f0153fe0 0x7f73f0145180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73f4a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f0153fe0 0x7f73f0145180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52048/0 (socket says 172.21.15.29:52048) 2024-04-17T19:48:06.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73f4a00640 1 -- 172.21.15.29:0/3608030329 learned_addr learned my addr 172.21.15.29:0/3608030329 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:06.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73e7e00640 1 --2- 172.21.15.29:0/3608030329 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73f00ca520 0x7f73f0144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73efe00640 1 --2- 172.21.15.29:0/3608030329 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73f00c4fb0 0x7f73f0152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73f4a00640 1 -- 172.21.15.29:0/3608030329 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73f00c4fb0 msgr2=0x7f73f0152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:06.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73f4a00640 1 --2- 172.21.15.29:0/3608030329 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73f00c4fb0 0x7f73f0152b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73f4a00640 1 -- 172.21.15.29:0/3608030329 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73f00ca520 msgr2=0x7f73f0144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:06.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73f4a00640 1 --2- 172.21.15.29:0/3608030329 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73f00ca520 0x7f73f0144c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73f4a00640 1 -- 172.21.15.29:0/3608030329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73f0071d80 con 0x7f73f0153fe0 2024-04-17T19:48:06.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73f4a00640 1 --2- 172.21.15.29:0/3608030329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f0153fe0 0x7f73f0145180 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f73e800cad0 tx=0x7f73e800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:06.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73ecc00640 1 -- 172.21.15.29:0/3608030329 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e8017070 con 0x7f73f0153fe0 2024-04-17T19:48:06.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73ecc00640 1 -- 172.21.15.29:0/3608030329 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73e800ac30 con 0x7f73f0153fe0 2024-04-17T19:48:06.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73f6a00640 1 -- 172.21.15.29:0/3608030329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73f0159080 con 0x7f73f0153fe0 2024-04-17T19:48:06.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73ecc00640 1 -- 172.21.15.29:0/3608030329 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73e8012400 con 0x7f73f0153fe0 2024-04-17T19:48:06.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73f6a00640 1 -- 172.21.15.29:0/3608030329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73f0072300 con 0x7f73f0153fe0 2024-04-17T19:48:06.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.481+0000 7f73f6a00640 1 -- 172.21.15.29:0/3608030329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73f0159bf0 con 0x7f73f0153fe0 2024-04-17T19:48:06.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.485+0000 7f73ecc00640 1 -- 172.21.15.29:0/3608030329 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f73e801a070 con 0x7f73f0153fe0 2024-04-17T19:48:06.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.485+0000 7f73ecc00640 1 --2- 172.21.15.29:0/3608030329 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f73cc068380 0x7f73cc06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.485+0000 7f73ecc00640 1 -- 172.21.15.29:0/3608030329 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f73e8086920 con 0x7f73f0153fe0 2024-04-17T19:48:06.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.485+0000 7f73efe00640 1 --2- 172.21.15.29:0/3608030329 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f73cc068380 0x7f73cc06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.485+0000 7f73efe00640 1 --2- 172.21.15.29:0/3608030329 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f73cc068380 0x7f73cc06a890 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f73e0009a50 tx=0x7f73e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:06.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.489+0000 7f73ecc00640 1 -- 172.21.15.29:0/3608030329 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f73e80564b0 con 0x7f73f0153fe0 2024-04-17T19:48:06.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.653+0000 7f73f6a00640 1 -- 172.21.15.29:0/3608030329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f73f00c8f30 con 0x7f73f0153fe0 2024-04-17T19:48:06.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.653+0000 7f73ecc00640 1 -- 172.21.15.29:0/3608030329 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f73e805a7b0 con 0x7f73f0153fe0 2024-04-17T19:48:06.644 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-04-17T19:48:06.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.657+0000 7f73f6a00640 1 -- 172.21.15.29:0/3608030329 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f73cc068380 msgr2=0x7f73cc06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:06.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.657+0000 7f73f6a00640 1 --2- 172.21.15.29:0/3608030329 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f73cc068380 0x7f73cc06a890 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f73e0009a50 tx=0x7f73e0008040 comp rx=0 tx=0).stop 2024-04-17T19:48:06.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.657+0000 7f73f6a00640 1 -- 172.21.15.29:0/3608030329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f0153fe0 msgr2=0x7f73f0145180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:06.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.657+0000 7f73f6a00640 1 --2- 172.21.15.29:0/3608030329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f0153fe0 0x7f73f0145180 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f73e800cad0 tx=0x7f73e800cf90 comp rx=0 tx=0).stop 2024-04-17T19:48:06.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.657+0000 7f73f6a00640 1 -- 172.21.15.29:0/3608030329 shutdown_connections 2024-04-17T19:48:06.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.657+0000 7f73f6a00640 1 --2- 172.21.15.29:0/3608030329 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f73cc068380 0x7f73cc06a890 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.657+0000 7f73f6a00640 1 --2- 172.21.15.29:0/3608030329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f0153fe0 0x7f73f0145180 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.657+0000 7f73f6a00640 1 --2- 172.21.15.29:0/3608030329 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73f00ca520 0x7f73f0144c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.657+0000 7f73f6a00640 1 --2- 172.21.15.29:0/3608030329 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73f00c4fb0 0x7f73f0152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.657+0000 7f73f6a00640 1 -- 172.21.15.29:0/3608030329 >> 172.21.15.29:0/3608030329 conn(0x7f73f006bf90 msgr2=0x7f73f01518a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:06.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.657+0000 7f73f6a00640 1 -- 172.21.15.29:0/3608030329 shutdown_connections 2024-04-17T19:48:06.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.657+0000 7f73f6a00640 1 -- 172.21.15.29:0/3608030329 wait complete. 2024-04-17T19:48:06.666 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T19:48:06.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.833+0000 7f9d83200640 1 Processor -- start 2024-04-17T19:48:06.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.833+0000 7f9d83200640 1 -- start start 2024-04-17T19:48:06.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.833+0000 7f9d83200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9d840c9c90 0x7f9d840ca0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.833+0000 7f9d83200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d841603e0 con 0x7f9d840c5ac0 2024-04-17T19:48:06.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.833+0000 7f9d83200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d840afba0 con 0x7f9d840c42b0 2024-04-17T19:48:06.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.833+0000 7f9d83200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9d8415ff50 con 0x7f9d840c9c90 2024-04-17T19:48:06.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.833+0000 7f9d82800640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9d840c5ac0 0x7f9d84166810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49722/0 (socket says 172.21.15.29:49722) 2024-04-17T19:48:06.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.833+0000 7f9d82800640 1 -- 172.21.15.29:0/3956046780 learned_addr learned my addr 172.21.15.29:0/3956046780 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:06.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.833+0000 7f9d79400640 1 --2- 172.21.15.29:0/3956046780 >> v2:172.21.15.37:3300/0 conn(0x7f9d840c9c90 0x7f9d840ca0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.833+0000 7f9d81400640 1 -- 172.21.15.29:0/3956046780 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4223440409 0 0) 0x7f9d841603e0 con 0x7f9d840c5ac0 2024-04-17T19:48:06.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.833+0000 7f9d81400640 1 -- 172.21.15.29:0/3956046780 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d68003530 con 0x7f9d840c5ac0 2024-04-17T19:48:06.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81400640 1 -- 172.21.15.29:0/3956046780 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1809603336 0 0) 0x7f9d68003530 con 0x7f9d840c5ac0 2024-04-17T19:48:06.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81400640 1 -- 172.21.15.29:0/3956046780 >> v2:172.21.15.37:3300/0 conn(0x7f9d840c9c90 msgr2=0x7f9d840ca0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:06.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81400640 1 --2- 172.21.15.29:0/3956046780 >> v2:172.21.15.37:3300/0 conn(0x7f9d840c9c90 0x7f9d840ca0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81400640 1 -- 172.21.15.29:0/3956046780 >> v1:172.21.15.37:6789/0 conn(0x7f9d840c42b0 legacy=0x7f9d84156030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81400640 1 -- 172.21.15.29:0/3956046780 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d8415f620 con 0x7f9d840c5ac0 2024-04-17T19:48:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81400640 1 -- 172.21.15.29:0/3956046780 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f9d7c001680 con 0x7f9d840c5ac0 2024-04-17T19:48:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81400640 1 -- 172.21.15.29:0/3956046780 >> v1:172.21.15.29:6789/0 conn(0x7f9d840c5ac0 legacy=0x7f9d84166810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81400640 1 --2- 172.21.15.29:0/3956046780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d680039c0 0x7f9d68003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81400640 1 --2- 172.21.15.29:0/3956046780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d68004810 0x7f9d68004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81400640 1 --2- 172.21.15.29:0/3956046780 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d680051d0 0x7f9d6800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81400640 1 -- 172.21.15.29:0/3956046780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8415ff50 con 0x7f9d680039c0 2024-04-17T19:48:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81400640 1 -- 172.21.15.29:0/3956046780 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9d6800c160 con 0x7f9d68004810 2024-04-17T19:48:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81400640 1 -- 172.21.15.29:0/3956046780 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9d6800c360 con 0x7f9d680051d0 2024-04-17T19:48:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81e00640 1 --2- 172.21.15.29:0/3956046780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d680039c0 0x7f9d68003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d79400640 1 --2- 172.21.15.29:0/3956046780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d68004810 0x7f9d68004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d82800640 1 --2- 172.21.15.29:0/3956046780 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d680051d0 0x7f9d6800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81e00640 1 -- 172.21.15.29:0/3956046780 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d680051d0 msgr2=0x7f9d6800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:06.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81e00640 1 --2- 172.21.15.29:0/3956046780 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d680051d0 0x7f9d6800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81e00640 1 -- 172.21.15.29:0/3956046780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d68004810 msgr2=0x7f9d68004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:06.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81e00640 1 --2- 172.21.15.29:0/3956046780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d68004810 0x7f9d68004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d81e00640 1 -- 172.21.15.29:0/3956046780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d68003530 con 0x7f9d680039c0 2024-04-17T19:48:06.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d79400640 1 --2- 172.21.15.29:0/3956046780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d68004810 0x7f9d68004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:06.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d82800640 1 --2- 172.21.15.29:0/3956046780 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d680051d0 0x7f9d6800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:06.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.837+0000 7f9d82800640 1 -- 172.21.15.29:0/3956046780 reap_dead start 2024-04-17T19:48:06.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.841+0000 7f9d81e00640 1 --2- 172.21.15.29:0/3956046780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d680039c0 0x7f9d68003e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9d74001780 tx=0x7f9d7401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a56200c4f4c1631 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:06.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.841+0000 7f9d81400640 1 -- 172.21.15.29:0/3956046780 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d740013a0 con 0x7f9d680039c0 2024-04-17T19:48:06.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.841+0000 7f9d81400640 1 -- 172.21.15.29:0/3956046780 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d740052f0 con 0x7f9d680039c0 2024-04-17T19:48:06.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.841+0000 7f9d81400640 1 -- 172.21.15.29:0/3956046780 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d74003840 con 0x7f9d680039c0 2024-04-17T19:48:06.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.841+0000 7f9d83200640 1 -- 172.21.15.29:0/3956046780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d680039c0 msgr2=0x7f9d68003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:06.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.841+0000 7f9d83200640 1 --2- 172.21.15.29:0/3956046780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d680039c0 0x7f9d68003e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9d74001780 tx=0x7f9d7401c340 comp rx=0 tx=0).stop 2024-04-17T19:48:06.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.841+0000 7f9d83200640 1 -- 172.21.15.29:0/3956046780 shutdown_connections 2024-04-17T19:48:06.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.841+0000 7f9d83200640 1 --2- 172.21.15.29:0/3956046780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d680039c0 0x7f9d68003e30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.841+0000 7f9d83200640 1 -- 172.21.15.29:0/3956046780 >> 172.21.15.29:0/3956046780 conn(0x7f9d84072060 msgr2=0x7f9d8406f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:06.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.841+0000 7f9d83200640 1 -- 172.21.15.29:0/3956046780 shutdown_connections 2024-04-17T19:48:06.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.841+0000 7f9d83200640 1 -- 172.21.15.29:0/3956046780 wait complete. 2024-04-17T19:48:06.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.841+0000 7f9d83200640 1 Processor -- start 2024-04-17T19:48:06.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.841+0000 7f9d83200640 1 -- start start 2024-04-17T19:48:06.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d83200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d680039c0 0x7f9d840b4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d83200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d84158af0 0x7f9d8415a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d83200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d84153000 0x7f9d840b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d83200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9d840af7f0 con 0x7f9d84153000 2024-04-17T19:48:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d83200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8415fbe0 con 0x7f9d680039c0 2024-04-17T19:48:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d83200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9d8415acd0 con 0x7f9d84158af0 2024-04-17T19:48:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d82800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d84153000 0x7f9d840b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d82800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d84153000 0x7f9d840b34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52072/0 (socket says 172.21.15.29:52072) 2024-04-17T19:48:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d82800640 1 -- 172.21.15.29:0/1700312440 learned_addr learned my addr 172.21.15.29:0/1700312440 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d81e00640 1 --2- 172.21.15.29:0/1700312440 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d680039c0 0x7f9d840b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d79400640 1 --2- 172.21.15.29:0/1700312440 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d84158af0 0x7f9d8415a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d82800640 1 -- 172.21.15.29:0/1700312440 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d84158af0 msgr2=0x7f9d8415a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d82800640 1 --2- 172.21.15.29:0/1700312440 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d84158af0 0x7f9d8415a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d82800640 1 -- 172.21.15.29:0/1700312440 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d680039c0 msgr2=0x7f9d840b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d82800640 1 --2- 172.21.15.29:0/1700312440 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d680039c0 0x7f9d840b4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d82800640 1 -- 172.21.15.29:0/1700312440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d84157a60 con 0x7f9d84153000 2024-04-17T19:48:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d82800640 1 --2- 172.21.15.29:0/1700312440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d84153000 0x7f9d840b34a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c004880 tx=0x7f9d7c011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d7aa00640 1 -- 172.21.15.29:0/1700312440 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c010040 con 0x7f9d84153000 2024-04-17T19:48:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d7aa00640 1 -- 172.21.15.29:0/1700312440 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d7c01a070 con 0x7f9d84153000 2024-04-17T19:48:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d83200640 1 -- 172.21.15.29:0/1700312440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d8415a800 con 0x7f9d84153000 2024-04-17T19:48:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d7aa00640 1 -- 172.21.15.29:0/1700312440 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c00fa40 con 0x7f9d84153000 2024-04-17T19:48:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.845+0000 7f9d83200640 1 -- 172.21.15.29:0/1700312440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d84157cd0 con 0x7f9d84153000 2024-04-17T19:48:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.849+0000 7f9d83200640 1 -- 172.21.15.29:0/1700312440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d8404ff10 con 0x7f9d84153000 2024-04-17T19:48:06.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.849+0000 7f9d7aa00640 1 -- 172.21.15.29:0/1700312440 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9d7c00fbe0 con 0x7f9d84153000 2024-04-17T19:48:06.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.849+0000 7f9d7aa00640 1 --2- 172.21.15.29:0/1700312440 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9d60068380 0x7f9d6006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:06.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.849+0000 7f9d7aa00640 1 -- 172.21.15.29:0/1700312440 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9d7c089ac0 con 0x7f9d84153000 2024-04-17T19:48:06.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.849+0000 7f9d81e00640 1 --2- 172.21.15.29:0/1700312440 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9d60068380 0x7f9d6006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:06.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.849+0000 7f9d81e00640 1 --2- 172.21.15.29:0/1700312440 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9d60068380 0x7f9d6006a890 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9d74001750 tx=0x7f9d740011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:06.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:06.853+0000 7f9d7aa00640 1 -- 172.21.15.29:0/1700312440 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9d7c0596f0 con 0x7f9d84153000 2024-04-17T19:48:07.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.021+0000 7f9d83200640 1 -- 172.21.15.29:0/1700312440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9d8415d400 con 0x7f9d84153000 2024-04-17T19:48:07.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.025+0000 7f9d7aa00640 1 -- 172.21.15.29:0/1700312440 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f9d7c05d9f0 con 0x7f9d84153000 2024-04-17T19:48:07.012 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-04-17T19:48:07.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.025+0000 7f9d83200640 1 -- 172.21.15.29:0/1700312440 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9d60068380 msgr2=0x7f9d6006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:07.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.025+0000 7f9d83200640 1 --2- 172.21.15.29:0/1700312440 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9d60068380 0x7f9d6006a890 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9d74001750 tx=0x7f9d740011a0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.025+0000 7f9d83200640 1 -- 172.21.15.29:0/1700312440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d84153000 msgr2=0x7f9d840b34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:07.014 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.025+0000 7f9d83200640 1 --2- 172.21.15.29:0/1700312440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d84153000 0x7f9d840b34a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c004880 tx=0x7f9d7c011c20 comp rx=0 tx=0).stop 2024-04-17T19:48:07.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.025+0000 7f9d83200640 1 -- 172.21.15.29:0/1700312440 shutdown_connections 2024-04-17T19:48:07.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.025+0000 7f9d83200640 1 --2- 172.21.15.29:0/1700312440 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9d60068380 0x7f9d6006a890 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.025+0000 7f9d83200640 1 --2- 172.21.15.29:0/1700312440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d84153000 0x7f9d840b34a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.025+0000 7f9d83200640 1 --2- 172.21.15.29:0/1700312440 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9d84158af0 0x7f9d8415a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.025+0000 7f9d83200640 1 --2- 172.21.15.29:0/1700312440 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9d680039c0 0x7f9d840b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.025+0000 7f9d83200640 1 -- 172.21.15.29:0/1700312440 >> 172.21.15.29:0/1700312440 conn(0x7f9d84072060 msgr2=0x7f9d841554b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:07.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.025+0000 7f9d83200640 1 -- 172.21.15.29:0/1700312440 shutdown_connections 2024-04-17T19:48:07.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.025+0000 7f9d83200640 1 -- 172.21.15.29:0/1700312440 wait complete. 2024-04-17T19:48:07.034 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T19:48:07.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.201+0000 7f9b03400640 1 Processor -- start 2024-04-17T19:48:07.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.201+0000 7f9b03400640 1 -- start start 2024-04-17T19:48:07.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.201+0000 7f9b03400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9afc15bf20 0x7f9afc1561e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:07.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.201+0000 7f9b03400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9afc06f8e0 0x7f9afc06fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:07.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.201+0000 7f9b03400640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9afc0c8210 con 0x7f9afc0ce260 2024-04-17T19:48:07.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.201+0000 7f9b03400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9afc0c9590 con 0x7f9afc15bf20 2024-04-17T19:48:07.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.201+0000 7f9b03400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9afc0c9100 con 0x7f9afc06f8e0 2024-04-17T19:48:07.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.201+0000 7f9afbe00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9afc15bf20 0x7f9afc1561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:07.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.201+0000 7f9b01400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9afc06f8e0 0x7f9afc06fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:07.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.201+0000 7f9b00a00640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f9afc0ce260 0x7f9afc15d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:51530/0 (socket says 172.21.15.29:51530) 2024-04-17T19:48:07.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.201+0000 7f9b00a00640 1 -- 172.21.15.29:0/3367731422 learned_addr learned my addr 172.21.15.29:0/3367731422 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:07.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.201+0000 7f9afbe00640 1 -- 172.21.15.29:0/3367731422 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9afc06f8e0 msgr2=0x7f9afc06fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:07.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.201+0000 7f9afbe00640 1 --2- 172.21.15.29:0/3367731422 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9afc06f8e0 0x7f9afc06fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.201+0000 7f9afbe00640 1 -- 172.21.15.29:0/3367731422 >> v1:172.21.15.37:6789/0 conn(0x7f9afc0ce260 legacy=0x7f9afc15d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:07.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.201+0000 7f9afbe00640 1 -- 172.21.15.29:0/3367731422 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9afc0c84f0 con 0x7f9afc15bf20 2024-04-17T19:48:07.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.205+0000 7f9afbe00640 1 --2- 172.21.15.29:0/3367731422 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9afc15bf20 0x7f9afc1561e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9ae80097c0 tx=0x7f9ae801ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea2b25fdc4ed761c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:07.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.205+0000 7f9afb400640 1 -- 172.21.15.29:0/3367731422 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae80b5070 con 0x7f9afc15bf20 2024-04-17T19:48:07.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.205+0000 7f9afb400640 1 -- 172.21.15.29:0/3367731422 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ae80a8990 con 0x7f9afc15bf20 2024-04-17T19:48:07.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.205+0000 7f9afb400640 1 -- 172.21.15.29:0/3367731422 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ae80a8d60 con 0x7f9afc15bf20 2024-04-17T19:48:07.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.205+0000 7f9b03400640 1 -- 172.21.15.29:0/3367731422 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9afc15bf20 msgr2=0x7f9afc1561e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:07.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.205+0000 7f9b03400640 1 --2- 172.21.15.29:0/3367731422 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9afc15bf20 0x7f9afc1561e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9ae80097c0 tx=0x7f9ae801ec90 comp rx=0 tx=0).stop 2024-04-17T19:48:07.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.205+0000 7f9b03400640 1 -- 172.21.15.29:0/3367731422 shutdown_connections 2024-04-17T19:48:07.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.205+0000 7f9b03400640 1 --2- 172.21.15.29:0/3367731422 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9afc06f8e0 0x7f9afc06fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.205+0000 7f9b03400640 1 --2- 172.21.15.29:0/3367731422 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9afc15bf20 0x7f9afc1561e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.205+0000 7f9b03400640 1 -- 172.21.15.29:0/3367731422 >> 172.21.15.29:0/3367731422 conn(0x7f9afc06c470 msgr2=0x7f9afc0cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:07.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.205+0000 7f9b03400640 1 -- 172.21.15.29:0/3367731422 shutdown_connections 2024-04-17T19:48:07.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.205+0000 7f9b03400640 1 -- 172.21.15.29:0/3367731422 wait complete. 2024-04-17T19:48:07.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9b03400640 1 Processor -- start 2024-04-17T19:48:07.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9b03400640 1 -- start start 2024-04-17T19:48:07.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9b03400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9afc06f8e0 0x7f9afc1f6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:07.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9b03400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9afc0ce260 0x7f9afc143560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:07.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9b03400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9afc15bf20 0x7f9afc13cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:07.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9b03400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9afc0c8d90 con 0x7f9afc06f8e0 2024-04-17T19:48:07.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9b03400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9afc0c8ab0 con 0x7f9afc0ce260 2024-04-17T19:48:07.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9b03400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9afc15db20 con 0x7f9afc15bf20 2024-04-17T19:48:07.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9b00a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9afc06f8e0 0x7f9afc1f6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:07.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9b00a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9afc06f8e0 0x7f9afc1f6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52076/0 (socket says 172.21.15.29:52076) 2024-04-17T19:48:07.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9b00a00640 1 -- 172.21.15.29:0/4074657286 learned_addr learned my addr 172.21.15.29:0/4074657286 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:07.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9b01400640 1 --2- 172.21.15.29:0/4074657286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9afc15bf20 0x7f9afc13cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:07.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9afbe00640 1 --2- 172.21.15.29:0/4074657286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9afc0ce260 0x7f9afc143560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:07.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9b00a00640 1 -- 172.21.15.29:0/4074657286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9afc15bf20 msgr2=0x7f9afc13cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:07.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9b00a00640 1 --2- 172.21.15.29:0/4074657286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9afc15bf20 0x7f9afc13cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9b00a00640 1 -- 172.21.15.29:0/4074657286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9afc0ce260 msgr2=0x7f9afc143560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:07.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9b00a00640 1 --2- 172.21.15.29:0/4074657286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9afc0ce260 0x7f9afc143560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9b00a00640 1 -- 172.21.15.29:0/4074657286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9afc142cc0 con 0x7f9afc06f8e0 2024-04-17T19:48:07.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.209+0000 7f9afbe00640 1 --2- 172.21.15.29:0/4074657286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9afc0ce260 0x7f9afc143560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:07.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.213+0000 7f9b00a00640 1 --2- 172.21.15.29:0/4074657286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9afc06f8e0 0x7f9afc1f6ac0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9af000c3a0 tx=0x7f9af000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:07.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.213+0000 7f9af8c00640 1 -- 172.21.15.29:0/4074657286 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af0017070 con 0x7f9afc06f8e0 2024-04-17T19:48:07.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.213+0000 7f9af8c00640 1 -- 172.21.15.29:0/4074657286 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9af0005cf0 con 0x7f9afc06f8e0 2024-04-17T19:48:07.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.213+0000 7f9b03400640 1 -- 172.21.15.29:0/4074657286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9afc142fa0 con 0x7f9afc06f8e0 2024-04-17T19:48:07.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.213+0000 7f9af8c00640 1 -- 172.21.15.29:0/4074657286 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9af00072f0 con 0x7f9afc06f8e0 2024-04-17T19:48:07.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.213+0000 7f9b03400640 1 -- 172.21.15.29:0/4074657286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9afc141330 con 0x7f9afc06f8e0 2024-04-17T19:48:07.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.213+0000 7f9b03400640 1 -- 172.21.15.29:0/4074657286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9afc069b90 con 0x7f9afc06f8e0 2024-04-17T19:48:07.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.213+0000 7f9af8c00640 1 -- 172.21.15.29:0/4074657286 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9af001a070 con 0x7f9afc06f8e0 2024-04-17T19:48:07.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.213+0000 7f9af8c00640 1 --2- 172.21.15.29:0/4074657286 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9adc068380 0x7f9adc06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:07.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.213+0000 7f9af8c00640 1 -- 172.21.15.29:0/4074657286 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9af00860b0 con 0x7f9afc06f8e0 2024-04-17T19:48:07.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.213+0000 7f9afbe00640 1 --2- 172.21.15.29:0/4074657286 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9adc068380 0x7f9adc06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:07.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.217+0000 7f9afbe00640 1 --2- 172.21.15.29:0/4074657286 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9adc068380 0x7f9adc06a890 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9ae80b2000 tx=0x7f9ae8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:07.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.217+0000 7f9af8c00640 1 -- 172.21.15.29:0/4074657286 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9af0055bb0 con 0x7f9afc06f8e0 2024-04-17T19:48:07.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.381+0000 7f9b03400640 1 -- 172.21.15.29:0/4074657286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9afc149440 con 0x7f9afc06f8e0 2024-04-17T19:48:07.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.385+0000 7f9af8c00640 1 -- 172.21.15.29:0/4074657286 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f9af0059eb0 con 0x7f9afc06f8e0 2024-04-17T19:48:07.372 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-04-17T19:48:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.385+0000 7f9b03400640 1 -- 172.21.15.29:0/4074657286 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9adc068380 msgr2=0x7f9adc06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.385+0000 7f9b03400640 1 --2- 172.21.15.29:0/4074657286 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9adc068380 0x7f9adc06a890 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9ae80b2000 tx=0x7f9ae8008040 comp rx=0 tx=0).stop 2024-04-17T19:48:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.385+0000 7f9b03400640 1 -- 172.21.15.29:0/4074657286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9afc06f8e0 msgr2=0x7f9afc1f6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.385+0000 7f9b03400640 1 --2- 172.21.15.29:0/4074657286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9afc06f8e0 0x7f9afc1f6ac0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9af000c3a0 tx=0x7f9af000c860 comp rx=0 tx=0).stop 2024-04-17T19:48:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.385+0000 7f9b03400640 1 -- 172.21.15.29:0/4074657286 shutdown_connections 2024-04-17T19:48:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.385+0000 7f9b03400640 1 --2- 172.21.15.29:0/4074657286 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9adc068380 0x7f9adc06a890 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.385+0000 7f9b03400640 1 --2- 172.21.15.29:0/4074657286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9afc15bf20 0x7f9afc13cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.385+0000 7f9b03400640 1 --2- 172.21.15.29:0/4074657286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9afc0ce260 0x7f9afc143560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.385+0000 7f9b03400640 1 --2- 172.21.15.29:0/4074657286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9afc06f8e0 0x7f9afc1f6ac0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.385+0000 7f9b03400640 1 -- 172.21.15.29:0/4074657286 >> 172.21.15.29:0/4074657286 conn(0x7f9afc06c470 msgr2=0x7f9afc15f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.385+0000 7f9b03400640 1 -- 172.21.15.29:0/4074657286 shutdown_connections 2024-04-17T19:48:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.385+0000 7f9b03400640 1 -- 172.21.15.29:0/4074657286 wait complete. 2024-04-17T19:48:07.395 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T19:48:07.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.561+0000 7f6c07400640 1 Processor -- start 2024-04-17T19:48:07.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.561+0000 7f6c07400640 1 -- start start 2024-04-17T19:48:07.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.561+0000 7f6c07400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6c00072e70 0x7f6c001527c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:07.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.561+0000 7f6c07400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6c000c97c0 0x7f6c000733c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:07.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.561+0000 7f6c07400640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c00073900 con 0x7f6c0006f6d0 2024-04-17T19:48:07.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.561+0000 7f6c07400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f6c00072c30 con 0x7f6c000c97c0 2024-04-17T19:48:07.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.561+0000 7f6c07400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6c000c7250 con 0x7f6c00072e70 2024-04-17T19:48:07.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.561+0000 7f6c05400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6c0006f6d0 0x7f6c000c7bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49730/0 (socket says 172.21.15.29:49730) 2024-04-17T19:48:07.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.561+0000 7f6c05400640 1 -- 172.21.15.29:0/3947519681 learned_addr learned my addr 172.21.15.29:0/3947519681 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:07.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.561+0000 7f6c04a00640 1 --2- 172.21.15.29:0/3947519681 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6c00072e70 0x7f6c001527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:07.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.561+0000 7f6bffe00640 1 --2- 172.21.15.29:0/3947519681 >> v2:172.21.15.37:3300/0 conn(0x7f6c000c97c0 0x7f6c000733c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:07.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.561+0000 7f6bff400640 1 -- 172.21.15.29:0/3947519681 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4169801360 0 0) 0x7f6c00073900 con 0x7f6c0006f6d0 2024-04-17T19:48:07.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.561+0000 7f6bff400640 1 -- 172.21.15.29:0/3947519681 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6be8003530 con 0x7f6c0006f6d0 2024-04-17T19:48:07.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.565+0000 7f6c04a00640 1 -- 172.21.15.29:0/3947519681 >> v2:172.21.15.37:3300/0 conn(0x7f6c000c97c0 msgr2=0x7f6c000733c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:07.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.565+0000 7f6c04a00640 1 --2- 172.21.15.29:0/3947519681 >> v2:172.21.15.37:3300/0 conn(0x7f6c000c97c0 0x7f6c000733c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.565+0000 7f6c04a00640 1 -- 172.21.15.29:0/3947519681 >> v1:172.21.15.29:6789/0 conn(0x7f6c0006f6d0 legacy=0x7f6c000c7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:07.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.565+0000 7f6bff400640 1 -- 172.21.15.29:0/3947519681 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 995725105 0 0) 0x7f6be8003530 con 0x7f6c0006f6d0 2024-04-17T19:48:07.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.565+0000 7f6c04a00640 1 -- 172.21.15.29:0/3947519681 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c000c7530 con 0x7f6c00072e70 2024-04-17T19:48:07.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.565+0000 7f6c04a00640 1 --2- 172.21.15.29:0/3947519681 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6c00072e70 0x7f6c001527c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6bf40097c0 tx=0x7f6bf401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=913fd1afa4c1cbc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:07.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.565+0000 7f6bff400640 1 -- 172.21.15.29:0/3947519681 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf40b0070 con 0x7f6c00072e70 2024-04-17T19:48:07.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.565+0000 7f6bff400640 1 -- 172.21.15.29:0/3947519681 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bf40ab400 con 0x7f6c00072e70 2024-04-17T19:48:07.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.565+0000 7f6bff400640 1 -- 172.21.15.29:0/3947519681 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8001a00 con 0x7f6c00072e70 2024-04-17T19:48:07.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.569+0000 7f6c07400640 1 -- 172.21.15.29:0/3947519681 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6c00072e70 msgr2=0x7f6c001527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:07.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.569+0000 7f6c07400640 1 --2- 172.21.15.29:0/3947519681 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6c00072e70 0x7f6c001527c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6bf40097c0 tx=0x7f6bf401c3c0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.569+0000 7f6c07400640 1 -- 172.21.15.29:0/3947519681 shutdown_connections 2024-04-17T19:48:07.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.569+0000 7f6c07400640 1 --2- 172.21.15.29:0/3947519681 >> v2:172.21.15.37:3300/0 conn(0x7f6c000c97c0 0x7f6c000733c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.569+0000 7f6c07400640 1 --2- 172.21.15.29:0/3947519681 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6c00072e70 0x7f6c001527c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.569+0000 7f6c07400640 1 -- 172.21.15.29:0/3947519681 >> 172.21.15.29:0/3947519681 conn(0x7f6c0006bf90 msgr2=0x7f6c0006c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:07.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.569+0000 7f6c07400640 1 -- 172.21.15.29:0/3947519681 shutdown_connections 2024-04-17T19:48:07.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.569+0000 7f6c07400640 1 -- 172.21.15.29:0/3947519681 wait complete. 2024-04-17T19:48:07.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.569+0000 7f6c07400640 1 Processor -- start 2024-04-17T19:48:07.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.569+0000 7f6c07400640 1 -- start start 2024-04-17T19:48:07.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.569+0000 7f6c07400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c0006f6d0 0x7f6c0014be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:07.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6c07400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6c00072e70 0x7f6c001569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:07.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6c07400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6c000c97c0 0x7f6c000cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6c07400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6c00072880 con 0x7f6c0006f6d0 2024-04-17T19:48:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6c07400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6c000c47f0 con 0x7f6c000c97c0 2024-04-17T19:48:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6c07400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6c001588f0 con 0x7f6c00072e70 2024-04-17T19:48:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6c04a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c0006f6d0 0x7f6c0014be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6c04a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c0006f6d0 0x7f6c0014be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52078/0 (socket says 172.21.15.29:52078) 2024-04-17T19:48:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6c04a00640 1 -- 172.21.15.29:0/2933410951 learned_addr learned my addr 172.21.15.29:0/2933410951 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6bffe00640 1 --2- 172.21.15.29:0/2933410951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6c00072e70 0x7f6c001569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6c05400640 1 --2- 172.21.15.29:0/2933410951 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6c000c97c0 0x7f6c000cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6c04a00640 1 -- 172.21.15.29:0/2933410951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6c00072e70 msgr2=0x7f6c001569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6c04a00640 1 --2- 172.21.15.29:0/2933410951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6c00072e70 0x7f6c001569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6c04a00640 1 -- 172.21.15.29:0/2933410951 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6c000c97c0 msgr2=0x7f6c000cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6c04a00640 1 --2- 172.21.15.29:0/2933410951 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6c000c97c0 0x7f6c000cce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6c04a00640 1 -- 172.21.15.29:0/2933410951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c00156f00 con 0x7f6c0006f6d0 2024-04-17T19:48:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6bffe00640 1 --2- 172.21.15.29:0/2933410951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6c00072e70 0x7f6c001569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:07.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6c04a00640 1 --2- 172.21.15.29:0/2933410951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c0006f6d0 0x7f6c0014be10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4000c10 tx=0x7f6bf401ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:07.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6bfcc00640 1 -- 172.21.15.29:0/2933410951 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf40b0050 con 0x7f6c0006f6d0 2024-04-17T19:48:07.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6bfcc00640 1 -- 172.21.15.29:0/2933410951 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bf40bc070 con 0x7f6c0006f6d0 2024-04-17T19:48:07.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6bfcc00640 1 -- 172.21.15.29:0/2933410951 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4006e20 con 0x7f6c0006f6d0 2024-04-17T19:48:07.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6c07400640 1 -- 172.21.15.29:0/2933410951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c00142430 con 0x7f6c0006f6d0 2024-04-17T19:48:07.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6c07400640 1 -- 172.21.15.29:0/2933410951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c00158050 con 0x7f6c0006f6d0 2024-04-17T19:48:07.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6c07400640 1 -- 172.21.15.29:0/2933410951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c00069b90 con 0x7f6c0006f6d0 2024-04-17T19:48:07.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.573+0000 7f6bfcc00640 1 -- 172.21.15.29:0/2933410951 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6bf40a4d10 con 0x7f6c0006f6d0 2024-04-17T19:48:07.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.577+0000 7f6bfcc00640 1 --2- 172.21.15.29:0/2933410951 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6be0068380 0x7f6be006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:07.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.577+0000 7f6bfcc00640 1 -- 172.21.15.29:0/2933410951 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6bf4120ff0 con 0x7f6c0006f6d0 2024-04-17T19:48:07.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.577+0000 7f6bffe00640 1 --2- 172.21.15.29:0/2933410951 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6be0068380 0x7f6be006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:07.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.577+0000 7f6bffe00640 1 --2- 172.21.15.29:0/2933410951 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6be0068380 0x7f6be006a890 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6c000cd970 tx=0x7f6bec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:07.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.581+0000 7f6bfcc00640 1 -- 172.21.15.29:0/2933410951 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6bf40c6020 con 0x7f6c0006f6d0 2024-04-17T19:48:07.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.745+0000 7f6c07400640 1 -- 172.21.15.29:0/2933410951 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6c0014fe60 con 0x7f6c0006f6d0 2024-04-17T19:48:07.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.745+0000 7f6bfcc00640 1 -- 172.21.15.29:0/2933410951 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f6bf4015fb0 con 0x7f6c0006f6d0 2024-04-17T19:48:07.733 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-04-17T19:48:07.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.745+0000 7f6c07400640 1 -- 172.21.15.29:0/2933410951 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6be0068380 msgr2=0x7f6be006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:07.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.745+0000 7f6c07400640 1 --2- 172.21.15.29:0/2933410951 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6be0068380 0x7f6be006a890 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6c000cd970 tx=0x7f6bec008040 comp rx=0 tx=0).stop 2024-04-17T19:48:07.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.745+0000 7f6c07400640 1 -- 172.21.15.29:0/2933410951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c0006f6d0 msgr2=0x7f6c0014be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:07.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.745+0000 7f6c07400640 1 --2- 172.21.15.29:0/2933410951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c0006f6d0 0x7f6c0014be10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4000c10 tx=0x7f6bf401ac00 comp rx=0 tx=0).stop 2024-04-17T19:48:07.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.745+0000 7f6c07400640 1 -- 172.21.15.29:0/2933410951 shutdown_connections 2024-04-17T19:48:07.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.745+0000 7f6c07400640 1 --2- 172.21.15.29:0/2933410951 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6be0068380 0x7f6be006a890 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.745+0000 7f6c07400640 1 --2- 172.21.15.29:0/2933410951 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6c000c97c0 0x7f6c000cce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.745+0000 7f6c07400640 1 --2- 172.21.15.29:0/2933410951 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6c00072e70 0x7f6c001569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.745+0000 7f6c07400640 1 --2- 172.21.15.29:0/2933410951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c0006f6d0 0x7f6c0014be10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.745+0000 7f6c07400640 1 -- 172.21.15.29:0/2933410951 >> 172.21.15.29:0/2933410951 conn(0x7f6c0006bf90 msgr2=0x7f6c00150080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:07.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.745+0000 7f6c07400640 1 -- 172.21.15.29:0/2933410951 shutdown_connections 2024-04-17T19:48:07.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.745+0000 7f6c07400640 1 -- 172.21.15.29:0/2933410951 wait complete. 2024-04-17T19:48:07.756 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T19:48:07.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.917+0000 7f3148400640 1 Processor -- start 2024-04-17T19:48:07.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.917+0000 7f3148400640 1 -- start start 2024-04-17T19:48:07.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.921+0000 7f3148400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f31401593d0 0x7f31401597e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:07.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.921+0000 7f3148400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31400c9f00 0x7f31400ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:07.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.921+0000 7f3148400640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f314015c6d0 con 0x7f3140071ce0 2024-04-17T19:48:07.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.921+0000 7f3148400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f31400c7bd0 con 0x7f31401593d0 2024-04-17T19:48:07.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.921+0000 7f3148400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f314015c9b0 con 0x7f31400c9f00 2024-04-17T19:48:07.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.921+0000 7f3146400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31400c9f00 0x7f31400ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:07.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.921+0000 7f3145000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f31401593d0 0x7f31401597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:07.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.921+0000 7f3146400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31400c9f00 0x7f31400ca310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:48928/0 (socket says 172.21.15.29:48928) 2024-04-17T19:48:07.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.921+0000 7f3146400640 1 -- 172.21.15.29:0/915177105 learned_addr learned my addr 172.21.15.29:0/915177105 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:07.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.921+0000 7f3145a00640 1 --1- 172.21.15.29:0/915177105 >> v1:172.21.15.37:6789/0 conn(0x7f3140071ce0 0x7f3140141de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:51544/0 (socket says 172.21.15.29:51544) 2024-04-17T19:48:07.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.921+0000 7f3145000640 1 -- 172.21.15.29:0/915177105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31400c9f00 msgr2=0x7f31400ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:07.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.921+0000 7f3145000640 1 --2- 172.21.15.29:0/915177105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31400c9f00 0x7f31400ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.921+0000 7f3145000640 1 -- 172.21.15.29:0/915177105 >> v1:172.21.15.37:6789/0 conn(0x7f3140071ce0 legacy=0x7f3140141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:07.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.921+0000 7f3145000640 1 -- 172.21.15.29:0/915177105 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314015cc90 con 0x7f31401593d0 2024-04-17T19:48:07.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.921+0000 7f3146400640 1 --2- 172.21.15.29:0/915177105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31400c9f00 0x7f31400ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:48:07.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.925+0000 7f3145000640 1 --2- 172.21.15.29:0/915177105 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31401593d0 0x7f31401597e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f312c0097c0 tx=0x7f312c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db47b7fd95e59ad7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:07.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.925+0000 7f313fe00640 1 -- 172.21.15.29:0/915177105 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312c0b1070 con 0x7f31401593d0 2024-04-17T19:48:07.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.925+0000 7f313fe00640 1 -- 172.21.15.29:0/915177105 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f312c0ac410 con 0x7f31401593d0 2024-04-17T19:48:07.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.925+0000 7f313fe00640 1 -- 172.21.15.29:0/915177105 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312c0ac7e0 con 0x7f31401593d0 2024-04-17T19:48:07.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.925+0000 7f3148400640 1 -- 172.21.15.29:0/915177105 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31401593d0 msgr2=0x7f31401597e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:07.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.925+0000 7f3148400640 1 --2- 172.21.15.29:0/915177105 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31401593d0 0x7f31401597e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f312c0097c0 tx=0x7f312c01c4a0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.925+0000 7f3148400640 1 -- 172.21.15.29:0/915177105 shutdown_connections 2024-04-17T19:48:07.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.925+0000 7f3148400640 1 --2- 172.21.15.29:0/915177105 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31400c9f00 0x7f31400ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.925+0000 7f3148400640 1 --2- 172.21.15.29:0/915177105 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31401593d0 0x7f31401597e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.925+0000 7f3148400640 1 -- 172.21.15.29:0/915177105 >> 172.21.15.29:0/915177105 conn(0x7f314006bf90 msgr2=0x7f314006c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:07.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.925+0000 7f3148400640 1 -- 172.21.15.29:0/915177105 shutdown_connections 2024-04-17T19:48:07.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.925+0000 7f3148400640 1 -- 172.21.15.29:0/915177105 wait complete. 2024-04-17T19:48:07.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3148400640 1 Processor -- start 2024-04-17T19:48:07.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3148400640 1 -- start start 2024-04-17T19:48:07.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3148400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3140071ce0 0x7f3140165210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3148400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31400c9f00 0x7f314014de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3148400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31401593d0 0x7f3140133970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3148400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f314015c3c0 con 0x7f31400c9f00 2024-04-17T19:48:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3148400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f314015c240 con 0x7f3140071ce0 2024-04-17T19:48:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3148400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3140163e30 con 0x7f31401593d0 2024-04-17T19:48:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3145000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31400c9f00 0x7f314014de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3145000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31400c9f00 0x7f314014de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52082/0 (socket says 172.21.15.29:52082) 2024-04-17T19:48:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3145000640 1 -- 172.21.15.29:0/3673417346 learned_addr learned my addr 172.21.15.29:0/3673417346 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3145a00640 1 --2- 172.21.15.29:0/3673417346 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3140071ce0 0x7f3140165210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3146400640 1 --2- 172.21.15.29:0/3673417346 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31401593d0 0x7f3140133970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3145000640 1 -- 172.21.15.29:0/3673417346 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31401593d0 msgr2=0x7f3140133970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3145000640 1 --2- 172.21.15.29:0/3673417346 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31401593d0 0x7f3140133970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3145000640 1 -- 172.21.15.29:0/3673417346 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3140071ce0 msgr2=0x7f3140165210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:07.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3145000640 1 --2- 172.21.15.29:0/3673417346 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3140071ce0 0x7f3140165210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:07.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3145000640 1 -- 172.21.15.29:0/3673417346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3140165750 con 0x7f31400c9f00 2024-04-17T19:48:07.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3146400640 1 --2- 172.21.15.29:0/3673417346 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31401593d0 0x7f3140133970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:07.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.929+0000 7f3145a00640 1 --2- 172.21.15.29:0/3673417346 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3140071ce0 0x7f3140165210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:07.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.933+0000 7f3145000640 1 --2- 172.21.15.29:0/3673417346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31400c9f00 0x7f314014de30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f312c01a0d0 tx=0x7f312c019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:07.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.933+0000 7f313d600640 1 -- 172.21.15.29:0/3673417346 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312c0b1040 con 0x7f31400c9f00 2024-04-17T19:48:07.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.933+0000 7f313d600640 1 -- 172.21.15.29:0/3673417346 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f312c0ba070 con 0x7f31400c9f00 2024-04-17T19:48:07.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.933+0000 7f313d600640 1 -- 172.21.15.29:0/3673417346 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312c006e20 con 0x7f31400c9f00 2024-04-17T19:48:07.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.933+0000 7f3148400640 1 -- 172.21.15.29:0/3673417346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f314015a910 con 0x7f31400c9f00 2024-04-17T19:48:07.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.933+0000 7f3148400640 1 -- 172.21.15.29:0/3673417346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f314015aed0 con 0x7f31400c9f00 2024-04-17T19:48:07.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.933+0000 7f3132a00640 1 -- 172.21.15.29:0/3673417346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3140069b90 con 0x7f31400c9f00 2024-04-17T19:48:07.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.933+0000 7f313d600640 1 -- 172.21.15.29:0/3673417346 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f312c0b5650 con 0x7f31400c9f00 2024-04-17T19:48:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.933+0000 7f313d600640 1 --2- 172.21.15.29:0/3673417346 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3120068380 0x7f312006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.937+0000 7f313d600640 1 -- 172.21.15.29:0/3673417346 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f312c1209b0 con 0x7f31400c9f00 2024-04-17T19:48:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.937+0000 7f3145a00640 1 --2- 172.21.15.29:0/3673417346 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3120068380 0x7f312006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:07.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.937+0000 7f3145a00640 1 --2- 172.21.15.29:0/3673417346 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3120068380 0x7f312006a890 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3134000f70 tx=0x7f3134007d70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:07.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:07.941+0000 7f313d600640 1 -- 172.21.15.29:0/3673417346 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f312c016090 con 0x7f31400c9f00 2024-04-17T19:48:08.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.113+0000 7f3132a00640 1 -- 172.21.15.29:0/3673417346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f31400700e0 con 0x7f31400c9f00 2024-04-17T19:48:08.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.113+0000 7f313d600640 1 -- 172.21.15.29:0/3673417346 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f312c0f0b90 con 0x7f31400c9f00 2024-04-17T19:48:08.103 INFO:teuthology.orchestra.run.smithi029.stderr:listed 0 entries 2024-04-17T19:48:08.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.117+0000 7f3132a00640 1 -- 172.21.15.29:0/3673417346 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3120068380 msgr2=0x7f312006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:08.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.117+0000 7f3132a00640 1 --2- 172.21.15.29:0/3673417346 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3120068380 0x7f312006a890 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3134000f70 tx=0x7f3134007d70 comp rx=0 tx=0).stop 2024-04-17T19:48:08.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.117+0000 7f3132a00640 1 -- 172.21.15.29:0/3673417346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31400c9f00 msgr2=0x7f314014de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:08.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.117+0000 7f3132a00640 1 --2- 172.21.15.29:0/3673417346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31400c9f00 0x7f314014de30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f312c01a0d0 tx=0x7f312c019240 comp rx=0 tx=0).stop 2024-04-17T19:48:08.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.117+0000 7f3132a00640 1 -- 172.21.15.29:0/3673417346 shutdown_connections 2024-04-17T19:48:08.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.117+0000 7f3132a00640 1 --2- 172.21.15.29:0/3673417346 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3120068380 0x7f312006a890 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.117+0000 7f3132a00640 1 --2- 172.21.15.29:0/3673417346 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31401593d0 0x7f3140133970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.117+0000 7f3132a00640 1 --2- 172.21.15.29:0/3673417346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31400c9f00 0x7f314014de30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.117+0000 7f3132a00640 1 --2- 172.21.15.29:0/3673417346 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3140071ce0 0x7f3140165210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.117+0000 7f3132a00640 1 -- 172.21.15.29:0/3673417346 >> 172.21.15.29:0/3673417346 conn(0x7f314006bf90 msgr2=0x7f31401408e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:08.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.117+0000 7f3132a00640 1 -- 172.21.15.29:0/3673417346 shutdown_connections 2024-04-17T19:48:08.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.117+0000 7f3132a00640 1 -- 172.21.15.29:0/3673417346 wait complete. 2024-04-17T19:48:08.125 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi029.front.sepia.ceph.com... 2024-04-17T19:48:08.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:08.126 DEBUG:teuthology.orchestra.run.smithi029:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-17T19:48:08.156 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi029.stderr:ceph-fuse[32615]: fuse finished with error 0 and tester_r 0 2024-04-17T19:48:08.158 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T19:48:08.235 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2024-04-17T19:48:08.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:08.235 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-17T19:48:08.262 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:48:08.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.469+0000 7fbc82a00640 1 Processor -- start 2024-04-17T19:48:08.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.469+0000 7fbc82a00640 1 -- start start 2024-04-17T19:48:08.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.473+0000 7fbc82a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbc7c0c69d0 0x7fbc7c0b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:08.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.473+0000 7fbc82a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc7c156250 0x7fbc7c0d17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:08.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.473+0000 7fbc82a00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc7c0c64d0 con 0x7fbc7c0d1d10 2024-04-17T19:48:08.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.473+0000 7fbc82a00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fbc7c15cf00 con 0x7fbc7c0c69d0 2024-04-17T19:48:08.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.473+0000 7fbc82a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbc7c156660 con 0x7fbc7c156250 2024-04-17T19:48:08.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.473+0000 7fbc7be00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbc7c0c69d0 0x7fbc7c0b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:08.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.473+0000 7fbc7be00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbc7c0c69d0 0x7fbc7c0b40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52086/0 (socket says 172.21.15.29:52086) 2024-04-17T19:48:08.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.473+0000 7fbc7be00640 1 -- 172.21.15.29:0/2175660160 learned_addr learned my addr 172.21.15.29:0/2175660160 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:08.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.473+0000 7fbc73e00640 1 --2- 172.21.15.29:0/2175660160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc7c156250 0x7fbc7c0d17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:08.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.473+0000 7fbc7be00640 1 -- 172.21.15.29:0/2175660160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc7c156250 msgr2=0x7fbc7c0d17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:08.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.473+0000 7fbc7be00640 1 --2- 172.21.15.29:0/2175660160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc7c156250 0x7fbc7c0d17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.473+0000 7fbc7be00640 1 -- 172.21.15.29:0/2175660160 >> v1:172.21.15.37:6789/0 conn(0x7fbc7c0d1d10 legacy=0x7fbc7c0c78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:08.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.473+0000 7fbc7be00640 1 -- 172.21.15.29:0/2175660160 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7c0ccce0 con 0x7fbc7c0c69d0 2024-04-17T19:48:08.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.473+0000 7fbc7be00640 1 --2- 172.21.15.29:0/2175660160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc7c0c69d0 0x7fbc7c0b40c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009870 tx=0x7fbc6c01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed62fe44c6be52ef server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:08.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.473+0000 7fbc7b400640 1 -- 172.21.15.29:0/2175660160 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c0b5070 con 0x7fbc7c0c69d0 2024-04-17T19:48:08.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.473+0000 7fbc7b400640 1 -- 172.21.15.29:0/2175660160 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc6c0a8a50 con 0x7fbc7c0c69d0 2024-04-17T19:48:08.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.473+0000 7fbc7b400640 1 -- 172.21.15.29:0/2175660160 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c0a8e20 con 0x7fbc7c0c69d0 2024-04-17T19:48:08.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.477+0000 7fbc82a00640 1 -- 172.21.15.29:0/2175660160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc7c0c69d0 msgr2=0x7fbc7c0b40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:08.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.477+0000 7fbc82a00640 1 --2- 172.21.15.29:0/2175660160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc7c0c69d0 0x7fbc7c0b40c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c009870 tx=0x7fbc6c01ed50 comp rx=0 tx=0).stop 2024-04-17T19:48:08.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.477+0000 7fbc82a00640 1 -- 172.21.15.29:0/2175660160 shutdown_connections 2024-04-17T19:48:08.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.477+0000 7fbc82a00640 1 --2- 172.21.15.29:0/2175660160 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc7c156250 0x7fbc7c0d17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.477+0000 7fbc82a00640 1 --2- 172.21.15.29:0/2175660160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc7c0c69d0 0x7fbc7c0b40c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.477+0000 7fbc82a00640 1 -- 172.21.15.29:0/2175660160 >> 172.21.15.29:0/2175660160 conn(0x7fbc7c06d080 msgr2=0x7fbc7c06d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:08.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.477+0000 7fbc82a00640 1 -- 172.21.15.29:0/2175660160 shutdown_connections 2024-04-17T19:48:08.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.477+0000 7fbc82a00640 1 -- 172.21.15.29:0/2175660160 wait complete. 2024-04-17T19:48:08.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.477+0000 7fbc82a00640 1 Processor -- start 2024-04-17T19:48:08.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.477+0000 7fbc82a00640 1 -- start start 2024-04-17T19:48:08.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc82a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc7c0c69d0 0x7fbc7c15e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:08.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc82a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc7c0d1d10 0x7fbc7c153d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:08.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc82a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc7c156250 0x7fbc7c150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:08.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc82a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c0c70c0 con 0x7fbc7c0d1d10 2024-04-17T19:48:08.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc82a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbc7c0cd2a0 con 0x7fbc7c0c69d0 2024-04-17T19:48:08.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc82a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbc7c0c73a0 con 0x7fbc7c156250 2024-04-17T19:48:08.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc73e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc7c0d1d10 0x7fbc7c153d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:08.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc73e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc7c0d1d10 0x7fbc7c153d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52088/0 (socket says 172.21.15.29:52088) 2024-04-17T19:48:08.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc73e00640 1 -- 172.21.15.29:0/1444154630 learned_addr learned my addr 172.21.15.29:0/1444154630 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:08.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc80a00640 1 --2- 172.21.15.29:0/1444154630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc7c156250 0x7fbc7c150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:08.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc7be00640 1 --2- 172.21.15.29:0/1444154630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc7c0c69d0 0x7fbc7c15e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:08.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc73e00640 1 -- 172.21.15.29:0/1444154630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc7c156250 msgr2=0x7fbc7c150020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:08.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc73e00640 1 --2- 172.21.15.29:0/1444154630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc7c156250 0x7fbc7c150020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc73e00640 1 -- 172.21.15.29:0/1444154630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc7c0c69d0 msgr2=0x7fbc7c15e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:08.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc73e00640 1 --2- 172.21.15.29:0/1444154630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc7c0c69d0 0x7fbc7c15e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc73e00640 1 -- 172.21.15.29:0/1444154630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc7c15eaa0 con 0x7fbc7c0d1d10 2024-04-17T19:48:08.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc80a00640 1 --2- 172.21.15.29:0/1444154630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc7c156250 0x7fbc7c150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:08.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc73e00640 1 --2- 172.21.15.29:0/1444154630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc7c0d1d10 0x7fbc7c153d60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400d620 tx=0x7fbc6400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:08.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc78c00640 1 -- 172.21.15.29:0/1444154630 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc640125f0 con 0x7fbc7c0d1d10 2024-04-17T19:48:08.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc78c00640 1 -- 172.21.15.29:0/1444154630 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc64019070 con 0x7fbc7c0d1d10 2024-04-17T19:48:08.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc82a00640 1 -- 172.21.15.29:0/1444154630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc7c1f2b30 con 0x7fbc7c0d1d10 2024-04-17T19:48:08.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc78c00640 1 -- 172.21.15.29:0/1444154630 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc640129b0 con 0x7fbc7c0d1d10 2024-04-17T19:48:08.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc82a00640 1 -- 172.21.15.29:0/1444154630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc7c1f30f0 con 0x7fbc7c0d1d10 2024-04-17T19:48:08.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc82a00640 1 -- 172.21.15.29:0/1444154630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc7c069b90 con 0x7fbc7c0d1d10 2024-04-17T19:48:08.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.481+0000 7fbc78c00640 1 -- 172.21.15.29:0/1444154630 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbc64012b50 con 0x7fbc7c0d1d10 2024-04-17T19:48:08.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.485+0000 7fbc78c00640 1 --2- 172.21.15.29:0/1444154630 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbc58068400 0x7fbc5806a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:08.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.485+0000 7fbc78c00640 1 -- 172.21.15.29:0/1444154630 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fbc64087060 con 0x7fbc7c0d1d10 2024-04-17T19:48:08.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.485+0000 7fbc7be00640 1 --2- 172.21.15.29:0/1444154630 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbc58068400 0x7fbc5806a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:08.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.485+0000 7fbc7be00640 1 --2- 172.21.15.29:0/1444154630 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbc58068400 0x7fbc5806a910 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0b2000 tx=0x7fbc6c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:08.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.489+0000 7fbc78c00640 1 -- 172.21.15.29:0/1444154630 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbc640572c0 con 0x7fbc7c0d1d10 2024-04-17T19:48:08.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.657+0000 7fbc82a00640 1 -- 172.21.15.29:0/1444154630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbc7c1f28b0 con 0x7fbc7c0d1d10 2024-04-17T19:48:08.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.657+0000 7fbc78c00640 1 -- 172.21.15.29:0/1444154630 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18394 (secure 0 0 0) 0x7fbc7c1f28b0 con 0x7fbc7c0d1d10 2024-04-17T19:48:08.645 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:08.645 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:47:54.321794+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T19:47:47.658449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T19:47:48.209047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T19:47:51.887909+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:48:08.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.657+0000 7fbc82a00640 1 -- 172.21.15.29:0/1444154630 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbc58068400 msgr2=0x7fbc5806a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:08.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.657+0000 7fbc82a00640 1 --2- 172.21.15.29:0/1444154630 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbc58068400 0x7fbc5806a910 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0b2000 tx=0x7fbc6c008040 comp rx=0 tx=0).stop 2024-04-17T19:48:08.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.657+0000 7fbc82a00640 1 -- 172.21.15.29:0/1444154630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc7c0d1d10 msgr2=0x7fbc7c153d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:08.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.657+0000 7fbc82a00640 1 --2- 172.21.15.29:0/1444154630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc7c0d1d10 0x7fbc7c153d60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbc6400d620 tx=0x7fbc6400dae0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.657+0000 7fbc82a00640 1 -- 172.21.15.29:0/1444154630 shutdown_connections 2024-04-17T19:48:08.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.657+0000 7fbc82a00640 1 --2- 172.21.15.29:0/1444154630 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbc58068400 0x7fbc5806a910 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.657+0000 7fbc82a00640 1 --2- 172.21.15.29:0/1444154630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbc7c156250 0x7fbc7c150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.657+0000 7fbc82a00640 1 --2- 172.21.15.29:0/1444154630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc7c0d1d10 0x7fbc7c153d60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.657+0000 7fbc82a00640 1 --2- 172.21.15.29:0/1444154630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbc7c0c69d0 0x7fbc7c15e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.657+0000 7fbc82a00640 1 -- 172.21.15.29:0/1444154630 >> 172.21.15.29:0/1444154630 conn(0x7fbc7c06d080 msgr2=0x7fbc7c0b58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:08.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.657+0000 7fbc82a00640 1 -- 172.21.15.29:0/1444154630 shutdown_connections 2024-04-17T19:48:08.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.657+0000 7fbc82a00640 1 -- 172.21.15.29:0/1444154630 wait complete. 2024-04-17T19:48:08.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:48:08.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.833+0000 7fcfb8600640 1 Processor -- start 2024-04-17T19:48:08.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.833+0000 7fcfb8600640 1 -- start start 2024-04-17T19:48:08.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.833+0000 7fcfb8600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfb00c4e10 0x7fcfb0153d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:08.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.833+0000 7fcfb8600640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fcfb015a460 0x7fcfb015a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:08.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.833+0000 7fcfb8600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcfb00cf730 0x7fcfb00cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:08.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.833+0000 7fcfb8600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fcfb00cbad0 con 0x7fcfb00cf730 2024-04-17T19:48:08.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.833+0000 7fcfb8600640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fcfb0159cc0 con 0x7fcfb015a460 2024-04-17T19:48:08.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.833+0000 7fcfb8600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcfb00c9e80 con 0x7fcfb00c4e10 2024-04-17T19:48:08.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.833+0000 7fcfb6600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcfb00cf730 0x7fcfb00cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:08.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.833+0000 7fcfb6600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcfb00cf730 0x7fcfb00cfb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52432/0 (socket says 172.21.15.29:52432) 2024-04-17T19:48:08.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.833+0000 7fcfb6600640 1 -- 172.21.15.29:0/287082627 learned_addr learned my addr 172.21.15.29:0/287082627 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:08.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.833+0000 7fcfb5200640 1 --2- 172.21.15.29:0/287082627 >> v2:172.21.15.37:3300/0 conn(0x7fcfb015a460 0x7fcfb015a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:08.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.833+0000 7fcfb5c00640 1 --2- 172.21.15.29:0/287082627 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfb00c4e10 0x7fcfb0153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:08.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.833+0000 7fcfb6600640 1 -- 172.21.15.29:0/287082627 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfb00c4e10 msgr2=0x7fcfb0153d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:08.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.833+0000 7fcfb6600640 1 --2- 172.21.15.29:0/287082627 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfb00c4e10 0x7fcfb0153d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.833+0000 7fcfb6600640 1 -- 172.21.15.29:0/287082627 >> v2:172.21.15.37:3300/0 conn(0x7fcfb015a460 msgr2=0x7fcfb015a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:08.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.833+0000 7fcfb6600640 1 --2- 172.21.15.29:0/287082627 >> v2:172.21.15.37:3300/0 conn(0x7fcfb015a460 0x7fcfb015a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.833+0000 7fcfb6600640 1 -- 172.21.15.29:0/287082627 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb00ca140 con 0x7fcfb00cf730 2024-04-17T19:48:08.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.837+0000 7fcfb6600640 1 --2- 172.21.15.29:0/287082627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00cf730 0x7fcfb00cfb40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcfb006eee0 tx=0x7fcfa801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8d3e7857cbd9b94 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:08.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.837+0000 7fcfafe00640 1 -- 172.21.15.29:0/287082627 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa80a5760 con 0x7fcfb00cf730 2024-04-17T19:48:08.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.837+0000 7fcfafe00640 1 -- 172.21.15.29:0/287082627 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfa80a5900 con 0x7fcfb00cf730 2024-04-17T19:48:08.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.837+0000 7fcfafe00640 1 -- 172.21.15.29:0/287082627 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa80a5cd0 con 0x7fcfb00cf730 2024-04-17T19:48:08.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.837+0000 7fcfb8600640 1 -- 172.21.15.29:0/287082627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00cf730 msgr2=0x7fcfb00cfb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:08.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.837+0000 7fcfb8600640 1 --2- 172.21.15.29:0/287082627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00cf730 0x7fcfb00cfb40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcfb006eee0 tx=0x7fcfa801c9f0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.837+0000 7fcfb8600640 1 -- 172.21.15.29:0/287082627 shutdown_connections 2024-04-17T19:48:08.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.837+0000 7fcfb8600640 1 --2- 172.21.15.29:0/287082627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00cf730 0x7fcfb00cfb40 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.837+0000 7fcfb8600640 1 --2- 172.21.15.29:0/287082627 >> v2:172.21.15.37:3300/0 conn(0x7fcfb015a460 0x7fcfb015a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.837+0000 7fcfb8600640 1 --2- 172.21.15.29:0/287082627 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfb00c4e10 0x7fcfb0153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.837+0000 7fcfb8600640 1 -- 172.21.15.29:0/287082627 >> 172.21.15.29:0/287082627 conn(0x7fcfb006e1a0 msgr2=0x7fcfb006e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:08.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.837+0000 7fcfb8600640 1 -- 172.21.15.29:0/287082627 shutdown_connections 2024-04-17T19:48:08.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.837+0000 7fcfb8600640 1 -- 172.21.15.29:0/287082627 wait complete. 2024-04-17T19:48:08.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb8600640 1 Processor -- start 2024-04-17T19:48:08.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb8600640 1 -- start start 2024-04-17T19:48:08.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb8600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00c4e10 0x7fcfb00cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:08.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb8600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcfb00cf730 0x7fcfb0144c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:08.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb8600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfb015a460 0x7fcfb01451b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:08.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb8600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb01542c0 con 0x7fcfb00c4e10 2024-04-17T19:48:08.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb8600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcfb0159f40 con 0x7fcfb00cf730 2024-04-17T19:48:08.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb8600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcfb00cb740 con 0x7fcfb015a460 2024-04-17T19:48:08.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb5c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00c4e10 0x7fcfb00cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:08.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb5c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00c4e10 0x7fcfb00cd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52440/0 (socket says 172.21.15.29:52440) 2024-04-17T19:48:08.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb5c00640 1 -- 172.21.15.29:0/2775530353 learned_addr learned my addr 172.21.15.29:0/2775530353 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:08.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb5200640 1 --2- 172.21.15.29:0/2775530353 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcfb00cf730 0x7fcfb0144c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:08.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb6600640 1 --2- 172.21.15.29:0/2775530353 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfb015a460 0x7fcfb01451b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:08.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb5c00640 1 -- 172.21.15.29:0/2775530353 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfb015a460 msgr2=0x7fcfb01451b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:08.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb5c00640 1 --2- 172.21.15.29:0/2775530353 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfb015a460 0x7fcfb01451b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb5c00640 1 -- 172.21.15.29:0/2775530353 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcfb00cf730 msgr2=0x7fcfb0144c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:08.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb5c00640 1 --2- 172.21.15.29:0/2775530353 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcfb00cf730 0x7fcfb0144c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:08.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb5c00640 1 -- 172.21.15.29:0/2775530353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfb00cd8d0 con 0x7fcfb00c4e10 2024-04-17T19:48:08.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb5200640 1 --2- 172.21.15.29:0/2775530353 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcfb00cf730 0x7fcfb0144c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:08.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfb5c00640 1 --2- 172.21.15.29:0/2775530353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00c4e10 0x7fcfb00cd390 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcfa400d800 tx=0x7fcfa400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:08.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfad600640 1 -- 172.21.15.29:0/2775530353 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa4018030 con 0x7fcfb00c4e10 2024-04-17T19:48:08.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.841+0000 7fcfad600640 1 -- 172.21.15.29:0/2775530353 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfa40096a0 con 0x7fcfb00c4e10 2024-04-17T19:48:08.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.845+0000 7fcfb8600640 1 -- 172.21.15.29:0/2775530353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfb01f95e0 con 0x7fcfb00c4e10 2024-04-17T19:48:08.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.845+0000 7fcfad600640 1 -- 172.21.15.29:0/2775530353 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa4009870 con 0x7fcfb00c4e10 2024-04-17T19:48:08.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.845+0000 7fcfb8600640 1 -- 172.21.15.29:0/2775530353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfb015c4f0 con 0x7fcfb00c4e10 2024-04-17T19:48:08.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.845+0000 7fcfb8600640 1 -- 172.21.15.29:0/2775530353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfb015cbb0 con 0x7fcfb00c4e10 2024-04-17T19:48:08.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.845+0000 7fcfad600640 1 -- 172.21.15.29:0/2775530353 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcfa401b070 con 0x7fcfb00c4e10 2024-04-17T19:48:08.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.845+0000 7fcfad600640 1 --2- 172.21.15.29:0/2775530353 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcf900683b0 0x7fcf9006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:08.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.845+0000 7fcfad600640 1 -- 172.21.15.29:0/2775530353 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fcfa4086eb0 con 0x7fcfb00c4e10 2024-04-17T19:48:08.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.845+0000 7fcfb5200640 1 --2- 172.21.15.29:0/2775530353 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcf900683b0 0x7fcf9006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:08.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.849+0000 7fcfb5200640 1 --2- 172.21.15.29:0/2775530353 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcf900683b0 0x7fcf9006a8c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcfb014a900 tx=0x7fcf9c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:08.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:08.849+0000 7fcfad600640 1 -- 172.21.15.29:0/2775530353 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fcfb015cbb0 con 0x7fcfb00c4e10 2024-04-17T19:48:09.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.017+0000 7fcfb8600640 1 -- 172.21.15.29:0/2775530353 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcfb0069b90 con 0x7fcfb00c4e10 2024-04-17T19:48:09.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.017+0000 7fcfad600640 1 -- 172.21.15.29:0/2775530353 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18394 (secure 0 0 0) 0x7fcfa40571a0 con 0x7fcfb00c4e10 2024-04-17T19:48:09.008 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:09.008 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:47:54.321794+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T19:47:47.658449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T19:47:48.209047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T19:47:51.887909+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:48:09.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.021+0000 7fcfb8600640 1 -- 172.21.15.29:0/2775530353 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcf900683b0 msgr2=0x7fcf9006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:09.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.021+0000 7fcfb8600640 1 --2- 172.21.15.29:0/2775530353 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcf900683b0 0x7fcf9006a8c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fcfb014a900 tx=0x7fcf9c008040 comp rx=0 tx=0).stop 2024-04-17T19:48:09.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.021+0000 7fcfb8600640 1 -- 172.21.15.29:0/2775530353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00c4e10 msgr2=0x7fcfb00cd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:09.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.021+0000 7fcfb8600640 1 --2- 172.21.15.29:0/2775530353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00c4e10 0x7fcfb00cd390 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcfa400d800 tx=0x7fcfa400dcc0 comp rx=0 tx=0).stop 2024-04-17T19:48:09.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.021+0000 7fcfb8600640 1 -- 172.21.15.29:0/2775530353 shutdown_connections 2024-04-17T19:48:09.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.021+0000 7fcfb8600640 1 --2- 172.21.15.29:0/2775530353 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcf900683b0 0x7fcf9006a8c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:09.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.021+0000 7fcfb8600640 1 --2- 172.21.15.29:0/2775530353 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcfb015a460 0x7fcfb01451b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:09.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.021+0000 7fcfb8600640 1 --2- 172.21.15.29:0/2775530353 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcfb00cf730 0x7fcfb0144c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:09.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.021+0000 7fcfb8600640 1 --2- 172.21.15.29:0/2775530353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcfb00c4e10 0x7fcfb00cd390 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:09.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.021+0000 7fcfb8600640 1 -- 172.21.15.29:0/2775530353 >> 172.21.15.29:0/2775530353 conn(0x7fcfb006e1a0 msgr2=0x7fcfb006e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:09.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.021+0000 7fcfb8600640 1 -- 172.21.15.29:0/2775530353 shutdown_connections 2024-04-17T19:48:09.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.021+0000 7fcfb8600640 1 -- 172.21.15.29:0/2775530353 wait complete. 2024-04-17T19:48:09.030 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T19:48:09.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.197+0000 7f388cc00640 1 Processor -- start 2024-04-17T19:48:09.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.197+0000 7f388cc00640 1 -- start start 2024-04-17T19:48:09.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.197+0000 7f388cc00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f388815d630 0x7f388815da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:09.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.197+0000 7f388cc00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3888159e70 con 0x7f38880c8920 2024-04-17T19:48:09.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.197+0000 7f388cc00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f388815a150 con 0x7f388806d0b0 2024-04-17T19:48:09.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.197+0000 7f388cc00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f388815a430 con 0x7f388815d630 2024-04-17T19:48:09.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.197+0000 7f3886a00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f38880c8920 0x7f38880c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38034/0 (socket says 172.21.15.29:38034) 2024-04-17T19:48:09.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.197+0000 7f3886a00640 1 -- 172.21.15.29:0/608982236 learned_addr learned my addr 172.21.15.29:0/608982236 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:09.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.197+0000 7f3887e00640 1 --2- 172.21.15.29:0/608982236 >> v2:172.21.15.37:3300/0 conn(0x7f388815d630 0x7f388815da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:09.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.197+0000 7f3886000640 1 -- 172.21.15.29:0/608982236 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2930100212 0 0) 0x7f3888159e70 con 0x7f38880c8920 2024-04-17T19:48:09.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.197+0000 7f3886000640 1 -- 172.21.15.29:0/608982236 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f386c003530 con 0x7f38880c8920 2024-04-17T19:48:09.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.197+0000 7f3886000640 1 -- 172.21.15.29:0/608982236 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1859233305 0 0) 0x7f386c003530 con 0x7f38880c8920 2024-04-17T19:48:09.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.197+0000 7f3887e00640 1 -- 172.21.15.29:0/608982236 >> v1:172.21.15.37:6789/0 conn(0x7f388806d0b0 legacy=0x7f38880ce210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:09.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.197+0000 7f3887e00640 1 -- 172.21.15.29:0/608982236 >> v1:172.21.15.29:6789/0 conn(0x7f38880c8920 legacy=0x7f38880c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:09.187 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.197+0000 7f3887e00640 1 -- 172.21.15.29:0/608982236 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f388815a710 con 0x7f388815d630 2024-04-17T19:48:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.201+0000 7f3887e00640 1 --2- 172.21.15.29:0/608982236 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f388815d630 0x7f388815da40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3880009d90 tx=0x7f388001c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8274ae9f6b4c370 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.201+0000 7f3886000640 1 -- 172.21.15.29:0/608982236 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3874001680 con 0x7f388815d630 2024-04-17T19:48:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.201+0000 7f3886000640 1 -- 172.21.15.29:0/608982236 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38800a6450 con 0x7f388815d630 2024-04-17T19:48:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.201+0000 7f3886000640 1 -- 172.21.15.29:0/608982236 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38800a6800 con 0x7f388815d630 2024-04-17T19:48:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.201+0000 7f388cc00640 1 -- 172.21.15.29:0/608982236 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f388815d630 msgr2=0x7f388815da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.201+0000 7f388cc00640 1 --2- 172.21.15.29:0/608982236 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f388815d630 0x7f388815da40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3880009d90 tx=0x7f388001c800 comp rx=0 tx=0).stop 2024-04-17T19:48:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.201+0000 7f388cc00640 1 -- 172.21.15.29:0/608982236 shutdown_connections 2024-04-17T19:48:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.201+0000 7f388cc00640 1 --2- 172.21.15.29:0/608982236 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f388815d630 0x7f388815da40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.201+0000 7f388cc00640 1 -- 172.21.15.29:0/608982236 >> 172.21.15.29:0/608982236 conn(0x7f3888070880 msgr2=0x7f3888070c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.201+0000 7f388cc00640 1 -- 172.21.15.29:0/608982236 shutdown_connections 2024-04-17T19:48:09.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.201+0000 7f388cc00640 1 -- 172.21.15.29:0/608982236 wait complete. 2024-04-17T19:48:09.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.205+0000 7f388cc00640 1 Processor -- start 2024-04-17T19:48:09.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.205+0000 7f388cc00640 1 -- start start 2024-04-17T19:48:09.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.205+0000 7f388cc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f388806d0b0 0x7f38881440b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:09.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.205+0000 7f388cc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38880c8920 0x7f388815ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:09.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.205+0000 7f388cc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f388815d630 0x7f38880c3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:09.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.205+0000 7f388cc00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f388806d640 con 0x7f38880c8920 2024-04-17T19:48:09.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.205+0000 7f388cc00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f388806d4c0 con 0x7f388815d630 2024-04-17T19:48:09.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.205+0000 7f388cc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3888167e10 con 0x7f388806d0b0 2024-04-17T19:48:09.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.205+0000 7f3886a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38880c8920 0x7f388815ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:09.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.205+0000 7f3886a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38880c8920 0x7f388815ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52454/0 (socket says 172.21.15.29:52454) 2024-04-17T19:48:09.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.205+0000 7f3886a00640 1 -- 172.21.15.29:0/2335986638 learned_addr learned my addr 172.21.15.29:0/2335986638 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:09.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.205+0000 7f3887e00640 1 --2- 172.21.15.29:0/2335986638 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f388815d630 0x7f38880c3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:09.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.205+0000 7f3887400640 1 --2- 172.21.15.29:0/2335986638 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f388806d0b0 0x7f38881440b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:09.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.209+0000 7f3886a00640 1 -- 172.21.15.29:0/2335986638 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f388806d0b0 msgr2=0x7f38881440b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:09.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.209+0000 7f3886a00640 1 --2- 172.21.15.29:0/2335986638 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f388806d0b0 0x7f38881440b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:09.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.209+0000 7f3886a00640 1 -- 172.21.15.29:0/2335986638 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f388815d630 msgr2=0x7f38880c3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:09.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.209+0000 7f3886a00640 1 --2- 172.21.15.29:0/2335986638 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f388815d630 0x7f38880c3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:09.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.209+0000 7f3886a00640 1 -- 172.21.15.29:0/2335986638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38881445f0 con 0x7f38880c8920 2024-04-17T19:48:09.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.209+0000 7f3886a00640 1 --2- 172.21.15.29:0/2335986638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38880c8920 0x7f388815ca70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f387400c530 tx=0x7f387400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:09.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.209+0000 7f387f400640 1 -- 172.21.15.29:0/2335986638 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3874017070 con 0x7f38880c8920 2024-04-17T19:48:09.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.209+0000 7f387f400640 1 -- 172.21.15.29:0/2335986638 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3874005cf0 con 0x7f38880c8920 2024-04-17T19:48:09.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.209+0000 7f388cc00640 1 -- 172.21.15.29:0/2335986638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38881435f0 con 0x7f38880c8920 2024-04-17T19:48:09.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.209+0000 7f387f400640 1 -- 172.21.15.29:0/2335986638 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38740072b0 con 0x7f38880c8920 2024-04-17T19:48:09.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.209+0000 7f388cc00640 1 -- 172.21.15.29:0/2335986638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f388815c310 con 0x7f38880c8920 2024-04-17T19:48:09.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.209+0000 7f388cc00640 1 -- 172.21.15.29:0/2335986638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f388804ff10 con 0x7f38880c8920 2024-04-17T19:48:09.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.209+0000 7f387f400640 1 -- 172.21.15.29:0/2335986638 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f387401a070 con 0x7f38880c8920 2024-04-17T19:48:09.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.209+0000 7f387f400640 1 --2- 172.21.15.29:0/2335986638 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3868068380 0x7f386806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:09.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.209+0000 7f387f400640 1 -- 172.21.15.29:0/2335986638 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3874086140 con 0x7f38880c8920 2024-04-17T19:48:09.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.213+0000 7f3887400640 1 --2- 172.21.15.29:0/2335986638 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3868068380 0x7f386806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:09.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.213+0000 7f3887400640 1 --2- 172.21.15.29:0/2335986638 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3868068380 0x7f386806a890 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3878007fd0 tx=0x7f3878009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:09.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.217+0000 7f387f400640 1 -- 172.21.15.29:0/2335986638 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3874055ce0 con 0x7f38880c8920 2024-04-17T19:48:09.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.417+0000 7f388cc00640 1 -- 172.21.15.29:0/2335986638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f3888158570 con 0x7f38880c8920 2024-04-17T19:48:09.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.713+0000 7f387f400640 1 -- 172.21.15.29:0/2335986638 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) ==== 118+0+0 (secure 0 0 0) 0x7f3874059fe0 con 0x7f38880c8920 2024-04-17T19:48:09.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.713+0000 7f388cc00640 1 -- 172.21.15.29:0/2335986638 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3868068380 msgr2=0x7f386806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:09.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.713+0000 7f388cc00640 1 --2- 172.21.15.29:0/2335986638 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3868068380 0x7f386806a890 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3878007fd0 tx=0x7f3878009040 comp rx=0 tx=0).stop 2024-04-17T19:48:09.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.713+0000 7f388cc00640 1 -- 172.21.15.29:0/2335986638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38880c8920 msgr2=0x7f388815ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:09.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.713+0000 7f388cc00640 1 --2- 172.21.15.29:0/2335986638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38880c8920 0x7f388815ca70 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f387400c530 tx=0x7f387400c9f0 comp rx=0 tx=0).stop 2024-04-17T19:48:09.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.713+0000 7f388cc00640 1 -- 172.21.15.29:0/2335986638 shutdown_connections 2024-04-17T19:48:09.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.713+0000 7f388cc00640 1 --2- 172.21.15.29:0/2335986638 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3868068380 0x7f386806a890 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:09.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.713+0000 7f388cc00640 1 --2- 172.21.15.29:0/2335986638 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f388815d630 0x7f38880c3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:09.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.713+0000 7f388cc00640 1 --2- 172.21.15.29:0/2335986638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38880c8920 0x7f388815ca70 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:09.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.713+0000 7f388cc00640 1 --2- 172.21.15.29:0/2335986638 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f388806d0b0 0x7f38881440b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:09.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.713+0000 7f388cc00640 1 -- 172.21.15.29:0/2335986638 >> 172.21.15.29:0/2335986638 conn(0x7f3888070880 msgr2=0x7f3888073230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:09.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.713+0000 7f388cc00640 1 -- 172.21.15.29:0/2335986638 shutdown_connections 2024-04-17T19:48:09.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.713+0000 7f388cc00640 1 -- 172.21.15.29:0/2335986638 wait complete. 2024-04-17T19:48:09.724 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300} 2024-04-17T19:48:09.725 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-17T19:48:09.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.893+0000 7f3832200640 1 Processor -- start 2024-04-17T19:48:09.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.893+0000 7f3832200640 1 -- start start 2024-04-17T19:48:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.893+0000 7f3832200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f382c0cbf80 0x7f382c0cc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.893+0000 7f3832200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f382c0c4b70 con 0x7f382c15a4b0 2024-04-17T19:48:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.893+0000 7f3832200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f382c0ca210 con 0x7f382c1541c0 2024-04-17T19:48:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.893+0000 7f3832200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f382c0ca790 con 0x7f382c0cbf80 2024-04-17T19:48:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.893+0000 7f3823e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f382c0cbf80 0x7f382c0cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.893+0000 7f3823e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f382c0cbf80 0x7f382c0cc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52462/0 (socket says 172.21.15.29:52462) 2024-04-17T19:48:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.893+0000 7f3823e00640 1 -- 172.21.15.29:0/3350678232 learned_addr learned my addr 172.21.15.29:0/3350678232 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.897+0000 7f3823e00640 1 -- 172.21.15.29:0/3350678232 >> v1:172.21.15.37:6789/0 conn(0x7f382c1541c0 legacy=0x7f382c0c5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.897+0000 7f3823e00640 1 -- 172.21.15.29:0/3350678232 >> v1:172.21.15.29:6789/0 conn(0x7f382c15a4b0 legacy=0x7f382c164790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.897+0000 7f3823e00640 1 -- 172.21.15.29:0/3350678232 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f382c0ca4b0 con 0x7f382c0cbf80 2024-04-17T19:48:09.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.897+0000 7f382aa00640 1 -- 172.21.15.29:0/3350678232 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2387866548 0 0) 0x7f382c0c4b70 con 0x7f382c15a4b0 2024-04-17T19:48:09.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.897+0000 7f3823e00640 1 --2- 172.21.15.29:0/3350678232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f382c0cbf80 0x7f382c0cc390 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f382c0ca3e0 tx=0x7f381401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd15053b09cf5f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:09.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.897+0000 7f382aa00640 1 -- 172.21.15.29:0/3350678232 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38140b0070 con 0x7f382c0cbf80 2024-04-17T19:48:09.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.897+0000 7f382aa00640 1 -- 172.21.15.29:0/3350678232 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38140ab600 con 0x7f382c0cbf80 2024-04-17T19:48:09.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.897+0000 7f382aa00640 1 -- 172.21.15.29:0/3350678232 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38140ab9d0 con 0x7f382c0cbf80 2024-04-17T19:48:09.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.897+0000 7f3832200640 1 -- 172.21.15.29:0/3350678232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f382c0cbf80 msgr2=0x7f382c0cc390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:09.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.897+0000 7f3832200640 1 --2- 172.21.15.29:0/3350678232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f382c0cbf80 0x7f382c0cc390 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f382c0ca3e0 tx=0x7f381401c4b0 comp rx=0 tx=0).stop 2024-04-17T19:48:09.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.897+0000 7f3832200640 1 -- 172.21.15.29:0/3350678232 shutdown_connections 2024-04-17T19:48:09.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.897+0000 7f3832200640 1 --2- 172.21.15.29:0/3350678232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f382c0cbf80 0x7f382c0cc390 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:09.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.897+0000 7f3832200640 1 -- 172.21.15.29:0/3350678232 >> 172.21.15.29:0/3350678232 conn(0x7f382c071a60 msgr2=0x7f382c071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:09.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.901+0000 7f3832200640 1 -- 172.21.15.29:0/3350678232 shutdown_connections 2024-04-17T19:48:09.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.901+0000 7f3832200640 1 -- 172.21.15.29:0/3350678232 wait complete. 2024-04-17T19:48:09.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.901+0000 7f3832200640 1 Processor -- start 2024-04-17T19:48:09.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.901+0000 7f3832200640 1 -- start start 2024-04-17T19:48:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f3832200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f382c0cbf80 0x7f382c1f6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f3832200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f382c1541c0 0x7f382c0b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f3832200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f382c15a4b0 0x7f382c0c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f3832200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f382c0caad0 con 0x7f382c0cbf80 2024-04-17T19:48:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f3832200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f382c15acd0 con 0x7f382c1541c0 2024-04-17T19:48:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f3832200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f382c0c8340 con 0x7f382c15a4b0 2024-04-17T19:48:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f382b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f382c0cbf80 0x7f382c1f6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f382b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f382c0cbf80 0x7f382c1f6280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52468/0 (socket says 172.21.15.29:52468) 2024-04-17T19:48:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f382b400640 1 -- 172.21.15.29:0/1278731211 learned_addr learned my addr 172.21.15.29:0/1278731211 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f3823e00640 1 --2- 172.21.15.29:0/1278731211 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f382c1541c0 0x7f382c0b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:09.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f382be00640 1 --2- 172.21.15.29:0/1278731211 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f382c15a4b0 0x7f382c0c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:09.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f382b400640 1 -- 172.21.15.29:0/1278731211 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f382c15a4b0 msgr2=0x7f382c0c8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:09.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f382b400640 1 --2- 172.21.15.29:0/1278731211 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f382c15a4b0 0x7f382c0c8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:09.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f382b400640 1 -- 172.21.15.29:0/1278731211 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f382c1541c0 msgr2=0x7f382c0b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:09.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f382b400640 1 --2- 172.21.15.29:0/1278731211 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f382c1541c0 0x7f382c0b0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:09.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f382b400640 1 -- 172.21.15.29:0/1278731211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f382c1f67c0 con 0x7f382c0cbf80 2024-04-17T19:48:09.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f382b400640 1 --2- 172.21.15.29:0/1278731211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f382c0cbf80 0x7f382c1f6280 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f381c00c3c0 tx=0x7f381c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f3823400640 1 -- 172.21.15.29:0/1278731211 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f381c017070 con 0x7f382c0cbf80 2024-04-17T19:48:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f3823400640 1 -- 172.21.15.29:0/1278731211 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f381c0052d0 con 0x7f382c0cbf80 2024-04-17T19:48:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f3832200640 1 -- 172.21.15.29:0/1278731211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f382c1f5aa0 con 0x7f382c0cbf80 2024-04-17T19:48:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f3823400640 1 -- 172.21.15.29:0/1278731211 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f381c003840 con 0x7f382c0cbf80 2024-04-17T19:48:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f3832200640 1 -- 172.21.15.29:0/1278731211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f382c0b4ee0 con 0x7f382c0cbf80 2024-04-17T19:48:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f3832200640 1 -- 172.21.15.29:0/1278731211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f382c069b90 con 0x7f382c0cbf80 2024-04-17T19:48:09.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.905+0000 7f3823400640 1 -- 172.21.15.29:0/1278731211 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f381c01a070 con 0x7f382c0cbf80 2024-04-17T19:48:09.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.909+0000 7f3823400640 1 --2- 172.21.15.29:0/1278731211 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3808068380 0x7f380806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:09.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.909+0000 7f3823400640 1 -- 172.21.15.29:0/1278731211 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f381c0860d0 con 0x7f382c0cbf80 2024-04-17T19:48:09.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.909+0000 7f3823e00640 1 --2- 172.21.15.29:0/1278731211 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3808068380 0x7f380806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:09.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.909+0000 7f3823e00640 1 --2- 172.21.15.29:0/1278731211 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3808068380 0x7f380806a890 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f38140ad000 tx=0x7f3814008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:09.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:09.913+0000 7f3823400640 1 -- 172.21.15.29:0/1278731211 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f381c005480 con 0x7f382c0cbf80 2024-04-17T19:48:10.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.117+0000 7f3832200640 1 -- 172.21.15.29:0/1278731211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f382c0ccec0 con 0x7f382c0cbf80 2024-04-17T19:48:10.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.729+0000 7f3823400640 1 -- 172.21.15.29:0/1278731211 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v9) ==== 105+0+0 (secure 0 0 0) 0x7f381c0562c0 con 0x7f382c0cbf80 2024-04-17T19:48:10.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.733+0000 7f3832200640 1 -- 172.21.15.29:0/1278731211 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3808068380 msgr2=0x7f380806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:10.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.733+0000 7f3832200640 1 --2- 172.21.15.29:0/1278731211 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3808068380 0x7f380806a890 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f38140ad000 tx=0x7f3814008040 comp rx=0 tx=0).stop 2024-04-17T19:48:10.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.733+0000 7f3832200640 1 -- 172.21.15.29:0/1278731211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f382c0cbf80 msgr2=0x7f382c1f6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:10.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.733+0000 7f3832200640 1 --2- 172.21.15.29:0/1278731211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f382c0cbf80 0x7f382c1f6280 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f381c00c3c0 tx=0x7f381c00c880 comp rx=0 tx=0).stop 2024-04-17T19:48:10.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.733+0000 7f3832200640 1 -- 172.21.15.29:0/1278731211 shutdown_connections 2024-04-17T19:48:10.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.733+0000 7f3832200640 1 --2- 172.21.15.29:0/1278731211 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3808068380 0x7f380806a890 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:10.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.733+0000 7f3832200640 1 --2- 172.21.15.29:0/1278731211 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f382c15a4b0 0x7f382c0c8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:10.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.733+0000 7f3832200640 1 --2- 172.21.15.29:0/1278731211 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f382c1541c0 0x7f382c0b0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:10.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.733+0000 7f3832200640 1 --2- 172.21.15.29:0/1278731211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f382c0cbf80 0x7f382c1f6280 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:10.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.733+0000 7f3832200640 1 -- 172.21.15.29:0/1278731211 >> 172.21.15.29:0/1278731211 conn(0x7f382c071a60 msgr2=0x7f382c15d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:10.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.733+0000 7f3832200640 1 -- 172.21.15.29:0/1278731211 shutdown_connections 2024-04-17T19:48:10.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.733+0000 7f3832200640 1 -- 172.21.15.29:0/1278731211 wait complete. 2024-04-17T19:48:10.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T19:48:10.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.913+0000 7f4e5f000640 1 Processor -- start 2024-04-17T19:48:10.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.913+0000 7f4e5f000640 1 -- start start 2024-04-17T19:48:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e5f000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f4e581593d0 0x7f4e581597e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e5f000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4e580c9f00 0x7f4e580ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e5f000640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e5815c6d0 con 0x7f4e58071ce0 2024-04-17T19:48:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e5f000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f4e580c7bd0 con 0x7f4e581593d0 2024-04-17T19:48:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e5f000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4e5815c9b0 con 0x7f4e580c9f00 2024-04-17T19:48:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4e58071ce0 0x7f4e58141de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38040/0 (socket says 172.21.15.29:38040) 2024-04-17T19:48:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57e00640 1 -- 172.21.15.29:0/3389466051 learned_addr learned my addr 172.21.15.29:0/3389466051 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e4fe00640 1 --2- 172.21.15.29:0/3389466051 >> v2:172.21.15.37:3300/0 conn(0x7f4e581593d0 0x7f4e581597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e5d000640 1 --2- 172.21.15.29:0/3389466051 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4e580c9f00 0x7f4e580ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57400640 1 -- 172.21.15.29:0/3389466051 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3724367489 0 0) 0x7f4e5815c6d0 con 0x7f4e58071ce0 2024-04-17T19:48:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57400640 1 -- 172.21.15.29:0/3389466051 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e3c005740 con 0x7f4e58071ce0 2024-04-17T19:48:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57400640 1 -- 172.21.15.29:0/3389466051 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1081835356 0 0) 0x7f4e3c005740 con 0x7f4e58071ce0 2024-04-17T19:48:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57400640 1 -- 172.21.15.29:0/3389466051 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4e580c9f00 msgr2=0x7f4e580ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57400640 1 --2- 172.21.15.29:0/3389466051 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4e580c9f00 0x7f4e580ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57400640 1 -- 172.21.15.29:0/3389466051 >> v2:172.21.15.37:3300/0 conn(0x7f4e581593d0 msgr2=0x7f4e581597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57400640 1 --2- 172.21.15.29:0/3389466051 >> v2:172.21.15.37:3300/0 conn(0x7f4e581593d0 0x7f4e581597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57400640 1 -- 172.21.15.29:0/3389466051 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e5815cc90 con 0x7f4e58071ce0 2024-04-17T19:48:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57400640 1 -- 172.21.15.29:0/3389466051 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f4e48001680 con 0x7f4e58071ce0 2024-04-17T19:48:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57400640 1 -- 172.21.15.29:0/3389466051 >> v1:172.21.15.29:6789/0 conn(0x7f4e58071ce0 legacy=0x7f4e58141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57400640 1 --2- 172.21.15.29:0/3389466051 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4e3c005bd0 0x7f4e3c006040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57400640 1 --2- 172.21.15.29:0/3389466051 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4e580c9f00 0x7f4e3c006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57400640 1 --2- 172.21.15.29:0/3389466051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e3c006f60 0x7f4e3c007370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57400640 1 -- 172.21.15.29:0/3389466051 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5815c9b0 con 0x7f4e3c006f60 2024-04-17T19:48:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57400640 1 -- 172.21.15.29:0/3389466051 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c00dbe0 con 0x7f4e3c005bd0 2024-04-17T19:48:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57400640 1 -- 172.21.15.29:0/3389466051 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4e3c00dd60 con 0x7f4e580c9f00 2024-04-17T19:48:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e57e00640 1 --2- 172.21.15.29:0/3389466051 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4e3c005bd0 0x7f4e3c006040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e5d000640 1 --2- 172.21.15.29:0/3389466051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e3c006f60 0x7f4e3c007370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e4fe00640 1 --2- 172.21.15.29:0/3389466051 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4e580c9f00 0x7f4e3c006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:10.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e5d000640 1 -- 172.21.15.29:0/3389466051 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4e580c9f00 msgr2=0x7f4e3c006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:10.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e5d000640 1 --2- 172.21.15.29:0/3389466051 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4e580c9f00 0x7f4e3c006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:10.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e5d000640 1 -- 172.21.15.29:0/3389466051 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4e3c005bd0 msgr2=0x7f4e3c006040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:10.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e5d000640 1 --2- 172.21.15.29:0/3389466051 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4e3c005bd0 0x7f4e3c006040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:10.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e5d000640 1 -- 172.21.15.29:0/3389466051 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e3c005740 con 0x7f4e3c006f60 2024-04-17T19:48:10.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.917+0000 7f4e4fe00640 1 --2- 172.21.15.29:0/3389466051 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4e580c9f00 0x7f4e3c006a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:10.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.921+0000 7f4e5d000640 1 --2- 172.21.15.29:0/3389466051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e3c006f60 0x7f4e3c007370 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4e50009f90 tx=0x7f4e5001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95dd712822ec92d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:10.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.921+0000 7f4e57400640 1 -- 172.21.15.29:0/3389466051 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e500b2070 con 0x7f4e3c006f60 2024-04-17T19:48:10.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.921+0000 7f4e57400640 1 -- 172.21.15.29:0/3389466051 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e500a5c30 con 0x7f4e3c006f60 2024-04-17T19:48:10.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.921+0000 7f4e57400640 1 -- 172.21.15.29:0/3389466051 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e50009c10 con 0x7f4e3c006f60 2024-04-17T19:48:10.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.921+0000 7f4e5f000640 1 -- 172.21.15.29:0/3389466051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e3c006f60 msgr2=0x7f4e3c007370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:10.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.921+0000 7f4e5f000640 1 --2- 172.21.15.29:0/3389466051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e3c006f60 0x7f4e3c007370 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4e50009f90 tx=0x7f4e5001cc20 comp rx=0 tx=0).stop 2024-04-17T19:48:10.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.921+0000 7f4e5d000640 1 -- 172.21.15.29:0/3389466051 reap_dead start 2024-04-17T19:48:10.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.921+0000 7f4e5f000640 1 -- 172.21.15.29:0/3389466051 shutdown_connections 2024-04-17T19:48:10.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.921+0000 7f4e5f000640 1 -- 172.21.15.29:0/3389466051 >> 172.21.15.29:0/3389466051 conn(0x7f4e5806bf90 msgr2=0x7f4e5806c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:10.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.921+0000 7f4e5f000640 1 -- 172.21.15.29:0/3389466051 shutdown_connections 2024-04-17T19:48:10.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.921+0000 7f4e5f000640 1 -- 172.21.15.29:0/3389466051 wait complete. 2024-04-17T19:48:10.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e5f000640 1 Processor -- start 2024-04-17T19:48:10.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e5f000640 1 -- start start 2024-04-17T19:48:10.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e5f000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4e3c006f60 0x7f4e5815a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:10.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e5f000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e58156650 0x7f4e5813df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:10.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e5f000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4e5815bb60 0x7f4e581f4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:10.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e5f000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4e580720f0 con 0x7f4e58156650 2024-04-17T19:48:10.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e5f000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4e58158eb0 con 0x7f4e3c006f60 2024-04-17T19:48:10.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e5f000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4e58072270 con 0x7f4e5815bb60 2024-04-17T19:48:10.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e4fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e58156650 0x7f4e5813df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:10.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e4fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e58156650 0x7f4e5813df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52472/0 (socket says 172.21.15.29:52472) 2024-04-17T19:48:10.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e4fe00640 1 -- 172.21.15.29:0/2796230106 learned_addr learned my addr 172.21.15.29:0/2796230106 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:10.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e57e00640 1 --2- 172.21.15.29:0/2796230106 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4e3c006f60 0x7f4e5815a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:10.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e5d000640 1 --2- 172.21.15.29:0/2796230106 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4e5815bb60 0x7f4e581f4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:10.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e4fe00640 1 -- 172.21.15.29:0/2796230106 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4e5815bb60 msgr2=0x7f4e581f4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:10.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e4fe00640 1 --2- 172.21.15.29:0/2796230106 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4e5815bb60 0x7f4e581f4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:10.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e4fe00640 1 -- 172.21.15.29:0/2796230106 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4e3c006f60 msgr2=0x7f4e5815a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:10.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e4fe00640 1 --2- 172.21.15.29:0/2796230106 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4e3c006f60 0x7f4e5815a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:10.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e4fe00640 1 -- 172.21.15.29:0/2796230106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e58140540 con 0x7f4e58156650 2024-04-17T19:48:10.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e4fe00640 1 --2- 172.21.15.29:0/2796230106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e58156650 0x7f4e5813df50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4e40011900 tx=0x7f4e40011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:10.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e54c00640 1 -- 172.21.15.29:0/2796230106 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e40004220 con 0x7f4e58156650 2024-04-17T19:48:10.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e54c00640 1 -- 172.21.15.29:0/2796230106 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e400043c0 con 0x7f4e58156650 2024-04-17T19:48:10.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e54c00640 1 -- 172.21.15.29:0/2796230106 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4000fbd0 con 0x7f4e58156650 2024-04-17T19:48:10.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e5f000640 1 -- 172.21.15.29:0/2796230106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e5815a6f0 con 0x7f4e58156650 2024-04-17T19:48:10.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.925+0000 7f4e5f000640 1 -- 172.21.15.29:0/2796230106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e581550d0 con 0x7f4e58156650 2024-04-17T19:48:10.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.929+0000 7f4e5f000640 1 -- 172.21.15.29:0/2796230106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e58069b90 con 0x7f4e58156650 2024-04-17T19:48:10.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.929+0000 7f4e54c00640 1 -- 172.21.15.29:0/2796230106 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4e40003620 con 0x7f4e58156650 2024-04-17T19:48:10.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.929+0000 7f4e54c00640 1 --2- 172.21.15.29:0/2796230106 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4e34068380 0x7f4e3406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:10.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.929+0000 7f4e54c00640 1 -- 172.21.15.29:0/2796230106 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4e400878e0 con 0x7f4e58156650 2024-04-17T19:48:10.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.929+0000 7f4e57e00640 1 --2- 172.21.15.29:0/2796230106 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4e34068380 0x7f4e3406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:10.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.933+0000 7f4e57e00640 1 --2- 172.21.15.29:0/2796230106 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4e34068380 0x7f4e3406a890 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4e48000c10 tx=0x7f4e48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:10.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:10.933+0000 7f4e54c00640 1 -- 172.21.15.29:0/2796230106 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4e40019030 con 0x7f4e58156650 2024-04-17T19:48:11.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.137+0000 7f4e5f000640 1 -- 172.21.15.29:0/2796230106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f4e58159910 con 0x7f4e58156650 2024-04-17T19:48:11.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.737+0000 7f4e54c00640 1 -- 172.21.15.29:0/2796230106 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v11) ==== 122+0+0 (secure 0 0 0) 0x7f4e40057ac0 con 0x7f4e58156650 2024-04-17T19:48:11.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.741+0000 7f4e5f000640 1 -- 172.21.15.29:0/2796230106 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4e34068380 msgr2=0x7f4e3406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:11.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.741+0000 7f4e5f000640 1 --2- 172.21.15.29:0/2796230106 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4e34068380 0x7f4e3406a890 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4e48000c10 tx=0x7f4e48009040 comp rx=0 tx=0).stop 2024-04-17T19:48:11.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.741+0000 7f4e5f000640 1 -- 172.21.15.29:0/2796230106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e58156650 msgr2=0x7f4e5813df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:11.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.741+0000 7f4e5f000640 1 --2- 172.21.15.29:0/2796230106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e58156650 0x7f4e5813df50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4e40011900 tx=0x7f4e40011dc0 comp rx=0 tx=0).stop 2024-04-17T19:48:11.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.741+0000 7f4e5f000640 1 -- 172.21.15.29:0/2796230106 shutdown_connections 2024-04-17T19:48:11.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.741+0000 7f4e5f000640 1 --2- 172.21.15.29:0/2796230106 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4e34068380 0x7f4e3406a890 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:11.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.741+0000 7f4e5f000640 1 --2- 172.21.15.29:0/2796230106 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4e5815bb60 0x7f4e581f4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:11.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.741+0000 7f4e5f000640 1 --2- 172.21.15.29:0/2796230106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e58156650 0x7f4e5813df50 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:11.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.741+0000 7f4e5f000640 1 --2- 172.21.15.29:0/2796230106 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4e3c006f60 0x7f4e5815a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:11.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.741+0000 7f4e5f000640 1 -- 172.21.15.29:0/2796230106 >> 172.21.15.29:0/2796230106 conn(0x7f4e5806bf90 msgr2=0x7f4e5806c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:11.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.741+0000 7f4e5f000640 1 -- 172.21.15.29:0/2796230106 shutdown_connections 2024-04-17T19:48:11.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.741+0000 7f4e5f000640 1 -- 172.21.15.29:0/2796230106 wait complete. 2024-04-17T19:48:11.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-17T19:48:11.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.929+0000 7f4f94600640 1 Processor -- start 2024-04-17T19:48:11.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.929+0000 7f4f94600640 1 -- start start 2024-04-17T19:48:11.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.929+0000 7f4f94600640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f4f8c073630 0x7f4f8c154380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:11.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.929+0000 7f4f94600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f8c06cea0 con 0x7f4f8c0c78f0 2024-04-17T19:48:11.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.929+0000 7f4f94600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f8c06d180 con 0x7f4f8c0cca50 2024-04-17T19:48:11.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.929+0000 7f4f94600640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f4f8c15d610 con 0x7f4f8c073630 2024-04-17T19:48:11.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.929+0000 7f4f92600640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4f8c0c78f0 0x7f4f8c1649a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38056/0 (socket says 172.21.15.29:38056) 2024-04-17T19:48:11.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.929+0000 7f4f92600640 1 -- 172.21.15.29:0/2602347193 learned_addr learned my addr 172.21.15.29:0/2602347193 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:11.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.929+0000 7f4f91200640 1 --2- 172.21.15.29:0/2602347193 >> v2:172.21.15.37:3300/0 conn(0x7f4f8c073630 0x7f4f8c154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f8be00640 1 -- 172.21.15.29:0/2602347193 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2122759621 0 0) 0x7f4f8c06cea0 con 0x7f4f8c0c78f0 2024-04-17T19:48:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f8be00640 1 -- 172.21.15.29:0/2602347193 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f74003530 con 0x7f4f8c0c78f0 2024-04-17T19:48:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f8be00640 1 -- 172.21.15.29:0/2602347193 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1464951895 0 0) 0x7f4f74003530 con 0x7f4f8c0c78f0 2024-04-17T19:48:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f8be00640 1 -- 172.21.15.29:0/2602347193 >> v2:172.21.15.37:3300/0 conn(0x7f4f8c073630 msgr2=0x7f4f8c154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f8be00640 1 --2- 172.21.15.29:0/2602347193 >> v2:172.21.15.37:3300/0 conn(0x7f4f8c073630 0x7f4f8c154380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f8be00640 1 -- 172.21.15.29:0/2602347193 >> v1:172.21.15.37:6789/0 conn(0x7f4f8c0cca50 legacy=0x7f4f8c15a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f8be00640 1 -- 172.21.15.29:0/2602347193 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8c06d460 con 0x7f4f8c0c78f0 2024-04-17T19:48:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f8be00640 1 -- 172.21.15.29:0/2602347193 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f4f84001a00 con 0x7f4f8c0c78f0 2024-04-17T19:48:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f8be00640 1 -- 172.21.15.29:0/2602347193 >> v1:172.21.15.29:6789/0 conn(0x7f4f8c0c78f0 legacy=0x7f4f8c1649a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f8be00640 1 --2- 172.21.15.29:0/2602347193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f740039c0 0x7f4f74003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f8be00640 1 --2- 172.21.15.29:0/2602347193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f74004810 0x7f4f74004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f8be00640 1 --2- 172.21.15.29:0/2602347193 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f74005260 0x7f4f7400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f8be00640 1 -- 172.21.15.29:0/2602347193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7400c1c0 con 0x7f4f740039c0 2024-04-17T19:48:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f8be00640 1 -- 172.21.15.29:0/2602347193 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7400c370 con 0x7f4f74005260 2024-04-17T19:48:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f8be00640 1 -- 172.21.15.29:0/2602347193 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4f7400c570 con 0x7f4f74004810 2024-04-17T19:48:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f91c00640 1 --2- 172.21.15.29:0/2602347193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f740039c0 0x7f4f74003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f92600640 1 --2- 172.21.15.29:0/2602347193 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f74005260 0x7f4f7400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f91200640 1 --2- 172.21.15.29:0/2602347193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f74004810 0x7f4f74004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f91c00640 1 -- 172.21.15.29:0/2602347193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f74004810 msgr2=0x7f4f74004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f91c00640 1 --2- 172.21.15.29:0/2602347193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f74004810 0x7f4f74004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f91c00640 1 -- 172.21.15.29:0/2602347193 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f74005260 msgr2=0x7f4f7400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f91c00640 1 --2- 172.21.15.29:0/2602347193 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f74005260 0x7f4f7400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f91c00640 1 -- 172.21.15.29:0/2602347193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f74003530 con 0x7f4f740039c0 2024-04-17T19:48:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.933+0000 7f4f92600640 1 -- 172.21.15.29:0/2602347193 reap_dead start 2024-04-17T19:48:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.937+0000 7f4f91c00640 1 --2- 172.21.15.29:0/2602347193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f740039c0 0x7f4f74003e30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4f80001780 tx=0x7f4f8001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398cd1e3321cd498 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.937+0000 7f4f8be00640 1 -- 172.21.15.29:0/2602347193 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f800013a0 con 0x7f4f740039c0 2024-04-17T19:48:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.937+0000 7f4f8be00640 1 -- 172.21.15.29:0/2602347193 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f800052f0 con 0x7f4f740039c0 2024-04-17T19:48:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.937+0000 7f4f8be00640 1 -- 172.21.15.29:0/2602347193 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f80003840 con 0x7f4f740039c0 2024-04-17T19:48:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.937+0000 7f4f94600640 1 -- 172.21.15.29:0/2602347193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f740039c0 msgr2=0x7f4f74003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.937+0000 7f4f94600640 1 --2- 172.21.15.29:0/2602347193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f740039c0 0x7f4f74003e30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4f80001780 tx=0x7f4f8001c400 comp rx=0 tx=0).stop 2024-04-17T19:48:11.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.937+0000 7f4f94600640 1 -- 172.21.15.29:0/2602347193 shutdown_connections 2024-04-17T19:48:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.937+0000 7f4f94600640 1 --2- 172.21.15.29:0/2602347193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f740039c0 0x7f4f74003e30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.937+0000 7f4f94600640 1 -- 172.21.15.29:0/2602347193 >> 172.21.15.29:0/2602347193 conn(0x7f4f8c0cace0 msgr2=0x7f4f8c0cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.937+0000 7f4f94600640 1 -- 172.21.15.29:0/2602347193 shutdown_connections 2024-04-17T19:48:11.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.937+0000 7f4f94600640 1 -- 172.21.15.29:0/2602347193 wait complete. 2024-04-17T19:48:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.937+0000 7f4f94600640 1 Processor -- start 2024-04-17T19:48:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.937+0000 7f4f94600640 1 -- start start 2024-04-17T19:48:11.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f94600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f740039c0 0x7f4f8c156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:11.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f94600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f8c15d0d0 0x7f4f8c155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:11.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f94600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f8c155a70 0x7f4f8c0c56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:11.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f94600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8c15a2c0 con 0x7f4f8c155a70 2024-04-17T19:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f94600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8c15af60 con 0x7f4f8c15d0d0 2024-04-17T19:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f94600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4f8c067ce0 con 0x7f4f740039c0 2024-04-17T19:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f92600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f8c155a70 0x7f4f8c0c56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f92600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f8c155a70 0x7f4f8c0c56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52482/0 (socket says 172.21.15.29:52482) 2024-04-17T19:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f92600640 1 -- 172.21.15.29:0/2020498784 learned_addr learned my addr 172.21.15.29:0/2020498784 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f91c00640 1 --2- 172.21.15.29:0/2020498784 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f740039c0 0x7f4f8c156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f91200640 1 --2- 172.21.15.29:0/2020498784 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f8c15d0d0 0x7f4f8c155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f92600640 1 -- 172.21.15.29:0/2020498784 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f740039c0 msgr2=0x7f4f8c156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f92600640 1 --2- 172.21.15.29:0/2020498784 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f740039c0 0x7f4f8c156410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f92600640 1 -- 172.21.15.29:0/2020498784 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f8c15d0d0 msgr2=0x7f4f8c155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f92600640 1 --2- 172.21.15.29:0/2020498784 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f8c15d0d0 0x7f4f8c155530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f92600640 1 -- 172.21.15.29:0/2020498784 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8c157c50 con 0x7f4f8c155a70 2024-04-17T19:48:11.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f92600640 1 --2- 172.21.15.29:0/2020498784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f8c155a70 0x7f4f8c0c56e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c06c4e0 tx=0x7f4f84007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:11.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f89600640 1 -- 172.21.15.29:0/2020498784 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8400ac50 con 0x7f4f8c155a70 2024-04-17T19:48:11.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f89600640 1 -- 172.21.15.29:0/2020498784 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f84018070 con 0x7f4f8c155a70 2024-04-17T19:48:11.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f89600640 1 -- 172.21.15.29:0/2020498784 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f840133f0 con 0x7f4f8c155a70 2024-04-17T19:48:11.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f94600640 1 -- 172.21.15.29:0/2020498784 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f8c0c1990 con 0x7f4f8c155a70 2024-04-17T19:48:11.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.941+0000 7f4f94600640 1 -- 172.21.15.29:0/2020498784 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f8c0c3ad0 con 0x7f4f8c155a70 2024-04-17T19:48:11.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.945+0000 7f4f94600640 1 -- 172.21.15.29:0/2020498784 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f8c154cf0 con 0x7f4f8c155a70 2024-04-17T19:48:11.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.945+0000 7f4f89600640 1 -- 172.21.15.29:0/2020498784 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4f84013590 con 0x7f4f8c155a70 2024-04-17T19:48:11.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.945+0000 7f4f89600640 1 --2- 172.21.15.29:0/2020498784 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4f6c068380 0x7f4f6c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:11.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.945+0000 7f4f89600640 1 -- 172.21.15.29:0/2020498784 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4f84087200 con 0x7f4f8c155a70 2024-04-17T19:48:11.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.945+0000 7f4f91c00640 1 --2- 172.21.15.29:0/2020498784 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4f6c068380 0x7f4f6c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:11.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.949+0000 7f4f91c00640 1 --2- 172.21.15.29:0/2020498784 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4f6c068380 0x7f4f6c06a890 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4f80001750 tx=0x7f4f800011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:11.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:11.949+0000 7f4f89600640 1 -- 172.21.15.29:0/2020498784 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4f84056e20 con 0x7f4f8c155a70 2024-04-17T19:48:12.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.149+0000 7f4f94600640 1 -- 172.21.15.29:0/2020498784 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f4f8c154a50 con 0x7f4f8c155a70 2024-04-17T19:48:12.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.745+0000 7f4f89600640 1 -- 172.21.15.29:0/2020498784 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v12) ==== 115+0+0 (secure 0 0 0) 0x7f4f8405b120 con 0x7f4f8c155a70 2024-04-17T19:48:12.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.749+0000 7f4f94600640 1 -- 172.21.15.29:0/2020498784 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4f6c068380 msgr2=0x7f4f6c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:12.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.749+0000 7f4f94600640 1 --2- 172.21.15.29:0/2020498784 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4f6c068380 0x7f4f6c06a890 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4f80001750 tx=0x7f4f800011a0 comp rx=0 tx=0).stop 2024-04-17T19:48:12.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.749+0000 7f4f94600640 1 -- 172.21.15.29:0/2020498784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f8c155a70 msgr2=0x7f4f8c0c56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:12.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.749+0000 7f4f94600640 1 --2- 172.21.15.29:0/2020498784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f8c155a70 0x7f4f8c0c56e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c06c4e0 tx=0x7f4f84007bd0 comp rx=0 tx=0).stop 2024-04-17T19:48:12.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.749+0000 7f4f94600640 1 -- 172.21.15.29:0/2020498784 shutdown_connections 2024-04-17T19:48:12.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.749+0000 7f4f94600640 1 --2- 172.21.15.29:0/2020498784 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4f6c068380 0x7f4f6c06a890 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:12.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.749+0000 7f4f94600640 1 --2- 172.21.15.29:0/2020498784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f8c155a70 0x7f4f8c0c56e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:12.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.749+0000 7f4f94600640 1 --2- 172.21.15.29:0/2020498784 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f8c15d0d0 0x7f4f8c155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:12.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.749+0000 7f4f94600640 1 --2- 172.21.15.29:0/2020498784 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f740039c0 0x7f4f8c156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:12.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.749+0000 7f4f94600640 1 -- 172.21.15.29:0/2020498784 >> 172.21.15.29:0/2020498784 conn(0x7f4f8c0cace0 msgr2=0x7f4f8c159a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:12.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.749+0000 7f4f94600640 1 -- 172.21.15.29:0/2020498784 shutdown_connections 2024-04-17T19:48:12.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.749+0000 7f4f94600640 1 -- 172.21.15.29:0/2020498784 wait complete. 2024-04-17T19:48:12.761 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T19:48:12.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:48:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.929+0000 7f6920800640 1 Processor -- start 2024-04-17T19:48:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.929+0000 7f6920800640 1 -- start start 2024-04-17T19:48:12.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.929+0000 7f6920800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69180cfde0 0x7f6918071940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:12.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.929+0000 7f6920800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6918071e80 0x7f69181647e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.929+0000 7f6920800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69180c4930 con 0x7f6918159e60 2024-04-17T19:48:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.929+0000 7f6920800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f69180cf620 con 0x7f6918071e80 2024-04-17T19:48:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.929+0000 7f6920800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f69180c5190 con 0x7f69180cfde0 2024-04-17T19:48:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.929+0000 7f691d400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69180cfde0 0x7f6918071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.929+0000 7f691e800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6918071e80 0x7f69181647e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.929+0000 7f691d400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69180cfde0 0x7f6918071940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36090/0 (socket says 172.21.15.29:36090) 2024-04-17T19:48:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.929+0000 7f691de00640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f6918159e60 0x7f6918153d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:49748/0 (socket says 172.21.15.29:49748) 2024-04-17T19:48:12.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.929+0000 7f691d400640 1 -- 172.21.15.29:0/1379762420 learned_addr learned my addr 172.21.15.29:0/1379762420 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:12.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.933+0000 7f691d400640 1 -- 172.21.15.29:0/1379762420 >> v2:172.21.15.37:3300/0 conn(0x7f6918071e80 msgr2=0x7f69181647e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:12.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.933+0000 7f691d400640 1 --2- 172.21.15.29:0/1379762420 >> v2:172.21.15.37:3300/0 conn(0x7f6918071e80 0x7f69181647e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:12.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.933+0000 7f691d400640 1 -- 172.21.15.29:0/1379762420 >> v1:172.21.15.37:6789/0 conn(0x7f6918159e60 legacy=0x7f6918153d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:12.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.933+0000 7f691d400640 1 -- 172.21.15.29:0/1379762420 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69180c4bd0 con 0x7f69180cfde0 2024-04-17T19:48:12.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.933+0000 7f691e800640 1 --2- 172.21.15.29:0/1379762420 >> v2:172.21.15.37:3300/0 conn(0x7f6918071e80 0x7f69181647e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:48:12.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.933+0000 7f691d400640 1 --2- 172.21.15.29:0/1379762420 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69180cfde0 0x7f6918071940 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6908009870 tx=0x7f690801c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cd91768cdbcd4c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:12.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.933+0000 7f691ca00640 1 -- 172.21.15.29:0/1379762420 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69080b2070 con 0x7f69180cfde0 2024-04-17T19:48:12.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.937+0000 7f691ca00640 1 -- 172.21.15.29:0/1379762420 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69080a9070 con 0x7f69180cfde0 2024-04-17T19:48:12.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.937+0000 7f691ca00640 1 -- 172.21.15.29:0/1379762420 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69080ad880 con 0x7f69180cfde0 2024-04-17T19:48:12.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.937+0000 7f6920800640 1 -- 172.21.15.29:0/1379762420 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69180cfde0 msgr2=0x7f6918071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:12.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.937+0000 7f6920800640 1 --2- 172.21.15.29:0/1379762420 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69180cfde0 0x7f6918071940 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6908009870 tx=0x7f690801c560 comp rx=0 tx=0).stop 2024-04-17T19:48:12.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.937+0000 7f6920800640 1 -- 172.21.15.29:0/1379762420 shutdown_connections 2024-04-17T19:48:12.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.937+0000 7f6920800640 1 --2- 172.21.15.29:0/1379762420 >> v2:172.21.15.37:3300/0 conn(0x7f6918071e80 0x7f69181647e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:12.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.937+0000 7f6920800640 1 --2- 172.21.15.29:0/1379762420 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69180cfde0 0x7f6918071940 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:12.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.937+0000 7f6920800640 1 -- 172.21.15.29:0/1379762420 >> 172.21.15.29:0/1379762420 conn(0x7f6918067eb0 msgr2=0x7f6918070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:12.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.937+0000 7f6920800640 1 -- 172.21.15.29:0/1379762420 shutdown_connections 2024-04-17T19:48:12.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.937+0000 7f6920800640 1 -- 172.21.15.29:0/1379762420 wait complete. 2024-04-17T19:48:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.937+0000 7f6920800640 1 Processor -- start 2024-04-17T19:48:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.937+0000 7f6920800640 1 -- start start 2024-04-17T19:48:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f6920800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6918071e80 0x7f69181557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f6920800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69180cfde0 0x7f6918156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f6920800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6918159e60 0x7f69180c8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f6920800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f69180cbd10 con 0x7f6918071e80 2024-04-17T19:48:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f6920800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f69180cbb90 con 0x7f6918159e60 2024-04-17T19:48:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f6920800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f69180c4510 con 0x7f69180cfde0 2024-04-17T19:48:12.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f691de00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6918071e80 0x7f69181557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f691de00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6918071e80 0x7f69181557d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52498/0 (socket says 172.21.15.29:52498) 2024-04-17T19:48:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f691de00640 1 -- 172.21.15.29:0/2873814696 learned_addr learned my addr 172.21.15.29:0/2873814696 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f691d400640 1 --2- 172.21.15.29:0/2873814696 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69180cfde0 0x7f6918156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f691e800640 1 --2- 172.21.15.29:0/2873814696 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6918159e60 0x7f69180c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f691de00640 1 -- 172.21.15.29:0/2873814696 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69180cfde0 msgr2=0x7f6918156d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f691de00640 1 --2- 172.21.15.29:0/2873814696 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69180cfde0 0x7f6918156d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f691de00640 1 -- 172.21.15.29:0/2873814696 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6918159e60 msgr2=0x7f69180c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f691de00640 1 --2- 172.21.15.29:0/2873814696 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6918159e60 0x7f69180c8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:12.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f691de00640 1 -- 172.21.15.29:0/2873814696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6918155d10 con 0x7f6918071e80 2024-04-17T19:48:12.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f691de00640 1 --2- 172.21.15.29:0/2873814696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6918071e80 0x7f69181557d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f690c00c3a0 tx=0x7f690c00c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:12.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f6912000640 1 -- 172.21.15.29:0/2873814696 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690c0074e0 con 0x7f6918071e80 2024-04-17T19:48:12.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f6912000640 1 -- 172.21.15.29:0/2873814696 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f690c0076a0 con 0x7f6918071e80 2024-04-17T19:48:12.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f6912000640 1 -- 172.21.15.29:0/2873814696 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690c004a30 con 0x7f6918071e80 2024-04-17T19:48:12.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f6920800640 1 -- 172.21.15.29:0/2873814696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69181572c0 con 0x7f6918071e80 2024-04-17T19:48:12.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f6920800640 1 -- 172.21.15.29:0/2873814696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6918154b60 con 0x7f6918071e80 2024-04-17T19:48:12.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.941+0000 7f6920800640 1 -- 172.21.15.29:0/2873814696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6918069b90 con 0x7f6918071e80 2024-04-17T19:48:12.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.945+0000 7f6912000640 1 -- 172.21.15.29:0/2873814696 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f690c01a020 con 0x7f6918071e80 2024-04-17T19:48:12.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.945+0000 7f6912000640 1 --2- 172.21.15.29:0/2873814696 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f68fc068380 0x7f68fc06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:12.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.945+0000 7f6912000640 1 -- 172.21.15.29:0/2873814696 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f690c0863b0 con 0x7f6918071e80 2024-04-17T19:48:12.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.945+0000 7f691d400640 1 --2- 172.21.15.29:0/2873814696 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f68fc068380 0x7f68fc06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:12.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.949+0000 7f691d400640 1 --2- 172.21.15.29:0/2873814696 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f68fc068380 0x7f68fc06a890 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f69080af000 tx=0x7f6908008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:12.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:12.949+0000 7f6912000640 1 -- 172.21.15.29:0/2873814696 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f690c0565a0 con 0x7f6918071e80 2024-04-17T19:48:13.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.153+0000 7f6920800640 1 -- 172.21.15.29:0/2873814696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69180cce20 con 0x7f6918071e80 2024-04-17T19:48:13.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.153+0000 7f6912000640 1 -- 172.21.15.29:0/2873814696 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+6836 (secure 0 0 0) 0x7f690c055ea0 con 0x7f6918071e80 2024-04-17T19:48:13.141 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:13.141 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4226,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6838/332907250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":332907250},{"type":"v1","addr":"172.21.15.37:6838","nonce":332907250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:48:12.151320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1],"up":{"mds_0":4305,"mds_1":4302},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:creating","state_seq":1,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305]},"id":1}]} 2024-04-17T19:48:13.142 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 12 2024-04-17T19:48:13.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.153+0000 7f6920800640 1 -- 172.21.15.29:0/2873814696 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f68fc068380 msgr2=0x7f68fc06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.153+0000 7f6920800640 1 --2- 172.21.15.29:0/2873814696 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f68fc068380 0x7f68fc06a890 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f69080af000 tx=0x7f6908008040 comp rx=0 tx=0).stop 2024-04-17T19:48:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.153+0000 7f6920800640 1 -- 172.21.15.29:0/2873814696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6918071e80 msgr2=0x7f69181557d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.153+0000 7f6920800640 1 --2- 172.21.15.29:0/2873814696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6918071e80 0x7f69181557d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f690c00c3a0 tx=0x7f690c00c7a0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.153+0000 7f6920800640 1 -- 172.21.15.29:0/2873814696 shutdown_connections 2024-04-17T19:48:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.153+0000 7f6920800640 1 --2- 172.21.15.29:0/2873814696 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f68fc068380 0x7f68fc06a890 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.153+0000 7f6920800640 1 --2- 172.21.15.29:0/2873814696 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6918159e60 0x7f69180c8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.153+0000 7f6920800640 1 --2- 172.21.15.29:0/2873814696 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69180cfde0 0x7f6918156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.153+0000 7f6920800640 1 --2- 172.21.15.29:0/2873814696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6918071e80 0x7f69181557d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.153+0000 7f6920800640 1 -- 172.21.15.29:0/2873814696 >> 172.21.15.29:0/2873814696 conn(0x7f6918067eb0 msgr2=0x7f6918158a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.153+0000 7f6920800640 1 -- 172.21.15.29:0/2873814696 shutdown_connections 2024-04-17T19:48:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.153+0000 7f6920800640 1 -- 172.21.15.29:0/2873814696 wait complete. 2024-04-17T19:48:13.163 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:48:13.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.325+0000 7ffbea400640 1 Processor -- start 2024-04-17T19:48:13.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.325+0000 7ffbea400640 1 -- start start 2024-04-17T19:48:13.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.329+0000 7ffbea400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ffbe40c6f70 0x7ffbe40c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:13.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.329+0000 7ffbea400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbe406c660 0x7ffbe406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:13.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.329+0000 7ffbea400640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbe415a0a0 con 0x7ffbe4159c90 2024-04-17T19:48:13.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.329+0000 7ffbea400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ffbe415abe0 con 0x7ffbe40c6f70 2024-04-17T19:48:13.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.329+0000 7ffbea400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ffbe415a340 con 0x7ffbe406c660 2024-04-17T19:48:13.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.329+0000 7ffbe3e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbe406c660 0x7ffbe406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:13.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.329+0000 7ffbdaa00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ffbe40c6f70 0x7ffbe40c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:13.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.329+0000 7ffbe3400640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7ffbe4159c90 0x7ffbe40ca2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:49762/0 (socket says 172.21.15.29:49762) 2024-04-17T19:48:13.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.329+0000 7ffbdaa00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ffbe40c6f70 0x7ffbe40c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46550/0 (socket says 172.21.15.29:46550) 2024-04-17T19:48:13.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.329+0000 7ffbe3400640 1 -- 172.21.15.29:0/2794017083 learned_addr learned my addr 172.21.15.29:0/2794017083 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:13.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.329+0000 7ffbe3e00640 1 -- 172.21.15.29:0/2794017083 >> v2:172.21.15.37:3300/0 conn(0x7ffbe40c6f70 msgr2=0x7ffbe40c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:13.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.329+0000 7ffbe3e00640 1 --2- 172.21.15.29:0/2794017083 >> v2:172.21.15.37:3300/0 conn(0x7ffbe40c6f70 0x7ffbe40c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.329+0000 7ffbe3e00640 1 -- 172.21.15.29:0/2794017083 >> v1:172.21.15.37:6789/0 conn(0x7ffbe4159c90 legacy=0x7ffbe40ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:13.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.329+0000 7ffbe3e00640 1 -- 172.21.15.29:0/2794017083 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbe415a620 con 0x7ffbe406c660 2024-04-17T19:48:13.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.333+0000 7ffbe3e00640 1 --2- 172.21.15.29:0/2794017083 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbe406c660 0x7ffbe406ca70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc009d90 tx=0x7ffbdc01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8141b63dfd6d081 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:13.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.333+0000 7ffbe2a00640 1 -- 172.21.15.29:0/2794017083 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc0b2070 con 0x7ffbe406c660 2024-04-17T19:48:13.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.333+0000 7ffbe2a00640 1 -- 172.21.15.29:0/2794017083 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbdc0a5830 con 0x7ffbe406c660 2024-04-17T19:48:13.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.333+0000 7ffbe2a00640 1 -- 172.21.15.29:0/2794017083 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc0a5c00 con 0x7ffbe406c660 2024-04-17T19:48:13.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.333+0000 7ffbea400640 1 -- 172.21.15.29:0/2794017083 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbe406c660 msgr2=0x7ffbe406ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:13.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.333+0000 7ffbea400640 1 --2- 172.21.15.29:0/2794017083 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbe406c660 0x7ffbe406ca70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc009d90 tx=0x7ffbdc01c970 comp rx=0 tx=0).stop 2024-04-17T19:48:13.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.333+0000 7ffbea400640 1 -- 172.21.15.29:0/2794017083 shutdown_connections 2024-04-17T19:48:13.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.333+0000 7ffbea400640 1 --2- 172.21.15.29:0/2794017083 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbe406c660 0x7ffbe406ca70 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.333+0000 7ffbea400640 1 --2- 172.21.15.29:0/2794017083 >> v2:172.21.15.37:3300/0 conn(0x7ffbe40c6f70 0x7ffbe40c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.333+0000 7ffbea400640 1 -- 172.21.15.29:0/2794017083 >> 172.21.15.29:0/2794017083 conn(0x7ffbe406fde0 msgr2=0x7ffbe40701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:13.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.333+0000 7ffbea400640 1 -- 172.21.15.29:0/2794017083 shutdown_connections 2024-04-17T19:48:13.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.333+0000 7ffbea400640 1 -- 172.21.15.29:0/2794017083 wait complete. 2024-04-17T19:48:13.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbea400640 1 Processor -- start 2024-04-17T19:48:13.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbea400640 1 -- start start 2024-04-17T19:48:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbea400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbe406c660 0x7ffbe40739f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbea400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbe40c6f70 0x7ffbe4151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbea400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffbe4159c90 0x7ffbe4146f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbea400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe40ca9c0 con 0x7ffbe40c6f70 2024-04-17T19:48:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbea400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ffbe406cfb0 con 0x7ffbe4159c90 2024-04-17T19:48:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbea400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ffbe40cab40 con 0x7ffbe406c660 2024-04-17T19:48:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbdaa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbe40c6f70 0x7ffbe4151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbdaa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbe40c6f70 0x7ffbe4151b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52504/0 (socket says 172.21.15.29:52504) 2024-04-17T19:48:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbdaa00640 1 -- 172.21.15.29:0/1947731923 learned_addr learned my addr 172.21.15.29:0/1947731923 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:13.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbe3400640 1 --2- 172.21.15.29:0/1947731923 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbe406c660 0x7ffbe40739f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbe3e00640 1 --2- 172.21.15.29:0/1947731923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffbe4159c90 0x7ffbe4146f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbdaa00640 1 -- 172.21.15.29:0/1947731923 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbe406c660 msgr2=0x7ffbe40739f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbdaa00640 1 --2- 172.21.15.29:0/1947731923 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbe406c660 0x7ffbe40739f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbdaa00640 1 -- 172.21.15.29:0/1947731923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffbe4159c90 msgr2=0x7ffbe4146f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbdaa00640 1 --2- 172.21.15.29:0/1947731923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffbe4159c90 0x7ffbe4146f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbdaa00640 1 -- 172.21.15.29:0/1947731923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbe41520d0 con 0x7ffbe40c6f70 2024-04-17T19:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbe3400640 1 --2- 172.21.15.29:0/1947731923 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbe406c660 0x7ffbe40739f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbe3e00640 1 --2- 172.21.15.29:0/1947731923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffbe4159c90 0x7ffbe4146f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbdaa00640 1 --2- 172.21.15.29:0/1947731923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbe40c6f70 0x7ffbe4151b90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc00e870 tx=0x7ffbcc00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbdbe00640 1 -- 172.21.15.29:0/1947731923 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc017070 con 0x7ffbe40c6f70 2024-04-17T19:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbdbe00640 1 -- 172.21.15.29:0/1947731923 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbcc00c940 con 0x7ffbe40c6f70 2024-04-17T19:48:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbea400640 1 -- 172.21.15.29:0/1947731923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbe41f8f40 con 0x7ffbe40c6f70 2024-04-17T19:48:13.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbdbe00640 1 -- 172.21.15.29:0/1947731923 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbcc00cd30 con 0x7ffbe40c6f70 2024-04-17T19:48:13.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.337+0000 7ffbea400640 1 -- 172.21.15.29:0/1947731923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbe41f9500 con 0x7ffbe40c6f70 2024-04-17T19:48:13.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.341+0000 7ffbea400640 1 -- 172.21.15.29:0/1947731923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbe41f8c90 con 0x7ffbe40c6f70 2024-04-17T19:48:13.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.341+0000 7ffbdbe00640 1 -- 172.21.15.29:0/1947731923 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ffbcc01a070 con 0x7ffbe40c6f70 2024-04-17T19:48:13.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.341+0000 7ffbdbe00640 1 --2- 172.21.15.29:0/1947731923 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ffbc00683b0 0x7ffbc006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:13.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.341+0000 7ffbdbe00640 1 -- 172.21.15.29:0/1947731923 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ffbcc087080 con 0x7ffbe40c6f70 2024-04-17T19:48:13.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.341+0000 7ffbe3400640 1 --2- 172.21.15.29:0/1947731923 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ffbc00683b0 0x7ffbc006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:13.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.345+0000 7ffbe3400640 1 --2- 172.21.15.29:0/1947731923 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ffbc00683b0 0x7ffbc006a8c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4000f70 tx=0x7ffbd4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:13.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.345+0000 7ffbdbe00640 1 -- 172.21.15.29:0/1947731923 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ffbcc0571d0 con 0x7ffbe40c6f70 2024-04-17T19:48:13.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.513+0000 7ffbea400640 1 -- 172.21.15.29:0/1947731923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffbe4069b90 con 0x7ffbe40c6f70 2024-04-17T19:48:13.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.513+0000 7ffbdbe00640 1 -- 172.21.15.29:0/1947731923 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18394 (secure 0 0 0) 0x7ffbcc056ad0 con 0x7ffbe40c6f70 2024-04-17T19:48:13.502 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:13.502 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:47:54.321794+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T19:47:47.658449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T19:47:48.209047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T19:47:51.887909+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:48:13.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.517+0000 7ffbea400640 1 -- 172.21.15.29:0/1947731923 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ffbc00683b0 msgr2=0x7ffbc006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:13.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.517+0000 7ffbea400640 1 --2- 172.21.15.29:0/1947731923 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ffbc00683b0 0x7ffbc006a8c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4000f70 tx=0x7ffbd4009040 comp rx=0 tx=0).stop 2024-04-17T19:48:13.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.517+0000 7ffbea400640 1 -- 172.21.15.29:0/1947731923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbe40c6f70 msgr2=0x7ffbe4151b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:13.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.517+0000 7ffbea400640 1 --2- 172.21.15.29:0/1947731923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbe40c6f70 0x7ffbe4151b90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc00e870 tx=0x7ffbcc00ec70 comp rx=0 tx=0).stop 2024-04-17T19:48:13.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.517+0000 7ffbea400640 1 -- 172.21.15.29:0/1947731923 shutdown_connections 2024-04-17T19:48:13.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.517+0000 7ffbea400640 1 --2- 172.21.15.29:0/1947731923 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ffbc00683b0 0x7ffbc006a8c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.517+0000 7ffbea400640 1 --2- 172.21.15.29:0/1947731923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffbe4159c90 0x7ffbe4146f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.517+0000 7ffbea400640 1 --2- 172.21.15.29:0/1947731923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbe40c6f70 0x7ffbe4151b90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.517+0000 7ffbea400640 1 --2- 172.21.15.29:0/1947731923 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffbe406c660 0x7ffbe40739f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.517+0000 7ffbea400640 1 -- 172.21.15.29:0/1947731923 >> 172.21.15.29:0/1947731923 conn(0x7ffbe406fde0 msgr2=0x7ffbe4148ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:13.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.517+0000 7ffbea400640 1 -- 172.21.15.29:0/1947731923 shutdown_connections 2024-04-17T19:48:13.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.517+0000 7ffbea400640 1 -- 172.21.15.29:0/1947731923 wait complete. 2024-04-17T19:48:13.524 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T19:48:13.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T19:48:13.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.685+0000 7fa9ec800640 1 Processor -- start 2024-04-17T19:48:13.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.685+0000 7fa9ec800640 1 -- start start 2024-04-17T19:48:13.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9ec800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa9e41625e0 0x7fa9e4164a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:13.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9ec800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9e40cd0e0 con 0x7fa9e40cc8c0 2024-04-17T19:48:13.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9ec800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9e40c88b0 con 0x7fa9e40cccd0 2024-04-17T19:48:13.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9ec800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa9e40c6ce0 con 0x7fa9e41625e0 2024-04-17T19:48:13.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e9e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa9e40cc8c0 0x7fa9e4159390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38058/0 (socket says 172.21.15.29:38058) 2024-04-17T19:48:13.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e9e00640 1 -- 172.21.15.29:0/3039724912 learned_addr learned my addr 172.21.15.29:0/3039724912 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:13.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9ea800640 1 --2- 172.21.15.29:0/3039724912 >> v2:172.21.15.37:3300/0 conn(0x7fa9e41625e0 0x7fa9e4164a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e8a00640 1 -- 172.21.15.29:0/3039724912 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 262040 0 0) 0x7fa9e40cd0e0 con 0x7fa9e40cc8c0 2024-04-17T19:48:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e8a00640 1 -- 172.21.15.29:0/3039724912 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9cc005740 con 0x7fa9e40cc8c0 2024-04-17T19:48:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e8a00640 1 -- 172.21.15.29:0/3039724912 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1849825428 0 0) 0x7fa9cc005740 con 0x7fa9e40cc8c0 2024-04-17T19:48:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e8a00640 1 -- 172.21.15.29:0/3039724912 >> v2:172.21.15.37:3300/0 conn(0x7fa9e41625e0 msgr2=0x7fa9e4164a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e8a00640 1 --2- 172.21.15.29:0/3039724912 >> v2:172.21.15.37:3300/0 conn(0x7fa9e41625e0 0x7fa9e4164a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e8a00640 1 -- 172.21.15.29:0/3039724912 >> v1:172.21.15.37:6789/0 conn(0x7fa9e40cccd0 legacy=0x7fa9e40ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e8a00640 1 -- 172.21.15.29:0/3039724912 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e40c6fc0 con 0x7fa9e40cc8c0 2024-04-17T19:48:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e8a00640 1 -- 172.21.15.29:0/3039724912 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fa9d8001680 con 0x7fa9e40cc8c0 2024-04-17T19:48:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e8a00640 1 -- 172.21.15.29:0/3039724912 >> v1:172.21.15.29:6789/0 conn(0x7fa9e40cc8c0 legacy=0x7fa9e4159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e8a00640 1 --2- 172.21.15.29:0/3039724912 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9cc005c40 0x7fa9cc0060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e8a00640 1 --2- 172.21.15.29:0/3039724912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9cc006a20 0x7fa9cc006e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e8a00640 1 --2- 172.21.15.29:0/3039724912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9cc007440 0x7fa9cc00dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e8a00640 1 -- 172.21.15.29:0/3039724912 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e40c6ce0 con 0x7fa9cc007440 2024-04-17T19:48:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e8a00640 1 -- 172.21.15.29:0/3039724912 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa9cc00e3d0 con 0x7fa9cc006a20 2024-04-17T19:48:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e8a00640 1 -- 172.21.15.29:0/3039724912 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa9cc00e5d0 con 0x7fa9cc005c40 2024-04-17T19:48:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9ea800640 1 --2- 172.21.15.29:0/3039724912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9cc007440 0x7fa9cc00dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e9e00640 1 --2- 172.21.15.29:0/3039724912 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9cc005c40 0x7fa9cc0060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9e9400640 1 --2- 172.21.15.29:0/3039724912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9cc006a20 0x7fa9cc006e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:13.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9ea800640 1 -- 172.21.15.29:0/3039724912 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9cc005c40 msgr2=0x7fa9cc0060b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:13.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9ea800640 1 --2- 172.21.15.29:0/3039724912 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9cc005c40 0x7fa9cc0060b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9ea800640 1 -- 172.21.15.29:0/3039724912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9cc006a20 msgr2=0x7fa9cc006e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:13.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9ea800640 1 --2- 172.21.15.29:0/3039724912 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9cc006a20 0x7fa9cc006e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.689+0000 7fa9ea800640 1 -- 172.21.15.29:0/3039724912 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9cc005740 con 0x7fa9cc007440 2024-04-17T19:48:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.693+0000 7fa9ea800640 1 -- 172.21.15.29:0/3039724912 reap_dead start 2024-04-17T19:48:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.693+0000 7fa9ea800640 1 --2- 172.21.15.29:0/3039724912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9cc007440 0x7fa9cc00dbe0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0009f90 tx=0x7fa9e001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa8fcd6c03d9ae0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.693+0000 7fa9e8a00640 1 -- 172.21.15.29:0/3039724912 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e00b2070 con 0x7fa9cc007440 2024-04-17T19:48:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.693+0000 7fa9e8a00640 1 -- 172.21.15.29:0/3039724912 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9e00a5c90 con 0x7fa9cc007440 2024-04-17T19:48:13.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.693+0000 7fa9e8a00640 1 -- 172.21.15.29:0/3039724912 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e0009c10 con 0x7fa9cc007440 2024-04-17T19:48:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.693+0000 7fa9ec800640 1 -- 172.21.15.29:0/3039724912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9cc007440 msgr2=0x7fa9cc00dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.693+0000 7fa9ec800640 1 --2- 172.21.15.29:0/3039724912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9cc007440 0x7fa9cc00dbe0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa9e0009f90 tx=0x7fa9e001cc80 comp rx=0 tx=0).stop 2024-04-17T19:48:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.693+0000 7fa9ec800640 1 -- 172.21.15.29:0/3039724912 shutdown_connections 2024-04-17T19:48:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.693+0000 7fa9ec800640 1 --2- 172.21.15.29:0/3039724912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9cc007440 0x7fa9cc00dbe0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.693+0000 7fa9ec800640 1 -- 172.21.15.29:0/3039724912 >> 172.21.15.29:0/3039724912 conn(0x7fa9e406fde0 msgr2=0x7fa9e40701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.693+0000 7fa9ec800640 1 -- 172.21.15.29:0/3039724912 shutdown_connections 2024-04-17T19:48:13.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.693+0000 7fa9ec800640 1 -- 172.21.15.29:0/3039724912 wait complete. 2024-04-17T19:48:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9ec800640 1 Processor -- start 2024-04-17T19:48:13.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9ec800640 1 -- start start 2024-04-17T19:48:13.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9ec800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9cc007440 0x7fa9e4073450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:13.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9ec800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9e4154d40 0x7fa9e41f1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:13.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9ec800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9e40c8d40 0x7fa9e4072320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:13.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9ec800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e4158e70 con 0x7fa9cc007440 2024-04-17T19:48:13.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9ec800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e40c7580 con 0x7fa9e40c8d40 2024-04-17T19:48:13.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9ec800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa9e40c6880 con 0x7fa9e4154d40 2024-04-17T19:48:13.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9e9e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9cc007440 0x7fa9e4073450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:13.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9e9e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9cc007440 0x7fa9e4073450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52526/0 (socket says 172.21.15.29:52526) 2024-04-17T19:48:13.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9e9e00640 1 -- 172.21.15.29:0/2013382987 learned_addr learned my addr 172.21.15.29:0/2013382987 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:13.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9e9400640 1 --2- 172.21.15.29:0/2013382987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9e4154d40 0x7fa9e41f1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:13.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9ea800640 1 --2- 172.21.15.29:0/2013382987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9e40c8d40 0x7fa9e4072320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:13.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9e9e00640 1 -- 172.21.15.29:0/2013382987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9e4154d40 msgr2=0x7fa9e41f1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:13.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9e9e00640 1 --2- 172.21.15.29:0/2013382987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9e4154d40 0x7fa9e41f1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9e9e00640 1 -- 172.21.15.29:0/2013382987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9e40c8d40 msgr2=0x7fa9e4072320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:13.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9e9e00640 1 --2- 172.21.15.29:0/2013382987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9e40c8d40 0x7fa9e4072320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9e9e00640 1 -- 172.21.15.29:0/2013382987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e41f6040 con 0x7fa9cc007440 2024-04-17T19:48:13.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9e9e00640 1 --2- 172.21.15.29:0/2013382987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9cc007440 0x7fa9e4073450 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa9d800e7b0 tx=0x7fa9d800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:13.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9de000640 1 -- 172.21.15.29:0/2013382987 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d800ee60 con 0x7fa9cc007440 2024-04-17T19:48:13.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9de000640 1 -- 172.21.15.29:0/2013382987 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9d8018070 con 0x7fa9cc007440 2024-04-17T19:48:13.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9de000640 1 -- 172.21.15.29:0/2013382987 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d800cab0 con 0x7fa9cc007440 2024-04-17T19:48:13.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.697+0000 7fa9ec800640 1 -- 172.21.15.29:0/2013382987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e41f5280 con 0x7fa9cc007440 2024-04-17T19:48:13.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.701+0000 7fa9ec800640 1 -- 172.21.15.29:0/2013382987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9e41f5840 con 0x7fa9cc007440 2024-04-17T19:48:13.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.701+0000 7fa9ec800640 1 -- 172.21.15.29:0/2013382987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9e4069b90 con 0x7fa9cc007440 2024-04-17T19:48:13.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.701+0000 7fa9de000640 1 -- 172.21.15.29:0/2013382987 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa9d800cc50 con 0x7fa9cc007440 2024-04-17T19:48:13.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.701+0000 7fa9de000640 1 --2- 172.21.15.29:0/2013382987 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa9c8068380 0x7fa9c806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:13.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.701+0000 7fa9de000640 1 -- 172.21.15.29:0/2013382987 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa9d8086b90 con 0x7fa9cc007440 2024-04-17T19:48:13.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.701+0000 7fa9e9400640 1 --2- 172.21.15.29:0/2013382987 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa9c8068380 0x7fa9c806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:13.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.705+0000 7fa9e9400640 1 --2- 172.21.15.29:0/2013382987 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa9c8068380 0x7fa9c806a890 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4000c10 tx=0x7fa9d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:13.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.705+0000 7fa9de000640 1 -- 172.21.15.29:0/2013382987 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa9d8056d80 con 0x7fa9cc007440 2024-04-17T19:48:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.873+0000 7fa9ec800640 1 -- 172.21.15.29:0/2013382987 --> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa9e4157000 con 0x7fa9c8068380 2024-04-17T19:48:13.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.905+0000 7fa9de000640 1 -- 172.21.15.29:0/2013382987 <== mgr.4105 v2:172.21.15.37:6832/1588862286 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332558 (secure 0 0 0) 0x7fa9d4007c80 con 0x7fa9c8068380 2024-04-17T19:48:13.896 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:13.898 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-04-17T19:48:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.909+0000 7fa9ec800640 1 -- 172.21.15.29:0/2013382987 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa9c8068380 msgr2=0x7fa9c806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.909+0000 7fa9ec800640 1 --2- 172.21.15.29:0/2013382987 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa9c8068380 0x7fa9c806a890 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4000c10 tx=0x7fa9d4009040 comp rx=0 tx=0).stop 2024-04-17T19:48:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.909+0000 7fa9ec800640 1 -- 172.21.15.29:0/2013382987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9cc007440 msgr2=0x7fa9e4073450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.909+0000 7fa9ec800640 1 --2- 172.21.15.29:0/2013382987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9cc007440 0x7fa9e4073450 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa9d800e7b0 tx=0x7fa9d800ec70 comp rx=0 tx=0).stop 2024-04-17T19:48:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.909+0000 7fa9ec800640 1 -- 172.21.15.29:0/2013382987 shutdown_connections 2024-04-17T19:48:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.909+0000 7fa9ec800640 1 --2- 172.21.15.29:0/2013382987 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa9c8068380 0x7fa9c806a890 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.909+0000 7fa9ec800640 1 --2- 172.21.15.29:0/2013382987 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa9e40c8d40 0x7fa9e4072320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.909+0000 7fa9ec800640 1 --2- 172.21.15.29:0/2013382987 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa9e4154d40 0x7fa9e41f1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.909+0000 7fa9ec800640 1 --2- 172.21.15.29:0/2013382987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9cc007440 0x7fa9e4073450 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.909+0000 7fa9ec800640 1 -- 172.21.15.29:0/2013382987 >> 172.21.15.29:0/2013382987 conn(0x7fa9e406fde0 msgr2=0x7fa9e40cb970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.909+0000 7fa9ec800640 1 -- 172.21.15.29:0/2013382987 shutdown_connections 2024-04-17T19:48:13.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:13.909+0000 7fa9ec800640 1 -- 172.21.15.29:0/2013382987 wait complete. 2024-04-17T19:48:13.917 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-17T19:48:12.634241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240172,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754734548,"statfs":{"total":773094113280,"available":772848177152,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005447"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842100+0000","last_change":"2024-04-17T19:47:48.885235+0000","last_active":"2024-04-17T19:47:55.842100+0000","last_peered":"2024-04-17T19:47:55.842100+0000","last_clean":"2024-04-17T19:47:55.842100+0000","last_became_active":"2024-04-17T19:47:48.884844+0000","last_became_peered":"2024-04-17T19:47:48.884844+0000","last_unstale":"2024-04-17T19:47:55.842100+0000","last_undegraded":"2024-04-17T19:47:55.842100+0000","last_fullsized":"2024-04-17T19:47:55.842100+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:01:52.854763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924264+0000","last_change":"2024-04-17T19:47:49.872314+0000","last_active":"2024-04-17T19:47:54.924264+0000","last_peered":"2024-04-17T19:47:54.924264+0000","last_clean":"2024-04-17T19:47:54.924264+0000","last_became_active":"2024-04-17T19:47:49.871990+0000","last_became_peered":"2024-04-17T19:47:49.871990+0000","last_unstale":"2024-04-17T19:47:54.924264+0000","last_undegraded":"2024-04-17T19:47:54.924264+0000","last_fullsized":"2024-04-17T19:47:54.924264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:12:09.196610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923798+0000","last_change":"2024-04-17T19:47:53.929840+0000","last_active":"2024-04-17T19:47:54.923798+0000","last_peered":"2024-04-17T19:47:54.923798+0000","last_clean":"2024-04-17T19:47:54.923798+0000","last_became_active":"2024-04-17T19:47:53.928272+0000","last_became_peered":"2024-04-17T19:47:53.928272+0000","last_unstale":"2024-04-17T19:47:54.923798+0000","last_undegraded":"2024-04-17T19:47:54.923798+0000","last_fullsized":"2024-04-17T19:47:54.923798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:27:29.734327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924636+0000","last_change":"2024-04-17T19:47:48.862315+0000","last_active":"2024-04-17T19:47:54.924636+0000","last_peered":"2024-04-17T19:47:54.924636+0000","last_clean":"2024-04-17T19:47:54.924636+0000","last_became_active":"2024-04-17T19:47:48.860408+0000","last_became_peered":"2024-04-17T19:47:48.860408+0000","last_unstale":"2024-04-17T19:47:54.924636+0000","last_undegraded":"2024-04-17T19:47:54.924636+0000","last_fullsized":"2024-04-17T19:47:54.924636+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:10:26.336063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841873+0000","last_change":"2024-04-17T19:47:49.902423+0000","last_active":"2024-04-17T19:47:55.841873+0000","last_peered":"2024-04-17T19:47:55.841873+0000","last_clean":"2024-04-17T19:47:55.841873+0000","last_became_active":"2024-04-17T19:47:49.902096+0000","last_became_peered":"2024-04-17T19:47:49.902096+0000","last_unstale":"2024-04-17T19:47:55.841873+0000","last_undegraded":"2024-04-17T19:47:55.841873+0000","last_fullsized":"2024-04-17T19:47:55.841873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:40.805247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955520+0000","last_change":"2024-04-17T19:47:53.956851+0000","last_active":"2024-04-17T19:47:56.955520+0000","last_peered":"2024-04-17T19:47:56.955520+0000","last_clean":"2024-04-17T19:47:56.955520+0000","last_became_active":"2024-04-17T19:47:53.954197+0000","last_became_peered":"2024-04-17T19:47:53.954197+0000","last_unstale":"2024-04-17T19:47:56.955520+0000","last_undegraded":"2024-04-17T19:47:56.955520+0000","last_fullsized":"2024-04-17T19:47:56.955520+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:16:18.379831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049047+0000","last_change":"2024-04-17T19:47:48.893969+0000","last_active":"2024-04-17T19:47:56.049047+0000","last_peered":"2024-04-17T19:47:56.049047+0000","last_clean":"2024-04-17T19:47:56.049047+0000","last_became_active":"2024-04-17T19:47:48.893229+0000","last_became_peered":"2024-04-17T19:47:48.893229+0000","last_unstale":"2024-04-17T19:47:56.049047+0000","last_undegraded":"2024-04-17T19:47:56.049047+0000","last_fullsized":"2024-04-17T19:47:56.049047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:38:42.472634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832673+0000","last_change":"2024-04-17T19:47:49.897987+0000","last_active":"2024-04-17T19:47:56.832673+0000","last_peered":"2024-04-17T19:47:56.832673+0000","last_clean":"2024-04-17T19:47:56.832673+0000","last_became_active":"2024-04-17T19:47:49.897096+0000","last_became_peered":"2024-04-17T19:47:49.897096+0000","last_unstale":"2024-04-17T19:47:56.832673+0000","last_undegraded":"2024-04-17T19:47:56.832673+0000","last_fullsized":"2024-04-17T19:47:56.832673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:36:36.856454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955695+0000","last_change":"2024-04-17T19:47:53.930784+0000","last_active":"2024-04-17T19:47:56.955695+0000","last_peered":"2024-04-17T19:47:56.955695+0000","last_clean":"2024-04-17T19:47:56.955695+0000","last_became_active":"2024-04-17T19:47:53.928281+0000","last_became_peered":"2024-04-17T19:47:53.928281+0000","last_unstale":"2024-04-17T19:47:56.955695+0000","last_undegraded":"2024-04-17T19:47:56.955695+0000","last_fullsized":"2024-04-17T19:47:56.955695+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:15:55.064622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048308+0000","last_change":"2024-04-17T19:47:48.893184+0000","last_active":"2024-04-17T19:47:56.048308+0000","last_peered":"2024-04-17T19:47:56.048308+0000","last_clean":"2024-04-17T19:47:56.048308+0000","last_became_active":"2024-04-17T19:47:48.892375+0000","last_became_peered":"2024-04-17T19:47:48.892375+0000","last_unstale":"2024-04-17T19:47:56.048308+0000","last_undegraded":"2024-04-17T19:47:56.048308+0000","last_fullsized":"2024-04-17T19:47:56.048308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:56:01.211666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832916+0000","last_change":"2024-04-17T19:47:49.896040+0000","last_active":"2024-04-17T19:47:56.832916+0000","last_peered":"2024-04-17T19:47:56.832916+0000","last_clean":"2024-04-17T19:47:56.832916+0000","last_became_active":"2024-04-17T19:47:49.894779+0000","last_became_peered":"2024-04-17T19:47:49.894779+0000","last_unstale":"2024-04-17T19:47:56.832916+0000","last_undegraded":"2024-04-17T19:47:56.832916+0000","last_fullsized":"2024-04-17T19:47:56.832916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:30:17.930203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955688+0000","last_change":"2024-04-17T19:47:53.930402+0000","last_active":"2024-04-17T19:47:56.955688+0000","last_peered":"2024-04-17T19:47:56.955688+0000","last_clean":"2024-04-17T19:47:56.955688+0000","last_became_active":"2024-04-17T19:47:53.929166+0000","last_became_peered":"2024-04-17T19:47:53.929166+0000","last_unstale":"2024-04-17T19:47:56.955688+0000","last_undegraded":"2024-04-17T19:47:56.955688+0000","last_fullsized":"2024-04-17T19:47:56.955688+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:34:08.282697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927108+0000","last_change":"2024-04-17T19:47:49.900142+0000","last_active":"2024-04-17T19:47:54.927108+0000","last_peered":"2024-04-17T19:47:54.927108+0000","last_clean":"2024-04-17T19:47:54.927108+0000","last_became_active":"2024-04-17T19:47:49.899365+0000","last_became_peered":"2024-04-17T19:47:49.899365+0000","last_unstale":"2024-04-17T19:47:54.927108+0000","last_undegraded":"2024-04-17T19:47:54.927108+0000","last_fullsized":"2024-04-17T19:47:54.927108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:34:49.616539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048674+0000","last_change":"2024-04-17T19:47:48.891513+0000","last_active":"2024-04-17T19:47:56.048674+0000","last_peered":"2024-04-17T19:47:56.048674+0000","last_clean":"2024-04-17T19:47:56.048674+0000","last_became_active":"2024-04-17T19:47:48.886561+0000","last_became_peered":"2024-04-17T19:47:48.886561+0000","last_unstale":"2024-04-17T19:47:56.048674+0000","last_undegraded":"2024-04-17T19:47:56.048674+0000","last_fullsized":"2024-04-17T19:47:56.048674+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:59:14.039700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.957430+0000","last_change":"2024-04-17T19:47:53.925058+0000","last_active":"2024-04-17T19:47:56.957430+0000","last_peered":"2024-04-17T19:47:56.957430+0000","last_clean":"2024-04-17T19:47:56.957430+0000","last_became_active":"2024-04-17T19:47:53.923721+0000","last_became_peered":"2024-04-17T19:47:53.923721+0000","last_unstale":"2024-04-17T19:47:56.957430+0000","last_undegraded":"2024-04-17T19:47:56.957430+0000","last_fullsized":"2024-04-17T19:47:56.957430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:22:42.106657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924842+0000","last_change":"2024-04-17T19:47:49.852183+0000","last_active":"2024-04-17T19:47:54.924842+0000","last_peered":"2024-04-17T19:47:54.924842+0000","last_clean":"2024-04-17T19:47:54.924842+0000","last_became_active":"2024-04-17T19:47:49.851765+0000","last_became_peered":"2024-04-17T19:47:49.851765+0000","last_unstale":"2024-04-17T19:47:54.924842+0000","last_undegraded":"2024-04-17T19:47:54.924842+0000","last_fullsized":"2024-04-17T19:47:54.924842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:02:03.101317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051463+0000","last_change":"2024-04-17T19:47:48.862865+0000","last_active":"2024-04-17T19:47:55.051463+0000","last_peered":"2024-04-17T19:47:55.051463+0000","last_clean":"2024-04-17T19:47:55.051463+0000","last_became_active":"2024-04-17T19:47:48.862524+0000","last_became_peered":"2024-04-17T19:47:48.862524+0000","last_unstale":"2024-04-17T19:47:55.051463+0000","last_undegraded":"2024-04-17T19:47:55.051463+0000","last_fullsized":"2024-04-17T19:47:55.051463+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:01:21.080029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956615+0000","last_change":"2024-04-17T19:47:53.943674+0000","last_active":"2024-04-17T19:47:56.956615+0000","last_peered":"2024-04-17T19:47:56.956615+0000","last_clean":"2024-04-17T19:47:56.956615+0000","last_became_active":"2024-04-17T19:47:53.942549+0000","last_became_peered":"2024-04-17T19:47:53.942549+0000","last_unstale":"2024-04-17T19:47:56.956615+0000","last_undegraded":"2024-04-17T19:47:56.956615+0000","last_fullsized":"2024-04-17T19:47:56.956615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:00:42.300738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924513+0000","last_change":"2024-04-17T19:47:49.862312+0000","last_active":"2024-04-17T19:47:54.924513+0000","last_peered":"2024-04-17T19:47:54.924513+0000","last_clean":"2024-04-17T19:47:54.924513+0000","last_became_active":"2024-04-17T19:47:49.860664+0000","last_became_peered":"2024-04-17T19:47:49.860664+0000","last_unstale":"2024-04-17T19:47:54.924513+0000","last_undegraded":"2024-04-17T19:47:54.924513+0000","last_fullsized":"2024-04-17T19:47:54.924513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:07:47.676973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051042+0000","last_change":"2024-04-17T19:47:48.870638+0000","last_active":"2024-04-17T19:47:55.051042+0000","last_peered":"2024-04-17T19:47:55.051042+0000","last_clean":"2024-04-17T19:47:55.051042+0000","last_became_active":"2024-04-17T19:47:48.870353+0000","last_became_peered":"2024-04-17T19:47:48.870353+0000","last_unstale":"2024-04-17T19:47:55.051042+0000","last_undegraded":"2024-04-17T19:47:55.051042+0000","last_fullsized":"2024-04-17T19:47:55.051042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:12:19.927824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.052170+0000","last_change":"2024-04-17T19:47:53.929285+0000","last_active":"2024-04-17T19:47:55.052170+0000","last_peered":"2024-04-17T19:47:55.052170+0000","last_clean":"2024-04-17T19:47:55.052170+0000","last_became_active":"2024-04-17T19:47:53.928189+0000","last_became_peered":"2024-04-17T19:47:53.928189+0000","last_unstale":"2024-04-17T19:47:55.052170+0000","last_undegraded":"2024-04-17T19:47:55.052170+0000","last_fullsized":"2024-04-17T19:47:55.052170+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:53:26.112634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956382+0000","last_change":"2024-04-17T19:47:49.888118+0000","last_active":"2024-04-17T19:47:56.956382+0000","last_peered":"2024-04-17T19:47:56.956382+0000","last_clean":"2024-04-17T19:47:56.956382+0000","last_became_active":"2024-04-17T19:47:49.884037+0000","last_became_peered":"2024-04-17T19:47:49.884037+0000","last_unstale":"2024-04-17T19:47:56.956382+0000","last_undegraded":"2024-04-17T19:47:56.956382+0000","last_fullsized":"2024-04-17T19:47:56.956382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:44:27.180380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.047604+0000","last_change":"2024-04-17T19:47:48.883686+0000","last_active":"2024-04-17T19:47:56.047604+0000","last_peered":"2024-04-17T19:47:56.047604+0000","last_clean":"2024-04-17T19:47:56.047604+0000","last_became_active":"2024-04-17T19:47:48.882375+0000","last_became_peered":"2024-04-17T19:47:48.882375+0000","last_unstale":"2024-04-17T19:47:56.047604+0000","last_undegraded":"2024-04-17T19:47:56.047604+0000","last_fullsized":"2024-04-17T19:47:56.047604+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:38:31.225374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.957175+0000","last_change":"2024-04-17T19:47:53.935396+0000","last_active":"2024-04-17T19:47:56.957175+0000","last_peered":"2024-04-17T19:47:56.957175+0000","last_clean":"2024-04-17T19:47:56.957175+0000","last_became_active":"2024-04-17T19:47:53.934094+0000","last_became_peered":"2024-04-17T19:47:53.934094+0000","last_unstale":"2024-04-17T19:47:56.957175+0000","last_undegraded":"2024-04-17T19:47:56.957175+0000","last_fullsized":"2024-04-17T19:47:56.957175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:58.334357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842041+0000","last_change":"2024-04-17T19:47:49.870297+0000","last_active":"2024-04-17T19:47:55.842041+0000","last_peered":"2024-04-17T19:47:55.842041+0000","last_clean":"2024-04-17T19:47:55.842041+0000","last_became_active":"2024-04-17T19:47:49.870113+0000","last_became_peered":"2024-04-17T19:47:49.870113+0000","last_unstale":"2024-04-17T19:47:55.842041+0000","last_undegraded":"2024-04-17T19:47:55.842041+0000","last_fullsized":"2024-04-17T19:47:55.842041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:50:54.517273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929723+0000","last_change":"2024-04-17T19:47:48.895905+0000","last_active":"2024-04-17T19:47:54.929723+0000","last_peered":"2024-04-17T19:47:54.929723+0000","last_clean":"2024-04-17T19:47:54.929723+0000","last_became_active":"2024-04-17T19:47:48.895263+0000","last_became_peered":"2024-04-17T19:47:48.895263+0000","last_unstale":"2024-04-17T19:47:54.929723+0000","last_undegraded":"2024-04-17T19:47:54.929723+0000","last_fullsized":"2024-04-17T19:47:54.929723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:07:23.179031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956013+0000","last_change":"2024-04-17T19:47:53.943286+0000","last_active":"2024-04-17T19:47:56.956013+0000","last_peered":"2024-04-17T19:47:56.956013+0000","last_clean":"2024-04-17T19:47:56.956013+0000","last_became_active":"2024-04-17T19:47:53.942325+0000","last_became_peered":"2024-04-17T19:47:53.942325+0000","last_unstale":"2024-04-17T19:47:56.956013+0000","last_undegraded":"2024-04-17T19:47:56.956013+0000","last_fullsized":"2024-04-17T19:47:56.956013+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:47:18.853991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924723+0000","last_change":"2024-04-17T19:47:49.872925+0000","last_active":"2024-04-17T19:47:54.924723+0000","last_peered":"2024-04-17T19:47:54.924723+0000","last_clean":"2024-04-17T19:47:54.924723+0000","last_became_active":"2024-04-17T19:47:49.872668+0000","last_became_peered":"2024-04-17T19:47:49.872668+0000","last_unstale":"2024-04-17T19:47:54.924723+0000","last_undegraded":"2024-04-17T19:47:54.924723+0000","last_fullsized":"2024-04-17T19:47:54.924723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:16:21.209634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051807+0000","last_change":"2024-04-17T19:47:48.896669+0000","last_active":"2024-04-17T19:47:55.051807+0000","last_peered":"2024-04-17T19:47:55.051807+0000","last_clean":"2024-04-17T19:47:55.051807+0000","last_became_active":"2024-04-17T19:47:48.896397+0000","last_became_peered":"2024-04-17T19:47:48.896397+0000","last_unstale":"2024-04-17T19:47:55.051807+0000","last_undegraded":"2024-04-17T19:47:55.051807+0000","last_fullsized":"2024-04-17T19:47:55.051807+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:07:23.179031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923702+0000","last_change":"2024-04-17T19:47:53.906261+0000","last_active":"2024-04-17T19:47:54.923702+0000","last_peered":"2024-04-17T19:47:54.923702+0000","last_clean":"2024-04-17T19:47:54.923702+0000","last_became_active":"2024-04-17T19:47:53.903702+0000","last_became_peered":"2024-04-17T19:47:53.903702+0000","last_unstale":"2024-04-17T19:47:54.923702+0000","last_undegraded":"2024-04-17T19:47:54.923702+0000","last_fullsized":"2024-04-17T19:47:54.923702+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:35:53.892917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051477+0000","last_change":"2024-04-17T19:47:49.870272+0000","last_active":"2024-04-17T19:47:55.051477+0000","last_peered":"2024-04-17T19:47:55.051477+0000","last_clean":"2024-04-17T19:47:55.051477+0000","last_became_active":"2024-04-17T19:47:49.869914+0000","last_became_peered":"2024-04-17T19:47:49.869914+0000","last_unstale":"2024-04-17T19:47:55.051477+0000","last_undegraded":"2024-04-17T19:47:55.051477+0000","last_fullsized":"2024-04-17T19:47:55.051477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:44:30.553286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049327+0000","last_change":"2024-04-17T19:47:48.892675+0000","last_active":"2024-04-17T19:47:56.049327+0000","last_peered":"2024-04-17T19:47:56.049327+0000","last_clean":"2024-04-17T19:47:56.049327+0000","last_became_active":"2024-04-17T19:47:48.892057+0000","last_became_peered":"2024-04-17T19:47:48.892057+0000","last_unstale":"2024-04-17T19:47:56.049327+0000","last_undegraded":"2024-04-17T19:47:56.049327+0000","last_fullsized":"2024-04-17T19:47:56.049327+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:18:35.735350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841981+0000","last_change":"2024-04-17T19:47:53.925135+0000","last_active":"2024-04-17T19:47:55.841981+0000","last_peered":"2024-04-17T19:47:55.841981+0000","last_clean":"2024-04-17T19:47:55.841981+0000","last_became_active":"2024-04-17T19:47:53.924617+0000","last_became_peered":"2024-04-17T19:47:53.924617+0000","last_unstale":"2024-04-17T19:47:55.841981+0000","last_undegraded":"2024-04-17T19:47:55.841981+0000","last_fullsized":"2024-04-17T19:47:55.841981+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:37:02.082648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.833132+0000","last_change":"2024-04-17T19:47:49.895815+0000","last_active":"2024-04-17T19:47:56.833132+0000","last_peered":"2024-04-17T19:47:56.833132+0000","last_clean":"2024-04-17T19:47:56.833132+0000","last_became_active":"2024-04-17T19:47:49.894978+0000","last_became_peered":"2024-04-17T19:47:49.894978+0000","last_unstale":"2024-04-17T19:47:56.833132+0000","last_undegraded":"2024-04-17T19:47:56.833132+0000","last_fullsized":"2024-04-17T19:47:56.833132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:32:12.184723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051589+0000","last_change":"2024-04-17T19:47:48.859247+0000","last_active":"2024-04-17T19:47:55.051589+0000","last_peered":"2024-04-17T19:47:55.051589+0000","last_clean":"2024-04-17T19:47:55.051589+0000","last_became_active":"2024-04-17T19:47:48.858756+0000","last_became_peered":"2024-04-17T19:47:48.858756+0000","last_unstale":"2024-04-17T19:47:55.051589+0000","last_undegraded":"2024-04-17T19:47:55.051589+0000","last_fullsized":"2024-04-17T19:47:55.051589+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:19:56.987957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051118+0000","last_change":"2024-04-17T19:47:53.929952+0000","last_active":"2024-04-17T19:47:55.051118+0000","last_peered":"2024-04-17T19:47:55.051118+0000","last_clean":"2024-04-17T19:47:55.051118+0000","last_became_active":"2024-04-17T19:47:53.929528+0000","last_became_peered":"2024-04-17T19:47:53.929528+0000","last_unstale":"2024-04-17T19:47:55.051118+0000","last_undegraded":"2024-04-17T19:47:55.051118+0000","last_fullsized":"2024-04-17T19:47:55.051118+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:11:38.041301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956659+0000","last_change":"2024-04-17T19:47:49.874150+0000","last_active":"2024-04-17T19:47:56.956659+0000","last_peered":"2024-04-17T19:47:56.956659+0000","last_clean":"2024-04-17T19:47:56.956659+0000","last_became_active":"2024-04-17T19:47:49.872859+0000","last_became_peered":"2024-04-17T19:47:49.872859+0000","last_unstale":"2024-04-17T19:47:56.956659+0000","last_undegraded":"2024-04-17T19:47:56.956659+0000","last_fullsized":"2024-04-17T19:47:56.956659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:57:54.666279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832613+0000","last_change":"2024-04-17T19:47:48.876672+0000","last_active":"2024-04-17T19:47:56.832613+0000","last_peered":"2024-04-17T19:47:56.832613+0000","last_clean":"2024-04-17T19:47:56.832613+0000","last_became_active":"2024-04-17T19:47:48.875263+0000","last_became_peered":"2024-04-17T19:47:48.875263+0000","last_unstale":"2024-04-17T19:47:56.832613+0000","last_undegraded":"2024-04-17T19:47:56.832613+0000","last_fullsized":"2024-04-17T19:47:56.832613+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:45:46.633540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051584+0000","last_change":"2024-04-17T19:47:53.932330+0000","last_active":"2024-04-17T19:47:55.051584+0000","last_peered":"2024-04-17T19:47:55.051584+0000","last_clean":"2024-04-17T19:47:55.051584+0000","last_became_active":"2024-04-17T19:47:53.931873+0000","last_became_peered":"2024-04-17T19:47:53.931873+0000","last_unstale":"2024-04-17T19:47:55.051584+0000","last_undegraded":"2024-04-17T19:47:55.051584+0000","last_fullsized":"2024-04-17T19:47:55.051584+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:05:42.346878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051044+0000","last_change":"2024-04-17T19:47:49.865020+0000","last_active":"2024-04-17T19:47:55.051044+0000","last_peered":"2024-04-17T19:47:55.051044+0000","last_clean":"2024-04-17T19:47:55.051044+0000","last_became_active":"2024-04-17T19:47:49.864764+0000","last_became_peered":"2024-04-17T19:47:49.864764+0000","last_unstale":"2024-04-17T19:47:55.051044+0000","last_undegraded":"2024-04-17T19:47:55.051044+0000","last_fullsized":"2024-04-17T19:47:55.051044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:45:36.325765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926588+0000","last_change":"2024-04-17T19:47:48.846570+0000","last_active":"2024-04-17T19:47:54.926588+0000","last_peered":"2024-04-17T19:47:54.926588+0000","last_clean":"2024-04-17T19:47:54.926588+0000","last_became_active":"2024-04-17T19:47:48.846175+0000","last_became_peered":"2024-04-17T19:47:48.846175+0000","last_unstale":"2024-04-17T19:47:54.926588+0000","last_undegraded":"2024-04-17T19:47:54.926588+0000","last_fullsized":"2024-04-17T19:47:54.926588+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:20:22.737016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.052346+0000","last_change":"2024-04-17T19:47:53.904590+0000","last_active":"2024-04-17T19:47:55.052346+0000","last_peered":"2024-04-17T19:47:55.052346+0000","last_clean":"2024-04-17T19:47:55.052346+0000","last_became_active":"2024-04-17T19:47:53.904026+0000","last_became_peered":"2024-04-17T19:47:53.904026+0000","last_unstale":"2024-04-17T19:47:55.052346+0000","last_undegraded":"2024-04-17T19:47:55.052346+0000","last_fullsized":"2024-04-17T19:47:55.052346+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:16:22.837570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841469+0000","last_change":"2024-04-17T19:47:49.860010+0000","last_active":"2024-04-17T19:47:55.841469+0000","last_peered":"2024-04-17T19:47:55.841469+0000","last_clean":"2024-04-17T19:47:55.841469+0000","last_became_active":"2024-04-17T19:47:49.859593+0000","last_became_peered":"2024-04-17T19:47:49.859593+0000","last_unstale":"2024-04-17T19:47:55.841469+0000","last_undegraded":"2024-04-17T19:47:55.841469+0000","last_fullsized":"2024-04-17T19:47:55.841469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:33:24.594397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.047579+0000","last_change":"2024-04-17T19:47:48.883126+0000","last_active":"2024-04-17T19:47:56.047579+0000","last_peered":"2024-04-17T19:47:56.047579+0000","last_clean":"2024-04-17T19:47:56.047579+0000","last_became_active":"2024-04-17T19:47:48.882173+0000","last_became_peered":"2024-04-17T19:47:48.882173+0000","last_unstale":"2024-04-17T19:47:56.047579+0000","last_undegraded":"2024-04-17T19:47:56.047579+0000","last_fullsized":"2024-04-17T19:47:56.047579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:11:59.395316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841972+0000","last_change":"2024-04-17T19:47:53.931258+0000","last_active":"2024-04-17T19:47:55.841972+0000","last_peered":"2024-04-17T19:47:55.841972+0000","last_clean":"2024-04-17T19:47:55.841972+0000","last_became_active":"2024-04-17T19:47:53.930773+0000","last_became_peered":"2024-04-17T19:47:53.930773+0000","last_unstale":"2024-04-17T19:47:55.841972+0000","last_undegraded":"2024-04-17T19:47:55.841972+0000","last_fullsized":"2024-04-17T19:47:55.841972+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:10:47.583714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924520+0000","last_change":"2024-04-17T19:47:49.909404+0000","last_active":"2024-04-17T19:47:54.924520+0000","last_peered":"2024-04-17T19:47:54.924520+0000","last_clean":"2024-04-17T19:47:54.924520+0000","last_became_active":"2024-04-17T19:47:49.909154+0000","last_became_peered":"2024-04-17T19:47:49.909154+0000","last_unstale":"2024-04-17T19:47:54.924520+0000","last_undegraded":"2024-04-17T19:47:54.924520+0000","last_fullsized":"2024-04-17T19:47:54.924520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:14:39.682888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832291+0000","last_change":"2024-04-17T19:47:48.878592+0000","last_active":"2024-04-17T19:47:56.832291+0000","last_peered":"2024-04-17T19:47:56.832291+0000","last_clean":"2024-04-17T19:47:56.832291+0000","last_became_active":"2024-04-17T19:47:48.877617+0000","last_became_peered":"2024-04-17T19:47:48.877617+0000","last_unstale":"2024-04-17T19:47:56.832291+0000","last_undegraded":"2024-04-17T19:47:56.832291+0000","last_fullsized":"2024-04-17T19:47:56.832291+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:30:17.013176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048011+0000","last_change":"2024-04-17T19:47:38.277798+0000","last_active":"2024-04-17T19:47:56.048011+0000","last_peered":"2024-04-17T19:47:56.048011+0000","last_clean":"2024-04-17T19:47:56.048011+0000","last_became_active":"2024-04-17T19:47:38.274697+0000","last_became_peered":"2024-04-17T19:47:38.274697+0000","last_unstale":"2024-04-17T19:47:56.048011+0000","last_undegraded":"2024-04-17T19:47:56.048011+0000","last_fullsized":"2024-04-17T19:47:56.048011+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:36.897071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:36.897071+0000","last_clean_scrub_stamp":"2024-04-17T19:47:36.897071+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:31:35.366600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929077+0000","last_change":"2024-04-17T19:47:53.953415+0000","last_active":"2024-04-17T19:47:54.929077+0000","last_peered":"2024-04-17T19:47:54.929077+0000","last_clean":"2024-04-17T19:47:54.929077+0000","last_became_active":"2024-04-17T19:47:53.952644+0000","last_became_peered":"2024-04-17T19:47:53.952644+0000","last_unstale":"2024-04-17T19:47:54.929077+0000","last_undegraded":"2024-04-17T19:47:54.929077+0000","last_fullsized":"2024-04-17T19:47:54.929077+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:00:25.923909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048406+0000","last_change":"2024-04-17T19:47:49.901572+0000","last_active":"2024-04-17T19:47:56.048406+0000","last_peered":"2024-04-17T19:47:56.048406+0000","last_clean":"2024-04-17T19:47:56.048406+0000","last_became_active":"2024-04-17T19:47:49.900707+0000","last_became_peered":"2024-04-17T19:47:49.900707+0000","last_unstale":"2024-04-17T19:47:56.048406+0000","last_undegraded":"2024-04-17T19:47:56.048406+0000","last_fullsized":"2024-04-17T19:47:56.048406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:58:02.897564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926753+0000","last_change":"2024-04-17T19:47:48.874721+0000","last_active":"2024-04-17T19:47:54.926753+0000","last_peered":"2024-04-17T19:47:54.926753+0000","last_clean":"2024-04-17T19:47:54.926753+0000","last_became_active":"2024-04-17T19:47:48.874364+0000","last_became_peered":"2024-04-17T19:47:48.874364+0000","last_unstale":"2024-04-17T19:47:54.926753+0000","last_undegraded":"2024-04-17T19:47:54.926753+0000","last_fullsized":"2024-04-17T19:47:54.926753+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:47:08.856753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927738+0000","last_change":"2024-04-17T19:47:53.942791+0000","last_active":"2024-04-17T19:47:54.927738+0000","last_peered":"2024-04-17T19:47:54.927738+0000","last_clean":"2024-04-17T19:47:54.927738+0000","last_became_active":"2024-04-17T19:47:53.941569+0000","last_became_peered":"2024-04-17T19:47:53.941569+0000","last_unstale":"2024-04-17T19:47:54.927738+0000","last_undegraded":"2024-04-17T19:47:54.927738+0000","last_fullsized":"2024-04-17T19:47:54.927738+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:23:35.452056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832644+0000","last_change":"2024-04-17T19:47:49.895046+0000","last_active":"2024-04-17T19:47:56.832644+0000","last_peered":"2024-04-17T19:47:56.832644+0000","last_clean":"2024-04-17T19:47:56.832644+0000","last_became_active":"2024-04-17T19:47:49.894250+0000","last_became_peered":"2024-04-17T19:47:49.894250+0000","last_unstale":"2024-04-17T19:47:56.832644+0000","last_undegraded":"2024-04-17T19:47:56.832644+0000","last_fullsized":"2024-04-17T19:47:56.832644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:30:17.210135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051734+0000","last_change":"2024-04-17T19:47:48.861802+0000","last_active":"2024-04-17T19:47:55.051734+0000","last_peered":"2024-04-17T19:47:55.051734+0000","last_clean":"2024-04-17T19:47:55.051734+0000","last_became_active":"2024-04-17T19:47:48.860418+0000","last_became_peered":"2024-04-17T19:47:48.860418+0000","last_unstale":"2024-04-17T19:47:55.051734+0000","last_undegraded":"2024-04-17T19:47:55.051734+0000","last_fullsized":"2024-04-17T19:47:55.051734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:54:41.949746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842288+0000","last_change":"2024-04-17T19:47:53.917732+0000","last_active":"2024-04-17T19:47:55.842288+0000","last_peered":"2024-04-17T19:47:55.842288+0000","last_clean":"2024-04-17T19:47:55.842288+0000","last_became_active":"2024-04-17T19:47:53.917160+0000","last_became_peered":"2024-04-17T19:47:53.917160+0000","last_unstale":"2024-04-17T19:47:55.842288+0000","last_undegraded":"2024-04-17T19:47:55.842288+0000","last_fullsized":"2024-04-17T19:47:55.842288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:19:20.394166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049773+0000","last_change":"2024-04-17T19:47:49.893077+0000","last_active":"2024-04-17T19:47:56.049773+0000","last_peered":"2024-04-17T19:47:56.049773+0000","last_clean":"2024-04-17T19:47:56.049773+0000","last_became_active":"2024-04-17T19:47:49.892280+0000","last_became_peered":"2024-04-17T19:47:49.892280+0000","last_unstale":"2024-04-17T19:47:56.049773+0000","last_undegraded":"2024-04-17T19:47:56.049773+0000","last_fullsized":"2024-04-17T19:47:56.049773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:58:24.218821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926706+0000","last_change":"2024-04-17T19:47:48.849027+0000","last_active":"2024-04-17T19:47:54.926706+0000","last_peered":"2024-04-17T19:47:54.926706+0000","last_clean":"2024-04-17T19:47:54.926706+0000","last_became_active":"2024-04-17T19:47:48.848485+0000","last_became_peered":"2024-04-17T19:47:48.848485+0000","last_unstale":"2024-04-17T19:47:54.926706+0000","last_undegraded":"2024-04-17T19:47:54.926706+0000","last_fullsized":"2024-04-17T19:47:54.926706+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:58:14.548891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.831438+0000","last_change":"2024-04-17T19:47:53.946933+0000","last_active":"2024-04-17T19:47:56.831438+0000","last_peered":"2024-04-17T19:47:56.831438+0000","last_clean":"2024-04-17T19:47:56.831438+0000","last_became_active":"2024-04-17T19:47:53.943030+0000","last_became_peered":"2024-04-17T19:47:53.943030+0000","last_unstale":"2024-04-17T19:47:56.831438+0000","last_undegraded":"2024-04-17T19:47:56.831438+0000","last_fullsized":"2024-04-17T19:47:56.831438+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:37:03.489250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049281+0000","last_change":"2024-04-17T19:47:49.893137+0000","last_active":"2024-04-17T19:47:56.049281+0000","last_peered":"2024-04-17T19:47:56.049281+0000","last_clean":"2024-04-17T19:47:56.049281+0000","last_became_active":"2024-04-17T19:47:49.892379+0000","last_became_peered":"2024-04-17T19:47:49.892379+0000","last_unstale":"2024-04-17T19:47:56.049281+0000","last_undegraded":"2024-04-17T19:47:56.049281+0000","last_fullsized":"2024-04-17T19:47:56.049281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:55:08.562395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.954554+0000","last_change":"2024-04-17T19:47:48.861154+0000","last_active":"2024-04-17T19:47:56.954554+0000","last_peered":"2024-04-17T19:47:56.954554+0000","last_clean":"2024-04-17T19:47:56.954554+0000","last_became_active":"2024-04-17T19:47:48.859268+0000","last_became_peered":"2024-04-17T19:47:48.859268+0000","last_unstale":"2024-04-17T19:47:56.954554+0000","last_undegraded":"2024-04-17T19:47:56.954554+0000","last_fullsized":"2024-04-17T19:47:56.954554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:04:19.867115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955200+0000","last_change":"2024-04-17T19:47:53.950580+0000","last_active":"2024-04-17T19:47:56.955200+0000","last_peered":"2024-04-17T19:47:56.955200+0000","last_clean":"2024-04-17T19:47:56.955200+0000","last_became_active":"2024-04-17T19:47:53.943798+0000","last_became_peered":"2024-04-17T19:47:53.943798+0000","last_unstale":"2024-04-17T19:47:56.955200+0000","last_undegraded":"2024-04-17T19:47:56.955200+0000","last_fullsized":"2024-04-17T19:47:56.955200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:52:14.763799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048020+0000","last_change":"2024-04-17T19:47:49.893774+0000","last_active":"2024-04-17T19:47:56.048020+0000","last_peered":"2024-04-17T19:47:56.048020+0000","last_clean":"2024-04-17T19:47:56.048020+0000","last_became_active":"2024-04-17T19:47:49.892780+0000","last_became_peered":"2024-04-17T19:47:49.892780+0000","last_unstale":"2024-04-17T19:47:56.048020+0000","last_undegraded":"2024-04-17T19:47:56.048020+0000","last_fullsized":"2024-04-17T19:47:56.048020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:50:47.658650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"23'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926204+0000","last_change":"2024-04-17T19:47:48.852970+0000","last_active":"2024-04-17T19:47:54.926204+0000","last_peered":"2024-04-17T19:47:54.926204+0000","last_clean":"2024-04-17T19:47:54.926204+0000","last_became_active":"2024-04-17T19:47:48.852676+0000","last_became_peered":"2024-04-17T19:47:48.852676+0000","last_unstale":"2024-04-17T19:47:54.926204+0000","last_undegraded":"2024-04-17T19:47:54.926204+0000","last_fullsized":"2024-04-17T19:47:54.926204+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:32:14.529596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841883+0000","last_change":"2024-04-17T19:47:53.915397+0000","last_active":"2024-04-17T19:47:55.841883+0000","last_peered":"2024-04-17T19:47:55.841883+0000","last_clean":"2024-04-17T19:47:55.841883+0000","last_became_active":"2024-04-17T19:47:53.914913+0000","last_became_peered":"2024-04-17T19:47:53.914913+0000","last_unstale":"2024-04-17T19:47:55.841883+0000","last_undegraded":"2024-04-17T19:47:55.841883+0000","last_fullsized":"2024-04-17T19:47:55.841883+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:02:01.175158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927357+0000","last_change":"2024-04-17T19:47:49.880484+0000","last_active":"2024-04-17T19:47:54.927357+0000","last_peered":"2024-04-17T19:47:54.927357+0000","last_clean":"2024-04-17T19:47:54.927357+0000","last_became_active":"2024-04-17T19:47:49.879625+0000","last_became_peered":"2024-04-17T19:47:49.879625+0000","last_unstale":"2024-04-17T19:47:54.927357+0000","last_undegraded":"2024-04-17T19:47:54.927357+0000","last_fullsized":"2024-04-17T19:47:54.927357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:50:00.611796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927417+0000","last_change":"2024-04-17T19:47:48.899160+0000","last_active":"2024-04-17T19:47:54.927417+0000","last_peered":"2024-04-17T19:47:54.927417+0000","last_clean":"2024-04-17T19:47:54.927417+0000","last_became_active":"2024-04-17T19:47:48.889831+0000","last_became_peered":"2024-04-17T19:47:48.889831+0000","last_unstale":"2024-04-17T19:47:54.927417+0000","last_undegraded":"2024-04-17T19:47:54.927417+0000","last_fullsized":"2024-04-17T19:47:54.927417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:55:20.167510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924391+0000","last_change":"2024-04-17T19:47:53.921093+0000","last_active":"2024-04-17T19:47:54.924391+0000","last_peered":"2024-04-17T19:47:54.924391+0000","last_clean":"2024-04-17T19:47:54.924391+0000","last_became_active":"2024-04-17T19:47:53.920479+0000","last_became_peered":"2024-04-17T19:47:53.920479+0000","last_unstale":"2024-04-17T19:47:54.924391+0000","last_undegraded":"2024-04-17T19:47:54.924391+0000","last_fullsized":"2024-04-17T19:47:54.924391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:08:09.569820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048418+0000","last_change":"2024-04-17T19:47:49.895275+0000","last_active":"2024-04-17T19:47:56.048418+0000","last_peered":"2024-04-17T19:47:56.048418+0000","last_clean":"2024-04-17T19:47:56.048418+0000","last_became_active":"2024-04-17T19:47:49.894534+0000","last_became_peered":"2024-04-17T19:47:49.894534+0000","last_unstale":"2024-04-17T19:47:56.048418+0000","last_undegraded":"2024-04-17T19:47:56.048418+0000","last_fullsized":"2024-04-17T19:47:56.048418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:31:40.818807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832491+0000","last_change":"2024-04-17T19:47:48.881715+0000","last_active":"2024-04-17T19:47:56.832491+0000","last_peered":"2024-04-17T19:47:56.832491+0000","last_clean":"2024-04-17T19:47:56.832491+0000","last_became_active":"2024-04-17T19:47:48.876203+0000","last_became_peered":"2024-04-17T19:47:48.876203+0000","last_unstale":"2024-04-17T19:47:56.832491+0000","last_undegraded":"2024-04-17T19:47:56.832491+0000","last_fullsized":"2024-04-17T19:47:56.832491+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:03:16.656528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924597+0000","last_change":"2024-04-17T19:47:53.905149+0000","last_active":"2024-04-17T19:47:54.924597+0000","last_peered":"2024-04-17T19:47:54.924597+0000","last_clean":"2024-04-17T19:47:54.924597+0000","last_became_active":"2024-04-17T19:47:53.904718+0000","last_became_peered":"2024-04-17T19:47:53.904718+0000","last_unstale":"2024-04-17T19:47:54.924597+0000","last_undegraded":"2024-04-17T19:47:54.924597+0000","last_fullsized":"2024-04-17T19:47:54.924597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:09:31.908752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841674+0000","last_change":"2024-04-17T19:47:49.875819+0000","last_active":"2024-04-17T19:47:55.841674+0000","last_peered":"2024-04-17T19:47:55.841674+0000","last_clean":"2024-04-17T19:47:55.841674+0000","last_became_active":"2024-04-17T19:47:49.872075+0000","last_became_peered":"2024-04-17T19:47:49.872075+0000","last_unstale":"2024-04-17T19:47:55.841674+0000","last_undegraded":"2024-04-17T19:47:55.841674+0000","last_fullsized":"2024-04-17T19:47:55.841674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:06:48.823135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842429+0000","last_change":"2024-04-17T19:47:48.867633+0000","last_active":"2024-04-17T19:47:55.842429+0000","last_peered":"2024-04-17T19:47:55.842429+0000","last_clean":"2024-04-17T19:47:55.842429+0000","last_became_active":"2024-04-17T19:47:48.867150+0000","last_became_peered":"2024-04-17T19:47:48.867150+0000","last_unstale":"2024-04-17T19:47:55.842429+0000","last_undegraded":"2024-04-17T19:47:55.842429+0000","last_fullsized":"2024-04-17T19:47:55.842429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:33:34.275816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832910+0000","last_change":"2024-04-17T19:47:53.925785+0000","last_active":"2024-04-17T19:47:56.832910+0000","last_peered":"2024-04-17T19:47:56.832910+0000","last_clean":"2024-04-17T19:47:56.832910+0000","last_became_active":"2024-04-17T19:47:53.924006+0000","last_became_peered":"2024-04-17T19:47:53.924006+0000","last_unstale":"2024-04-17T19:47:56.832910+0000","last_undegraded":"2024-04-17T19:47:56.832910+0000","last_fullsized":"2024-04-17T19:47:56.832910+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:46:31.965271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048194+0000","last_change":"2024-04-17T19:47:49.869829+0000","last_active":"2024-04-17T19:47:56.048194+0000","last_peered":"2024-04-17T19:47:56.048194+0000","last_clean":"2024-04-17T19:47:56.048194+0000","last_became_active":"2024-04-17T19:47:49.868933+0000","last_became_peered":"2024-04-17T19:47:49.868933+0000","last_unstale":"2024-04-17T19:47:56.048194+0000","last_undegraded":"2024-04-17T19:47:56.048194+0000","last_fullsized":"2024-04-17T19:47:56.048194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:36:08.866159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:03.662022+0000","last_change":"2024-04-17T19:47:48.887461+0000","last_active":"2024-04-17T19:48:03.662022+0000","last_peered":"2024-04-17T19:48:03.662022+0000","last_clean":"2024-04-17T19:48:03.662022+0000","last_became_active":"2024-04-17T19:47:48.886689+0000","last_became_peered":"2024-04-17T19:47:48.886689+0000","last_unstale":"2024-04-17T19:48:03.662022+0000","last_undegraded":"2024-04-17T19:48:03.662022+0000","last_fullsized":"2024-04-17T19:48:03.662022+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:55:35.262995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051405+0000","last_change":"2024-04-17T19:47:53.941607+0000","last_active":"2024-04-17T19:47:55.051405+0000","last_peered":"2024-04-17T19:47:55.051405+0000","last_clean":"2024-04-17T19:47:55.051405+0000","last_became_active":"2024-04-17T19:47:53.941050+0000","last_became_peered":"2024-04-17T19:47:53.941050+0000","last_unstale":"2024-04-17T19:47:55.051405+0000","last_undegraded":"2024-04-17T19:47:55.051405+0000","last_fullsized":"2024-04-17T19:47:55.051405+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:26:08.964058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.957037+0000","last_change":"2024-04-17T19:47:49.879524+0000","last_active":"2024-04-17T19:47:56.957037+0000","last_peered":"2024-04-17T19:47:56.957037+0000","last_clean":"2024-04-17T19:47:56.957037+0000","last_became_active":"2024-04-17T19:47:49.877563+0000","last_became_peered":"2024-04-17T19:47:49.877563+0000","last_unstale":"2024-04-17T19:47:56.957037+0000","last_undegraded":"2024-04-17T19:47:56.957037+0000","last_fullsized":"2024-04-17T19:47:56.957037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:34:58.539940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"23'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051676+0000","last_change":"2024-04-17T19:47:48.870100+0000","last_active":"2024-04-17T19:47:55.051676+0000","last_peered":"2024-04-17T19:47:55.051676+0000","last_clean":"2024-04-17T19:47:55.051676+0000","last_became_active":"2024-04-17T19:47:48.869797+0000","last_became_peered":"2024-04-17T19:47:48.869797+0000","last_unstale":"2024-04-17T19:47:55.051676+0000","last_undegraded":"2024-04-17T19:47:55.051676+0000","last_fullsized":"2024-04-17T19:47:55.051676+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:51:21.463081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956297+0000","last_change":"2024-04-17T19:47:53.944568+0000","last_active":"2024-04-17T19:47:56.956297+0000","last_peered":"2024-04-17T19:47:56.956297+0000","last_clean":"2024-04-17T19:47:56.956297+0000","last_became_active":"2024-04-17T19:47:53.943151+0000","last_became_peered":"2024-04-17T19:47:53.943151+0000","last_unstale":"2024-04-17T19:47:56.956297+0000","last_undegraded":"2024-04-17T19:47:56.956297+0000","last_fullsized":"2024-04-17T19:47:56.956297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:19:00.150450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842029+0000","last_change":"2024-04-17T19:47:49.902401+0000","last_active":"2024-04-17T19:47:55.842029+0000","last_peered":"2024-04-17T19:47:55.842029+0000","last_clean":"2024-04-17T19:47:55.842029+0000","last_became_active":"2024-04-17T19:47:49.902166+0000","last_became_peered":"2024-04-17T19:47:49.902166+0000","last_unstale":"2024-04-17T19:47:55.842029+0000","last_undegraded":"2024-04-17T19:47:55.842029+0000","last_fullsized":"2024-04-17T19:47:55.842029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:54:13.129748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832891+0000","last_change":"2024-04-17T19:47:48.900800+0000","last_active":"2024-04-17T19:47:56.832891+0000","last_peered":"2024-04-17T19:47:56.832891+0000","last_clean":"2024-04-17T19:47:56.832891+0000","last_became_active":"2024-04-17T19:47:48.900188+0000","last_became_peered":"2024-04-17T19:47:48.900188+0000","last_unstale":"2024-04-17T19:47:56.832891+0000","last_undegraded":"2024-04-17T19:47:56.832891+0000","last_fullsized":"2024-04-17T19:47:56.832891+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:05:04.305871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923962+0000","last_change":"2024-04-17T19:47:53.928721+0000","last_active":"2024-04-17T19:47:54.923962+0000","last_peered":"2024-04-17T19:47:54.923962+0000","last_clean":"2024-04-17T19:47:54.923962+0000","last_became_active":"2024-04-17T19:47:53.928311+0000","last_became_peered":"2024-04-17T19:47:53.928311+0000","last_unstale":"2024-04-17T19:47:54.923962+0000","last_undegraded":"2024-04-17T19:47:54.923962+0000","last_fullsized":"2024-04-17T19:47:54.923962+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:35:28.870803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832516+0000","last_change":"2024-04-17T19:47:49.905421+0000","last_active":"2024-04-17T19:47:56.832516+0000","last_peered":"2024-04-17T19:47:56.832516+0000","last_clean":"2024-04-17T19:47:56.832516+0000","last_became_active":"2024-04-17T19:47:49.904646+0000","last_became_peered":"2024-04-17T19:47:49.904646+0000","last_unstale":"2024-04-17T19:47:56.832516+0000","last_undegraded":"2024-04-17T19:47:56.832516+0000","last_fullsized":"2024-04-17T19:47:56.832516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:02:23.936080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929405+0000","last_change":"2024-04-17T19:47:48.895356+0000","last_active":"2024-04-17T19:47:54.929405+0000","last_peered":"2024-04-17T19:47:54.929405+0000","last_clean":"2024-04-17T19:47:54.929405+0000","last_became_active":"2024-04-17T19:47:48.889225+0000","last_became_peered":"2024-04-17T19:47:48.889225+0000","last_unstale":"2024-04-17T19:47:54.929405+0000","last_undegraded":"2024-04-17T19:47:54.929405+0000","last_fullsized":"2024-04-17T19:47:54.929405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:00:22.587917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923872+0000","last_change":"2024-04-17T19:47:53.940206+0000","last_active":"2024-04-17T19:47:54.923872+0000","last_peered":"2024-04-17T19:47:54.923872+0000","last_clean":"2024-04-17T19:47:54.923872+0000","last_became_active":"2024-04-17T19:47:53.939843+0000","last_became_peered":"2024-04-17T19:47:53.939843+0000","last_unstale":"2024-04-17T19:47:54.923872+0000","last_undegraded":"2024-04-17T19:47:54.923872+0000","last_fullsized":"2024-04-17T19:47:54.923872+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:18:55.562573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841764+0000","last_change":"2024-04-17T19:47:53.920258+0000","last_active":"2024-04-17T19:47:55.841764+0000","last_peered":"2024-04-17T19:47:55.841764+0000","last_clean":"2024-04-17T19:47:55.841764+0000","last_became_active":"2024-04-17T19:47:53.918932+0000","last_became_peered":"2024-04-17T19:47:53.918932+0000","last_unstale":"2024-04-17T19:47:55.841764+0000","last_undegraded":"2024-04-17T19:47:55.841764+0000","last_fullsized":"2024-04-17T19:47:55.841764+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:45:55.456803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.050900+0000","last_change":"2024-04-17T19:47:49.861692+0000","last_active":"2024-04-17T19:47:55.050900+0000","last_peered":"2024-04-17T19:47:55.050900+0000","last_clean":"2024-04-17T19:47:55.050900+0000","last_became_active":"2024-04-17T19:47:49.861333+0000","last_became_peered":"2024-04-17T19:47:49.861333+0000","last_unstale":"2024-04-17T19:47:55.050900+0000","last_undegraded":"2024-04-17T19:47:55.050900+0000","last_fullsized":"2024-04-17T19:47:55.050900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:47:13.317679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832138+0000","last_change":"2024-04-17T19:47:48.873355+0000","last_active":"2024-04-17T19:47:56.832138+0000","last_peered":"2024-04-17T19:47:56.832138+0000","last_clean":"2024-04-17T19:47:56.832138+0000","last_became_active":"2024-04-17T19:47:48.872427+0000","last_became_peered":"2024-04-17T19:47:48.872427+0000","last_unstale":"2024-04-17T19:47:56.832138+0000","last_undegraded":"2024-04-17T19:47:56.832138+0000","last_fullsized":"2024-04-17T19:47:56.832138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:32:41.792605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929095+0000","last_change":"2024-04-17T19:47:53.952602+0000","last_active":"2024-04-17T19:47:54.929095+0000","last_peered":"2024-04-17T19:47:54.929095+0000","last_clean":"2024-04-17T19:47:54.929095+0000","last_became_active":"2024-04-17T19:47:53.951750+0000","last_became_peered":"2024-04-17T19:47:53.951750+0000","last_unstale":"2024-04-17T19:47:54.929095+0000","last_undegraded":"2024-04-17T19:47:54.929095+0000","last_fullsized":"2024-04-17T19:47:54.929095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:13:08.485238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923863+0000","last_change":"2024-04-17T19:47:49.862071+0000","last_active":"2024-04-17T19:47:54.923863+0000","last_peered":"2024-04-17T19:47:54.923863+0000","last_clean":"2024-04-17T19:47:54.923863+0000","last_became_active":"2024-04-17T19:47:49.860809+0000","last_became_peered":"2024-04-17T19:47:49.860809+0000","last_unstale":"2024-04-17T19:47:54.923863+0000","last_undegraded":"2024-04-17T19:47:54.923863+0000","last_fullsized":"2024-04-17T19:47:54.923863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:42:34.451839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924173+0000","last_change":"2024-04-17T19:47:48.867855+0000","last_active":"2024-04-17T19:47:54.924173+0000","last_peered":"2024-04-17T19:47:54.924173+0000","last_clean":"2024-04-17T19:47:54.924173+0000","last_became_active":"2024-04-17T19:47:48.867532+0000","last_became_peered":"2024-04-17T19:47:48.867532+0000","last_unstale":"2024-04-17T19:47:54.924173+0000","last_undegraded":"2024-04-17T19:47:54.924173+0000","last_fullsized":"2024-04-17T19:47:54.924173+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:21:26.903245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927964+0000","last_change":"2024-04-17T19:47:48.894946+0000","last_active":"2024-04-17T19:47:54.927964+0000","last_peered":"2024-04-17T19:47:54.927964+0000","last_clean":"2024-04-17T19:47:54.927964+0000","last_became_active":"2024-04-17T19:47:48.893407+0000","last_became_peered":"2024-04-17T19:47:48.893407+0000","last_unstale":"2024-04-17T19:47:54.927964+0000","last_undegraded":"2024-04-17T19:47:54.927964+0000","last_fullsized":"2024-04-17T19:47:54.927964+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:09:56.284638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842374+0000","last_change":"2024-04-17T19:47:49.859985+0000","last_active":"2024-04-17T19:47:55.842374+0000","last_peered":"2024-04-17T19:47:55.842374+0000","last_clean":"2024-04-17T19:47:55.842374+0000","last_became_active":"2024-04-17T19:47:49.859635+0000","last_became_peered":"2024-04-17T19:47:49.859635+0000","last_unstale":"2024-04-17T19:47:55.842374+0000","last_undegraded":"2024-04-17T19:47:55.842374+0000","last_fullsized":"2024-04-17T19:47:55.842374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:04:14.834218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926673+0000","last_change":"2024-04-17T19:47:53.925444+0000","last_active":"2024-04-17T19:47:54.926673+0000","last_peered":"2024-04-17T19:47:54.926673+0000","last_clean":"2024-04-17T19:47:54.926673+0000","last_became_active":"2024-04-17T19:47:53.924805+0000","last_became_peered":"2024-04-17T19:47:53.924805+0000","last_unstale":"2024-04-17T19:47:54.926673+0000","last_undegraded":"2024-04-17T19:47:54.926673+0000","last_fullsized":"2024-04-17T19:47:54.926673+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:19:24.022368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924894+0000","last_change":"2024-04-17T19:47:48.867176+0000","last_active":"2024-04-17T19:47:54.924894+0000","last_peered":"2024-04-17T19:47:54.924894+0000","last_clean":"2024-04-17T19:47:54.924894+0000","last_became_active":"2024-04-17T19:47:48.866817+0000","last_became_peered":"2024-04-17T19:47:48.866817+0000","last_unstale":"2024-04-17T19:47:54.924894+0000","last_undegraded":"2024-04-17T19:47:54.924894+0000","last_fullsized":"2024-04-17T19:47:54.924894+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:44:30.842583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955878+0000","last_change":"2024-04-17T19:47:49.858924+0000","last_active":"2024-04-17T19:47:56.955878+0000","last_peered":"2024-04-17T19:47:56.955878+0000","last_clean":"2024-04-17T19:47:56.955878+0000","last_became_active":"2024-04-17T19:47:49.857820+0000","last_became_peered":"2024-04-17T19:47:49.857820+0000","last_unstale":"2024-04-17T19:47:56.955878+0000","last_undegraded":"2024-04-17T19:47:56.955878+0000","last_fullsized":"2024-04-17T19:47:56.955878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:09:28.706054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924516+0000","last_change":"2024-04-17T19:47:53.933337+0000","last_active":"2024-04-17T19:47:54.924516+0000","last_peered":"2024-04-17T19:47:54.924516+0000","last_clean":"2024-04-17T19:47:54.924516+0000","last_became_active":"2024-04-17T19:47:53.931623+0000","last_became_peered":"2024-04-17T19:47:53.931623+0000","last_unstale":"2024-04-17T19:47:54.924516+0000","last_undegraded":"2024-04-17T19:47:54.924516+0000","last_fullsized":"2024-04-17T19:47:54.924516+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:57:37.600957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927056+0000","last_change":"2024-04-17T19:47:49.881093+0000","last_active":"2024-04-17T19:47:54.927056+0000","last_peered":"2024-04-17T19:47:54.927056+0000","last_clean":"2024-04-17T19:47:54.927056+0000","last_became_active":"2024-04-17T19:47:49.880210+0000","last_became_peered":"2024-04-17T19:47:49.880210+0000","last_unstale":"2024-04-17T19:47:54.927056+0000","last_undegraded":"2024-04-17T19:47:54.927056+0000","last_fullsized":"2024-04-17T19:47:54.927056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:12:15.522651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926335+0000","last_change":"2024-04-17T19:47:49.886936+0000","last_active":"2024-04-17T19:47:54.926335+0000","last_peered":"2024-04-17T19:47:54.926335+0000","last_clean":"2024-04-17T19:47:54.926335+0000","last_became_active":"2024-04-17T19:47:49.886580+0000","last_became_peered":"2024-04-17T19:47:49.886580+0000","last_unstale":"2024-04-17T19:47:54.926335+0000","last_undegraded":"2024-04-17T19:47:54.926335+0000","last_fullsized":"2024-04-17T19:47:54.926335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:00:44.599463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832155+0000","last_change":"2024-04-17T19:47:49.905445+0000","last_active":"2024-04-17T19:47:56.832155+0000","last_peered":"2024-04-17T19:47:56.832155+0000","last_clean":"2024-04-17T19:47:56.832155+0000","last_became_active":"2024-04-17T19:47:49.896510+0000","last_became_peered":"2024-04-17T19:47:49.896510+0000","last_unstale":"2024-04-17T19:47:56.832155+0000","last_undegraded":"2024-04-17T19:47:56.832155+0000","last_fullsized":"2024-04-17T19:47:56.832155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:23:27.401861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842320+0000","last_change":"2024-04-17T19:47:49.850490+0000","last_active":"2024-04-17T19:47:55.842320+0000","last_peered":"2024-04-17T19:47:55.842320+0000","last_clean":"2024-04-17T19:47:55.842320+0000","last_became_active":"2024-04-17T19:47:49.849344+0000","last_became_peered":"2024-04-17T19:47:49.849344+0000","last_unstale":"2024-04-17T19:47:55.842320+0000","last_undegraded":"2024-04-17T19:47:55.842320+0000","last_fullsized":"2024-04-17T19:47:55.842320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:44:32.231058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924753+0000","last_change":"2024-04-17T19:47:49.857737+0000","last_active":"2024-04-17T19:47:54.924753+0000","last_peered":"2024-04-17T19:47:54.924753+0000","last_clean":"2024-04-17T19:47:54.924753+0000","last_became_active":"2024-04-17T19:47:49.857476+0000","last_became_peered":"2024-04-17T19:47:49.857476+0000","last_unstale":"2024-04-17T19:47:54.924753+0000","last_undegraded":"2024-04-17T19:47:54.924753+0000","last_fullsized":"2024-04-17T19:47:54.924753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:50:13.070628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051526+0000","last_change":"2024-04-17T19:47:49.864985+0000","last_active":"2024-04-17T19:47:55.051526+0000","last_peered":"2024-04-17T19:47:55.051526+0000","last_clean":"2024-04-17T19:47:55.051526+0000","last_became_active":"2024-04-17T19:47:49.864329+0000","last_became_peered":"2024-04-17T19:47:49.864329+0000","last_unstale":"2024-04-17T19:47:55.051526+0000","last_undegraded":"2024-04-17T19:47:55.051526+0000","last_fullsized":"2024-04-17T19:47:55.051526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:43:57.448269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929577+0000","last_change":"2024-04-17T19:47:49.881785+0000","last_active":"2024-04-17T19:47:54.929577+0000","last_peered":"2024-04-17T19:47:54.929577+0000","last_clean":"2024-04-17T19:47:54.929577+0000","last_became_active":"2024-04-17T19:47:49.880939+0000","last_became_peered":"2024-04-17T19:47:49.880939+0000","last_unstale":"2024-04-17T19:47:54.929577+0000","last_undegraded":"2024-04-17T19:47:54.929577+0000","last_fullsized":"2024-04-17T19:47:54.929577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:49:07.194643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924461+0000","last_change":"2024-04-17T19:47:49.884944+0000","last_active":"2024-04-17T19:47:54.924461+0000","last_peered":"2024-04-17T19:47:54.924461+0000","last_clean":"2024-04-17T19:47:54.924461+0000","last_became_active":"2024-04-17T19:47:49.884559+0000","last_became_peered":"2024-04-17T19:47:49.884559+0000","last_unstale":"2024-04-17T19:47:54.924461+0000","last_undegraded":"2024-04-17T19:47:54.924461+0000","last_fullsized":"2024-04-17T19:47:54.924461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:00.485472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.831772+0000","last_change":"2024-04-17T19:47:49.891230+0000","last_active":"2024-04-17T19:47:56.831772+0000","last_peered":"2024-04-17T19:47:56.831772+0000","last_clean":"2024-04-17T19:47:56.831772+0000","last_became_active":"2024-04-17T19:47:49.886001+0000","last_became_peered":"2024-04-17T19:47:49.886001+0000","last_unstale":"2024-04-17T19:47:56.831772+0000","last_undegraded":"2024-04-17T19:47:56.831772+0000","last_fullsized":"2024-04-17T19:47:56.831772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:00.485472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923958+0000","last_change":"2024-04-17T19:47:49.859382+0000","last_active":"2024-04-17T19:47:54.923958+0000","last_peered":"2024-04-17T19:47:54.923958+0000","last_clean":"2024-04-17T19:47:54.923958+0000","last_became_active":"2024-04-17T19:47:49.859170+0000","last_became_peered":"2024-04-17T19:47:49.859170+0000","last_unstale":"2024-04-17T19:47:54.923958+0000","last_undegraded":"2024-04-17T19:47:54.923958+0000","last_fullsized":"2024-04-17T19:47:54.923958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:58:51.111053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832143+0000","last_change":"2024-04-17T19:47:49.909188+0000","last_active":"2024-04-17T19:47:56.832143+0000","last_peered":"2024-04-17T19:47:56.832143+0000","last_clean":"2024-04-17T19:47:56.832143+0000","last_became_active":"2024-04-17T19:47:49.908432+0000","last_became_peered":"2024-04-17T19:47:49.908432+0000","last_unstale":"2024-04-17T19:47:56.832143+0000","last_undegraded":"2024-04-17T19:47:56.832143+0000","last_fullsized":"2024-04-17T19:47:56.832143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:16:21.209634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842710+0000","last_change":"2024-04-17T19:47:49.876406+0000","last_active":"2024-04-17T19:47:55.842710+0000","last_peered":"2024-04-17T19:47:55.842710+0000","last_clean":"2024-04-17T19:47:55.842710+0000","last_became_active":"2024-04-17T19:47:49.876132+0000","last_became_peered":"2024-04-17T19:47:49.876132+0000","last_unstale":"2024-04-17T19:47:55.842710+0000","last_undegraded":"2024-04-17T19:47:55.842710+0000","last_fullsized":"2024-04-17T19:47:55.842710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:51:20.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.928888+0000","last_change":"2024-04-17T19:47:49.896244+0000","last_active":"2024-04-17T19:47:54.928888+0000","last_peered":"2024-04-17T19:47:54.928888+0000","last_clean":"2024-04-17T19:47:54.928888+0000","last_became_active":"2024-04-17T19:47:49.895380+0000","last_became_peered":"2024-04-17T19:47:49.895380+0000","last_unstale":"2024-04-17T19:47:54.928888+0000","last_undegraded":"2024-04-17T19:47:54.928888+0000","last_fullsized":"2024-04-17T19:47:54.928888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:04:43.091307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049138+0000","last_change":"2024-04-17T19:47:49.870551+0000","last_active":"2024-04-17T19:47:56.049138+0000","last_peered":"2024-04-17T19:47:56.049138+0000","last_clean":"2024-04-17T19:47:56.049138+0000","last_became_active":"2024-04-17T19:47:49.869315+0000","last_became_peered":"2024-04-17T19:47:49.869315+0000","last_unstale":"2024-04-17T19:47:56.049138+0000","last_undegraded":"2024-04-17T19:47:56.049138+0000","last_fullsized":"2024-04-17T19:47:56.049138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:06.341123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926618+0000","last_change":"2024-04-17T19:47:49.880060+0000","last_active":"2024-04-17T19:47:54.926618+0000","last_peered":"2024-04-17T19:47:54.926618+0000","last_clean":"2024-04-17T19:47:54.926618+0000","last_became_active":"2024-04-17T19:47:49.879711+0000","last_became_peered":"2024-04-17T19:47:49.879711+0000","last_unstale":"2024-04-17T19:47:54.926618+0000","last_undegraded":"2024-04-17T19:47:54.926618+0000","last_fullsized":"2024-04-17T19:47:54.926618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:41:25.628870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.833444+0000","last_change":"2024-04-17T19:47:49.895452+0000","last_active":"2024-04-17T19:47:56.833444+0000","last_peered":"2024-04-17T19:47:56.833444+0000","last_clean":"2024-04-17T19:47:56.833444+0000","last_became_active":"2024-04-17T19:47:49.894684+0000","last_became_peered":"2024-04-17T19:47:49.894684+0000","last_unstale":"2024-04-17T19:47:56.833444+0000","last_undegraded":"2024-04-17T19:47:56.833444+0000","last_fullsized":"2024-04-17T19:47:56.833444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:15:55.224238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.831437+0000","last_change":"2024-04-17T19:47:49.886902+0000","last_active":"2024-04-17T19:47:56.831437+0000","last_peered":"2024-04-17T19:47:56.831437+0000","last_clean":"2024-04-17T19:47:56.831437+0000","last_became_active":"2024-04-17T19:47:49.885451+0000","last_became_peered":"2024-04-17T19:47:49.885451+0000","last_unstale":"2024-04-17T19:47:56.831437+0000","last_undegraded":"2024-04-17T19:47:56.831437+0000","last_fullsized":"2024-04-17T19:47:56.831437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:12:09.196610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.050989+0000","last_change":"2024-04-17T19:47:49.858908+0000","last_active":"2024-04-17T19:47:55.050989+0000","last_peered":"2024-04-17T19:47:55.050989+0000","last_clean":"2024-04-17T19:47:55.050989+0000","last_became_active":"2024-04-17T19:47:49.858457+0000","last_became_peered":"2024-04-17T19:47:49.858457+0000","last_unstale":"2024-04-17T19:47:55.050989+0000","last_undegraded":"2024-04-17T19:47:55.050989+0000","last_fullsized":"2024-04-17T19:47:55.050989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:09.467859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841618+0000","last_change":"2024-04-17T19:47:49.856371+0000","last_active":"2024-04-17T19:47:55.841618+0000","last_peered":"2024-04-17T19:47:55.841618+0000","last_clean":"2024-04-17T19:47:55.841618+0000","last_became_active":"2024-04-17T19:47:49.856062+0000","last_became_peered":"2024-04-17T19:47:49.856062+0000","last_unstale":"2024-04-17T19:47:55.841618+0000","last_undegraded":"2024-04-17T19:47:55.841618+0000","last_fullsized":"2024-04-17T19:47:55.841618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:39:54.502619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924769+0000","last_change":"2024-04-17T19:47:49.873350+0000","last_active":"2024-04-17T19:47:54.924769+0000","last_peered":"2024-04-17T19:47:54.924769+0000","last_clean":"2024-04-17T19:47:54.924769+0000","last_became_active":"2024-04-17T19:47:49.873019+0000","last_became_peered":"2024-04-17T19:47:49.873019+0000","last_unstale":"2024-04-17T19:47:54.924769+0000","last_undegraded":"2024-04-17T19:47:54.924769+0000","last_fullsized":"2024-04-17T19:47:54.924769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:06:58.641727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956704+0000","last_change":"2024-04-17T19:47:49.891551+0000","last_active":"2024-04-17T19:47:56.956704+0000","last_peered":"2024-04-17T19:47:56.956704+0000","last_clean":"2024-04-17T19:47:56.956704+0000","last_became_active":"2024-04-17T19:47:49.890132+0000","last_became_peered":"2024-04-17T19:47:49.890132+0000","last_unstale":"2024-04-17T19:47:56.956704+0000","last_undegraded":"2024-04-17T19:47:56.956704+0000","last_fullsized":"2024-04-17T19:47:56.956704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:26:22.971489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.928586+0000","last_change":"2024-04-17T19:47:49.903854+0000","last_active":"2024-04-17T19:47:54.928586+0000","last_peered":"2024-04-17T19:47:54.928586+0000","last_clean":"2024-04-17T19:47:54.928586+0000","last_became_active":"2024-04-17T19:47:49.903178+0000","last_became_peered":"2024-04-17T19:47:49.903178+0000","last_unstale":"2024-04-17T19:47:54.928586+0000","last_undegraded":"2024-04-17T19:47:54.928586+0000","last_fullsized":"2024-04-17T19:47:54.928586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:24:52.699532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929394+0000","last_change":"2024-04-17T19:47:49.900848+0000","last_active":"2024-04-17T19:47:54.929394+0000","last_peered":"2024-04-17T19:47:54.929394+0000","last_clean":"2024-04-17T19:47:54.929394+0000","last_became_active":"2024-04-17T19:47:49.900052+0000","last_became_peered":"2024-04-17T19:47:49.900052+0000","last_unstale":"2024-04-17T19:47:54.929394+0000","last_undegraded":"2024-04-17T19:47:54.929394+0000","last_fullsized":"2024-04-17T19:47:54.929394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:28:41.115173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049711+0000","last_change":"2024-04-17T19:47:49.906531+0000","last_active":"2024-04-17T19:47:56.049711+0000","last_peered":"2024-04-17T19:47:56.049711+0000","last_clean":"2024-04-17T19:47:56.049711+0000","last_became_active":"2024-04-17T19:47:49.905217+0000","last_became_peered":"2024-04-17T19:47:49.905217+0000","last_unstale":"2024-04-17T19:47:56.049711+0000","last_undegraded":"2024-04-17T19:47:56.049711+0000","last_fullsized":"2024-04-17T19:47:56.049711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:22.078473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923726+0000","last_change":"2024-04-17T19:47:49.864577+0000","last_active":"2024-04-17T19:47:54.923726+0000","last_peered":"2024-04-17T19:47:54.923726+0000","last_clean":"2024-04-17T19:47:54.923726+0000","last_became_active":"2024-04-17T19:47:49.864251+0000","last_became_peered":"2024-04-17T19:47:49.864251+0000","last_unstale":"2024-04-17T19:47:54.923726+0000","last_undegraded":"2024-04-17T19:47:54.923726+0000","last_fullsized":"2024-04-17T19:47:54.923726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:32:10.329543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.047770+0000","last_change":"2024-04-17T19:47:49.869245+0000","last_active":"2024-04-17T19:47:56.047770+0000","last_peered":"2024-04-17T19:47:56.047770+0000","last_clean":"2024-04-17T19:47:56.047770+0000","last_became_active":"2024-04-17T19:47:49.868156+0000","last_became_peered":"2024-04-17T19:47:49.868156+0000","last_unstale":"2024-04-17T19:47:56.047770+0000","last_undegraded":"2024-04-17T19:47:56.047770+0000","last_fullsized":"2024-04-17T19:47:56.047770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:58:39.814933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.831810+0000","last_change":"2024-04-17T19:47:49.907573+0000","last_active":"2024-04-17T19:47:56.831810+0000","last_peered":"2024-04-17T19:47:56.831810+0000","last_clean":"2024-04-17T19:47:56.831810+0000","last_became_active":"2024-04-17T19:47:49.897308+0000","last_became_peered":"2024-04-17T19:47:49.897308+0000","last_unstale":"2024-04-17T19:47:56.831810+0000","last_undegraded":"2024-04-17T19:47:56.831810+0000","last_fullsized":"2024-04-17T19:47:56.831810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:02:03.101317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.047866+0000","last_change":"2024-04-17T19:47:49.876052+0000","last_active":"2024-04-17T19:47:56.047866+0000","last_peered":"2024-04-17T19:47:56.047866+0000","last_clean":"2024-04-17T19:47:56.047866+0000","last_became_active":"2024-04-17T19:47:49.875008+0000","last_became_peered":"2024-04-17T19:47:49.875008+0000","last_unstale":"2024-04-17T19:47:56.047866+0000","last_undegraded":"2024-04-17T19:47:56.047866+0000","last_fullsized":"2024-04-17T19:47:56.047866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:36:14.486150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832276+0000","last_change":"2024-04-17T19:47:49.906840+0000","last_active":"2024-04-17T19:47:56.832276+0000","last_peered":"2024-04-17T19:47:56.832276+0000","last_clean":"2024-04-17T19:47:56.832276+0000","last_became_active":"2024-04-17T19:47:49.905703+0000","last_became_peered":"2024-04-17T19:47:49.905703+0000","last_unstale":"2024-04-17T19:47:56.832276+0000","last_undegraded":"2024-04-17T19:47:56.832276+0000","last_fullsized":"2024-04-17T19:47:56.832276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:28:26.686472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841783+0000","last_change":"2024-04-17T19:47:49.869425+0000","last_active":"2024-04-17T19:47:55.841783+0000","last_peered":"2024-04-17T19:47:55.841783+0000","last_clean":"2024-04-17T19:47:55.841783+0000","last_became_active":"2024-04-17T19:47:49.868758+0000","last_became_peered":"2024-04-17T19:47:49.868758+0000","last_unstale":"2024-04-17T19:47:55.841783+0000","last_undegraded":"2024-04-17T19:47:55.841783+0000","last_fullsized":"2024-04-17T19:47:55.841783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:57:27.214150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051778+0000","last_change":"2024-04-17T19:47:49.856309+0000","last_active":"2024-04-17T19:47:55.051778+0000","last_peered":"2024-04-17T19:47:55.051778+0000","last_clean":"2024-04-17T19:47:55.051778+0000","last_became_active":"2024-04-17T19:47:49.856026+0000","last_became_peered":"2024-04-17T19:47:49.856026+0000","last_unstale":"2024-04-17T19:47:55.051778+0000","last_undegraded":"2024-04-17T19:47:55.051778+0000","last_fullsized":"2024-04-17T19:47:55.051778+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:51:57.963637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049403+0000","last_change":"2024-04-17T19:47:49.890343+0000","last_active":"2024-04-17T19:47:56.049403+0000","last_peered":"2024-04-17T19:47:56.049403+0000","last_clean":"2024-04-17T19:47:56.049403+0000","last_became_active":"2024-04-17T19:47:49.880566+0000","last_became_peered":"2024-04-17T19:47:49.880566+0000","last_unstale":"2024-04-17T19:47:56.049403+0000","last_undegraded":"2024-04-17T19:47:56.049403+0000","last_fullsized":"2024-04-17T19:47:56.049403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:19:29.359024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T19:48:13.924 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T19:48:14.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.121+0000 7fb9ba200640 1 Processor -- start 2024-04-17T19:48:14.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.121+0000 7fb9ba200640 1 -- start start 2024-04-17T19:48:14.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.121+0000 7fb9ba200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb9b40b4f20 0x7fb9b40c5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.121+0000 7fb9ba200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9b40c4720 con 0x7fb9b415d9c0 2024-04-17T19:48:14.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.121+0000 7fb9ba200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9b40cb7d0 con 0x7fb9b40cc340 2024-04-17T19:48:14.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.121+0000 7fb9ba200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb9b40ca880 con 0x7fb9b40b4f20 2024-04-17T19:48:14.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b3e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb9b415d9c0 0x7fb9b4164640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38064/0 (socket says 172.21.15.29:38064) 2024-04-17T19:48:14.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b3e00640 1 -- 172.21.15.29:0/2894660914 learned_addr learned my addr 172.21.15.29:0/2894660914 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:14.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b3400640 1 --2- 172.21.15.29:0/2894660914 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb9b40b4f20 0x7fb9b40c5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b2a00640 1 -- 172.21.15.29:0/2894660914 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1263761573 0 0) 0x7fb9ac0014a0 con 0x7fb9b415d9c0 2024-04-17T19:48:14.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b2a00640 1 -- 172.21.15.29:0/2894660914 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb998003530 con 0x7fb9b415d9c0 2024-04-17T19:48:14.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b2a00640 1 -- 172.21.15.29:0/2894660914 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1845395660 0 0) 0x7fb998003530 con 0x7fb9b415d9c0 2024-04-17T19:48:14.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b2a00640 1 -- 172.21.15.29:0/2894660914 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb9b40b4f20 msgr2=0x7fb9b40c5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:14.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b2a00640 1 --2- 172.21.15.29:0/2894660914 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb9b40b4f20 0x7fb9b40c5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b2a00640 1 -- 172.21.15.29:0/2894660914 >> v1:172.21.15.37:6789/0 conn(0x7fb9b40cc340 legacy=0x7fb9b415d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:14.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b2a00640 1 -- 172.21.15.29:0/2894660914 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b40cbab0 con 0x7fb9b415d9c0 2024-04-17T19:48:14.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b2a00640 1 -- 172.21.15.29:0/2894660914 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fb9ac001bd0 con 0x7fb9b415d9c0 2024-04-17T19:48:14.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b2a00640 1 -- 172.21.15.29:0/2894660914 >> v1:172.21.15.29:6789/0 conn(0x7fb9b415d9c0 legacy=0x7fb9b4164640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:14.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b2a00640 1 --2- 172.21.15.29:0/2894660914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb998003a60 0x7fb998003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b2a00640 1 --2- 172.21.15.29:0/2894660914 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb998004810 0x7fb998004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b2a00640 1 --2- 172.21.15.29:0/2894660914 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb9b40b4f20 0x7fb9980052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b2a00640 1 -- 172.21.15.29:0/2894660914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb99800bb60 con 0x7fb998003a60 2024-04-17T19:48:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b2a00640 1 -- 172.21.15.29:0/2894660914 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb99800bce0 con 0x7fb998004810 2024-04-17T19:48:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b2a00640 1 -- 172.21.15.29:0/2894660914 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb99800bee0 con 0x7fb9b40b4f20 2024-04-17T19:48:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b3400640 1 --2- 172.21.15.29:0/2894660914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb998003a60 0x7fb998003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9abe00640 1 --2- 172.21.15.29:0/2894660914 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb998004810 0x7fb998004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b3e00640 1 --2- 172.21.15.29:0/2894660914 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb9b40b4f20 0x7fb9980052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b3400640 1 -- 172.21.15.29:0/2894660914 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb9b40b4f20 msgr2=0x7fb9980052e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b3400640 1 --2- 172.21.15.29:0/2894660914 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb9b40b4f20 0x7fb9980052e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b3400640 1 -- 172.21.15.29:0/2894660914 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb998004810 msgr2=0x7fb998004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b3400640 1 --2- 172.21.15.29:0/2894660914 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb998004810 0x7fb998004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9b3400640 1 -- 172.21.15.29:0/2894660914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb998003530 con 0x7fb998003a60 2024-04-17T19:48:14.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.125+0000 7fb9abe00640 1 --2- 172.21.15.29:0/2894660914 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb998004810 0x7fb998004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:14.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.129+0000 7fb9b3400640 1 --2- 172.21.15.29:0/2894660914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb998003a60 0x7fb998003ed0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb9a4009ae0 tx=0x7fb9a401ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7244eb6c4c665052 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:14.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.129+0000 7fb9b2a00640 1 -- 172.21.15.29:0/2894660914 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a40096b0 con 0x7fb998003a60 2024-04-17T19:48:14.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.129+0000 7fb9b2a00640 1 -- 172.21.15.29:0/2894660914 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9a40a6470 con 0x7fb998003a60 2024-04-17T19:48:14.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.129+0000 7fb9b2a00640 1 -- 172.21.15.29:0/2894660914 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9a40a6860 con 0x7fb998003a60 2024-04-17T19:48:14.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.129+0000 7fb9ba200640 1 -- 172.21.15.29:0/2894660914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb998003a60 msgr2=0x7fb998003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:14.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.129+0000 7fb9ba200640 1 --2- 172.21.15.29:0/2894660914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb998003a60 0x7fb998003ed0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb9a4009ae0 tx=0x7fb9a401ca90 comp rx=0 tx=0).stop 2024-04-17T19:48:14.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.129+0000 7fb9b3e00640 1 -- 172.21.15.29:0/2894660914 reap_dead start 2024-04-17T19:48:14.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.129+0000 7fb9ba200640 1 -- 172.21.15.29:0/2894660914 shutdown_connections 2024-04-17T19:48:14.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.129+0000 7fb9ba200640 1 -- 172.21.15.29:0/2894660914 >> 172.21.15.29:0/2894660914 conn(0x7fb9b406e200 msgr2=0x7fb9b406e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:14.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.129+0000 7fb9ba200640 1 -- 172.21.15.29:0/2894660914 shutdown_connections 2024-04-17T19:48:14.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.129+0000 7fb9ba200640 1 -- 172.21.15.29:0/2894660914 wait complete. 2024-04-17T19:48:14.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.129+0000 7fb9ba200640 1 Processor -- start 2024-04-17T19:48:14.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.129+0000 7fb9ba200640 1 -- start start 2024-04-17T19:48:14.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9ba200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb998003a60 0x7fb9b41f6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9ba200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b406c6f0 0x7fb9b4164c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9ba200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9b414ba30 0x7fb9b414be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9ba200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b40c5ba0 con 0x7fb9b406c6f0 2024-04-17T19:48:14.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9ba200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb9b40b5330 con 0x7fb9b414ba30 2024-04-17T19:48:14.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9ba200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb9b40c5d20 con 0x7fb998003a60 2024-04-17T19:48:14.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9abe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b406c6f0 0x7fb9b4164c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9abe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b406c6f0 0x7fb9b4164c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52540/0 (socket says 172.21.15.29:52540) 2024-04-17T19:48:14.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9abe00640 1 -- 172.21.15.29:0/2746850884 learned_addr learned my addr 172.21.15.29:0/2746850884 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:14.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9b3400640 1 --2- 172.21.15.29:0/2746850884 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb998003a60 0x7fb9b41f6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9b3e00640 1 --2- 172.21.15.29:0/2746850884 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9b414ba30 0x7fb9b414be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9abe00640 1 -- 172.21.15.29:0/2746850884 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb998003a60 msgr2=0x7fb9b41f6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:14.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9abe00640 1 --2- 172.21.15.29:0/2746850884 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb998003a60 0x7fb9b41f6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9abe00640 1 -- 172.21.15.29:0/2746850884 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9b414ba30 msgr2=0x7fb9b414be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:14.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9abe00640 1 --2- 172.21.15.29:0/2746850884 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9b414ba30 0x7fb9b414be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9abe00640 1 -- 172.21.15.29:0/2746850884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9b406d040 con 0x7fb9b406c6f0 2024-04-17T19:48:14.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9abe00640 1 --2- 172.21.15.29:0/2746850884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b406c6f0 0x7fb9b4164c00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb99c00e510 tx=0x7fb99c00e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:14.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9ab400640 1 -- 172.21.15.29:0/2746850884 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c0052e0 con 0x7fb9b406c6f0 2024-04-17T19:48:14.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9ab400640 1 -- 172.21.15.29:0/2746850884 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb99c018070 con 0x7fb9b406c6f0 2024-04-17T19:48:14.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9ab400640 1 -- 172.21.15.29:0/2746850884 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99c003840 con 0x7fb9b406c6f0 2024-04-17T19:48:14.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9ba200640 1 -- 172.21.15.29:0/2746850884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9b406d2b0 con 0x7fb9b406c6f0 2024-04-17T19:48:14.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9ba200640 1 -- 172.21.15.29:0/2746850884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9b406db40 con 0x7fb9b406c6f0 2024-04-17T19:48:14.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.133+0000 7fb9ba200640 1 -- 172.21.15.29:0/2746850884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9b415d530 con 0x7fb9b406c6f0 2024-04-17T19:48:14.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.137+0000 7fb9ab400640 1 -- 172.21.15.29:0/2746850884 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb99c0039e0 con 0x7fb9b406c6f0 2024-04-17T19:48:14.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.137+0000 7fb9ab400640 1 --2- 172.21.15.29:0/2746850884 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb990068380 0x7fb99006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.137+0000 7fb9ab400640 1 -- 172.21.15.29:0/2746850884 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb99c086ac0 con 0x7fb9b406c6f0 2024-04-17T19:48:14.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.137+0000 7fb9b3400640 1 --2- 172.21.15.29:0/2746850884 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb990068380 0x7fb99006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.137+0000 7fb9b3400640 1 --2- 172.21.15.29:0/2746850884 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb990068380 0x7fb99006a890 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb9a40ad000 tx=0x7fb9a40046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:14.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.141+0000 7fb9ab400640 1 -- 172.21.15.29:0/2746850884 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb99c056ca0 con 0x7fb9b406c6f0 2024-04-17T19:48:14.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.305+0000 7fb9ba200640 1 -- 172.21.15.29:0/2746850884 --> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb9b414c380 con 0x7fb990068380 2024-04-17T19:48:14.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.329+0000 7fb9ab400640 1 -- 172.21.15.29:0/2746850884 <== mgr.4105 v2:172.21.15.37:6832/1588862286 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332558 (secure 0 0 0) 0x7fb9b414c380 con 0x7fb990068380 2024-04-17T19:48:14.321 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:14.323 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-04-17T19:48:14.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.333+0000 7fb9ba200640 1 -- 172.21.15.29:0/2746850884 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb990068380 msgr2=0x7fb99006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:14.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.333+0000 7fb9ba200640 1 --2- 172.21.15.29:0/2746850884 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb990068380 0x7fb99006a890 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb9a40ad000 tx=0x7fb9a40046d0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.333+0000 7fb9ba200640 1 -- 172.21.15.29:0/2746850884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b406c6f0 msgr2=0x7fb9b4164c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:14.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.333+0000 7fb9ba200640 1 --2- 172.21.15.29:0/2746850884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b406c6f0 0x7fb9b4164c00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb99c00e510 tx=0x7fb99c00e9d0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.333+0000 7fb9ba200640 1 -- 172.21.15.29:0/2746850884 shutdown_connections 2024-04-17T19:48:14.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.333+0000 7fb9ba200640 1 --2- 172.21.15.29:0/2746850884 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb990068380 0x7fb99006a890 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.333+0000 7fb9ba200640 1 --2- 172.21.15.29:0/2746850884 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb9b414ba30 0x7fb9b414be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.333+0000 7fb9ba200640 1 --2- 172.21.15.29:0/2746850884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b406c6f0 0x7fb9b4164c00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.333+0000 7fb9ba200640 1 --2- 172.21.15.29:0/2746850884 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb998003a60 0x7fb9b41f6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.333+0000 7fb9ba200640 1 -- 172.21.15.29:0/2746850884 >> 172.21.15.29:0/2746850884 conn(0x7fb9b406e200 msgr2=0x7fb9b4152850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:14.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.333+0000 7fb9ba200640 1 -- 172.21.15.29:0/2746850884 shutdown_connections 2024-04-17T19:48:14.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.333+0000 7fb9ba200640 1 -- 172.21.15.29:0/2746850884 wait complete. 2024-04-17T19:48:14.341 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-17T19:48:12.634241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240172,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754734548,"statfs":{"total":773094113280,"available":772848177152,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005447"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842100+0000","last_change":"2024-04-17T19:47:48.885235+0000","last_active":"2024-04-17T19:47:55.842100+0000","last_peered":"2024-04-17T19:47:55.842100+0000","last_clean":"2024-04-17T19:47:55.842100+0000","last_became_active":"2024-04-17T19:47:48.884844+0000","last_became_peered":"2024-04-17T19:47:48.884844+0000","last_unstale":"2024-04-17T19:47:55.842100+0000","last_undegraded":"2024-04-17T19:47:55.842100+0000","last_fullsized":"2024-04-17T19:47:55.842100+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:01:52.854763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924264+0000","last_change":"2024-04-17T19:47:49.872314+0000","last_active":"2024-04-17T19:47:54.924264+0000","last_peered":"2024-04-17T19:47:54.924264+0000","last_clean":"2024-04-17T19:47:54.924264+0000","last_became_active":"2024-04-17T19:47:49.871990+0000","last_became_peered":"2024-04-17T19:47:49.871990+0000","last_unstale":"2024-04-17T19:47:54.924264+0000","last_undegraded":"2024-04-17T19:47:54.924264+0000","last_fullsized":"2024-04-17T19:47:54.924264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:12:09.196610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923798+0000","last_change":"2024-04-17T19:47:53.929840+0000","last_active":"2024-04-17T19:47:54.923798+0000","last_peered":"2024-04-17T19:47:54.923798+0000","last_clean":"2024-04-17T19:47:54.923798+0000","last_became_active":"2024-04-17T19:47:53.928272+0000","last_became_peered":"2024-04-17T19:47:53.928272+0000","last_unstale":"2024-04-17T19:47:54.923798+0000","last_undegraded":"2024-04-17T19:47:54.923798+0000","last_fullsized":"2024-04-17T19:47:54.923798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:27:29.734327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924636+0000","last_change":"2024-04-17T19:47:48.862315+0000","last_active":"2024-04-17T19:47:54.924636+0000","last_peered":"2024-04-17T19:47:54.924636+0000","last_clean":"2024-04-17T19:47:54.924636+0000","last_became_active":"2024-04-17T19:47:48.860408+0000","last_became_peered":"2024-04-17T19:47:48.860408+0000","last_unstale":"2024-04-17T19:47:54.924636+0000","last_undegraded":"2024-04-17T19:47:54.924636+0000","last_fullsized":"2024-04-17T19:47:54.924636+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:10:26.336063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841873+0000","last_change":"2024-04-17T19:47:49.902423+0000","last_active":"2024-04-17T19:47:55.841873+0000","last_peered":"2024-04-17T19:47:55.841873+0000","last_clean":"2024-04-17T19:47:55.841873+0000","last_became_active":"2024-04-17T19:47:49.902096+0000","last_became_peered":"2024-04-17T19:47:49.902096+0000","last_unstale":"2024-04-17T19:47:55.841873+0000","last_undegraded":"2024-04-17T19:47:55.841873+0000","last_fullsized":"2024-04-17T19:47:55.841873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:40.805247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955520+0000","last_change":"2024-04-17T19:47:53.956851+0000","last_active":"2024-04-17T19:47:56.955520+0000","last_peered":"2024-04-17T19:47:56.955520+0000","last_clean":"2024-04-17T19:47:56.955520+0000","last_became_active":"2024-04-17T19:47:53.954197+0000","last_became_peered":"2024-04-17T19:47:53.954197+0000","last_unstale":"2024-04-17T19:47:56.955520+0000","last_undegraded":"2024-04-17T19:47:56.955520+0000","last_fullsized":"2024-04-17T19:47:56.955520+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:16:18.379831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049047+0000","last_change":"2024-04-17T19:47:48.893969+0000","last_active":"2024-04-17T19:47:56.049047+0000","last_peered":"2024-04-17T19:47:56.049047+0000","last_clean":"2024-04-17T19:47:56.049047+0000","last_became_active":"2024-04-17T19:47:48.893229+0000","last_became_peered":"2024-04-17T19:47:48.893229+0000","last_unstale":"2024-04-17T19:47:56.049047+0000","last_undegraded":"2024-04-17T19:47:56.049047+0000","last_fullsized":"2024-04-17T19:47:56.049047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:38:42.472634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832673+0000","last_change":"2024-04-17T19:47:49.897987+0000","last_active":"2024-04-17T19:47:56.832673+0000","last_peered":"2024-04-17T19:47:56.832673+0000","last_clean":"2024-04-17T19:47:56.832673+0000","last_became_active":"2024-04-17T19:47:49.897096+0000","last_became_peered":"2024-04-17T19:47:49.897096+0000","last_unstale":"2024-04-17T19:47:56.832673+0000","last_undegraded":"2024-04-17T19:47:56.832673+0000","last_fullsized":"2024-04-17T19:47:56.832673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:36:36.856454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955695+0000","last_change":"2024-04-17T19:47:53.930784+0000","last_active":"2024-04-17T19:47:56.955695+0000","last_peered":"2024-04-17T19:47:56.955695+0000","last_clean":"2024-04-17T19:47:56.955695+0000","last_became_active":"2024-04-17T19:47:53.928281+0000","last_became_peered":"2024-04-17T19:47:53.928281+0000","last_unstale":"2024-04-17T19:47:56.955695+0000","last_undegraded":"2024-04-17T19:47:56.955695+0000","last_fullsized":"2024-04-17T19:47:56.955695+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:15:55.064622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048308+0000","last_change":"2024-04-17T19:47:48.893184+0000","last_active":"2024-04-17T19:47:56.048308+0000","last_peered":"2024-04-17T19:47:56.048308+0000","last_clean":"2024-04-17T19:47:56.048308+0000","last_became_active":"2024-04-17T19:47:48.892375+0000","last_became_peered":"2024-04-17T19:47:48.892375+0000","last_unstale":"2024-04-17T19:47:56.048308+0000","last_undegraded":"2024-04-17T19:47:56.048308+0000","last_fullsized":"2024-04-17T19:47:56.048308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:56:01.211666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832916+0000","last_change":"2024-04-17T19:47:49.896040+0000","last_active":"2024-04-17T19:47:56.832916+0000","last_peered":"2024-04-17T19:47:56.832916+0000","last_clean":"2024-04-17T19:47:56.832916+0000","last_became_active":"2024-04-17T19:47:49.894779+0000","last_became_peered":"2024-04-17T19:47:49.894779+0000","last_unstale":"2024-04-17T19:47:56.832916+0000","last_undegraded":"2024-04-17T19:47:56.832916+0000","last_fullsized":"2024-04-17T19:47:56.832916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:30:17.930203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955688+0000","last_change":"2024-04-17T19:47:53.930402+0000","last_active":"2024-04-17T19:47:56.955688+0000","last_peered":"2024-04-17T19:47:56.955688+0000","last_clean":"2024-04-17T19:47:56.955688+0000","last_became_active":"2024-04-17T19:47:53.929166+0000","last_became_peered":"2024-04-17T19:47:53.929166+0000","last_unstale":"2024-04-17T19:47:56.955688+0000","last_undegraded":"2024-04-17T19:47:56.955688+0000","last_fullsized":"2024-04-17T19:47:56.955688+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:34:08.282697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927108+0000","last_change":"2024-04-17T19:47:49.900142+0000","last_active":"2024-04-17T19:47:54.927108+0000","last_peered":"2024-04-17T19:47:54.927108+0000","last_clean":"2024-04-17T19:47:54.927108+0000","last_became_active":"2024-04-17T19:47:49.899365+0000","last_became_peered":"2024-04-17T19:47:49.899365+0000","last_unstale":"2024-04-17T19:47:54.927108+0000","last_undegraded":"2024-04-17T19:47:54.927108+0000","last_fullsized":"2024-04-17T19:47:54.927108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:34:49.616539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048674+0000","last_change":"2024-04-17T19:47:48.891513+0000","last_active":"2024-04-17T19:47:56.048674+0000","last_peered":"2024-04-17T19:47:56.048674+0000","last_clean":"2024-04-17T19:47:56.048674+0000","last_became_active":"2024-04-17T19:47:48.886561+0000","last_became_peered":"2024-04-17T19:47:48.886561+0000","last_unstale":"2024-04-17T19:47:56.048674+0000","last_undegraded":"2024-04-17T19:47:56.048674+0000","last_fullsized":"2024-04-17T19:47:56.048674+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:59:14.039700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.957430+0000","last_change":"2024-04-17T19:47:53.925058+0000","last_active":"2024-04-17T19:47:56.957430+0000","last_peered":"2024-04-17T19:47:56.957430+0000","last_clean":"2024-04-17T19:47:56.957430+0000","last_became_active":"2024-04-17T19:47:53.923721+0000","last_became_peered":"2024-04-17T19:47:53.923721+0000","last_unstale":"2024-04-17T19:47:56.957430+0000","last_undegraded":"2024-04-17T19:47:56.957430+0000","last_fullsized":"2024-04-17T19:47:56.957430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:22:42.106657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924842+0000","last_change":"2024-04-17T19:47:49.852183+0000","last_active":"2024-04-17T19:47:54.924842+0000","last_peered":"2024-04-17T19:47:54.924842+0000","last_clean":"2024-04-17T19:47:54.924842+0000","last_became_active":"2024-04-17T19:47:49.851765+0000","last_became_peered":"2024-04-17T19:47:49.851765+0000","last_unstale":"2024-04-17T19:47:54.924842+0000","last_undegraded":"2024-04-17T19:47:54.924842+0000","last_fullsized":"2024-04-17T19:47:54.924842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:02:03.101317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051463+0000","last_change":"2024-04-17T19:47:48.862865+0000","last_active":"2024-04-17T19:47:55.051463+0000","last_peered":"2024-04-17T19:47:55.051463+0000","last_clean":"2024-04-17T19:47:55.051463+0000","last_became_active":"2024-04-17T19:47:48.862524+0000","last_became_peered":"2024-04-17T19:47:48.862524+0000","last_unstale":"2024-04-17T19:47:55.051463+0000","last_undegraded":"2024-04-17T19:47:55.051463+0000","last_fullsized":"2024-04-17T19:47:55.051463+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:01:21.080029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956615+0000","last_change":"2024-04-17T19:47:53.943674+0000","last_active":"2024-04-17T19:47:56.956615+0000","last_peered":"2024-04-17T19:47:56.956615+0000","last_clean":"2024-04-17T19:47:56.956615+0000","last_became_active":"2024-04-17T19:47:53.942549+0000","last_became_peered":"2024-04-17T19:47:53.942549+0000","last_unstale":"2024-04-17T19:47:56.956615+0000","last_undegraded":"2024-04-17T19:47:56.956615+0000","last_fullsized":"2024-04-17T19:47:56.956615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:00:42.300738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924513+0000","last_change":"2024-04-17T19:47:49.862312+0000","last_active":"2024-04-17T19:47:54.924513+0000","last_peered":"2024-04-17T19:47:54.924513+0000","last_clean":"2024-04-17T19:47:54.924513+0000","last_became_active":"2024-04-17T19:47:49.860664+0000","last_became_peered":"2024-04-17T19:47:49.860664+0000","last_unstale":"2024-04-17T19:47:54.924513+0000","last_undegraded":"2024-04-17T19:47:54.924513+0000","last_fullsized":"2024-04-17T19:47:54.924513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:07:47.676973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051042+0000","last_change":"2024-04-17T19:47:48.870638+0000","last_active":"2024-04-17T19:47:55.051042+0000","last_peered":"2024-04-17T19:47:55.051042+0000","last_clean":"2024-04-17T19:47:55.051042+0000","last_became_active":"2024-04-17T19:47:48.870353+0000","last_became_peered":"2024-04-17T19:47:48.870353+0000","last_unstale":"2024-04-17T19:47:55.051042+0000","last_undegraded":"2024-04-17T19:47:55.051042+0000","last_fullsized":"2024-04-17T19:47:55.051042+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:12:19.927824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.052170+0000","last_change":"2024-04-17T19:47:53.929285+0000","last_active":"2024-04-17T19:47:55.052170+0000","last_peered":"2024-04-17T19:47:55.052170+0000","last_clean":"2024-04-17T19:47:55.052170+0000","last_became_active":"2024-04-17T19:47:53.928189+0000","last_became_peered":"2024-04-17T19:47:53.928189+0000","last_unstale":"2024-04-17T19:47:55.052170+0000","last_undegraded":"2024-04-17T19:47:55.052170+0000","last_fullsized":"2024-04-17T19:47:55.052170+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:53:26.112634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956382+0000","last_change":"2024-04-17T19:47:49.888118+0000","last_active":"2024-04-17T19:47:56.956382+0000","last_peered":"2024-04-17T19:47:56.956382+0000","last_clean":"2024-04-17T19:47:56.956382+0000","last_became_active":"2024-04-17T19:47:49.884037+0000","last_became_peered":"2024-04-17T19:47:49.884037+0000","last_unstale":"2024-04-17T19:47:56.956382+0000","last_undegraded":"2024-04-17T19:47:56.956382+0000","last_fullsized":"2024-04-17T19:47:56.956382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:44:27.180380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.047604+0000","last_change":"2024-04-17T19:47:48.883686+0000","last_active":"2024-04-17T19:47:56.047604+0000","last_peered":"2024-04-17T19:47:56.047604+0000","last_clean":"2024-04-17T19:47:56.047604+0000","last_became_active":"2024-04-17T19:47:48.882375+0000","last_became_peered":"2024-04-17T19:47:48.882375+0000","last_unstale":"2024-04-17T19:47:56.047604+0000","last_undegraded":"2024-04-17T19:47:56.047604+0000","last_fullsized":"2024-04-17T19:47:56.047604+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:38:31.225374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.957175+0000","last_change":"2024-04-17T19:47:53.935396+0000","last_active":"2024-04-17T19:47:56.957175+0000","last_peered":"2024-04-17T19:47:56.957175+0000","last_clean":"2024-04-17T19:47:56.957175+0000","last_became_active":"2024-04-17T19:47:53.934094+0000","last_became_peered":"2024-04-17T19:47:53.934094+0000","last_unstale":"2024-04-17T19:47:56.957175+0000","last_undegraded":"2024-04-17T19:47:56.957175+0000","last_fullsized":"2024-04-17T19:47:56.957175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:58.334357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842041+0000","last_change":"2024-04-17T19:47:49.870297+0000","last_active":"2024-04-17T19:47:55.842041+0000","last_peered":"2024-04-17T19:47:55.842041+0000","last_clean":"2024-04-17T19:47:55.842041+0000","last_became_active":"2024-04-17T19:47:49.870113+0000","last_became_peered":"2024-04-17T19:47:49.870113+0000","last_unstale":"2024-04-17T19:47:55.842041+0000","last_undegraded":"2024-04-17T19:47:55.842041+0000","last_fullsized":"2024-04-17T19:47:55.842041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:50:54.517273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929723+0000","last_change":"2024-04-17T19:47:48.895905+0000","last_active":"2024-04-17T19:47:54.929723+0000","last_peered":"2024-04-17T19:47:54.929723+0000","last_clean":"2024-04-17T19:47:54.929723+0000","last_became_active":"2024-04-17T19:47:48.895263+0000","last_became_peered":"2024-04-17T19:47:48.895263+0000","last_unstale":"2024-04-17T19:47:54.929723+0000","last_undegraded":"2024-04-17T19:47:54.929723+0000","last_fullsized":"2024-04-17T19:47:54.929723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:07:23.179031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956013+0000","last_change":"2024-04-17T19:47:53.943286+0000","last_active":"2024-04-17T19:47:56.956013+0000","last_peered":"2024-04-17T19:47:56.956013+0000","last_clean":"2024-04-17T19:47:56.956013+0000","last_became_active":"2024-04-17T19:47:53.942325+0000","last_became_peered":"2024-04-17T19:47:53.942325+0000","last_unstale":"2024-04-17T19:47:56.956013+0000","last_undegraded":"2024-04-17T19:47:56.956013+0000","last_fullsized":"2024-04-17T19:47:56.956013+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:47:18.853991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924723+0000","last_change":"2024-04-17T19:47:49.872925+0000","last_active":"2024-04-17T19:47:54.924723+0000","last_peered":"2024-04-17T19:47:54.924723+0000","last_clean":"2024-04-17T19:47:54.924723+0000","last_became_active":"2024-04-17T19:47:49.872668+0000","last_became_peered":"2024-04-17T19:47:49.872668+0000","last_unstale":"2024-04-17T19:47:54.924723+0000","last_undegraded":"2024-04-17T19:47:54.924723+0000","last_fullsized":"2024-04-17T19:47:54.924723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:16:21.209634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051807+0000","last_change":"2024-04-17T19:47:48.896669+0000","last_active":"2024-04-17T19:47:55.051807+0000","last_peered":"2024-04-17T19:47:55.051807+0000","last_clean":"2024-04-17T19:47:55.051807+0000","last_became_active":"2024-04-17T19:47:48.896397+0000","last_became_peered":"2024-04-17T19:47:48.896397+0000","last_unstale":"2024-04-17T19:47:55.051807+0000","last_undegraded":"2024-04-17T19:47:55.051807+0000","last_fullsized":"2024-04-17T19:47:55.051807+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:07:23.179031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923702+0000","last_change":"2024-04-17T19:47:53.906261+0000","last_active":"2024-04-17T19:47:54.923702+0000","last_peered":"2024-04-17T19:47:54.923702+0000","last_clean":"2024-04-17T19:47:54.923702+0000","last_became_active":"2024-04-17T19:47:53.903702+0000","last_became_peered":"2024-04-17T19:47:53.903702+0000","last_unstale":"2024-04-17T19:47:54.923702+0000","last_undegraded":"2024-04-17T19:47:54.923702+0000","last_fullsized":"2024-04-17T19:47:54.923702+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:35:53.892917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051477+0000","last_change":"2024-04-17T19:47:49.870272+0000","last_active":"2024-04-17T19:47:55.051477+0000","last_peered":"2024-04-17T19:47:55.051477+0000","last_clean":"2024-04-17T19:47:55.051477+0000","last_became_active":"2024-04-17T19:47:49.869914+0000","last_became_peered":"2024-04-17T19:47:49.869914+0000","last_unstale":"2024-04-17T19:47:55.051477+0000","last_undegraded":"2024-04-17T19:47:55.051477+0000","last_fullsized":"2024-04-17T19:47:55.051477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:44:30.553286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049327+0000","last_change":"2024-04-17T19:47:48.892675+0000","last_active":"2024-04-17T19:47:56.049327+0000","last_peered":"2024-04-17T19:47:56.049327+0000","last_clean":"2024-04-17T19:47:56.049327+0000","last_became_active":"2024-04-17T19:47:48.892057+0000","last_became_peered":"2024-04-17T19:47:48.892057+0000","last_unstale":"2024-04-17T19:47:56.049327+0000","last_undegraded":"2024-04-17T19:47:56.049327+0000","last_fullsized":"2024-04-17T19:47:56.049327+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:18:35.735350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841981+0000","last_change":"2024-04-17T19:47:53.925135+0000","last_active":"2024-04-17T19:47:55.841981+0000","last_peered":"2024-04-17T19:47:55.841981+0000","last_clean":"2024-04-17T19:47:55.841981+0000","last_became_active":"2024-04-17T19:47:53.924617+0000","last_became_peered":"2024-04-17T19:47:53.924617+0000","last_unstale":"2024-04-17T19:47:55.841981+0000","last_undegraded":"2024-04-17T19:47:55.841981+0000","last_fullsized":"2024-04-17T19:47:55.841981+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:37:02.082648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.833132+0000","last_change":"2024-04-17T19:47:49.895815+0000","last_active":"2024-04-17T19:47:56.833132+0000","last_peered":"2024-04-17T19:47:56.833132+0000","last_clean":"2024-04-17T19:47:56.833132+0000","last_became_active":"2024-04-17T19:47:49.894978+0000","last_became_peered":"2024-04-17T19:47:49.894978+0000","last_unstale":"2024-04-17T19:47:56.833132+0000","last_undegraded":"2024-04-17T19:47:56.833132+0000","last_fullsized":"2024-04-17T19:47:56.833132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:32:12.184723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051589+0000","last_change":"2024-04-17T19:47:48.859247+0000","last_active":"2024-04-17T19:47:55.051589+0000","last_peered":"2024-04-17T19:47:55.051589+0000","last_clean":"2024-04-17T19:47:55.051589+0000","last_became_active":"2024-04-17T19:47:48.858756+0000","last_became_peered":"2024-04-17T19:47:48.858756+0000","last_unstale":"2024-04-17T19:47:55.051589+0000","last_undegraded":"2024-04-17T19:47:55.051589+0000","last_fullsized":"2024-04-17T19:47:55.051589+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:19:56.987957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051118+0000","last_change":"2024-04-17T19:47:53.929952+0000","last_active":"2024-04-17T19:47:55.051118+0000","last_peered":"2024-04-17T19:47:55.051118+0000","last_clean":"2024-04-17T19:47:55.051118+0000","last_became_active":"2024-04-17T19:47:53.929528+0000","last_became_peered":"2024-04-17T19:47:53.929528+0000","last_unstale":"2024-04-17T19:47:55.051118+0000","last_undegraded":"2024-04-17T19:47:55.051118+0000","last_fullsized":"2024-04-17T19:47:55.051118+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:11:38.041301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956659+0000","last_change":"2024-04-17T19:47:49.874150+0000","last_active":"2024-04-17T19:47:56.956659+0000","last_peered":"2024-04-17T19:47:56.956659+0000","last_clean":"2024-04-17T19:47:56.956659+0000","last_became_active":"2024-04-17T19:47:49.872859+0000","last_became_peered":"2024-04-17T19:47:49.872859+0000","last_unstale":"2024-04-17T19:47:56.956659+0000","last_undegraded":"2024-04-17T19:47:56.956659+0000","last_fullsized":"2024-04-17T19:47:56.956659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:57:54.666279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832613+0000","last_change":"2024-04-17T19:47:48.876672+0000","last_active":"2024-04-17T19:47:56.832613+0000","last_peered":"2024-04-17T19:47:56.832613+0000","last_clean":"2024-04-17T19:47:56.832613+0000","last_became_active":"2024-04-17T19:47:48.875263+0000","last_became_peered":"2024-04-17T19:47:48.875263+0000","last_unstale":"2024-04-17T19:47:56.832613+0000","last_undegraded":"2024-04-17T19:47:56.832613+0000","last_fullsized":"2024-04-17T19:47:56.832613+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:45:46.633540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051584+0000","last_change":"2024-04-17T19:47:53.932330+0000","last_active":"2024-04-17T19:47:55.051584+0000","last_peered":"2024-04-17T19:47:55.051584+0000","last_clean":"2024-04-17T19:47:55.051584+0000","last_became_active":"2024-04-17T19:47:53.931873+0000","last_became_peered":"2024-04-17T19:47:53.931873+0000","last_unstale":"2024-04-17T19:47:55.051584+0000","last_undegraded":"2024-04-17T19:47:55.051584+0000","last_fullsized":"2024-04-17T19:47:55.051584+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:05:42.346878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051044+0000","last_change":"2024-04-17T19:47:49.865020+0000","last_active":"2024-04-17T19:47:55.051044+0000","last_peered":"2024-04-17T19:47:55.051044+0000","last_clean":"2024-04-17T19:47:55.051044+0000","last_became_active":"2024-04-17T19:47:49.864764+0000","last_became_peered":"2024-04-17T19:47:49.864764+0000","last_unstale":"2024-04-17T19:47:55.051044+0000","last_undegraded":"2024-04-17T19:47:55.051044+0000","last_fullsized":"2024-04-17T19:47:55.051044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:45:36.325765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926588+0000","last_change":"2024-04-17T19:47:48.846570+0000","last_active":"2024-04-17T19:47:54.926588+0000","last_peered":"2024-04-17T19:47:54.926588+0000","last_clean":"2024-04-17T19:47:54.926588+0000","last_became_active":"2024-04-17T19:47:48.846175+0000","last_became_peered":"2024-04-17T19:47:48.846175+0000","last_unstale":"2024-04-17T19:47:54.926588+0000","last_undegraded":"2024-04-17T19:47:54.926588+0000","last_fullsized":"2024-04-17T19:47:54.926588+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:20:22.737016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.052346+0000","last_change":"2024-04-17T19:47:53.904590+0000","last_active":"2024-04-17T19:47:55.052346+0000","last_peered":"2024-04-17T19:47:55.052346+0000","last_clean":"2024-04-17T19:47:55.052346+0000","last_became_active":"2024-04-17T19:47:53.904026+0000","last_became_peered":"2024-04-17T19:47:53.904026+0000","last_unstale":"2024-04-17T19:47:55.052346+0000","last_undegraded":"2024-04-17T19:47:55.052346+0000","last_fullsized":"2024-04-17T19:47:55.052346+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:16:22.837570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841469+0000","last_change":"2024-04-17T19:47:49.860010+0000","last_active":"2024-04-17T19:47:55.841469+0000","last_peered":"2024-04-17T19:47:55.841469+0000","last_clean":"2024-04-17T19:47:55.841469+0000","last_became_active":"2024-04-17T19:47:49.859593+0000","last_became_peered":"2024-04-17T19:47:49.859593+0000","last_unstale":"2024-04-17T19:47:55.841469+0000","last_undegraded":"2024-04-17T19:47:55.841469+0000","last_fullsized":"2024-04-17T19:47:55.841469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:33:24.594397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.047579+0000","last_change":"2024-04-17T19:47:48.883126+0000","last_active":"2024-04-17T19:47:56.047579+0000","last_peered":"2024-04-17T19:47:56.047579+0000","last_clean":"2024-04-17T19:47:56.047579+0000","last_became_active":"2024-04-17T19:47:48.882173+0000","last_became_peered":"2024-04-17T19:47:48.882173+0000","last_unstale":"2024-04-17T19:47:56.047579+0000","last_undegraded":"2024-04-17T19:47:56.047579+0000","last_fullsized":"2024-04-17T19:47:56.047579+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:11:59.395316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841972+0000","last_change":"2024-04-17T19:47:53.931258+0000","last_active":"2024-04-17T19:47:55.841972+0000","last_peered":"2024-04-17T19:47:55.841972+0000","last_clean":"2024-04-17T19:47:55.841972+0000","last_became_active":"2024-04-17T19:47:53.930773+0000","last_became_peered":"2024-04-17T19:47:53.930773+0000","last_unstale":"2024-04-17T19:47:55.841972+0000","last_undegraded":"2024-04-17T19:47:55.841972+0000","last_fullsized":"2024-04-17T19:47:55.841972+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:10:47.583714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924520+0000","last_change":"2024-04-17T19:47:49.909404+0000","last_active":"2024-04-17T19:47:54.924520+0000","last_peered":"2024-04-17T19:47:54.924520+0000","last_clean":"2024-04-17T19:47:54.924520+0000","last_became_active":"2024-04-17T19:47:49.909154+0000","last_became_peered":"2024-04-17T19:47:49.909154+0000","last_unstale":"2024-04-17T19:47:54.924520+0000","last_undegraded":"2024-04-17T19:47:54.924520+0000","last_fullsized":"2024-04-17T19:47:54.924520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:14:39.682888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832291+0000","last_change":"2024-04-17T19:47:48.878592+0000","last_active":"2024-04-17T19:47:56.832291+0000","last_peered":"2024-04-17T19:47:56.832291+0000","last_clean":"2024-04-17T19:47:56.832291+0000","last_became_active":"2024-04-17T19:47:48.877617+0000","last_became_peered":"2024-04-17T19:47:48.877617+0000","last_unstale":"2024-04-17T19:47:56.832291+0000","last_undegraded":"2024-04-17T19:47:56.832291+0000","last_fullsized":"2024-04-17T19:47:56.832291+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:30:17.013176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048011+0000","last_change":"2024-04-17T19:47:38.277798+0000","last_active":"2024-04-17T19:47:56.048011+0000","last_peered":"2024-04-17T19:47:56.048011+0000","last_clean":"2024-04-17T19:47:56.048011+0000","last_became_active":"2024-04-17T19:47:38.274697+0000","last_became_peered":"2024-04-17T19:47:38.274697+0000","last_unstale":"2024-04-17T19:47:56.048011+0000","last_undegraded":"2024-04-17T19:47:56.048011+0000","last_fullsized":"2024-04-17T19:47:56.048011+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:36.897071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:36.897071+0000","last_clean_scrub_stamp":"2024-04-17T19:47:36.897071+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:31:35.366600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929077+0000","last_change":"2024-04-17T19:47:53.953415+0000","last_active":"2024-04-17T19:47:54.929077+0000","last_peered":"2024-04-17T19:47:54.929077+0000","last_clean":"2024-04-17T19:47:54.929077+0000","last_became_active":"2024-04-17T19:47:53.952644+0000","last_became_peered":"2024-04-17T19:47:53.952644+0000","last_unstale":"2024-04-17T19:47:54.929077+0000","last_undegraded":"2024-04-17T19:47:54.929077+0000","last_fullsized":"2024-04-17T19:47:54.929077+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:00:25.923909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048406+0000","last_change":"2024-04-17T19:47:49.901572+0000","last_active":"2024-04-17T19:47:56.048406+0000","last_peered":"2024-04-17T19:47:56.048406+0000","last_clean":"2024-04-17T19:47:56.048406+0000","last_became_active":"2024-04-17T19:47:49.900707+0000","last_became_peered":"2024-04-17T19:47:49.900707+0000","last_unstale":"2024-04-17T19:47:56.048406+0000","last_undegraded":"2024-04-17T19:47:56.048406+0000","last_fullsized":"2024-04-17T19:47:56.048406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:58:02.897564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926753+0000","last_change":"2024-04-17T19:47:48.874721+0000","last_active":"2024-04-17T19:47:54.926753+0000","last_peered":"2024-04-17T19:47:54.926753+0000","last_clean":"2024-04-17T19:47:54.926753+0000","last_became_active":"2024-04-17T19:47:48.874364+0000","last_became_peered":"2024-04-17T19:47:48.874364+0000","last_unstale":"2024-04-17T19:47:54.926753+0000","last_undegraded":"2024-04-17T19:47:54.926753+0000","last_fullsized":"2024-04-17T19:47:54.926753+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:47:08.856753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927738+0000","last_change":"2024-04-17T19:47:53.942791+0000","last_active":"2024-04-17T19:47:54.927738+0000","last_peered":"2024-04-17T19:47:54.927738+0000","last_clean":"2024-04-17T19:47:54.927738+0000","last_became_active":"2024-04-17T19:47:53.941569+0000","last_became_peered":"2024-04-17T19:47:53.941569+0000","last_unstale":"2024-04-17T19:47:54.927738+0000","last_undegraded":"2024-04-17T19:47:54.927738+0000","last_fullsized":"2024-04-17T19:47:54.927738+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:23:35.452056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832644+0000","last_change":"2024-04-17T19:47:49.895046+0000","last_active":"2024-04-17T19:47:56.832644+0000","last_peered":"2024-04-17T19:47:56.832644+0000","last_clean":"2024-04-17T19:47:56.832644+0000","last_became_active":"2024-04-17T19:47:49.894250+0000","last_became_peered":"2024-04-17T19:47:49.894250+0000","last_unstale":"2024-04-17T19:47:56.832644+0000","last_undegraded":"2024-04-17T19:47:56.832644+0000","last_fullsized":"2024-04-17T19:47:56.832644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:30:17.210135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051734+0000","last_change":"2024-04-17T19:47:48.861802+0000","last_active":"2024-04-17T19:47:55.051734+0000","last_peered":"2024-04-17T19:47:55.051734+0000","last_clean":"2024-04-17T19:47:55.051734+0000","last_became_active":"2024-04-17T19:47:48.860418+0000","last_became_peered":"2024-04-17T19:47:48.860418+0000","last_unstale":"2024-04-17T19:47:55.051734+0000","last_undegraded":"2024-04-17T19:47:55.051734+0000","last_fullsized":"2024-04-17T19:47:55.051734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:54:41.949746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842288+0000","last_change":"2024-04-17T19:47:53.917732+0000","last_active":"2024-04-17T19:47:55.842288+0000","last_peered":"2024-04-17T19:47:55.842288+0000","last_clean":"2024-04-17T19:47:55.842288+0000","last_became_active":"2024-04-17T19:47:53.917160+0000","last_became_peered":"2024-04-17T19:47:53.917160+0000","last_unstale":"2024-04-17T19:47:55.842288+0000","last_undegraded":"2024-04-17T19:47:55.842288+0000","last_fullsized":"2024-04-17T19:47:55.842288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:19:20.394166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049773+0000","last_change":"2024-04-17T19:47:49.893077+0000","last_active":"2024-04-17T19:47:56.049773+0000","last_peered":"2024-04-17T19:47:56.049773+0000","last_clean":"2024-04-17T19:47:56.049773+0000","last_became_active":"2024-04-17T19:47:49.892280+0000","last_became_peered":"2024-04-17T19:47:49.892280+0000","last_unstale":"2024-04-17T19:47:56.049773+0000","last_undegraded":"2024-04-17T19:47:56.049773+0000","last_fullsized":"2024-04-17T19:47:56.049773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:58:24.218821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926706+0000","last_change":"2024-04-17T19:47:48.849027+0000","last_active":"2024-04-17T19:47:54.926706+0000","last_peered":"2024-04-17T19:47:54.926706+0000","last_clean":"2024-04-17T19:47:54.926706+0000","last_became_active":"2024-04-17T19:47:48.848485+0000","last_became_peered":"2024-04-17T19:47:48.848485+0000","last_unstale":"2024-04-17T19:47:54.926706+0000","last_undegraded":"2024-04-17T19:47:54.926706+0000","last_fullsized":"2024-04-17T19:47:54.926706+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:58:14.548891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.831438+0000","last_change":"2024-04-17T19:47:53.946933+0000","last_active":"2024-04-17T19:47:56.831438+0000","last_peered":"2024-04-17T19:47:56.831438+0000","last_clean":"2024-04-17T19:47:56.831438+0000","last_became_active":"2024-04-17T19:47:53.943030+0000","last_became_peered":"2024-04-17T19:47:53.943030+0000","last_unstale":"2024-04-17T19:47:56.831438+0000","last_undegraded":"2024-04-17T19:47:56.831438+0000","last_fullsized":"2024-04-17T19:47:56.831438+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:37:03.489250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049281+0000","last_change":"2024-04-17T19:47:49.893137+0000","last_active":"2024-04-17T19:47:56.049281+0000","last_peered":"2024-04-17T19:47:56.049281+0000","last_clean":"2024-04-17T19:47:56.049281+0000","last_became_active":"2024-04-17T19:47:49.892379+0000","last_became_peered":"2024-04-17T19:47:49.892379+0000","last_unstale":"2024-04-17T19:47:56.049281+0000","last_undegraded":"2024-04-17T19:47:56.049281+0000","last_fullsized":"2024-04-17T19:47:56.049281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:55:08.562395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.954554+0000","last_change":"2024-04-17T19:47:48.861154+0000","last_active":"2024-04-17T19:47:56.954554+0000","last_peered":"2024-04-17T19:47:56.954554+0000","last_clean":"2024-04-17T19:47:56.954554+0000","last_became_active":"2024-04-17T19:47:48.859268+0000","last_became_peered":"2024-04-17T19:47:48.859268+0000","last_unstale":"2024-04-17T19:47:56.954554+0000","last_undegraded":"2024-04-17T19:47:56.954554+0000","last_fullsized":"2024-04-17T19:47:56.954554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:04:19.867115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955200+0000","last_change":"2024-04-17T19:47:53.950580+0000","last_active":"2024-04-17T19:47:56.955200+0000","last_peered":"2024-04-17T19:47:56.955200+0000","last_clean":"2024-04-17T19:47:56.955200+0000","last_became_active":"2024-04-17T19:47:53.943798+0000","last_became_peered":"2024-04-17T19:47:53.943798+0000","last_unstale":"2024-04-17T19:47:56.955200+0000","last_undegraded":"2024-04-17T19:47:56.955200+0000","last_fullsized":"2024-04-17T19:47:56.955200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:52:14.763799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048020+0000","last_change":"2024-04-17T19:47:49.893774+0000","last_active":"2024-04-17T19:47:56.048020+0000","last_peered":"2024-04-17T19:47:56.048020+0000","last_clean":"2024-04-17T19:47:56.048020+0000","last_became_active":"2024-04-17T19:47:49.892780+0000","last_became_peered":"2024-04-17T19:47:49.892780+0000","last_unstale":"2024-04-17T19:47:56.048020+0000","last_undegraded":"2024-04-17T19:47:56.048020+0000","last_fullsized":"2024-04-17T19:47:56.048020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:50:47.658650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"23'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926204+0000","last_change":"2024-04-17T19:47:48.852970+0000","last_active":"2024-04-17T19:47:54.926204+0000","last_peered":"2024-04-17T19:47:54.926204+0000","last_clean":"2024-04-17T19:47:54.926204+0000","last_became_active":"2024-04-17T19:47:48.852676+0000","last_became_peered":"2024-04-17T19:47:48.852676+0000","last_unstale":"2024-04-17T19:47:54.926204+0000","last_undegraded":"2024-04-17T19:47:54.926204+0000","last_fullsized":"2024-04-17T19:47:54.926204+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:32:14.529596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841883+0000","last_change":"2024-04-17T19:47:53.915397+0000","last_active":"2024-04-17T19:47:55.841883+0000","last_peered":"2024-04-17T19:47:55.841883+0000","last_clean":"2024-04-17T19:47:55.841883+0000","last_became_active":"2024-04-17T19:47:53.914913+0000","last_became_peered":"2024-04-17T19:47:53.914913+0000","last_unstale":"2024-04-17T19:47:55.841883+0000","last_undegraded":"2024-04-17T19:47:55.841883+0000","last_fullsized":"2024-04-17T19:47:55.841883+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:02:01.175158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927357+0000","last_change":"2024-04-17T19:47:49.880484+0000","last_active":"2024-04-17T19:47:54.927357+0000","last_peered":"2024-04-17T19:47:54.927357+0000","last_clean":"2024-04-17T19:47:54.927357+0000","last_became_active":"2024-04-17T19:47:49.879625+0000","last_became_peered":"2024-04-17T19:47:49.879625+0000","last_unstale":"2024-04-17T19:47:54.927357+0000","last_undegraded":"2024-04-17T19:47:54.927357+0000","last_fullsized":"2024-04-17T19:47:54.927357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:50:00.611796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927417+0000","last_change":"2024-04-17T19:47:48.899160+0000","last_active":"2024-04-17T19:47:54.927417+0000","last_peered":"2024-04-17T19:47:54.927417+0000","last_clean":"2024-04-17T19:47:54.927417+0000","last_became_active":"2024-04-17T19:47:48.889831+0000","last_became_peered":"2024-04-17T19:47:48.889831+0000","last_unstale":"2024-04-17T19:47:54.927417+0000","last_undegraded":"2024-04-17T19:47:54.927417+0000","last_fullsized":"2024-04-17T19:47:54.927417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:55:20.167510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924391+0000","last_change":"2024-04-17T19:47:53.921093+0000","last_active":"2024-04-17T19:47:54.924391+0000","last_peered":"2024-04-17T19:47:54.924391+0000","last_clean":"2024-04-17T19:47:54.924391+0000","last_became_active":"2024-04-17T19:47:53.920479+0000","last_became_peered":"2024-04-17T19:47:53.920479+0000","last_unstale":"2024-04-17T19:47:54.924391+0000","last_undegraded":"2024-04-17T19:47:54.924391+0000","last_fullsized":"2024-04-17T19:47:54.924391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:08:09.569820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048418+0000","last_change":"2024-04-17T19:47:49.895275+0000","last_active":"2024-04-17T19:47:56.048418+0000","last_peered":"2024-04-17T19:47:56.048418+0000","last_clean":"2024-04-17T19:47:56.048418+0000","last_became_active":"2024-04-17T19:47:49.894534+0000","last_became_peered":"2024-04-17T19:47:49.894534+0000","last_unstale":"2024-04-17T19:47:56.048418+0000","last_undegraded":"2024-04-17T19:47:56.048418+0000","last_fullsized":"2024-04-17T19:47:56.048418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:31:40.818807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832491+0000","last_change":"2024-04-17T19:47:48.881715+0000","last_active":"2024-04-17T19:47:56.832491+0000","last_peered":"2024-04-17T19:47:56.832491+0000","last_clean":"2024-04-17T19:47:56.832491+0000","last_became_active":"2024-04-17T19:47:48.876203+0000","last_became_peered":"2024-04-17T19:47:48.876203+0000","last_unstale":"2024-04-17T19:47:56.832491+0000","last_undegraded":"2024-04-17T19:47:56.832491+0000","last_fullsized":"2024-04-17T19:47:56.832491+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:03:16.656528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924597+0000","last_change":"2024-04-17T19:47:53.905149+0000","last_active":"2024-04-17T19:47:54.924597+0000","last_peered":"2024-04-17T19:47:54.924597+0000","last_clean":"2024-04-17T19:47:54.924597+0000","last_became_active":"2024-04-17T19:47:53.904718+0000","last_became_peered":"2024-04-17T19:47:53.904718+0000","last_unstale":"2024-04-17T19:47:54.924597+0000","last_undegraded":"2024-04-17T19:47:54.924597+0000","last_fullsized":"2024-04-17T19:47:54.924597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:09:31.908752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841674+0000","last_change":"2024-04-17T19:47:49.875819+0000","last_active":"2024-04-17T19:47:55.841674+0000","last_peered":"2024-04-17T19:47:55.841674+0000","last_clean":"2024-04-17T19:47:55.841674+0000","last_became_active":"2024-04-17T19:47:49.872075+0000","last_became_peered":"2024-04-17T19:47:49.872075+0000","last_unstale":"2024-04-17T19:47:55.841674+0000","last_undegraded":"2024-04-17T19:47:55.841674+0000","last_fullsized":"2024-04-17T19:47:55.841674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:06:48.823135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842429+0000","last_change":"2024-04-17T19:47:48.867633+0000","last_active":"2024-04-17T19:47:55.842429+0000","last_peered":"2024-04-17T19:47:55.842429+0000","last_clean":"2024-04-17T19:47:55.842429+0000","last_became_active":"2024-04-17T19:47:48.867150+0000","last_became_peered":"2024-04-17T19:47:48.867150+0000","last_unstale":"2024-04-17T19:47:55.842429+0000","last_undegraded":"2024-04-17T19:47:55.842429+0000","last_fullsized":"2024-04-17T19:47:55.842429+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:33:34.275816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832910+0000","last_change":"2024-04-17T19:47:53.925785+0000","last_active":"2024-04-17T19:47:56.832910+0000","last_peered":"2024-04-17T19:47:56.832910+0000","last_clean":"2024-04-17T19:47:56.832910+0000","last_became_active":"2024-04-17T19:47:53.924006+0000","last_became_peered":"2024-04-17T19:47:53.924006+0000","last_unstale":"2024-04-17T19:47:56.832910+0000","last_undegraded":"2024-04-17T19:47:56.832910+0000","last_fullsized":"2024-04-17T19:47:56.832910+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:46:31.965271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048194+0000","last_change":"2024-04-17T19:47:49.869829+0000","last_active":"2024-04-17T19:47:56.048194+0000","last_peered":"2024-04-17T19:47:56.048194+0000","last_clean":"2024-04-17T19:47:56.048194+0000","last_became_active":"2024-04-17T19:47:49.868933+0000","last_became_peered":"2024-04-17T19:47:49.868933+0000","last_unstale":"2024-04-17T19:47:56.048194+0000","last_undegraded":"2024-04-17T19:47:56.048194+0000","last_fullsized":"2024-04-17T19:47:56.048194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:36:08.866159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:03.662022+0000","last_change":"2024-04-17T19:47:48.887461+0000","last_active":"2024-04-17T19:48:03.662022+0000","last_peered":"2024-04-17T19:48:03.662022+0000","last_clean":"2024-04-17T19:48:03.662022+0000","last_became_active":"2024-04-17T19:47:48.886689+0000","last_became_peered":"2024-04-17T19:47:48.886689+0000","last_unstale":"2024-04-17T19:48:03.662022+0000","last_undegraded":"2024-04-17T19:48:03.662022+0000","last_fullsized":"2024-04-17T19:48:03.662022+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:55:35.262995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051405+0000","last_change":"2024-04-17T19:47:53.941607+0000","last_active":"2024-04-17T19:47:55.051405+0000","last_peered":"2024-04-17T19:47:55.051405+0000","last_clean":"2024-04-17T19:47:55.051405+0000","last_became_active":"2024-04-17T19:47:53.941050+0000","last_became_peered":"2024-04-17T19:47:53.941050+0000","last_unstale":"2024-04-17T19:47:55.051405+0000","last_undegraded":"2024-04-17T19:47:55.051405+0000","last_fullsized":"2024-04-17T19:47:55.051405+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:26:08.964058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.957037+0000","last_change":"2024-04-17T19:47:49.879524+0000","last_active":"2024-04-17T19:47:56.957037+0000","last_peered":"2024-04-17T19:47:56.957037+0000","last_clean":"2024-04-17T19:47:56.957037+0000","last_became_active":"2024-04-17T19:47:49.877563+0000","last_became_peered":"2024-04-17T19:47:49.877563+0000","last_unstale":"2024-04-17T19:47:56.957037+0000","last_undegraded":"2024-04-17T19:47:56.957037+0000","last_fullsized":"2024-04-17T19:47:56.957037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:34:58.539940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"23'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051676+0000","last_change":"2024-04-17T19:47:48.870100+0000","last_active":"2024-04-17T19:47:55.051676+0000","last_peered":"2024-04-17T19:47:55.051676+0000","last_clean":"2024-04-17T19:47:55.051676+0000","last_became_active":"2024-04-17T19:47:48.869797+0000","last_became_peered":"2024-04-17T19:47:48.869797+0000","last_unstale":"2024-04-17T19:47:55.051676+0000","last_undegraded":"2024-04-17T19:47:55.051676+0000","last_fullsized":"2024-04-17T19:47:55.051676+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:51:21.463081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956297+0000","last_change":"2024-04-17T19:47:53.944568+0000","last_active":"2024-04-17T19:47:56.956297+0000","last_peered":"2024-04-17T19:47:56.956297+0000","last_clean":"2024-04-17T19:47:56.956297+0000","last_became_active":"2024-04-17T19:47:53.943151+0000","last_became_peered":"2024-04-17T19:47:53.943151+0000","last_unstale":"2024-04-17T19:47:56.956297+0000","last_undegraded":"2024-04-17T19:47:56.956297+0000","last_fullsized":"2024-04-17T19:47:56.956297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:19:00.150450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842029+0000","last_change":"2024-04-17T19:47:49.902401+0000","last_active":"2024-04-17T19:47:55.842029+0000","last_peered":"2024-04-17T19:47:55.842029+0000","last_clean":"2024-04-17T19:47:55.842029+0000","last_became_active":"2024-04-17T19:47:49.902166+0000","last_became_peered":"2024-04-17T19:47:49.902166+0000","last_unstale":"2024-04-17T19:47:55.842029+0000","last_undegraded":"2024-04-17T19:47:55.842029+0000","last_fullsized":"2024-04-17T19:47:55.842029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:54:13.129748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832891+0000","last_change":"2024-04-17T19:47:48.900800+0000","last_active":"2024-04-17T19:47:56.832891+0000","last_peered":"2024-04-17T19:47:56.832891+0000","last_clean":"2024-04-17T19:47:56.832891+0000","last_became_active":"2024-04-17T19:47:48.900188+0000","last_became_peered":"2024-04-17T19:47:48.900188+0000","last_unstale":"2024-04-17T19:47:56.832891+0000","last_undegraded":"2024-04-17T19:47:56.832891+0000","last_fullsized":"2024-04-17T19:47:56.832891+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:05:04.305871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923962+0000","last_change":"2024-04-17T19:47:53.928721+0000","last_active":"2024-04-17T19:47:54.923962+0000","last_peered":"2024-04-17T19:47:54.923962+0000","last_clean":"2024-04-17T19:47:54.923962+0000","last_became_active":"2024-04-17T19:47:53.928311+0000","last_became_peered":"2024-04-17T19:47:53.928311+0000","last_unstale":"2024-04-17T19:47:54.923962+0000","last_undegraded":"2024-04-17T19:47:54.923962+0000","last_fullsized":"2024-04-17T19:47:54.923962+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:35:28.870803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832516+0000","last_change":"2024-04-17T19:47:49.905421+0000","last_active":"2024-04-17T19:47:56.832516+0000","last_peered":"2024-04-17T19:47:56.832516+0000","last_clean":"2024-04-17T19:47:56.832516+0000","last_became_active":"2024-04-17T19:47:49.904646+0000","last_became_peered":"2024-04-17T19:47:49.904646+0000","last_unstale":"2024-04-17T19:47:56.832516+0000","last_undegraded":"2024-04-17T19:47:56.832516+0000","last_fullsized":"2024-04-17T19:47:56.832516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:02:23.936080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929405+0000","last_change":"2024-04-17T19:47:48.895356+0000","last_active":"2024-04-17T19:47:54.929405+0000","last_peered":"2024-04-17T19:47:54.929405+0000","last_clean":"2024-04-17T19:47:54.929405+0000","last_became_active":"2024-04-17T19:47:48.889225+0000","last_became_peered":"2024-04-17T19:47:48.889225+0000","last_unstale":"2024-04-17T19:47:54.929405+0000","last_undegraded":"2024-04-17T19:47:54.929405+0000","last_fullsized":"2024-04-17T19:47:54.929405+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:00:22.587917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923872+0000","last_change":"2024-04-17T19:47:53.940206+0000","last_active":"2024-04-17T19:47:54.923872+0000","last_peered":"2024-04-17T19:47:54.923872+0000","last_clean":"2024-04-17T19:47:54.923872+0000","last_became_active":"2024-04-17T19:47:53.939843+0000","last_became_peered":"2024-04-17T19:47:53.939843+0000","last_unstale":"2024-04-17T19:47:54.923872+0000","last_undegraded":"2024-04-17T19:47:54.923872+0000","last_fullsized":"2024-04-17T19:47:54.923872+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:18:55.562573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841764+0000","last_change":"2024-04-17T19:47:53.920258+0000","last_active":"2024-04-17T19:47:55.841764+0000","last_peered":"2024-04-17T19:47:55.841764+0000","last_clean":"2024-04-17T19:47:55.841764+0000","last_became_active":"2024-04-17T19:47:53.918932+0000","last_became_peered":"2024-04-17T19:47:53.918932+0000","last_unstale":"2024-04-17T19:47:55.841764+0000","last_undegraded":"2024-04-17T19:47:55.841764+0000","last_fullsized":"2024-04-17T19:47:55.841764+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:45:55.456803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.050900+0000","last_change":"2024-04-17T19:47:49.861692+0000","last_active":"2024-04-17T19:47:55.050900+0000","last_peered":"2024-04-17T19:47:55.050900+0000","last_clean":"2024-04-17T19:47:55.050900+0000","last_became_active":"2024-04-17T19:47:49.861333+0000","last_became_peered":"2024-04-17T19:47:49.861333+0000","last_unstale":"2024-04-17T19:47:55.050900+0000","last_undegraded":"2024-04-17T19:47:55.050900+0000","last_fullsized":"2024-04-17T19:47:55.050900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:47:13.317679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832138+0000","last_change":"2024-04-17T19:47:48.873355+0000","last_active":"2024-04-17T19:47:56.832138+0000","last_peered":"2024-04-17T19:47:56.832138+0000","last_clean":"2024-04-17T19:47:56.832138+0000","last_became_active":"2024-04-17T19:47:48.872427+0000","last_became_peered":"2024-04-17T19:47:48.872427+0000","last_unstale":"2024-04-17T19:47:56.832138+0000","last_undegraded":"2024-04-17T19:47:56.832138+0000","last_fullsized":"2024-04-17T19:47:56.832138+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:32:41.792605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929095+0000","last_change":"2024-04-17T19:47:53.952602+0000","last_active":"2024-04-17T19:47:54.929095+0000","last_peered":"2024-04-17T19:47:54.929095+0000","last_clean":"2024-04-17T19:47:54.929095+0000","last_became_active":"2024-04-17T19:47:53.951750+0000","last_became_peered":"2024-04-17T19:47:53.951750+0000","last_unstale":"2024-04-17T19:47:54.929095+0000","last_undegraded":"2024-04-17T19:47:54.929095+0000","last_fullsized":"2024-04-17T19:47:54.929095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:13:08.485238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923863+0000","last_change":"2024-04-17T19:47:49.862071+0000","last_active":"2024-04-17T19:47:54.923863+0000","last_peered":"2024-04-17T19:47:54.923863+0000","last_clean":"2024-04-17T19:47:54.923863+0000","last_became_active":"2024-04-17T19:47:49.860809+0000","last_became_peered":"2024-04-17T19:47:49.860809+0000","last_unstale":"2024-04-17T19:47:54.923863+0000","last_undegraded":"2024-04-17T19:47:54.923863+0000","last_fullsized":"2024-04-17T19:47:54.923863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:42:34.451839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924173+0000","last_change":"2024-04-17T19:47:48.867855+0000","last_active":"2024-04-17T19:47:54.924173+0000","last_peered":"2024-04-17T19:47:54.924173+0000","last_clean":"2024-04-17T19:47:54.924173+0000","last_became_active":"2024-04-17T19:47:48.867532+0000","last_became_peered":"2024-04-17T19:47:48.867532+0000","last_unstale":"2024-04-17T19:47:54.924173+0000","last_undegraded":"2024-04-17T19:47:54.924173+0000","last_fullsized":"2024-04-17T19:47:54.924173+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:21:26.903245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927964+0000","last_change":"2024-04-17T19:47:48.894946+0000","last_active":"2024-04-17T19:47:54.927964+0000","last_peered":"2024-04-17T19:47:54.927964+0000","last_clean":"2024-04-17T19:47:54.927964+0000","last_became_active":"2024-04-17T19:47:48.893407+0000","last_became_peered":"2024-04-17T19:47:48.893407+0000","last_unstale":"2024-04-17T19:47:54.927964+0000","last_undegraded":"2024-04-17T19:47:54.927964+0000","last_fullsized":"2024-04-17T19:47:54.927964+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:09:56.284638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842374+0000","last_change":"2024-04-17T19:47:49.859985+0000","last_active":"2024-04-17T19:47:55.842374+0000","last_peered":"2024-04-17T19:47:55.842374+0000","last_clean":"2024-04-17T19:47:55.842374+0000","last_became_active":"2024-04-17T19:47:49.859635+0000","last_became_peered":"2024-04-17T19:47:49.859635+0000","last_unstale":"2024-04-17T19:47:55.842374+0000","last_undegraded":"2024-04-17T19:47:55.842374+0000","last_fullsized":"2024-04-17T19:47:55.842374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:04:14.834218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926673+0000","last_change":"2024-04-17T19:47:53.925444+0000","last_active":"2024-04-17T19:47:54.926673+0000","last_peered":"2024-04-17T19:47:54.926673+0000","last_clean":"2024-04-17T19:47:54.926673+0000","last_became_active":"2024-04-17T19:47:53.924805+0000","last_became_peered":"2024-04-17T19:47:53.924805+0000","last_unstale":"2024-04-17T19:47:54.926673+0000","last_undegraded":"2024-04-17T19:47:54.926673+0000","last_fullsized":"2024-04-17T19:47:54.926673+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:19:24.022368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924894+0000","last_change":"2024-04-17T19:47:48.867176+0000","last_active":"2024-04-17T19:47:54.924894+0000","last_peered":"2024-04-17T19:47:54.924894+0000","last_clean":"2024-04-17T19:47:54.924894+0000","last_became_active":"2024-04-17T19:47:48.866817+0000","last_became_peered":"2024-04-17T19:47:48.866817+0000","last_unstale":"2024-04-17T19:47:54.924894+0000","last_undegraded":"2024-04-17T19:47:54.924894+0000","last_fullsized":"2024-04-17T19:47:54.924894+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:44:30.842583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955878+0000","last_change":"2024-04-17T19:47:49.858924+0000","last_active":"2024-04-17T19:47:56.955878+0000","last_peered":"2024-04-17T19:47:56.955878+0000","last_clean":"2024-04-17T19:47:56.955878+0000","last_became_active":"2024-04-17T19:47:49.857820+0000","last_became_peered":"2024-04-17T19:47:49.857820+0000","last_unstale":"2024-04-17T19:47:56.955878+0000","last_undegraded":"2024-04-17T19:47:56.955878+0000","last_fullsized":"2024-04-17T19:47:56.955878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:09:28.706054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924516+0000","last_change":"2024-04-17T19:47:53.933337+0000","last_active":"2024-04-17T19:47:54.924516+0000","last_peered":"2024-04-17T19:47:54.924516+0000","last_clean":"2024-04-17T19:47:54.924516+0000","last_became_active":"2024-04-17T19:47:53.931623+0000","last_became_peered":"2024-04-17T19:47:53.931623+0000","last_unstale":"2024-04-17T19:47:54.924516+0000","last_undegraded":"2024-04-17T19:47:54.924516+0000","last_fullsized":"2024-04-17T19:47:54.924516+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:57:37.600957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927056+0000","last_change":"2024-04-17T19:47:49.881093+0000","last_active":"2024-04-17T19:47:54.927056+0000","last_peered":"2024-04-17T19:47:54.927056+0000","last_clean":"2024-04-17T19:47:54.927056+0000","last_became_active":"2024-04-17T19:47:49.880210+0000","last_became_peered":"2024-04-17T19:47:49.880210+0000","last_unstale":"2024-04-17T19:47:54.927056+0000","last_undegraded":"2024-04-17T19:47:54.927056+0000","last_fullsized":"2024-04-17T19:47:54.927056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:12:15.522651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926335+0000","last_change":"2024-04-17T19:47:49.886936+0000","last_active":"2024-04-17T19:47:54.926335+0000","last_peered":"2024-04-17T19:47:54.926335+0000","last_clean":"2024-04-17T19:47:54.926335+0000","last_became_active":"2024-04-17T19:47:49.886580+0000","last_became_peered":"2024-04-17T19:47:49.886580+0000","last_unstale":"2024-04-17T19:47:54.926335+0000","last_undegraded":"2024-04-17T19:47:54.926335+0000","last_fullsized":"2024-04-17T19:47:54.926335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:00:44.599463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832155+0000","last_change":"2024-04-17T19:47:49.905445+0000","last_active":"2024-04-17T19:47:56.832155+0000","last_peered":"2024-04-17T19:47:56.832155+0000","last_clean":"2024-04-17T19:47:56.832155+0000","last_became_active":"2024-04-17T19:47:49.896510+0000","last_became_peered":"2024-04-17T19:47:49.896510+0000","last_unstale":"2024-04-17T19:47:56.832155+0000","last_undegraded":"2024-04-17T19:47:56.832155+0000","last_fullsized":"2024-04-17T19:47:56.832155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:23:27.401861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842320+0000","last_change":"2024-04-17T19:47:49.850490+0000","last_active":"2024-04-17T19:47:55.842320+0000","last_peered":"2024-04-17T19:47:55.842320+0000","last_clean":"2024-04-17T19:47:55.842320+0000","last_became_active":"2024-04-17T19:47:49.849344+0000","last_became_peered":"2024-04-17T19:47:49.849344+0000","last_unstale":"2024-04-17T19:47:55.842320+0000","last_undegraded":"2024-04-17T19:47:55.842320+0000","last_fullsized":"2024-04-17T19:47:55.842320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:44:32.231058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924753+0000","last_change":"2024-04-17T19:47:49.857737+0000","last_active":"2024-04-17T19:47:54.924753+0000","last_peered":"2024-04-17T19:47:54.924753+0000","last_clean":"2024-04-17T19:47:54.924753+0000","last_became_active":"2024-04-17T19:47:49.857476+0000","last_became_peered":"2024-04-17T19:47:49.857476+0000","last_unstale":"2024-04-17T19:47:54.924753+0000","last_undegraded":"2024-04-17T19:47:54.924753+0000","last_fullsized":"2024-04-17T19:47:54.924753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:50:13.070628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051526+0000","last_change":"2024-04-17T19:47:49.864985+0000","last_active":"2024-04-17T19:47:55.051526+0000","last_peered":"2024-04-17T19:47:55.051526+0000","last_clean":"2024-04-17T19:47:55.051526+0000","last_became_active":"2024-04-17T19:47:49.864329+0000","last_became_peered":"2024-04-17T19:47:49.864329+0000","last_unstale":"2024-04-17T19:47:55.051526+0000","last_undegraded":"2024-04-17T19:47:55.051526+0000","last_fullsized":"2024-04-17T19:47:55.051526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:43:57.448269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929577+0000","last_change":"2024-04-17T19:47:49.881785+0000","last_active":"2024-04-17T19:47:54.929577+0000","last_peered":"2024-04-17T19:47:54.929577+0000","last_clean":"2024-04-17T19:47:54.929577+0000","last_became_active":"2024-04-17T19:47:49.880939+0000","last_became_peered":"2024-04-17T19:47:49.880939+0000","last_unstale":"2024-04-17T19:47:54.929577+0000","last_undegraded":"2024-04-17T19:47:54.929577+0000","last_fullsized":"2024-04-17T19:47:54.929577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:49:07.194643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924461+0000","last_change":"2024-04-17T19:47:49.884944+0000","last_active":"2024-04-17T19:47:54.924461+0000","last_peered":"2024-04-17T19:47:54.924461+0000","last_clean":"2024-04-17T19:47:54.924461+0000","last_became_active":"2024-04-17T19:47:49.884559+0000","last_became_peered":"2024-04-17T19:47:49.884559+0000","last_unstale":"2024-04-17T19:47:54.924461+0000","last_undegraded":"2024-04-17T19:47:54.924461+0000","last_fullsized":"2024-04-17T19:47:54.924461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:00.485472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.831772+0000","last_change":"2024-04-17T19:47:49.891230+0000","last_active":"2024-04-17T19:47:56.831772+0000","last_peered":"2024-04-17T19:47:56.831772+0000","last_clean":"2024-04-17T19:47:56.831772+0000","last_became_active":"2024-04-17T19:47:49.886001+0000","last_became_peered":"2024-04-17T19:47:49.886001+0000","last_unstale":"2024-04-17T19:47:56.831772+0000","last_undegraded":"2024-04-17T19:47:56.831772+0000","last_fullsized":"2024-04-17T19:47:56.831772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:00.485472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923958+0000","last_change":"2024-04-17T19:47:49.859382+0000","last_active":"2024-04-17T19:47:54.923958+0000","last_peered":"2024-04-17T19:47:54.923958+0000","last_clean":"2024-04-17T19:47:54.923958+0000","last_became_active":"2024-04-17T19:47:49.859170+0000","last_became_peered":"2024-04-17T19:47:49.859170+0000","last_unstale":"2024-04-17T19:47:54.923958+0000","last_undegraded":"2024-04-17T19:47:54.923958+0000","last_fullsized":"2024-04-17T19:47:54.923958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:58:51.111053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832143+0000","last_change":"2024-04-17T19:47:49.909188+0000","last_active":"2024-04-17T19:47:56.832143+0000","last_peered":"2024-04-17T19:47:56.832143+0000","last_clean":"2024-04-17T19:47:56.832143+0000","last_became_active":"2024-04-17T19:47:49.908432+0000","last_became_peered":"2024-04-17T19:47:49.908432+0000","last_unstale":"2024-04-17T19:47:56.832143+0000","last_undegraded":"2024-04-17T19:47:56.832143+0000","last_fullsized":"2024-04-17T19:47:56.832143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:16:21.209634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842710+0000","last_change":"2024-04-17T19:47:49.876406+0000","last_active":"2024-04-17T19:47:55.842710+0000","last_peered":"2024-04-17T19:47:55.842710+0000","last_clean":"2024-04-17T19:47:55.842710+0000","last_became_active":"2024-04-17T19:47:49.876132+0000","last_became_peered":"2024-04-17T19:47:49.876132+0000","last_unstale":"2024-04-17T19:47:55.842710+0000","last_undegraded":"2024-04-17T19:47:55.842710+0000","last_fullsized":"2024-04-17T19:47:55.842710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:51:20.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.928888+0000","last_change":"2024-04-17T19:47:49.896244+0000","last_active":"2024-04-17T19:47:54.928888+0000","last_peered":"2024-04-17T19:47:54.928888+0000","last_clean":"2024-04-17T19:47:54.928888+0000","last_became_active":"2024-04-17T19:47:49.895380+0000","last_became_peered":"2024-04-17T19:47:49.895380+0000","last_unstale":"2024-04-17T19:47:54.928888+0000","last_undegraded":"2024-04-17T19:47:54.928888+0000","last_fullsized":"2024-04-17T19:47:54.928888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:04:43.091307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049138+0000","last_change":"2024-04-17T19:47:49.870551+0000","last_active":"2024-04-17T19:47:56.049138+0000","last_peered":"2024-04-17T19:47:56.049138+0000","last_clean":"2024-04-17T19:47:56.049138+0000","last_became_active":"2024-04-17T19:47:49.869315+0000","last_became_peered":"2024-04-17T19:47:49.869315+0000","last_unstale":"2024-04-17T19:47:56.049138+0000","last_undegraded":"2024-04-17T19:47:56.049138+0000","last_fullsized":"2024-04-17T19:47:56.049138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:06.341123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926618+0000","last_change":"2024-04-17T19:47:49.880060+0000","last_active":"2024-04-17T19:47:54.926618+0000","last_peered":"2024-04-17T19:47:54.926618+0000","last_clean":"2024-04-17T19:47:54.926618+0000","last_became_active":"2024-04-17T19:47:49.879711+0000","last_became_peered":"2024-04-17T19:47:49.879711+0000","last_unstale":"2024-04-17T19:47:54.926618+0000","last_undegraded":"2024-04-17T19:47:54.926618+0000","last_fullsized":"2024-04-17T19:47:54.926618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:41:25.628870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.833444+0000","last_change":"2024-04-17T19:47:49.895452+0000","last_active":"2024-04-17T19:47:56.833444+0000","last_peered":"2024-04-17T19:47:56.833444+0000","last_clean":"2024-04-17T19:47:56.833444+0000","last_became_active":"2024-04-17T19:47:49.894684+0000","last_became_peered":"2024-04-17T19:47:49.894684+0000","last_unstale":"2024-04-17T19:47:56.833444+0000","last_undegraded":"2024-04-17T19:47:56.833444+0000","last_fullsized":"2024-04-17T19:47:56.833444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:15:55.224238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.831437+0000","last_change":"2024-04-17T19:47:49.886902+0000","last_active":"2024-04-17T19:47:56.831437+0000","last_peered":"2024-04-17T19:47:56.831437+0000","last_clean":"2024-04-17T19:47:56.831437+0000","last_became_active":"2024-04-17T19:47:49.885451+0000","last_became_peered":"2024-04-17T19:47:49.885451+0000","last_unstale":"2024-04-17T19:47:56.831437+0000","last_undegraded":"2024-04-17T19:47:56.831437+0000","last_fullsized":"2024-04-17T19:47:56.831437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:12:09.196610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.050989+0000","last_change":"2024-04-17T19:47:49.858908+0000","last_active":"2024-04-17T19:47:55.050989+0000","last_peered":"2024-04-17T19:47:55.050989+0000","last_clean":"2024-04-17T19:47:55.050989+0000","last_became_active":"2024-04-17T19:47:49.858457+0000","last_became_peered":"2024-04-17T19:47:49.858457+0000","last_unstale":"2024-04-17T19:47:55.050989+0000","last_undegraded":"2024-04-17T19:47:55.050989+0000","last_fullsized":"2024-04-17T19:47:55.050989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:09.467859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841618+0000","last_change":"2024-04-17T19:47:49.856371+0000","last_active":"2024-04-17T19:47:55.841618+0000","last_peered":"2024-04-17T19:47:55.841618+0000","last_clean":"2024-04-17T19:47:55.841618+0000","last_became_active":"2024-04-17T19:47:49.856062+0000","last_became_peered":"2024-04-17T19:47:49.856062+0000","last_unstale":"2024-04-17T19:47:55.841618+0000","last_undegraded":"2024-04-17T19:47:55.841618+0000","last_fullsized":"2024-04-17T19:47:55.841618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:39:54.502619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924769+0000","last_change":"2024-04-17T19:47:49.873350+0000","last_active":"2024-04-17T19:47:54.924769+0000","last_peered":"2024-04-17T19:47:54.924769+0000","last_clean":"2024-04-17T19:47:54.924769+0000","last_became_active":"2024-04-17T19:47:49.873019+0000","last_became_peered":"2024-04-17T19:47:49.873019+0000","last_unstale":"2024-04-17T19:47:54.924769+0000","last_undegraded":"2024-04-17T19:47:54.924769+0000","last_fullsized":"2024-04-17T19:47:54.924769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:06:58.641727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956704+0000","last_change":"2024-04-17T19:47:49.891551+0000","last_active":"2024-04-17T19:47:56.956704+0000","last_peered":"2024-04-17T19:47:56.956704+0000","last_clean":"2024-04-17T19:47:56.956704+0000","last_became_active":"2024-04-17T19:47:49.890132+0000","last_became_peered":"2024-04-17T19:47:49.890132+0000","last_unstale":"2024-04-17T19:47:56.956704+0000","last_undegraded":"2024-04-17T19:47:56.956704+0000","last_fullsized":"2024-04-17T19:47:56.956704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:26:22.971489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.928586+0000","last_change":"2024-04-17T19:47:49.903854+0000","last_active":"2024-04-17T19:47:54.928586+0000","last_peered":"2024-04-17T19:47:54.928586+0000","last_clean":"2024-04-17T19:47:54.928586+0000","last_became_active":"2024-04-17T19:47:49.903178+0000","last_became_peered":"2024-04-17T19:47:49.903178+0000","last_unstale":"2024-04-17T19:47:54.928586+0000","last_undegraded":"2024-04-17T19:47:54.928586+0000","last_fullsized":"2024-04-17T19:47:54.928586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:24:52.699532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929394+0000","last_change":"2024-04-17T19:47:49.900848+0000","last_active":"2024-04-17T19:47:54.929394+0000","last_peered":"2024-04-17T19:47:54.929394+0000","last_clean":"2024-04-17T19:47:54.929394+0000","last_became_active":"2024-04-17T19:47:49.900052+0000","last_became_peered":"2024-04-17T19:47:49.900052+0000","last_unstale":"2024-04-17T19:47:54.929394+0000","last_undegraded":"2024-04-17T19:47:54.929394+0000","last_fullsized":"2024-04-17T19:47:54.929394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:28:41.115173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049711+0000","last_change":"2024-04-17T19:47:49.906531+0000","last_active":"2024-04-17T19:47:56.049711+0000","last_peered":"2024-04-17T19:47:56.049711+0000","last_clean":"2024-04-17T19:47:56.049711+0000","last_became_active":"2024-04-17T19:47:49.905217+0000","last_became_peered":"2024-04-17T19:47:49.905217+0000","last_unstale":"2024-04-17T19:47:56.049711+0000","last_undegraded":"2024-04-17T19:47:56.049711+0000","last_fullsized":"2024-04-17T19:47:56.049711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:22.078473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923726+0000","last_change":"2024-04-17T19:47:49.864577+0000","last_active":"2024-04-17T19:47:54.923726+0000","last_peered":"2024-04-17T19:47:54.923726+0000","last_clean":"2024-04-17T19:47:54.923726+0000","last_became_active":"2024-04-17T19:47:49.864251+0000","last_became_peered":"2024-04-17T19:47:49.864251+0000","last_unstale":"2024-04-17T19:47:54.923726+0000","last_undegraded":"2024-04-17T19:47:54.923726+0000","last_fullsized":"2024-04-17T19:47:54.923726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:32:10.329543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.047770+0000","last_change":"2024-04-17T19:47:49.869245+0000","last_active":"2024-04-17T19:47:56.047770+0000","last_peered":"2024-04-17T19:47:56.047770+0000","last_clean":"2024-04-17T19:47:56.047770+0000","last_became_active":"2024-04-17T19:47:49.868156+0000","last_became_peered":"2024-04-17T19:47:49.868156+0000","last_unstale":"2024-04-17T19:47:56.047770+0000","last_undegraded":"2024-04-17T19:47:56.047770+0000","last_fullsized":"2024-04-17T19:47:56.047770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:58:39.814933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.831810+0000","last_change":"2024-04-17T19:47:49.907573+0000","last_active":"2024-04-17T19:47:56.831810+0000","last_peered":"2024-04-17T19:47:56.831810+0000","last_clean":"2024-04-17T19:47:56.831810+0000","last_became_active":"2024-04-17T19:47:49.897308+0000","last_became_peered":"2024-04-17T19:47:49.897308+0000","last_unstale":"2024-04-17T19:47:56.831810+0000","last_undegraded":"2024-04-17T19:47:56.831810+0000","last_fullsized":"2024-04-17T19:47:56.831810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:02:03.101317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.047866+0000","last_change":"2024-04-17T19:47:49.876052+0000","last_active":"2024-04-17T19:47:56.047866+0000","last_peered":"2024-04-17T19:47:56.047866+0000","last_clean":"2024-04-17T19:47:56.047866+0000","last_became_active":"2024-04-17T19:47:49.875008+0000","last_became_peered":"2024-04-17T19:47:49.875008+0000","last_unstale":"2024-04-17T19:47:56.047866+0000","last_undegraded":"2024-04-17T19:47:56.047866+0000","last_fullsized":"2024-04-17T19:47:56.047866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:36:14.486150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832276+0000","last_change":"2024-04-17T19:47:49.906840+0000","last_active":"2024-04-17T19:47:56.832276+0000","last_peered":"2024-04-17T19:47:56.832276+0000","last_clean":"2024-04-17T19:47:56.832276+0000","last_became_active":"2024-04-17T19:47:49.905703+0000","last_became_peered":"2024-04-17T19:47:49.905703+0000","last_unstale":"2024-04-17T19:47:56.832276+0000","last_undegraded":"2024-04-17T19:47:56.832276+0000","last_fullsized":"2024-04-17T19:47:56.832276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:28:26.686472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841783+0000","last_change":"2024-04-17T19:47:49.869425+0000","last_active":"2024-04-17T19:47:55.841783+0000","last_peered":"2024-04-17T19:47:55.841783+0000","last_clean":"2024-04-17T19:47:55.841783+0000","last_became_active":"2024-04-17T19:47:49.868758+0000","last_became_peered":"2024-04-17T19:47:49.868758+0000","last_unstale":"2024-04-17T19:47:55.841783+0000","last_undegraded":"2024-04-17T19:47:55.841783+0000","last_fullsized":"2024-04-17T19:47:55.841783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:57:27.214150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051778+0000","last_change":"2024-04-17T19:47:49.856309+0000","last_active":"2024-04-17T19:47:55.051778+0000","last_peered":"2024-04-17T19:47:55.051778+0000","last_clean":"2024-04-17T19:47:55.051778+0000","last_became_active":"2024-04-17T19:47:49.856026+0000","last_became_peered":"2024-04-17T19:47:49.856026+0000","last_unstale":"2024-04-17T19:47:55.051778+0000","last_undegraded":"2024-04-17T19:47:55.051778+0000","last_fullsized":"2024-04-17T19:47:55.051778+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:51:57.963637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049403+0000","last_change":"2024-04-17T19:47:49.890343+0000","last_active":"2024-04-17T19:47:56.049403+0000","last_peered":"2024-04-17T19:47:56.049403+0000","last_clean":"2024-04-17T19:47:56.049403+0000","last_became_active":"2024-04-17T19:47:49.880566+0000","last_became_peered":"2024-04-17T19:47:49.880566+0000","last_unstale":"2024-04-17T19:47:56.049403+0000","last_undegraded":"2024-04-17T19:47:56.049403+0000","last_fullsized":"2024-04-17T19:47:56.049403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:19:29.359024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T19:48:14.347 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T19:48:14.347 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-17T19:48:14.347 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-17T19:48:14.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-17T19:48:14.348 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-17T19:48:14.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.545+0000 7fc5dd600640 1 Processor -- start 2024-04-17T19:48:14.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.545+0000 7fc5dd600640 1 -- start start 2024-04-17T19:48:14.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5dd600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d80cb560 0x7fc5d80c4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5dd600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5d80c9ce0 con 0x7fc5d80703e0 2024-04-17T19:48:14.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5dd600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5d80c9f80 con 0x7fc5d815a750 2024-04-17T19:48:14.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5dd600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc5d8154390 con 0x7fc5d80cb560 2024-04-17T19:48:14.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d7400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc5d80703e0 0x7fc5d80cf810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38080/0 (socket says 172.21.15.29:38080) 2024-04-17T19:48:14.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d7400640 1 -- 172.21.15.29:0/674505074 learned_addr learned my addr 172.21.15.29:0/674505074 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:14.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d6a00640 1 --2- 172.21.15.29:0/674505074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d80cb560 0x7fc5d80c4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d6000640 1 -- 172.21.15.29:0/674505074 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1031743115 0 0) 0x7fc5d80c9ce0 con 0x7fc5d80703e0 2024-04-17T19:48:14.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d6000640 1 -- 172.21.15.29:0/674505074 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5bc003530 con 0x7fc5d80703e0 2024-04-17T19:48:14.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d6000640 1 -- 172.21.15.29:0/674505074 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1920981086 0 0) 0x7fc5bc003530 con 0x7fc5d80703e0 2024-04-17T19:48:14.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d6000640 1 -- 172.21.15.29:0/674505074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d80cb560 msgr2=0x7fc5d80c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:14.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d6000640 1 --2- 172.21.15.29:0/674505074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d80cb560 0x7fc5d80c4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d6000640 1 -- 172.21.15.29:0/674505074 >> v1:172.21.15.37:6789/0 conn(0x7fc5d815a750 legacy=0x7fc5d8153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:14.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d6000640 1 -- 172.21.15.29:0/674505074 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d80ca260 con 0x7fc5d80703e0 2024-04-17T19:48:14.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d6000640 1 -- 172.21.15.29:0/674505074 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fc5d0001a00 con 0x7fc5d80703e0 2024-04-17T19:48:14.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d6000640 1 -- 172.21.15.29:0/674505074 >> v1:172.21.15.29:6789/0 conn(0x7fc5d80703e0 legacy=0x7fc5d80cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:14.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d6000640 1 --2- 172.21.15.29:0/674505074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d80cb560 0x7fc5bc003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d6000640 1 --2- 172.21.15.29:0/674505074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5bc003fe0 0x7fc5bc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d6000640 1 --2- 172.21.15.29:0/674505074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5bc004d50 0x7fc5bc0051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d6000640 1 -- 172.21.15.29:0/674505074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d8154390 con 0x7fc5bc003fe0 2024-04-17T19:48:14.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d6000640 1 -- 172.21.15.29:0/674505074 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc5bc00bc80 con 0x7fc5bc004d50 2024-04-17T19:48:14.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d6000640 1 -- 172.21.15.29:0/674505074 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc5bc00be80 con 0x7fc5d80cb560 2024-04-17T19:48:14.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5cfe00640 1 --2- 172.21.15.29:0/674505074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5bc003fe0 0x7fc5bc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d6a00640 1 --2- 172.21.15.29:0/674505074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d80cb560 0x7fc5bc003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d7400640 1 --2- 172.21.15.29:0/674505074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5bc004d50 0x7fc5bc0051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5cfe00640 1 -- 172.21.15.29:0/674505074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d80cb560 msgr2=0x7fc5bc003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:14.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5cfe00640 1 --2- 172.21.15.29:0/674505074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d80cb560 0x7fc5bc003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5cfe00640 1 -- 172.21.15.29:0/674505074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5bc004d50 msgr2=0x7fc5bc0051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:14.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5cfe00640 1 --2- 172.21.15.29:0/674505074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5bc004d50 0x7fc5bc0051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5cfe00640 1 -- 172.21.15.29:0/674505074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5bc003530 con 0x7fc5bc003fe0 2024-04-17T19:48:14.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d7400640 1 --2- 172.21.15.29:0/674505074 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5bc004d50 0x7fc5bc0051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T19:48:14.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.549+0000 7fc5d6a00640 1 --2- 172.21.15.29:0/674505074 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d80cb560 0x7fc5bc003a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:14.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.553+0000 7fc5cfe00640 1 --2- 172.21.15.29:0/674505074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5bc003fe0 0x7fc5bc004810 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0001750 tx=0x7fc5c001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d168cda1a9887c79 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:14.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.553+0000 7fc5d6000640 1 -- 172.21.15.29:0/674505074 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c00013a0 con 0x7fc5bc003fe0 2024-04-17T19:48:14.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.553+0000 7fc5d6000640 1 -- 172.21.15.29:0/674505074 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5c00052d0 con 0x7fc5bc003fe0 2024-04-17T19:48:14.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.553+0000 7fc5d6000640 1 -- 172.21.15.29:0/674505074 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c00054c0 con 0x7fc5bc003fe0 2024-04-17T19:48:14.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.553+0000 7fc5dd600640 1 -- 172.21.15.29:0/674505074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5bc003fe0 msgr2=0x7fc5bc004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:14.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.553+0000 7fc5dd600640 1 --2- 172.21.15.29:0/674505074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5bc003fe0 0x7fc5bc004810 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc5c0001750 tx=0x7fc5c001c3e0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.553+0000 7fc5d7400640 1 -- 172.21.15.29:0/674505074 reap_dead start 2024-04-17T19:48:14.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.553+0000 7fc5dd600640 1 -- 172.21.15.29:0/674505074 shutdown_connections 2024-04-17T19:48:14.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.553+0000 7fc5dd600640 1 -- 172.21.15.29:0/674505074 >> 172.21.15.29:0/674505074 conn(0x7fc5d806e3d0 msgr2=0x7fc5d806e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:14.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.553+0000 7fc5dd600640 1 -- 172.21.15.29:0/674505074 shutdown_connections 2024-04-17T19:48:14.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.553+0000 7fc5dd600640 1 -- 172.21.15.29:0/674505074 wait complete. 2024-04-17T19:48:14.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5dd600640 1 Processor -- start 2024-04-17T19:48:14.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5dd600640 1 -- start start 2024-04-17T19:48:14.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5dd600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5bc003fe0 0x7fc5d8150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5dd600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d8155b90 0x7fc5d8148a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5dd600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5d81f4850 0x7fc5d81f4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5dd600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d80707f0 con 0x7fc5bc003fe0 2024-04-17T19:48:14.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5dd600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc5d8159c20 con 0x7fc5d81f4850 2024-04-17T19:48:14.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5dd600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc5d80cff20 con 0x7fc5d8155b90 2024-04-17T19:48:14.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5d6a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5bc003fe0 0x7fc5d8150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5d6a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5bc003fe0 0x7fc5d8150420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52564/0 (socket says 172.21.15.29:52564) 2024-04-17T19:48:14.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5d6a00640 1 -- 172.21.15.29:0/2685952472 learned_addr learned my addr 172.21.15.29:0/2685952472 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:14.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5d7400640 1 --2- 172.21.15.29:0/2685952472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5d81f4850 0x7fc5d81f4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5cfe00640 1 --2- 172.21.15.29:0/2685952472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d8155b90 0x7fc5d8148a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5d6a00640 1 -- 172.21.15.29:0/2685952472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d8155b90 msgr2=0x7fc5d8148a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:14.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5d6a00640 1 --2- 172.21.15.29:0/2685952472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d8155b90 0x7fc5d8148a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5d6a00640 1 -- 172.21.15.29:0/2685952472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5d81f4850 msgr2=0x7fc5d81f4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:14.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5d6a00640 1 --2- 172.21.15.29:0/2685952472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5d81f4850 0x7fc5d81f4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5d6a00640 1 -- 172.21.15.29:0/2685952472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d8155480 con 0x7fc5bc003fe0 2024-04-17T19:48:14.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5cfe00640 1 --2- 172.21.15.29:0/2685952472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d8155b90 0x7fc5d8148a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T19:48:14.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5d7400640 1 --2- 172.21.15.29:0/2685952472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5d81f4850 0x7fc5d81f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T19:48:14.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.557+0000 7fc5d6a00640 1 --2- 172.21.15.29:0/2685952472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5bc003fe0 0x7fc5d8150420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009fd0 tx=0x7fc5c8011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:14.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.561+0000 7fc5cea00640 1 -- 172.21.15.29:0/2685952472 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8004220 con 0x7fc5bc003fe0 2024-04-17T19:48:14.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.561+0000 7fc5cea00640 1 -- 172.21.15.29:0/2685952472 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5c80043c0 con 0x7fc5bc003fe0 2024-04-17T19:48:14.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.561+0000 7fc5cea00640 1 -- 172.21.15.29:0/2685952472 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5c800f9e0 con 0x7fc5bc003fe0 2024-04-17T19:48:14.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.561+0000 7fc5dd600640 1 -- 172.21.15.29:0/2685952472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5d8156480 con 0x7fc5bc003fe0 2024-04-17T19:48:14.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.561+0000 7fc5dd600640 1 -- 172.21.15.29:0/2685952472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5d8156a00 con 0x7fc5bc003fe0 2024-04-17T19:48:14.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.561+0000 7fc5dd600640 1 -- 172.21.15.29:0/2685952472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5d8069b90 con 0x7fc5bc003fe0 2024-04-17T19:48:14.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.561+0000 7fc5cea00640 1 -- 172.21.15.29:0/2685952472 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc5c800fb80 con 0x7fc5bc003fe0 2024-04-17T19:48:14.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.561+0000 7fc5cea00640 1 --2- 172.21.15.29:0/2685952472 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc5b4068380 0x7fc5b406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.561+0000 7fc5cea00640 1 -- 172.21.15.29:0/2685952472 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc5c80875b0 con 0x7fc5bc003fe0 2024-04-17T19:48:14.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.561+0000 7fc5cfe00640 1 --2- 172.21.15.29:0/2685952472 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc5b4068380 0x7fc5b406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.565+0000 7fc5cfe00640 1 --2- 172.21.15.29:0/2685952472 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc5b4068380 0x7fc5b406a890 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc5d80c9eb0 tx=0x7fc5c00011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:14.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.565+0000 7fc5cea00640 1 -- 172.21.15.29:0/2685952472 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc5c8019030 con 0x7fc5bc003fe0 2024-04-17T19:48:14.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.773+0000 7fc5dd600640 1 -- 172.21.15.29:0/2685952472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fc5d8157640 con 0x7fc5bc003fe0 2024-04-17T19:48:14.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.777+0000 7fc5cea00640 1 -- 172.21.15.29:0/2685952472 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+168085 (secure 0 0 0) 0x7fc5c8057700 con 0x7fc5bc003fe0 2024-04-17T19:48:14.768 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:14.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.781+0000 7fc5dd600640 1 -- 172.21.15.29:0/2685952472 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc5b4068380 msgr2=0x7fc5b406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:14.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.781+0000 7fc5dd600640 1 --2- 172.21.15.29:0/2685952472 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc5b4068380 0x7fc5b406a890 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc5d80c9eb0 tx=0x7fc5c00011a0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.781+0000 7fc5dd600640 1 -- 172.21.15.29:0/2685952472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5bc003fe0 msgr2=0x7fc5d8150420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:14.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.781+0000 7fc5dd600640 1 --2- 172.21.15.29:0/2685952472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5bc003fe0 0x7fc5d8150420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009fd0 tx=0x7fc5c8011b30 comp rx=0 tx=0).stop 2024-04-17T19:48:14.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.781+0000 7fc5dd600640 1 -- 172.21.15.29:0/2685952472 shutdown_connections 2024-04-17T19:48:14.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.781+0000 7fc5dd600640 1 --2- 172.21.15.29:0/2685952472 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc5b4068380 0x7fc5b406a890 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.781+0000 7fc5dd600640 1 --2- 172.21.15.29:0/2685952472 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc5d81f4850 0x7fc5d81f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.781+0000 7fc5dd600640 1 --2- 172.21.15.29:0/2685952472 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc5d8155b90 0x7fc5d8148a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.781+0000 7fc5dd600640 1 --2- 172.21.15.29:0/2685952472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5bc003fe0 0x7fc5d8150420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.781+0000 7fc5dd600640 1 -- 172.21.15.29:0/2685952472 >> 172.21.15.29:0/2685952472 conn(0x7fc5d806e3d0 msgr2=0x7fc5d806d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:14.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.781+0000 7fc5dd600640 1 -- 172.21.15.29:0/2685952472 shutdown_connections 2024-04-17T19:48:14.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.781+0000 7fc5dd600640 1 -- 172.21.15.29:0/2685952472 wait complete. 2024-04-17T19:48:14.789 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6832","nonce":1588862286},{"type":"v1","addr":"172.21.15.37:6833","nonce":1588862286}]},"active_addr":"172.21.15.37:6833/1588862286","active_change":"2024-04-17T19:47:28.608789+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4108,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.37:0","nonce":3743633940}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.37:0","nonce":1598471127}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.37:0","nonce":4077958058}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.37:0","nonce":3442162694}]}]} 2024-04-17T19:48:14.793 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-17T19:48:14.793 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-17T19:48:14.793 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:48:14.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.957+0000 7ffa3b200640 1 Processor -- start 2024-04-17T19:48:14.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.957+0000 7ffa3b200640 1 -- start start 2024-04-17T19:48:14.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.957+0000 7ffa3b200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ffa340c7a20 0x7ffa340710d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.957+0000 7ffa3b200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa340c7e30 0x7ffa3415a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.957+0000 7ffa3b200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffa3415ab20 0x7ffa3415db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.957+0000 7ffa3b200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7ffa3415a150 con 0x7ffa3415ab20 2024-04-17T19:48:14.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.957+0000 7ffa3b200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ffa340c9660 con 0x7ffa340c7a20 2024-04-17T19:48:14.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.957+0000 7ffa3b200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ffa340c8ae0 con 0x7ffa340c7e30 2024-04-17T19:48:14.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.957+0000 7ffa39200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffa3415ab20 0x7ffa3415db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.957+0000 7ffa39200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffa3415ab20 0x7ffa3415db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52580/0 (socket says 172.21.15.29:52580) 2024-04-17T19:48:14.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.957+0000 7ffa39200640 1 -- 172.21.15.29:0/3239127273 learned_addr learned my addr 172.21.15.29:0/3239127273 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:14.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.957+0000 7ffa33400640 1 --2- 172.21.15.29:0/3239127273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa340c7e30 0x7ffa3415a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.957+0000 7ffa33e00640 1 --2- 172.21.15.29:0/3239127273 >> v2:172.21.15.37:3300/0 conn(0x7ffa340c7a20 0x7ffa340710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.957+0000 7ffa39200640 1 -- 172.21.15.29:0/3239127273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa340c7e30 msgr2=0x7ffa3415a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:14.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.957+0000 7ffa39200640 1 --2- 172.21.15.29:0/3239127273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa340c7e30 0x7ffa3415a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.957+0000 7ffa39200640 1 -- 172.21.15.29:0/3239127273 >> v2:172.21.15.37:3300/0 conn(0x7ffa340c7a20 msgr2=0x7ffa340710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:14.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.957+0000 7ffa39200640 1 --2- 172.21.15.29:0/3239127273 >> v2:172.21.15.37:3300/0 conn(0x7ffa340c7a20 0x7ffa340710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.957+0000 7ffa39200640 1 -- 172.21.15.29:0/3239127273 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa340c8dc0 con 0x7ffa3415ab20 2024-04-17T19:48:14.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.961+0000 7ffa39200640 1 --2- 172.21.15.29:0/3239127273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3415ab20 0x7ffa3415db00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ffa34050520 tx=0x7ffa2c01f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7dd97e08ac998c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:14.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.961+0000 7ffa32a00640 1 -- 172.21.15.29:0/3239127273 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c0b4070 con 0x7ffa3415ab20 2024-04-17T19:48:14.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.961+0000 7ffa32a00640 1 -- 172.21.15.29:0/3239127273 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa2c0af400 con 0x7ffa3415ab20 2024-04-17T19:48:14.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.961+0000 7ffa32a00640 1 -- 172.21.15.29:0/3239127273 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c0af7d0 con 0x7ffa3415ab20 2024-04-17T19:48:14.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.961+0000 7ffa3b200640 1 -- 172.21.15.29:0/3239127273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3415ab20 msgr2=0x7ffa3415db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:14.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.961+0000 7ffa3b200640 1 --2- 172.21.15.29:0/3239127273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3415ab20 0x7ffa3415db00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ffa34050520 tx=0x7ffa2c01f1e0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.961+0000 7ffa3b200640 1 -- 172.21.15.29:0/3239127273 shutdown_connections 2024-04-17T19:48:14.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.961+0000 7ffa3b200640 1 --2- 172.21.15.29:0/3239127273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3415ab20 0x7ffa3415db00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.961+0000 7ffa3b200640 1 --2- 172.21.15.29:0/3239127273 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa340c7e30 0x7ffa3415a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.961+0000 7ffa3b200640 1 --2- 172.21.15.29:0/3239127273 >> v2:172.21.15.37:3300/0 conn(0x7ffa340c7a20 0x7ffa340710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.961+0000 7ffa3b200640 1 -- 172.21.15.29:0/3239127273 >> 172.21.15.29:0/3239127273 conn(0x7ffa34072e50 msgr2=0x7ffa34073260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:14.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.961+0000 7ffa3b200640 1 -- 172.21.15.29:0/3239127273 shutdown_connections 2024-04-17T19:48:14.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.961+0000 7ffa3b200640 1 -- 172.21.15.29:0/3239127273 wait complete. 2024-04-17T19:48:14.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.965+0000 7ffa3b200640 1 Processor -- start 2024-04-17T19:48:14.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.965+0000 7ffa3b200640 1 -- start start 2024-04-17T19:48:14.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.965+0000 7ffa3b200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa340c7a20 0x7ffa34141950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.965+0000 7ffa3b200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa340c7e30 0x7ffa3415bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.965+0000 7ffa3b200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa3415ab20 0x7ffa341e5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.965+0000 7ffa3b200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3406bfe0 con 0x7ffa340c7e30 2024-04-17T19:48:14.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.965+0000 7ffa3b200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ffa340c9380 con 0x7ffa3415ab20 2024-04-17T19:48:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.965+0000 7ffa3b200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ffa34071610 con 0x7ffa340c7a20 2024-04-17T19:48:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.965+0000 7ffa33400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa340c7e30 0x7ffa3415bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.965+0000 7ffa33400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa340c7e30 0x7ffa3415bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52584/0 (socket says 172.21.15.29:52584) 2024-04-17T19:48:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.965+0000 7ffa33400640 1 -- 172.21.15.29:0/1870704595 learned_addr learned my addr 172.21.15.29:0/1870704595 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.965+0000 7ffa33e00640 1 --2- 172.21.15.29:0/1870704595 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa340c7a20 0x7ffa34141950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.965+0000 7ffa39200640 1 --2- 172.21.15.29:0/1870704595 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa3415ab20 0x7ffa341e5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.969+0000 7ffa33400640 1 -- 172.21.15.29:0/1870704595 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa340c7a20 msgr2=0x7ffa34141950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.969+0000 7ffa33400640 1 --2- 172.21.15.29:0/1870704595 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa340c7a20 0x7ffa34141950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.969+0000 7ffa33400640 1 -- 172.21.15.29:0/1870704595 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa3415ab20 msgr2=0x7ffa341e5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.969+0000 7ffa33400640 1 --2- 172.21.15.29:0/1870704595 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa3415ab20 0x7ffa341e5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.969+0000 7ffa33400640 1 -- 172.21.15.29:0/1870704595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa34142270 con 0x7ffa340c7e30 2024-04-17T19:48:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.969+0000 7ffa33e00640 1 --2- 172.21.15.29:0/1870704595 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa340c7a20 0x7ffa34141950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.969+0000 7ffa33400640 1 --2- 172.21.15.29:0/1870704595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa340c7e30 0x7ffa3415bd50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ffa2000e870 tx=0x7ffa2000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.969+0000 7ffa2be00640 1 -- 172.21.15.29:0/1870704595 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa20017070 con 0x7ffa340c7e30 2024-04-17T19:48:14.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.969+0000 7ffa2be00640 1 -- 172.21.15.29:0/1870704595 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa2000c940 con 0x7ffa340c7e30 2024-04-17T19:48:14.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.969+0000 7ffa2be00640 1 -- 172.21.15.29:0/1870704595 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2000ccf0 con 0x7ffa340c7e30 2024-04-17T19:48:14.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.969+0000 7ffa3b200640 1 -- 172.21.15.29:0/1870704595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa34140d50 con 0x7ffa340c7e30 2024-04-17T19:48:14.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.969+0000 7ffa3b200640 1 -- 172.21.15.29:0/1870704595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa34141310 con 0x7ffa340c7e30 2024-04-17T19:48:14.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.969+0000 7ffa3b200640 1 -- 172.21.15.29:0/1870704595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa34142550 con 0x7ffa340c7e30 2024-04-17T19:48:14.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.969+0000 7ffa2be00640 1 -- 172.21.15.29:0/1870704595 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ffa2001a070 con 0x7ffa340c7e30 2024-04-17T19:48:14.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.969+0000 7ffa2be00640 1 --2- 172.21.15.29:0/1870704595 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ffa14068380 0x7ffa1406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:14.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.969+0000 7ffa2be00640 1 -- 172.21.15.29:0/1870704595 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ffa20086c80 con 0x7ffa340c7e30 2024-04-17T19:48:14.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.969+0000 7ffa33e00640 1 --2- 172.21.15.29:0/1870704595 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ffa14068380 0x7ffa1406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:14.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.973+0000 7ffa33e00640 1 --2- 172.21.15.29:0/1870704595 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ffa14068380 0x7ffa1406a890 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009b10 tx=0x7ffa24008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:14.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:14.977+0000 7ffa2be00640 1 -- 172.21.15.29:0/1870704595 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ffa34142550 con 0x7ffa340c7e30 2024-04-17T19:48:15.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.145+0000 7ffa3b200640 1 -- 172.21.15.29:0/1870704595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffa34140b40 con 0x7ffa340c7e30 2024-04-17T19:48:15.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.145+0000 7ffa2be00640 1 -- 172.21.15.29:0/1870704595 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18394 (secure 0 0 0) 0x7ffa20056dd0 con 0x7ffa340c7e30 2024-04-17T19:48:15.134 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:15.134 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:47:54.321794+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T19:47:47.658449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T19:47:48.209047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T19:47:51.887909+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:48:15.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.149+0000 7ffa3b200640 1 -- 172.21.15.29:0/1870704595 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ffa14068380 msgr2=0x7ffa1406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.149+0000 7ffa3b200640 1 --2- 172.21.15.29:0/1870704595 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ffa14068380 0x7ffa1406a890 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009b10 tx=0x7ffa24008040 comp rx=0 tx=0).stop 2024-04-17T19:48:15.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.149+0000 7ffa3b200640 1 -- 172.21.15.29:0/1870704595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa340c7e30 msgr2=0x7ffa3415bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.149+0000 7ffa3b200640 1 --2- 172.21.15.29:0/1870704595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa340c7e30 0x7ffa3415bd50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ffa2000e870 tx=0x7ffa2000ec70 comp rx=0 tx=0).stop 2024-04-17T19:48:15.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.149+0000 7ffa3b200640 1 -- 172.21.15.29:0/1870704595 shutdown_connections 2024-04-17T19:48:15.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.149+0000 7ffa3b200640 1 --2- 172.21.15.29:0/1870704595 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ffa14068380 0x7ffa1406a890 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.149+0000 7ffa3b200640 1 --2- 172.21.15.29:0/1870704595 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa3415ab20 0x7ffa341e5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.149+0000 7ffa3b200640 1 --2- 172.21.15.29:0/1870704595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa340c7e30 0x7ffa3415bd50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.149+0000 7ffa3b200640 1 --2- 172.21.15.29:0/1870704595 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa340c7a20 0x7ffa34141950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.149+0000 7ffa3b200640 1 -- 172.21.15.29:0/1870704595 >> 172.21.15.29:0/1870704595 conn(0x7ffa34072e50 msgr2=0x7ffa3406d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.149+0000 7ffa3b200640 1 -- 172.21.15.29:0/1870704595 shutdown_connections 2024-04-17T19:48:15.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.149+0000 7ffa3b200640 1 -- 172.21.15.29:0/1870704595 wait complete. 2024-04-17T19:48:15.155 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-17T19:48:15.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:48:15.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.317+0000 7fee52a00640 1 Processor -- start 2024-04-17T19:48:15.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.317+0000 7fee52a00640 1 -- start start 2024-04-17T19:48:15.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.321+0000 7fee52a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee4c1561c0 0x7fee4c0c7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.321+0000 7fee52a00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fee4c0cb870 0x7fee4c0cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.321+0000 7fee52a00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee4c15cdb0 con 0x7fee4c0c83d0 2024-04-17T19:48:15.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.321+0000 7fee52a00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fee4c0c9d00 con 0x7fee4c0cb870 2024-04-17T19:48:15.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.321+0000 7fee52a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fee4c0cdc10 con 0x7fee4c1561c0 2024-04-17T19:48:15.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.321+0000 7fee43e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fee4c0c83d0 0x7fee4c0cd180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38094/0 (socket says 172.21.15.29:38094) 2024-04-17T19:48:15.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.321+0000 7fee43e00640 1 -- 172.21.15.29:0/2706589287 learned_addr learned my addr 172.21.15.29:0/2706589287 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:15.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.321+0000 7fee4be00640 1 --2- 172.21.15.29:0/2706589287 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee4c1561c0 0x7fee4c0c7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.321+0000 7fee50a00640 1 --2- 172.21.15.29:0/2706589287 >> v2:172.21.15.37:3300/0 conn(0x7fee4c0cb870 0x7fee4c0cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.321+0000 7fee4b400640 1 -- 172.21.15.29:0/2706589287 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 387201531 0 0) 0x7fee4c15cdb0 con 0x7fee4c0c83d0 2024-04-17T19:48:15.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.321+0000 7fee4b400640 1 -- 172.21.15.29:0/2706589287 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee30003530 con 0x7fee4c0c83d0 2024-04-17T19:48:15.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.321+0000 7fee4be00640 1 -- 172.21.15.29:0/2706589287 >> v2:172.21.15.37:3300/0 conn(0x7fee4c0cb870 msgr2=0x7fee4c0cbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:15.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.321+0000 7fee4be00640 1 --2- 172.21.15.29:0/2706589287 >> v2:172.21.15.37:3300/0 conn(0x7fee4c0cb870 0x7fee4c0cbc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.321+0000 7fee4be00640 1 -- 172.21.15.29:0/2706589287 >> v1:172.21.15.29:6789/0 conn(0x7fee4c0c83d0 legacy=0x7fee4c0cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.321+0000 7fee4b400640 1 -- 172.21.15.29:0/2706589287 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4114950249 0 0) 0x7fee30003530 con 0x7fee4c0c83d0 2024-04-17T19:48:15.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.321+0000 7fee4be00640 1 -- 172.21.15.29:0/2706589287 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee4c0c9f60 con 0x7fee4c1561c0 2024-04-17T19:48:15.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.325+0000 7fee4be00640 1 --2- 172.21.15.29:0/2706589287 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee4c1561c0 0x7fee4c0c7e90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fee3c0097c0 tx=0x7fee3c01ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66683859eef8094e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.325+0000 7fee4b400640 1 -- 172.21.15.29:0/2706589287 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c0b5070 con 0x7fee4c1561c0 2024-04-17T19:48:15.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.325+0000 7fee4b400640 1 -- 172.21.15.29:0/2706589287 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee3c0a85f0 con 0x7fee4c1561c0 2024-04-17T19:48:15.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.325+0000 7fee4b400640 1 -- 172.21.15.29:0/2706589287 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee34001680 con 0x7fee4c1561c0 2024-04-17T19:48:15.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.325+0000 7fee52a00640 1 -- 172.21.15.29:0/2706589287 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee4c1561c0 msgr2=0x7fee4c0c7e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.325+0000 7fee52a00640 1 --2- 172.21.15.29:0/2706589287 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee4c1561c0 0x7fee4c0c7e90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fee3c0097c0 tx=0x7fee3c01ebb0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.325+0000 7fee52a00640 1 -- 172.21.15.29:0/2706589287 shutdown_connections 2024-04-17T19:48:15.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.325+0000 7fee52a00640 1 --2- 172.21.15.29:0/2706589287 >> v2:172.21.15.37:3300/0 conn(0x7fee4c0cb870 0x7fee4c0cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.325+0000 7fee52a00640 1 --2- 172.21.15.29:0/2706589287 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee4c1561c0 0x7fee4c0c7e90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.325+0000 7fee52a00640 1 -- 172.21.15.29:0/2706589287 >> 172.21.15.29:0/2706589287 conn(0x7fee4c06bdc0 msgr2=0x7fee4c06c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.325+0000 7fee52a00640 1 -- 172.21.15.29:0/2706589287 shutdown_connections 2024-04-17T19:48:15.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.325+0000 7fee52a00640 1 -- 172.21.15.29:0/2706589287 wait complete. 2024-04-17T19:48:15.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.325+0000 7fee52a00640 1 Processor -- start 2024-04-17T19:48:15.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee52a00640 1 -- start start 2024-04-17T19:48:15.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee52a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee4c0c83d0 0x7fee4c157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee52a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fee4c0cb870 0x7fee4c15d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee52a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee4c1561c0 0x7fee4c14cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee52a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c0ca800 con 0x7fee4c1561c0 2024-04-17T19:48:15.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee52a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c0ca520 con 0x7fee4c0cb870 2024-04-17T19:48:15.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee52a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fee4c1565d0 con 0x7fee4c0c83d0 2024-04-17T19:48:15.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee50a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee4c1561c0 0x7fee4c14cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee50a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee4c1561c0 0x7fee4c14cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52588/0 (socket says 172.21.15.29:52588) 2024-04-17T19:48:15.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee50a00640 1 -- 172.21.15.29:0/3692873709 learned_addr learned my addr 172.21.15.29:0/3692873709 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:15.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee43e00640 1 --2- 172.21.15.29:0/3692873709 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fee4c0cb870 0x7fee4c15d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee4be00640 1 --2- 172.21.15.29:0/3692873709 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee4c0c83d0 0x7fee4c157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee50a00640 1 -- 172.21.15.29:0/3692873709 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee4c0c83d0 msgr2=0x7fee4c157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee50a00640 1 --2- 172.21.15.29:0/3692873709 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee4c0c83d0 0x7fee4c157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee50a00640 1 -- 172.21.15.29:0/3692873709 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fee4c0cb870 msgr2=0x7fee4c15d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee50a00640 1 --2- 172.21.15.29:0/3692873709 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fee4c0cb870 0x7fee4c15d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee50a00640 1 -- 172.21.15.29:0/3692873709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee4c158180 con 0x7fee4c1561c0 2024-04-17T19:48:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee4be00640 1 --2- 172.21.15.29:0/3692873709 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee4c0c83d0 0x7fee4c157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee43e00640 1 --2- 172.21.15.29:0/3692873709 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fee4c0cb870 0x7fee4c15d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee50a00640 1 --2- 172.21.15.29:0/3692873709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee4c1561c0 0x7fee4c14cfa0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fee44009d30 tx=0x7fee4400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee48c00640 1 -- 172.21.15.29:0/3692873709 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee44018020 con 0x7fee4c1561c0 2024-04-17T19:48:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee48c00640 1 -- 172.21.15.29:0/3692873709 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee44014480 con 0x7fee4c1561c0 2024-04-17T19:48:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee52a00640 1 -- 172.21.15.29:0/3692873709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee4c15f5d0 con 0x7fee4c1561c0 2024-04-17T19:48:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee48c00640 1 -- 172.21.15.29:0/3692873709 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee44014870 con 0x7fee4c1561c0 2024-04-17T19:48:15.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.329+0000 7fee52a00640 1 -- 172.21.15.29:0/3692873709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee4c15fb70 con 0x7fee4c1561c0 2024-04-17T19:48:15.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.333+0000 7fee52a00640 1 -- 172.21.15.29:0/3692873709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee4c069b90 con 0x7fee4c1561c0 2024-04-17T19:48:15.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.333+0000 7fee48c00640 1 -- 172.21.15.29:0/3692873709 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fee4401b020 con 0x7fee4c1561c0 2024-04-17T19:48:15.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.333+0000 7fee48c00640 1 --2- 172.21.15.29:0/3692873709 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fee28068380 0x7fee2806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.333+0000 7fee48c00640 1 -- 172.21.15.29:0/3692873709 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fee44087840 con 0x7fee4c1561c0 2024-04-17T19:48:15.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.333+0000 7fee4be00640 1 --2- 172.21.15.29:0/3692873709 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fee28068380 0x7fee2806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.333+0000 7fee4be00640 1 --2- 172.21.15.29:0/3692873709 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fee28068380 0x7fee2806a890 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fee3c0b2000 tx=0x7fee3c0aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.337+0000 7fee48c00640 1 -- 172.21.15.29:0/3692873709 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fee44022070 con 0x7fee4c1561c0 2024-04-17T19:48:15.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.501+0000 7fee52a00640 1 -- 172.21.15.29:0/3692873709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fee4c15a090 con 0x7fee4c1561c0 2024-04-17T19:48:15.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.505+0000 7fee48c00640 1 -- 172.21.15.29:0/3692873709 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18394 (secure 0 0 0) 0x7fee44010040 con 0x7fee4c1561c0 2024-04-17T19:48:15.492 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:15.493 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:47:54.321794+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T19:47:47.658449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T19:47:48.209047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T19:47:51.887909+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:48:15.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.505+0000 7fee52a00640 1 -- 172.21.15.29:0/3692873709 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fee28068380 msgr2=0x7fee2806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.505+0000 7fee52a00640 1 --2- 172.21.15.29:0/3692873709 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fee28068380 0x7fee2806a890 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fee3c0b2000 tx=0x7fee3c0aa070 comp rx=0 tx=0).stop 2024-04-17T19:48:15.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.505+0000 7fee52a00640 1 -- 172.21.15.29:0/3692873709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee4c1561c0 msgr2=0x7fee4c14cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.505+0000 7fee52a00640 1 --2- 172.21.15.29:0/3692873709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee4c1561c0 0x7fee4c14cfa0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fee44009d30 tx=0x7fee4400c490 comp rx=0 tx=0).stop 2024-04-17T19:48:15.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.505+0000 7fee52a00640 1 -- 172.21.15.29:0/3692873709 shutdown_connections 2024-04-17T19:48:15.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.505+0000 7fee52a00640 1 --2- 172.21.15.29:0/3692873709 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fee28068380 0x7fee2806a890 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.505+0000 7fee52a00640 1 --2- 172.21.15.29:0/3692873709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee4c1561c0 0x7fee4c14cfa0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.505+0000 7fee52a00640 1 --2- 172.21.15.29:0/3692873709 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fee4c0cb870 0x7fee4c15d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.505+0000 7fee52a00640 1 --2- 172.21.15.29:0/3692873709 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee4c0c83d0 0x7fee4c157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.505+0000 7fee52a00640 1 -- 172.21.15.29:0/3692873709 >> 172.21.15.29:0/3692873709 conn(0x7fee4c06bdc0 msgr2=0x7fee4c06fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.505+0000 7fee52a00640 1 -- 172.21.15.29:0/3692873709 shutdown_connections 2024-04-17T19:48:15.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.505+0000 7fee52a00640 1 -- 172.21.15.29:0/3692873709 wait complete. 2024-04-17T19:48:15.515 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-17T19:48:15.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-17T19:48:15.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-17T19:48:15.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-17T19:48:15.518 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-17T19:48:15.518 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-17T19:48:15.522 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-17T19:48:15.522 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-17T19:48:15.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.813+0000 7f5d50200640 1 Processor -- start 2024-04-17T19:48:15.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.813+0000 7f5d50200640 1 -- start start 2024-04-17T19:48:15.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.813+0000 7fa755800640 1 Processor -- start 2024-04-17T19:48:15.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.813+0000 7fa755800640 1 -- start start 2024-04-17T19:48:15.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.817+0000 7f5d50200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d400dda20 0x7f5d400457e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.817+0000 7f5d50200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5d400060f0 0x7f5d40006500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.817+0000 7f5d50200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d400e7960 con 0x7f5d4004b690 2024-04-17T19:48:15.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.817+0000 7f5d50200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f5d400dde30 con 0x7f5d400060f0 2024-04-17T19:48:15.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.817+0000 7f5d50200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5d400e7c20 con 0x7f5d400dda20 2024-04-17T19:48:15.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.817+0000 7f5d4d800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d400dda20 0x7f5d400457e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.817+0000 7f5d4d800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d400dda20 0x7f5d400457e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36238/0 (socket says 172.21.15.29:36238) 2024-04-17T19:48:15.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.817+0000 7f5d4d800640 1 -- 172.21.15.29:0/1594174318 learned_addr learned my addr 172.21.15.29:0/1594174318 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:15.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.817+0000 7f5d4ce00640 1 --2- 172.21.15.29:0/1594174318 >> v2:172.21.15.37:3300/0 conn(0x7f5d400060f0 0x7f5d40006500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.817+0000 7fe042600640 1 Processor -- start 2024-04-17T19:48:15.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.817+0000 7fe042600640 1 -- start start 2024-04-17T19:48:15.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.817+0000 7f5d4d800640 1 -- 172.21.15.29:0/1594174318 >> v2:172.21.15.37:3300/0 conn(0x7f5d400060f0 msgr2=0x7f5d40006500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.817+0000 7fa0d8000640 1 Processor -- start 2024-04-17T19:48:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.817+0000 7fa755800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7500c8d20 0x7fa7500c9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.817+0000 7f5d4d800640 1 --2- 172.21.15.29:0/1594174318 >> v2:172.21.15.37:3300/0 conn(0x7f5d400060f0 0x7f5d40006500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.817+0000 7f5d4d800640 1 -- 172.21.15.29:0/1594174318 >> v1:172.21.15.37:6789/0 conn(0x7f5d4004b690 legacy=0x7f5d400462f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7f5d4d800640 1 -- 172.21.15.29:0/1594174318 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d400e4620 con 0x7f5d400dda20 2024-04-17T19:48:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fa755800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7500c8280 con 0x7fa75006df30 2024-04-17T19:48:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fa755800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7500cab90 con 0x7fa750160c80 2024-04-17T19:48:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.817+0000 7fa0d8000640 1 -- start start 2024-04-17T19:48:15.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fa755800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa750072550 con 0x7fa7500c8d20 2024-04-17T19:48:15.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fa74fe00640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa750160c80 0x7fa75015a340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:49792/0 (socket says 172.21.15.29:49792) 2024-04-17T19:48:15.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fa74fe00640 1 -- 172.21.15.29:0/3558700059 learned_addr learned my addr 172.21.15.29:0/3558700059 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:15.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fa74f400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7500c8d20 0x7fa7500c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fa74ea00640 1 -- 172.21.15.29:0/3558700059 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3463118556 0 0) 0x7fa7500cab90 con 0x7fa750160c80 2024-04-17T19:48:15.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fa74ea00640 1 -- 172.21.15.29:0/3558700059 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa734003530 con 0x7fa750160c80 2024-04-17T19:48:15.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fe042600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe03c15f480 0x7fe03c156200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fe042600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe03c15c920 0x7fe03c0c2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fe042600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe03c15f9e0 con 0x7fe03c0c32b0 2024-04-17T19:48:15.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fe042600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fe03c0c8ac0 con 0x7fe03c15c920 2024-04-17T19:48:15.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fe042600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe03c0c2600 con 0x7fe03c15f480 2024-04-17T19:48:15.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fa74f400640 1 -- 172.21.15.29:0/3558700059 >> v1:172.21.15.37:6789/0 conn(0x7fa750160c80 legacy=0x7fa75015a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fa74f400640 1 -- 172.21.15.29:0/3558700059 >> v1:172.21.15.29:6789/0 conn(0x7fa75006df30 legacy=0x7fa75016aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fa74f400640 1 -- 172.21.15.29:0/3558700059 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7500cae70 con 0x7fa7500c8d20 2024-04-17T19:48:15.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fe03be00640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe03c0c32b0 0x7fe03c0c7b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:49794/0 (socket says 172.21.15.29:49794) 2024-04-17T19:48:15.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fe03be00640 1 -- 172.21.15.29:0/2810439213 learned_addr learned my addr 172.21.15.29:0/2810439213 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:15.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fe03b400640 1 --2- 172.21.15.29:0/2810439213 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe03c15f480 0x7fe03c156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fe03aa00640 1 --2- 172.21.15.29:0/2810439213 >> v2:172.21.15.29:3300/0 conn(0x7fe03c15c920 0x7fe03c0c2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fe03a000640 1 -- 172.21.15.29:0/2810439213 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1217798134 0 0) 0x7fe03c15f9e0 con 0x7fe03c0c32b0 2024-04-17T19:48:15.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fe03a000640 1 -- 172.21.15.29:0/2810439213 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe024003530 con 0x7fe03c0c32b0 2024-04-17T19:48:15.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.821+0000 7fa0d8000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa0d0157f60 0x7fa0d006f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fe03aa00640 1 -- 172.21.15.29:0/2810439213 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe03c15f480 msgr2=0x7fe03c156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:15.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fe03aa00640 1 --2- 172.21.15.29:0/2810439213 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe03c15f480 0x7fe03c156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fe03aa00640 1 -- 172.21.15.29:0/2810439213 >> v1:172.21.15.37:6789/0 conn(0x7fe03c0c32b0 legacy=0x7fe03c0c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fe03aa00640 1 -- 172.21.15.29:0/2810439213 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe03c15fcc0 con 0x7fe03c15c920 2024-04-17T19:48:15.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0d8000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa0d006fc10 0x7fa0d01628e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0d8000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0d0071d50 con 0x7fa0d00c9160 2024-04-17T19:48:15.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0d8000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fa0d00c7de0 con 0x7fa0d006fc10 2024-04-17T19:48:15.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0d8000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa0d00c80c0 con 0x7fa0d0157f60 2024-04-17T19:48:15.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0d5600640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa0d00c9160 0x7fa0d0072a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:49804/0 (socket says 172.21.15.29:49804) 2024-04-17T19:48:15.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0d5600640 1 -- 172.21.15.29:0/3781153661 learned_addr learned my addr 172.21.15.29:0/3781153661 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:15.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0d4c00640 1 --2- 172.21.15.29:0/3781153661 >> v2:172.21.15.37:3300/0 conn(0x7fa0d0157f60 0x7fa0d006f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7f5d4d800640 1 --2- 172.21.15.29:0/1594174318 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d400dda20 0x7f5d400457e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c006fd0 tx=0x7f5d3c01c580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72c397d3433cdb9a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0d6000640 1 --2- 172.21.15.29:0/3781153661 >> v2:172.21.15.29:3300/0 conn(0x7fa0d006fc10 0x7fa0d01628e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0cfe00640 1 -- 172.21.15.29:0/3781153661 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1983649491 0 0) 0x7fa0d0071d50 con 0x7fa0d00c9160 2024-04-17T19:48:15.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0cfe00640 1 -- 172.21.15.29:0/3781153661 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0b8004640 con 0x7fa0d00c9160 2024-04-17T19:48:15.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7f5d47e00640 1 -- 172.21.15.29:0/1594174318 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c0ab650 con 0x7f5d400dda20 2024-04-17T19:48:15.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7f5d47e00640 1 -- 172.21.15.29:0/1594174318 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d3c0b0070 con 0x7f5d400dda20 2024-04-17T19:48:15.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0cfe00640 1 -- 172.21.15.29:0/3781153661 <== mon.1 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 22667970 0 0) 0x7fa0b8004640 con 0x7fa0d00c9160 2024-04-17T19:48:15.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0cfe00640 1 -- 172.21.15.29:0/3781153661 >> v2:172.21.15.37:3300/0 conn(0x7fa0d0157f60 msgr2=0x7fa0d006f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:15.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0cfe00640 1 --2- 172.21.15.29:0/3781153661 >> v2:172.21.15.37:3300/0 conn(0x7fa0d0157f60 0x7fa0d006f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7f5d47e00640 1 -- 172.21.15.29:0/1594174318 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c0aba10 con 0x7f5d400dda20 2024-04-17T19:48:15.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0cfe00640 1 -- 172.21.15.29:0/3781153661 >> v2:172.21.15.29:3300/0 conn(0x7fa0d006fc10 msgr2=0x7fa0d01628e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:15.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0cfe00640 1 --2- 172.21.15.29:0/3781153661 >> v2:172.21.15.29:3300/0 conn(0x7fa0d006fc10 0x7fa0d01628e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0cfe00640 1 -- 172.21.15.29:0/3781153661 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d00c83a0 con 0x7fa0d00c9160 2024-04-17T19:48:15.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0cfe00640 1 -- 172.21.15.29:0/3781153661 <== mon.1 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fa0c8005060 con 0x7fa0d00c9160 2024-04-17T19:48:15.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0cfe00640 1 -- 172.21.15.29:0/3781153661 >> v1:172.21.15.37:6789/0 conn(0x7fa0d00c9160 legacy=0x7fa0d0072a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0cfe00640 1 --2- 172.21.15.29:0/3781153661 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0b8004ad0 0x7fa0b8004f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0cfe00640 1 --2- 172.21.15.29:0/3781153661 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0b8005480 0x7fa0b80059a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0cfe00640 1 --2- 172.21.15.29:0/3781153661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0b8006010 0x7fa0b8003530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0cfe00640 1 -- 172.21.15.29:0/3781153661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d00c7de0 con 0x7fa0b8006010 2024-04-17T19:48:15.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0cfe00640 1 -- 172.21.15.29:0/3781153661 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d00c80c0 con 0x7fa0b8005480 2024-04-17T19:48:15.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0cfe00640 1 -- 172.21.15.29:0/3781153661 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa0b800cc10 con 0x7fa0b8004ad0 2024-04-17T19:48:15.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0d4c00640 1 --2- 172.21.15.29:0/3781153661 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0b8005480 0x7fa0b80059a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.825+0000 7fa0d5600640 1 --2- 172.21.15.29:0/3781153661 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0b8004ad0 0x7fa0b8004f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7f5d50200640 1 -- 172.21.15.29:0/1594174318 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d400dda20 msgr2=0x7f5d400457e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7f5d50200640 1 --2- 172.21.15.29:0/1594174318 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d400dda20 0x7f5d400457e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c006fd0 tx=0x7f5d3c01c580 comp rx=0 tx=0).stop 2024-04-17T19:48:15.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fa0d6000640 1 --2- 172.21.15.29:0/3781153661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0b8006010 0x7fa0b8003530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fa0d5600640 1 -- 172.21.15.29:0/3781153661 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0b8005480 msgr2=0x7fa0b80059a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:15.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fa0d5600640 1 --2- 172.21.15.29:0/3781153661 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0b8005480 0x7fa0b80059a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fe03aa00640 1 --2- 172.21.15.29:0/2810439213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03c15c920 0x7fe03c0c2d70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe0280097c0 tx=0x7fe02801ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68476deb31e6ba23 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fa0d5600640 1 -- 172.21.15.29:0/3781153661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0b8006010 msgr2=0x7fa0b8003530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:15.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fa0d5600640 1 --2- 172.21.15.29:0/3781153661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0b8006010 0x7fa0b8003530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fa0d5600640 1 -- 172.21.15.29:0/3781153661 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d00c83a0 con 0x7fa0b8004ad0 2024-04-17T19:48:15.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fa0d6000640 1 -- 172.21.15.29:0/3781153661 reap_dead start 2024-04-17T19:48:15.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7f5d50200640 1 -- 172.21.15.29:0/1594174318 shutdown_connections 2024-04-17T19:48:15.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7f5d50200640 1 --2- 172.21.15.29:0/1594174318 >> v2:172.21.15.37:3300/0 conn(0x7f5d400060f0 0x7f5d40006500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7f5d50200640 1 --2- 172.21.15.29:0/1594174318 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d400dda20 0x7f5d400457e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7f5d50200640 1 -- 172.21.15.29:0/1594174318 >> 172.21.15.29:0/1594174318 conn(0x7f5d400503c0 msgr2=0x7f5d400507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7f5d50200640 1 -- 172.21.15.29:0/1594174318 shutdown_connections 2024-04-17T19:48:15.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7f5d50200640 1 -- 172.21.15.29:0/1594174318 wait complete. 2024-04-17T19:48:15.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fa74f400640 1 --2- 172.21.15.29:0/3558700059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7500c8d20 0x7fa7500c9130 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa7400097f0 tx=0x7fa74001ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60ad9974c7b727f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fe03a000640 1 -- 172.21.15.29:0/2810439213 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0280b5070 con 0x7fe03c15c920 2024-04-17T19:48:15.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fe03a000640 1 -- 172.21.15.29:0/2810439213 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0280a8990 con 0x7fe03c15c920 2024-04-17T19:48:15.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fe03a000640 1 -- 172.21.15.29:0/2810439213 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0280a8d60 con 0x7fe03c15c920 2024-04-17T19:48:15.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fa74ea00640 1 -- 172.21.15.29:0/3558700059 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7400b5070 con 0x7fa7500c8d20 2024-04-17T19:48:15.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fa74ea00640 1 -- 172.21.15.29:0/3558700059 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7400a89d0 con 0x7fa7500c8d20 2024-04-17T19:48:15.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fa74ea00640 1 -- 172.21.15.29:0/3558700059 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7400a8da0 con 0x7fa7500c8d20 2024-04-17T19:48:15.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fe042600640 1 -- 172.21.15.29:0/2810439213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03c15c920 msgr2=0x7fe03c0c2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fa755800640 1 -- 172.21.15.29:0/3558700059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7500c8d20 msgr2=0x7fa7500c9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fa755800640 1 --2- 172.21.15.29:0/3558700059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7500c8d20 0x7fa7500c9130 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa7400097f0 tx=0x7fa74001ecd0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fe042600640 1 --2- 172.21.15.29:0/2810439213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03c15c920 0x7fe03c0c2d70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe0280097c0 tx=0x7fe02801ec90 comp rx=0 tx=0).stop 2024-04-17T19:48:15.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fe042600640 1 -- 172.21.15.29:0/2810439213 shutdown_connections 2024-04-17T19:48:15.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fe042600640 1 --2- 172.21.15.29:0/2810439213 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03c15c920 0x7fe03c0c2d70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fe042600640 1 --2- 172.21.15.29:0/2810439213 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe03c15f480 0x7fe03c156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fe042600640 1 -- 172.21.15.29:0/2810439213 >> 172.21.15.29:0/2810439213 conn(0x7fe03c06dd20 msgr2=0x7fe03c06f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fe042600640 1 -- 172.21.15.29:0/2810439213 shutdown_connections 2024-04-17T19:48:15.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.829+0000 7fe042600640 1 -- 172.21.15.29:0/2810439213 wait complete. 2024-04-17T19:48:15.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.833+0000 7fa755800640 1 -- 172.21.15.29:0/3558700059 shutdown_connections 2024-04-17T19:48:15.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.833+0000 7fa755800640 1 --2- 172.21.15.29:0/3558700059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7500c8d20 0x7fa7500c9130 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.833+0000 7fa755800640 1 -- 172.21.15.29:0/3558700059 >> 172.21.15.29:0/3558700059 conn(0x7fa7500cdde0 msgr2=0x7fa7500ca2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.833+0000 7fa755800640 1 -- 172.21.15.29:0/3558700059 shutdown_connections 2024-04-17T19:48:15.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.833+0000 7fa755800640 1 -- 172.21.15.29:0/3558700059 wait complete. 2024-04-17T19:48:15.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.833+0000 7f141ae00640 1 Processor -- start 2024-04-17T19:48:15.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.833+0000 7f5d50200640 1 Processor -- start 2024-04-17T19:48:15.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.833+0000 7f5d50200640 1 -- start start 2024-04-17T19:48:15.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.833+0000 7f141ae00640 1 -- start start 2024-04-17T19:48:15.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.833+0000 7fe042600640 1 Processor -- start 2024-04-17T19:48:15.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.833+0000 7fa0d5600640 1 --2- 172.21.15.29:0/3781153661 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0b8004ad0 0x7fa0b8004f40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8001830 tx=0x7fa0c801cfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66c4b8465f2ac7e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.833+0000 7fa0cfe00640 1 -- 172.21.15.29:0/3781153661 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c8008dd0 con 0x7fa0b8004ad0 2024-04-17T19:48:15.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.833+0000 7fe042600640 1 -- start start 2024-04-17T19:48:15.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.833+0000 7fa755800640 1 Processor -- start 2024-04-17T19:48:15.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fa0cfe00640 1 -- 172.21.15.29:0/3781153661 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0c80a9020 con 0x7fa0b8004ad0 2024-04-17T19:48:15.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fa0cfe00640 1 -- 172.21.15.29:0/3781153661 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0c80080d0 con 0x7fa0b8004ad0 2024-04-17T19:48:15.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fa755800640 1 -- start start 2024-04-17T19:48:15.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5d50200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d400060f0 0x7f5d4004e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5d50200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d4004b690 0x7f5d400cf460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5d4d800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d400060f0 0x7f5d4004e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5d4d800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d400060f0 0x7f5d4004e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36268/0 (socket says 172.21.15.29:36268) 2024-04-17T19:48:15.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5d4ce00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d4004b690 0x7f5d400cf460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5d4ce00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d4004b690 0x7f5d400cf460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46704/0 (socket says 172.21.15.29:46704) 2024-04-17T19:48:15.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f0580000640 1 Processor -- start 2024-04-17T19:48:15.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5d50200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d400dda20 0x7f5d400cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f0580000640 1 -- start start 2024-04-17T19:48:15.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5d4d800640 1 -- 172.21.15.29:0/3980054984 learned_addr learned my addr 172.21.15.29:0/3980054984 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:15.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f141ae00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14141618e0 0x7f1414163d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5d50200640 1 -- 172.21.15.29:0/3980054984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5d400e7670 con 0x7f5d400dda20 2024-04-17T19:48:15.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5410a00640 1 Processor -- start 2024-04-17T19:48:15.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5d50200640 1 -- 172.21.15.29:0/3980054984 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5d400e4100 con 0x7f5d4004b690 2024-04-17T19:48:15.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5d50200640 1 -- 172.21.15.29:0/3980054984 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5d4004beb0 con 0x7f5d400060f0 2024-04-17T19:48:15.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f1413e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1414157f60 0x7f14140c98d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38104/0 (socket says 172.21.15.29:38104) 2024-04-17T19:48:15.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f1413e00640 1 -- 172.21.15.29:0/2222824050 learned_addr learned my addr 172.21.15.29:0/2222824050 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:15.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7ff9b4a00640 1 Processor -- start 2024-04-17T19:48:15.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fe042600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe03c0c32b0 0x7fe03c15a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fa755800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75006df30 0x7fa7500cc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fa755800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7500c8d20 0x7fa7501f9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fa755800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa750160c80 0x7fa7501fa330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fa0d8000640 1 -- 172.21.15.29:0/3781153661 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0b8004ad0 msgr2=0x7fa0b8004f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fa0d8000640 1 --2- 172.21.15.29:0/3781153661 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0b8004ad0 0x7fa0b8004f40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8001830 tx=0x7fa0c801cfd0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5410a00640 1 -- start start 2024-04-17T19:48:15.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f1418e00640 1 --2- 172.21.15.29:0/2222824050 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14141618e0 0x7f1414163d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fa0d8000640 1 -- 172.21.15.29:0/3781153661 shutdown_connections 2024-04-17T19:48:15.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fa0d8000640 1 --2- 172.21.15.29:0/3781153661 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0b8004ad0 0x7fa0b8004f40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fa0d8000640 1 -- 172.21.15.29:0/3781153661 >> 172.21.15.29:0/3781153661 conn(0x7fa0d006bf90 msgr2=0x7fa0d006c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fe042600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe03c15c920 0x7fe03c14dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fe042600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03c15f480 0x7fe03c15d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7ff9b4a00640 1 -- start start 2024-04-17T19:48:15.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5d4d800640 1 -- 172.21.15.29:0/3980054984 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d4004b690 msgr2=0x7f5d400cf460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5d4d800640 1 --2- 172.21.15.29:0/3980054984 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d4004b690 0x7f5d400cf460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5d4d800640 1 -- 172.21.15.29:0/3980054984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d400dda20 msgr2=0x7f5d400cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5d4e200640 1 --2- 172.21.15.29:0/3980054984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d400dda20 0x7f5d400cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5d4d800640 1 --2- 172.21.15.29:0/3980054984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d400dda20 0x7f5d400cf9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f5d4d800640 1 -- 172.21.15.29:0/3980054984 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4004ecb0 con 0x7f5d400060f0 2024-04-17T19:48:15.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fa74f400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7500c8d20 0x7fa7501f9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fa74f400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7500c8d20 0x7fa7501f9df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36294/0 (socket says 172.21.15.29:36294) 2024-04-17T19:48:15.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fa74f400640 1 -- 172.21.15.29:0/3406969078 learned_addr learned my addr 172.21.15.29:0/3406969078 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:15.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fe03b400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe03c0c32b0 0x7fe03c15a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fe03b400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe03c0c32b0 0x7fe03c15a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46708/0 (socket says 172.21.15.29:46708) 2024-04-17T19:48:15.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fe03aa00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe03c15c920 0x7fe03c14dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f141ae00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14140732f0 con 0x7f1414157f60 2024-04-17T19:48:15.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fe042600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c160280 con 0x7fe03c15f480 2024-04-17T19:48:15.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fe042600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe03c0c28e0 con 0x7fe03c0c32b0 2024-04-17T19:48:15.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fe042600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe03c15cd30 con 0x7fe03c15c920 2024-04-17T19:48:15.847 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f5d4d800640 1 --2- 172.21.15.29:0/3980054984 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d400060f0 0x7f5d4004e770 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c01a1b0 tx=0x7f5d3c019320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fe03aa00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe03c15c920 0x7fe03c14dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36304/0 (socket says 172.21.15.29:36304) 2024-04-17T19:48:15.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fe03aa00640 1 -- 172.21.15.29:0/1153649630 learned_addr learned my addr 172.21.15.29:0/1153649630 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:15.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fe03be00640 1 --2- 172.21.15.29:0/1153649630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03c15f480 0x7fe03c15d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fa754e00640 1 --2- 172.21.15.29:0/3406969078 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa750160c80 0x7fa7501fa330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fa755800640 1 -- 172.21.15.29:0/3406969078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa750161090 con 0x7fa75006df30 2024-04-17T19:48:15.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fa755800640 1 -- 172.21.15.29:0/3406969078 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa7500c8800 con 0x7fa750160c80 2024-04-17T19:48:15.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7fa0d8000640 1 -- 172.21.15.29:0/3781153661 shutdown_connections 2024-04-17T19:48:15.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f141ae00640 1 -- 172.21.15.29:0/2222824050 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14140c7de0 con 0x7f1414073d70 2024-04-17T19:48:15.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.837+0000 7f141ae00640 1 -- 172.21.15.29:0/2222824050 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f14140c80c0 con 0x7f14141618e0 2024-04-17T19:48:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7fa755800640 1 -- 172.21.15.29:0/3406969078 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa7500cb150 con 0x7fa7500c8d20 2024-04-17T19:48:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0580000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f057815a880 0x7f057806e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0580000640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f057815d490 con 0x7f0578153ea0 2024-04-17T19:48:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0580000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05781542b0 con 0x7f057806e8f0 2024-04-17T19:48:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0580000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f05780ca1e0 con 0x7f057815a880 2024-04-17T19:48:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7fe03be00640 1 -- 172.21.15.29:0/1153649630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe03c15c920 msgr2=0x7fe03c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f5d45600640 1 -- 172.21.15.29:0/3980054984 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c003b70 con 0x7f5d400060f0 2024-04-17T19:48:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f5d45600640 1 -- 172.21.15.29:0/3980054984 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d3c0bd070 con 0x7f5d400060f0 2024-04-17T19:48:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7fe03be00640 1 --2- 172.21.15.29:0/1153649630 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe03c15c920 0x7fe03c14dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7fe03be00640 1 -- 172.21.15.29:0/1153649630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe03c0c32b0 msgr2=0x7fe03c15a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f5d50200640 1 -- 172.21.15.29:0/3980054984 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d40035760 con 0x7f5d400060f0 2024-04-17T19:48:15.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7fa74fe00640 1 --2- 172.21.15.29:0/3406969078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75006df30 0x7fa7500cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7fe03be00640 1 --2- 172.21.15.29:0/1153649630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe03c0c32b0 0x7fe03c15a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7fe03be00640 1 -- 172.21.15.29:0/1153649630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe03c15ab40 con 0x7fe03c15f480 2024-04-17T19:48:15.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7fe03b400640 1 --2- 172.21.15.29:0/1153649630 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe03c0c32b0 0x7fe03c15a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:48:15.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f057e000640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f057806e8f0 0x7f0578164780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:49824/0 (socket says 172.21.15.29:49824) 2024-04-17T19:48:15.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f057e000640 1 -- 172.21.15.29:0/342739685 learned_addr learned my addr 172.21.15.29:0/342739685 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:15.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f5d50200640 1 -- 172.21.15.29:0/3980054984 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d40035d20 con 0x7f5d400060f0 2024-04-17T19:48:15.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f5d45600640 1 -- 172.21.15.29:0/3980054984 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3c0ab450 con 0x7f5d400060f0 2024-04-17T19:48:15.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f5410a00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f540c0c7b10 0x7f540c15a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7fa754e00640 1 -- 172.21.15.29:0/3406969078 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7500c8d20 msgr2=0x7fa7501f9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7fa754e00640 1 --2- 172.21.15.29:0/3406969078 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7500c8d20 0x7fa7501f9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7fe03be00640 1 --2- 172.21.15.29:0/1153649630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03c15f480 0x7fe03c15d050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe03c0c7a30 tx=0x7fe03400cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f5410a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f540c0cce60 0x7f540c0cd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7fa754e00640 1 -- 172.21.15.29:0/3406969078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75006df30 msgr2=0x7fa7500cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f540b400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f540c0c7b10 0x7f540c15a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7fa754e00640 1 --2- 172.21.15.29:0/3406969078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75006df30 0x7fa7500cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7fa754e00640 1 -- 172.21.15.29:0/3406969078 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7500ccb40 con 0x7fa750160c80 2024-04-17T19:48:15.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7fa74fe00640 1 --2- 172.21.15.29:0/3406969078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa75006df30 0x7fa7500cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:48:15.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f540b400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f540c0c7b10 0x7f540c15a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46726/0 (socket says 172.21.15.29:46726) 2024-04-17T19:48:15.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f540b400640 1 -- 172.21.15.29:0/3316779836 learned_addr learned my addr 172.21.15.29:0/3316779836 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:15.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f5410a00640 1 -- 172.21.15.29:0/3316779836 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f540c0c9730 con 0x7f540c1608e0 2024-04-17T19:48:15.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f5410a00640 1 -- 172.21.15.29:0/3316779836 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f540c0c8bd0 con 0x7f540c0c7b10 2024-04-17T19:48:15.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f5410a00640 1 -- 172.21.15.29:0/3316779836 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f540c15a9a0 con 0x7f540c0cce60 2024-04-17T19:48:15.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0577e00640 1 -- 172.21.15.29:0/342739685 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3722627421 0 0) 0x7f05781542b0 con 0x7f057806e8f0 2024-04-17T19:48:15.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0577e00640 1 -- 172.21.15.29:0/342739685 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0560003530 con 0x7f057806e8f0 2024-04-17T19:48:15.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f5403e00640 1 --2- 172.21.15.29:0/3316779836 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f540c0cce60 0x7f540c0cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7fa754e00640 1 --2- 172.21.15.29:0/3406969078 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa750160c80 0x7fa7501fa330 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa750164830 tx=0x7fa73c00c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f540aa00640 1 -- 172.21.15.29:0/3316779836 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1164722935 0 0) 0x7f540c0c9730 con 0x7f540c1608e0 2024-04-17T19:48:15.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f540aa00640 1 -- 172.21.15.29:0/3316779836 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53f0003530 con 0x7f540c1608e0 2024-04-17T19:48:15.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f1412a00640 1 -- 172.21.15.29:0/2222824050 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2154111492 0 0) 0x7f14140c7de0 con 0x7f1414073d70 2024-04-17T19:48:15.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f1412a00640 1 -- 172.21.15.29:0/2222824050 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13f8003530 con 0x7f1414073d70 2024-04-17T19:48:15.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f1412a00640 1 -- 172.21.15.29:0/2222824050 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3841758972 0 0) 0x7f14140732f0 con 0x7f1414157f60 2024-04-17T19:48:15.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f1412a00640 1 -- 172.21.15.29:0/2222824050 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14140c7de0 con 0x7f1414157f60 2024-04-17T19:48:15.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f057cc00640 1 --2- 172.21.15.29:0/342739685 >> v2:172.21.15.29:3300/0 conn(0x7f057815a880 0x7f057806e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0577e00640 1 -- 172.21.15.29:0/342739685 <== mon.1 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1079709550 0 0) 0x7f0560003530 con 0x7f057806e8f0 2024-04-17T19:48:15.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0577e00640 1 -- 172.21.15.29:0/342739685 >> v2:172.21.15.29:3300/0 conn(0x7f057815a880 msgr2=0x7f057806e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:15.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0577e00640 1 --2- 172.21.15.29:0/342739685 >> v2:172.21.15.29:3300/0 conn(0x7f057815a880 0x7f057806e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0577e00640 1 -- 172.21.15.29:0/342739685 >> v1:172.21.15.29:6789/0 conn(0x7f0578153ea0 legacy=0x7f05780c4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:15.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0577e00640 1 -- 172.21.15.29:0/342739685 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05780caaf0 con 0x7f057806e8f0 2024-04-17T19:48:15.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0577e00640 1 -- 172.21.15.29:0/342739685 <== mon.1 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f0570001a00 con 0x7f057806e8f0 2024-04-17T19:48:15.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0577e00640 1 -- 172.21.15.29:0/342739685 >> v1:172.21.15.37:6789/0 conn(0x7f057806e8f0 legacy=0x7f0578164780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7ff9b4a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9b00c69a0 0x7ff9b00c6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0577e00640 1 --2- 172.21.15.29:0/342739685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05600039c0 0x7f0560003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0577e00640 1 --2- 172.21.15.29:0/342739685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0560004810 0x7f0560004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0577e00640 1 --2- 172.21.15.29:0/342739685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560005280 0x7f056000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0577e00640 1 -- 172.21.15.29:0/342739685 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f05780ca1e0 con 0x7f05600039c0 2024-04-17T19:48:15.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0577e00640 1 -- 172.21.15.29:0/342739685 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f056000c200 con 0x7f0560004810 2024-04-17T19:48:15.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f0577e00640 1 -- 172.21.15.29:0/342739685 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f056000c400 con 0x7f0560005280 2024-04-17T19:48:15.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f5d50200640 1 -- 172.21.15.29:0/3980054984 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5d4004e350 con 0x7f5d400060f0 2024-04-17T19:48:15.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f5d45600640 1 -- 172.21.15.29:0/3980054984 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5d3c0ab5f0 con 0x7f5d400060f0 2024-04-17T19:48:15.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f5d45600640 1 --2- 172.21.15.29:0/3980054984 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5d280683b0 0x7f5d2806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f5d4ce00640 1 --2- 172.21.15.29:0/3980054984 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5d280683b0 0x7f5d2806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7fe02f400640 1 -- 172.21.15.29:0/1153649630 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe034017070 con 0x7fe03c15f480 2024-04-17T19:48:15.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7fe042600640 1 -- 172.21.15.29:0/1153649630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe03c1517b0 con 0x7fe03c15f480 2024-04-17T19:48:15.864 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7fe042600640 1 -- 172.21.15.29:0/1153649630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe03c073d90 con 0x7fe03c15f480 2024-04-17T19:48:15.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7fa747e00640 1 -- 172.21.15.29:0/3406969078 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73c017070 con 0x7fa750160c80 2024-04-17T19:48:15.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7fa755800640 1 -- 172.21.15.29:0/3406969078 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa75015f690 con 0x7fa750160c80 2024-04-17T19:48:15.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7fa755800640 1 -- 172.21.15.29:0/3406969078 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7500cb840 con 0x7fa750160c80 2024-04-17T19:48:15.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7fa747e00640 1 -- 172.21.15.29:0/3406969078 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa73c0052d0 con 0x7fa750160c80 2024-04-17T19:48:15.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7fa747e00640 1 -- 172.21.15.29:0/3406969078 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa73c003840 con 0x7fa750160c80 2024-04-17T19:48:15.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7fa755800640 1 -- 172.21.15.29:0/3406969078 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa75004ff10 con 0x7fa750160c80 2024-04-17T19:48:15.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7fa0d8000640 1 -- 172.21.15.29:0/3781153661 wait complete. 2024-04-17T19:48:15.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7fa0d8000640 1 Processor -- start 2024-04-17T19:48:15.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f540aa00640 1 -- 172.21.15.29:0/3316779836 <== mon.1 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3285459398 0 0) 0x7f53f0003530 con 0x7f540c1608e0 2024-04-17T19:48:15.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f540aa00640 1 -- 172.21.15.29:0/3316779836 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f540c0cce60 msgr2=0x7f540c0cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:15.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f540aa00640 1 --2- 172.21.15.29:0/3316779836 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f540c0cce60 0x7f540c0cd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f540aa00640 1 -- 172.21.15.29:0/3316779836 >> v2:172.21.15.37:3300/0 conn(0x7f540c0c7b10 msgr2=0x7f540c15a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f540aa00640 1 --2- 172.21.15.29:0/3316779836 >> v2:172.21.15.37:3300/0 conn(0x7f540c0c7b10 0x7f540c15a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.841+0000 7f540aa00640 1 -- 172.21.15.29:0/3316779836 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f540c0c8e90 con 0x7f540c1608e0 2024-04-17T19:48:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f540aa00640 1 -- 172.21.15.29:0/3316779836 <== mon.1 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f5404001a00 con 0x7f540c1608e0 2024-04-17T19:48:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f540aa00640 1 -- 172.21.15.29:0/3316779836 >> v1:172.21.15.37:6789/0 conn(0x7f540c1608e0 legacy=0x7f540c0c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f540aa00640 1 --2- 172.21.15.29:0/3316779836 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53f00039c0 0x7f53f0003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f540aa00640 1 --2- 172.21.15.29:0/3316779836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53f0004810 0x7f53f0004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f540aa00640 1 --2- 172.21.15.29:0/3316779836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f0005160 0x7f53f000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f540aa00640 1 -- 172.21.15.29:0/3316779836 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f540c0c8bd0 con 0x7f53f0005160 2024-04-17T19:48:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f540aa00640 1 -- 172.21.15.29:0/3316779836 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f540c15a9a0 con 0x7f53f0004810 2024-04-17T19:48:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f540aa00640 1 -- 172.21.15.29:0/3316779836 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f53f000c0a0 con 0x7f53f00039c0 2024-04-17T19:48:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f540be00640 1 --2- 172.21.15.29:0/3316779836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f0005160 0x7f53f000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f5403e00640 1 --2- 172.21.15.29:0/3316779836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53f0004810 0x7f53f0004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f540b400640 1 --2- 172.21.15.29:0/3316779836 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53f00039c0 0x7f53f0003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f540b400640 1 -- 172.21.15.29:0/3316779836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53f0004810 msgr2=0x7f53f0004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:15.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f540b400640 1 --2- 172.21.15.29:0/3316779836 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53f0004810 0x7f53f0004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f540b400640 1 -- 172.21.15.29:0/3316779836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f0005160 msgr2=0x7f53f000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:15.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f540b400640 1 --2- 172.21.15.29:0/3316779836 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f0005160 0x7f53f000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f540b400640 1 -- 172.21.15.29:0/3316779836 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f0003530 con 0x7f53f00039c0 2024-04-17T19:48:15.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f057cc00640 1 --2- 172.21.15.29:0/342739685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0560004810 0x7f0560004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f057d600640 1 --2- 172.21.15.29:0/342739685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05600039c0 0x7f0560003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f057e000640 1 --2- 172.21.15.29:0/342739685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560005280 0x7f056000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f057d600640 1 -- 172.21.15.29:0/342739685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560005280 msgr2=0x7f056000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f057d600640 1 --2- 172.21.15.29:0/342739685 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0560005280 0x7f056000b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f057d600640 1 -- 172.21.15.29:0/342739685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0560004810 msgr2=0x7f0560004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f057d600640 1 --2- 172.21.15.29:0/342739685 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0560004810 0x7f0560004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f057d600640 1 -- 172.21.15.29:0/342739685 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05780caaf0 con 0x7f05600039c0 2024-04-17T19:48:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7ff9b4a00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9b015f6a0 con 0x7ff9b00729b0 2024-04-17T19:48:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7ff9b4a00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9b015f980 con 0x7ff9b015c790 2024-04-17T19:48:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7ff9b4a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff9b015f450 con 0x7ff9b00c69a0 2024-04-17T19:48:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7ff9af400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9b00c69a0 0x7ff9b00c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7ff9af400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9b00c69a0 0x7ff9b00c6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36320/0 (socket says 172.21.15.29:36320) 2024-04-17T19:48:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7ff9af400640 1 -- 172.21.15.29:0/2621728579 learned_addr learned my addr 172.21.15.29:0/2621728579 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:15.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7ff9aea00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff9b00729b0 0x7ff9b015c080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38112/0 (socket says 172.21.15.29:38112) 2024-04-17T19:48:15.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7ff9ae000640 1 -- 172.21.15.29:0/2621728579 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2244337988 0 0) 0x7ff9b015f6a0 con 0x7ff9b00729b0 2024-04-17T19:48:15.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7ff9ae000640 1 -- 172.21.15.29:0/2621728579 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff994003530 con 0x7ff9b00729b0 2024-04-17T19:48:15.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7ff9af400640 1 -- 172.21.15.29:0/2621728579 >> v1:172.21.15.37:6789/0 conn(0x7ff9b015c790 legacy=0x7ff9b01667e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:15.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7ff9af400640 1 -- 172.21.15.29:0/2621728579 >> v1:172.21.15.29:6789/0 conn(0x7ff9b00729b0 legacy=0x7ff9b015c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7ff9af400640 1 -- 172.21.15.29:0/2621728579 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b015fc60 con 0x7ff9b00c69a0 2024-04-17T19:48:15.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7f5d45600640 1 -- 172.21.15.29:0/3980054984 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5d3c121120 con 0x7f5d400060f0 2024-04-17T19:48:15.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7f5d4ce00640 1 --2- 172.21.15.29:0/3980054984 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5d280683b0 0x7f5d2806a8c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5d400353b0 tx=0x7f5d34009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7f5d45600640 1 --2- 172.21.15.29:0/3980054984 >> [v2:172.21.15.29:6800/515534509,v1:172.21.15.29:6801/515534509] conn(0x7f5d28071a70 0x7f5d28073f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f1418e00640 1 -- 172.21.15.29:0/2222824050 >> v1:172.21.15.37:6789/0 conn(0x7f1414073d70 legacy=0x7f141406f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f1418e00640 1 -- 172.21.15.29:0/2222824050 >> v1:172.21.15.29:6789/0 conn(0x7f1414157f60 legacy=0x7f14140c98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7f1418e00640 1 -- 172.21.15.29:0/2222824050 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14140c83a0 con 0x7f14141618e0 2024-04-17T19:48:15.871 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7f5d45600640 1 -- 172.21.15.29:0/3980054984 --> [v2:172.21.15.29:6800/515534509,v1:172.21.15.29:6801/515534509] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5d3c001d40 con 0x7f5d28071a70 2024-04-17T19:48:15.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7f5d45600640 1 -- 172.21.15.29:0/3980054984 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5d3c128050 con 0x7f5d400060f0 2024-04-17T19:48:15.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7fe02f400640 1 -- 172.21.15.29:0/1153649630 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe03400ab40 con 0x7fe03c15f480 2024-04-17T19:48:15.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.845+0000 7fe02f400640 1 -- 172.21.15.29:0/1153649630 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe034017070 con 0x7fe03c15f480 2024-04-17T19:48:15.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7fe042600640 1 -- 172.21.15.29:0/1153649630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe018001040 con 0x7fe03c15f480 2024-04-17T19:48:15.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7f5d4e200640 1 --2- 172.21.15.29:0/3980054984 >> [v2:172.21.15.29:6800/515534509,v1:172.21.15.29:6801/515534509] conn(0x7f5d28071a70 0x7f5d28073f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7fa747e00640 1 -- 172.21.15.29:0/3406969078 <== mon.1 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa73c01a070 con 0x7fa750160c80 2024-04-17T19:48:15.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7fa747e00640 1 --2- 172.21.15.29:0/3406969078 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa730068380 0x7fa73006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7fa747e00640 1 -- 172.21.15.29:0/3406969078 <== mon.1 v2:172.21.15.37:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa73c086710 con 0x7fa750160c80 2024-04-17T19:48:15.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7fa747e00640 1 --2- 172.21.15.29:0/3406969078 >> [v2:172.21.15.29:6808/868032589,v1:172.21.15.29:6809/868032589] conn(0x7fa730071b40 0x7fa730073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7ff9af400640 1 --2- 172.21.15.29:0/2621728579 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9b00c69a0 0x7ff9b00c6180 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8009e80 tx=0x7ff9a801f200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56ec101a6c7b96eb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7fa747e00640 1 -- 172.21.15.29:0/3406969078 --> [v2:172.21.15.29:6808/868032589,v1:172.21.15.29:6809/868032589] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7300746a0 con 0x7fa730071b40 2024-04-17T19:48:15.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7fa747e00640 1 -- 172.21.15.29:0/3406969078 <== mon.1 v2:172.21.15.37:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa73c086b00 con 0x7fa750160c80 2024-04-17T19:48:15.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7f057e000640 1 -- 172.21.15.29:0/342739685 reap_dead start 2024-04-17T19:48:15.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7fa0d8000640 1 -- start start 2024-04-17T19:48:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7fa0d8000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0b8004ad0 0x7fa0d01ee000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7fa0d8000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0d00c4420 0x7fa0d00cbb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7fa0d8000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0d00cafd0 0x7fa0d0149b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7fa0d8000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d00c8c40 con 0x7fa0b8004ad0 2024-04-17T19:48:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7fa0d8000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d00c8960 con 0x7fa0d00c4420 2024-04-17T19:48:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7fa0d8000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa0d006f6d0 con 0x7fa0d00cafd0 2024-04-17T19:48:15.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7f057d600640 1 --2- 172.21.15.29:0/342739685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05600039c0 0x7f0560003e30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f056c001670 tx=0x7f056c01c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e815705a0ecb1af server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7fa74fe00640 1 --2- 172.21.15.29:0/3406969078 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa730068380 0x7fa73006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7ff9ae000640 1 -- 172.21.15.29:0/2621728579 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a80b4070 con 0x7ff9b00c69a0 2024-04-17T19:48:15.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7ff9ae000640 1 -- 172.21.15.29:0/2621728579 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9a80af400 con 0x7ff9b00c69a0 2024-04-17T19:48:15.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.849+0000 7f540b400640 1 --2- 172.21.15.29:0/3316779836 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53f00039c0 0x7f53f0003e30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f53fc009b90 tx=0x7f53fc01cb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81f1b32abe8eda9e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7ff9ae000640 1 -- 172.21.15.29:0/2621728579 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a80af7b0 con 0x7ff9b00c69a0 2024-04-17T19:48:15.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f5d4e200640 1 --2- 172.21.15.29:0/3980054984 >> [v2:172.21.15.29:6800/515534509,v1:172.21.15.29:6801/515534509] conn(0x7f5d28071a70 0x7f5d28073f00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f0577e00640 1 -- 172.21.15.29:0/342739685 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f056c001270 con 0x7f05600039c0 2024-04-17T19:48:15.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f1418e00640 1 --2- 172.21.15.29:0/2222824050 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14141618e0 0x7f1414163d00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f140c009dc0 tx=0x7f140c01f190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff7cf11b49829ca7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f540aa00640 1 -- 172.21.15.29:0/3316779836 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53fc0b2070 con 0x7f53f00039c0 2024-04-17T19:48:15.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f540aa00640 1 -- 172.21.15.29:0/3316779836 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53fc0a5770 con 0x7f53f00039c0 2024-04-17T19:48:15.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f540aa00640 1 -- 172.21.15.29:0/3316779836 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53fc009760 con 0x7f53f00039c0 2024-04-17T19:48:15.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa0d6000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0d00cafd0 0x7fa0d0149b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa0d6000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0d00cafd0 0x7fa0d0149b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36358/0 (socket says 172.21.15.29:36358) 2024-04-17T19:48:15.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa0d6000640 1 -- 172.21.15.29:0/4245952925 learned_addr learned my addr 172.21.15.29:0/4245952925 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:15.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa0d4c00640 1 --2- 172.21.15.29:0/4245952925 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0d00c4420 0x7fa0d00cbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa0d5600640 1 --2- 172.21.15.29:0/4245952925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0b8004ad0 0x7fa0d01ee000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f0577e00640 1 -- 172.21.15.29:0/342739685 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f056c005270 con 0x7f05600039c0 2024-04-17T19:48:15.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2621728579 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9b00c69a0 msgr2=0x7ff9b00c6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7ff9b4a00640 1 --2- 172.21.15.29:0/2621728579 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9b00c69a0 0x7ff9b00c6180 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8009e80 tx=0x7ff9a801f200 comp rx=0 tx=0).stop 2024-04-17T19:48:15.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f0577e00640 1 -- 172.21.15.29:0/342739685 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f056c001270 con 0x7f05600039c0 2024-04-17T19:48:15.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f0580000640 1 -- 172.21.15.29:0/342739685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05600039c0 msgr2=0x7f0560003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f0580000640 1 --2- 172.21.15.29:0/342739685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05600039c0 0x7f0560003e30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f056c001670 tx=0x7f056c01c340 comp rx=0 tx=0).stop 2024-04-17T19:48:15.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa74f400640 1 --2- 172.21.15.29:0/3406969078 >> [v2:172.21.15.29:6808/868032589,v1:172.21.15.29:6809/868032589] conn(0x7fa730071b40 0x7fa730073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa0d6000640 1 -- 172.21.15.29:0/4245952925 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0d00c4420 msgr2=0x7fa0d00cbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa0d6000640 1 --2- 172.21.15.29:0/4245952925 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa0d00c4420 0x7fa0d00cbb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa0d6000640 1 -- 172.21.15.29:0/4245952925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0b8004ad0 msgr2=0x7fa0d01ee000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa0d6000640 1 --2- 172.21.15.29:0/4245952925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0b8004ad0 0x7fa0d01ee000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa0d6000640 1 -- 172.21.15.29:0/4245952925 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d01f2180 con 0x7fa0d00cafd0 2024-04-17T19:48:15.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa0d5600640 1 --2- 172.21.15.29:0/4245952925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0b8004ad0 0x7fa0d01ee000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:15.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f5410a00640 1 -- 172.21.15.29:0/3316779836 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53f00039c0 msgr2=0x7f53f0003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f5410a00640 1 --2- 172.21.15.29:0/3316779836 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53f00039c0 0x7f53f0003e30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f53fc009b90 tx=0x7f53fc01cb70 comp rx=0 tx=0).stop 2024-04-17T19:48:15.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f0580000640 1 -- 172.21.15.29:0/342739685 shutdown_connections 2024-04-17T19:48:15.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f0580000640 1 --2- 172.21.15.29:0/342739685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05600039c0 0x7f0560003e30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f0580000640 1 -- 172.21.15.29:0/342739685 >> 172.21.15.29:0/342739685 conn(0x7f05780cf770 msgr2=0x7f05780cfb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f540be00640 1 -- 172.21.15.29:0/3316779836 reap_dead start 2024-04-17T19:48:15.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2621728579 shutdown_connections 2024-04-17T19:48:15.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7ff9b4a00640 1 --2- 172.21.15.29:0/2621728579 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9b00c69a0 0x7ff9b00c6180 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2621728579 >> 172.21.15.29:0/2621728579 conn(0x7ff9b00d18e0 msgr2=0x7ff9b00d1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f1412a00640 1 -- 172.21.15.29:0/2222824050 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140c0b4070 con 0x7f14141618e0 2024-04-17T19:48:15.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f1412a00640 1 -- 172.21.15.29:0/2222824050 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f140c0af600 con 0x7f14141618e0 2024-04-17T19:48:15.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f1412a00640 1 -- 172.21.15.29:0/2222824050 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140c0af9d0 con 0x7f14141618e0 2024-04-17T19:48:15.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f5410a00640 1 -- 172.21.15.29:0/3316779836 shutdown_connections 2024-04-17T19:48:15.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f5410a00640 1 -- 172.21.15.29:0/3316779836 >> 172.21.15.29:0/3316779836 conn(0x7f540c06ddd0 msgr2=0x7f540c06bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fe02f400640 1 -- 172.21.15.29:0/1153649630 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe03401a070 con 0x7fe03c15f480 2024-04-17T19:48:15.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa74fe00640 1 --2- 172.21.15.29:0/3406969078 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa730068380 0x7fa73006a890 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa748001570 tx=0x7fa748001a00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa0d6000640 1 --2- 172.21.15.29:0/4245952925 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0d00cafd0 0x7fa0d0149b70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa0d00ca3c0 tx=0x7fa0bc011860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2621728579 shutdown_connections 2024-04-17T19:48:15.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2621728579 wait complete. 2024-04-17T19:48:15.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa0cd600640 1 -- 172.21.15.29:0/4245952925 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc003620 con 0x7fa0d00cafd0 2024-04-17T19:48:15.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa0cd600640 1 -- 172.21.15.29:0/4245952925 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0bc01b070 con 0x7fa0d00cafd0 2024-04-17T19:48:15.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa0cd600640 1 -- 172.21.15.29:0/4245952925 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc003a20 con 0x7fa0d00cafd0 2024-04-17T19:48:15.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f5410a00640 1 -- 172.21.15.29:0/3316779836 shutdown_connections 2024-04-17T19:48:15.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa0d8000640 1 -- 172.21.15.29:0/4245952925 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0d01ee540 con 0x7fa0d00cafd0 2024-04-17T19:48:15.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fa0d8000640 1 -- 172.21.15.29:0/4245952925 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0d01f1570 con 0x7fa0d00cafd0 2024-04-17T19:48:15.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f0580000640 1 -- 172.21.15.29:0/342739685 shutdown_connections 2024-04-17T19:48:15.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f0580000640 1 -- 172.21.15.29:0/342739685 wait complete. 2024-04-17T19:48:15.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f5410a00640 1 -- 172.21.15.29:0/3316779836 wait complete. 2024-04-17T19:48:15.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f141ae00640 1 -- 172.21.15.29:0/2222824050 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14141618e0 msgr2=0x7f1414163d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7f141ae00640 1 --2- 172.21.15.29:0/2222824050 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14141618e0 0x7f1414163d00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f140c009dc0 tx=0x7f140c01f190 comp rx=0 tx=0).stop 2024-04-17T19:48:15.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.853+0000 7fe02f400640 1 --2- 172.21.15.29:0/1153649630 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe01c068580 0x7fe01c06aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7f5d45600640 1 -- 172.21.15.29:0/3980054984 <== osd.0 v2:172.21.15.29:6800/515534509 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5d3c001d40 con 0x7f5d28071a70 2024-04-17T19:48:15.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7fa74f400640 1 --2- 172.21.15.29:0/3406969078 >> [v2:172.21.15.29:6808/868032589,v1:172.21.15.29:6809/868032589] conn(0x7fa730071b40 0x7fa730073fd0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7fe02f400640 1 -- 172.21.15.29:0/1153649630 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe034086c30 con 0x7fe03c15f480 2024-04-17T19:48:15.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7fe03b400640 1 --2- 172.21.15.29:0/1153649630 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe01c068580 0x7fe01c06aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7fe02f400640 1 --2- 172.21.15.29:0/1153649630 >> [v2:172.21.15.29:6824/843118748,v1:172.21.15.29:6825/843118748] conn(0x7fe01c071d40 0x7fe01c0741d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7fe02f400640 1 -- 172.21.15.29:0/1153649630 --> [v2:172.21.15.29:6824/843118748,v1:172.21.15.29:6825/843118748] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe03400de50 con 0x7fe01c071d40 2024-04-17T19:48:15.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7fe02f400640 1 -- 172.21.15.29:0/1153649630 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe034087020 con 0x7fe03c15f480 2024-04-17T19:48:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7f141ae00640 1 -- 172.21.15.29:0/2222824050 shutdown_connections 2024-04-17T19:48:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7f141ae00640 1 --2- 172.21.15.29:0/2222824050 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14141618e0 0x7f1414163d00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7f141ae00640 1 -- 172.21.15.29:0/2222824050 >> 172.21.15.29:0/2222824050 conn(0x7f141406bf90 msgr2=0x7f141406c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7f0580000640 1 Processor -- start 2024-04-17T19:48:15.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7f0580000640 1 -- start start 2024-04-17T19:48:15.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7fe03b400640 1 --2- 172.21.15.29:0/1153649630 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe01c068580 0x7fe01c06aa90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe030007db0 tx=0x7fe030008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7f5410a00640 1 Processor -- start 2024-04-17T19:48:15.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7fe03aa00640 1 --2- 172.21.15.29:0/1153649630 >> [v2:172.21.15.29:6824/843118748,v1:172.21.15.29:6825/843118748] conn(0x7fe01c071d40 0x7fe01c0741d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7fa0d8000640 1 -- 172.21.15.29:0/4245952925 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa0d0069b90 con 0x7fa0d00cafd0 2024-04-17T19:48:15.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7fa0cd600640 1 -- 172.21.15.29:0/4245952925 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa0bc003bc0 con 0x7fa0d00cafd0 2024-04-17T19:48:15.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7f5410a00640 1 -- start start 2024-04-17T19:48:15.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7fa0cd600640 1 --2- 172.21.15.29:0/4245952925 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa0b0068380 0x7fa0b006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7fa0cd600640 1 -- 172.21.15.29:0/4245952925 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa0bc086930 con 0x7fa0d00cafd0 2024-04-17T19:48:15.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7fa0cd600640 1 --2- 172.21.15.29:0/4245952925 >> [v2:172.21.15.29:6816/155534919,v1:172.21.15.29:6817/155534919] conn(0x7fa0b0071a40 0x7fa0b0073ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7fa0cd600640 1 -- 172.21.15.29:0/4245952925 --> [v2:172.21.15.29:6816/155534919,v1:172.21.15.29:6817/155534919] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa0b00745a0 con 0x7fa0b0071a40 2024-04-17T19:48:15.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7f141ae00640 1 -- 172.21.15.29:0/2222824050 shutdown_connections 2024-04-17T19:48:15.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.857+0000 7f141ae00640 1 -- 172.21.15.29:0/2222824050 wait complete. 2024-04-17T19:48:15.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7fe03aa00640 1 --2- 172.21.15.29:0/1153649630 >> [v2:172.21.15.29:6824/843118748,v1:172.21.15.29:6825/843118748] conn(0x7fe01c071d40 0x7fe01c0741d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f0580000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05600039c0 0x7f057806d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f0580000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05780c4560 0x7f05780c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f0580000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05780c4970 0x7f05780c8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f0580000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f057815da10 con 0x7f05600039c0 2024-04-17T19:48:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f0580000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f057815d730 con 0x7f05780c4970 2024-04-17T19:48:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f0580000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f057815dcf0 con 0x7f05780c4560 2024-04-17T19:48:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f057e000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05780c4970 0x7f05780c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f057e000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05780c4970 0x7f05780c8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46758/0 (socket says 172.21.15.29:46758) 2024-04-17T19:48:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f057e000640 1 -- 172.21.15.29:0/672165178 learned_addr learned my addr 172.21.15.29:0/672165178 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7ff9b4a00640 1 Processor -- start 2024-04-17T19:48:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7fe02f400640 1 -- 172.21.15.29:0/1153649630 <== osd.1 v2:172.21.15.29:6824/843118748 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe03400de50 con 0x7fe01c071d40 2024-04-17T19:48:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7fa0cd600640 1 -- 172.21.15.29:0/4245952925 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fa0bc0527e0 con 0x7fa0d00cafd0 2024-04-17T19:48:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7fa0d4c00640 1 --2- 172.21.15.29:0/4245952925 >> [v2:172.21.15.29:6816/155534919,v1:172.21.15.29:6817/155534919] conn(0x7fa0b0071a40 0x7fa0b0073ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7fa0d5600640 1 --2- 172.21.15.29:0/4245952925 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa0b0068380 0x7fa0b006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f5410a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f00039c0 0x7f540c1fa290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f5410a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f540c1ec290 0x7f540c0735c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f5410a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f540c14af20 0x7f540c14b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f5410a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f540c0c3630 con 0x7f53f00039c0 2024-04-17T19:48:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f5410a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f540c0c9450 con 0x7f540c14af20 2024-04-17T19:48:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f5410a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f540c0c8890 con 0x7f540c1ec290 2024-04-17T19:48:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f057cc00640 1 --2- 172.21.15.29:0/672165178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05780c4560 0x7f05780c7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7ff9b4a00640 1 -- start start 2024-04-17T19:48:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f141ae00640 1 Processor -- start 2024-04-17T19:48:15.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f141ae00640 1 -- start start 2024-04-17T19:48:15.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f540be00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f540c14af20 0x7f540c14b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f540be00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f540c14af20 0x7f540c14b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46770/0 (socket says 172.21.15.29:46770) 2024-04-17T19:48:15.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f540be00640 1 -- 172.21.15.29:0/2224789403 learned_addr learned my addr 172.21.15.29:0/2224789403 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:15.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f5403e00640 1 --2- 172.21.15.29:0/2224789403 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f540c1ec290 0x7f540c0735c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f057e000640 1 -- 172.21.15.29:0/672165178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05780c4560 msgr2=0x7f05780c7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f057e000640 1 --2- 172.21.15.29:0/672165178 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05780c4560 0x7f05780c7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f057e000640 1 -- 172.21.15.29:0/672165178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05600039c0 msgr2=0x7f057806d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f057e000640 1 --2- 172.21.15.29:0/672165178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05600039c0 0x7f057806d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f057e000640 1 -- 172.21.15.29:0/672165178 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05780c8550 con 0x7f05780c4970 2024-04-17T19:48:15.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f540b400640 1 --2- 172.21.15.29:0/2224789403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f00039c0 0x7f540c1fa290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7fa747e00640 1 -- 172.21.15.29:0/3406969078 <== osd.2 v2:172.21.15.29:6808/868032589 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7300746a0 con 0x7fa730071b40 2024-04-17T19:48:15.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f057e000640 1 --2- 172.21.15.29:0/672165178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05780c4970 0x7f05780c8010 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0578070cc0 tx=0x7f057000fe80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7fa0d5600640 1 --2- 172.21.15.29:0/4245952925 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa0b0068380 0x7fa0b006a890 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa0c80aa000 tx=0x7fa0c8005360 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7fa0d4c00640 1 --2- 172.21.15.29:0/4245952925 >> [v2:172.21.15.29:6816/155534919,v1:172.21.15.29:6817/155534919] conn(0x7fa0b0071a40 0x7fa0b0073ed0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f0575600640 1 -- 172.21.15.29:0/672165178 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057000d5b0 con 0x7f05780c4970 2024-04-17T19:48:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f0575600640 1 -- 172.21.15.29:0/672165178 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f057000d770 con 0x7f05780c4970 2024-04-17T19:48:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f5403e00640 1 -- 172.21.15.29:0/2224789403 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f540c14af20 msgr2=0x7f540c14b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f5403e00640 1 --2- 172.21.15.29:0/2224789403 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f540c14af20 0x7f540c14b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f5403e00640 1 -- 172.21.15.29:0/2224789403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f00039c0 msgr2=0x7f540c1fa290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f5403e00640 1 --2- 172.21.15.29:0/2224789403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f00039c0 0x7f540c1fa290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f5403e00640 1 -- 172.21.15.29:0/2224789403 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f540c1fa7d0 con 0x7f540c1ec290 2024-04-17T19:48:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7ff9b4a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9b00729b0 0x7ff9b01fa0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7ff9b4a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9b00c69a0 0x7ff9b01e7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f141ae00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1414073d70 0x7f1414159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f141ae00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1414157f60 0x7f14140cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.905 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f141ae00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14141618e0 0x7f141414d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7ff9b4a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9b015c790 0x7ff9b015b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7ff9af400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9b00729b0 0x7ff9b01fa0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7ff9af400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9b00729b0 0x7ff9b01fa0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36384/0 (socket says 172.21.15.29:36384) 2024-04-17T19:48:15.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f5403e00640 1 --2- 172.21.15.29:0/2224789403 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f540c1ec290 0x7f540c0735c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f540c06f5c0 tx=0x7f53f4010d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7ff9af400640 1 -- 172.21.15.29:0/2430975076 learned_addr learned my addr 172.21.15.29:0/2430975076 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2430975076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b01563e0 con 0x7ff9b00c69a0 2024-04-17T19:48:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2430975076 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff9b01602b0 con 0x7ff9b015c790 2024-04-17T19:48:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2430975076 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff9b0156560 con 0x7ff9b00729b0 2024-04-17T19:48:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f1413400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1414157f60 0x7f14140cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7ff9afe00640 1 --2- 172.21.15.29:0/2430975076 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9b015c790 0x7ff9b015b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f141ae00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f14140c8c40 con 0x7f14141618e0 2024-04-17T19:48:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f141ae00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f14140c8960 con 0x7f1414157f60 2024-04-17T19:48:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f141ae00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1414074180 con 0x7f1414073d70 2024-04-17T19:48:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f5403400640 1 -- 172.21.15.29:0/2224789403 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f401a020 con 0x7f540c1ec290 2024-04-17T19:48:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f5410a00640 1 -- 172.21.15.29:0/2224789403 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f540c1e9ed0 con 0x7f540c1ec290 2024-04-17T19:48:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f5403400640 1 -- 172.21.15.29:0/2224789403 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53f400be10 con 0x7f540c1ec290 2024-04-17T19:48:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f5403400640 1 -- 172.21.15.29:0/2224789403 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53f400ebe0 con 0x7f540c1ec290 2024-04-17T19:48:15.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f1413400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1414157f60 0x7f14140cd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46778/0 (socket says 172.21.15.29:46778) 2024-04-17T19:48:15.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f1418e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14141618e0 0x7f141414d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f1413400640 1 -- 172.21.15.29:0/4186490848 learned_addr learned my addr 172.21.15.29:0/4186490848 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:15.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f1413e00640 1 --2- 172.21.15.29:0/4186490848 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1414073d70 0x7f1414159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7ff9af400640 1 -- 172.21.15.29:0/2430975076 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9b015c790 msgr2=0x7ff9b015b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7ff9af400640 1 --2- 172.21.15.29:0/2430975076 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff9b015c790 0x7ff9b015b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7ff9af400640 1 -- 172.21.15.29:0/2430975076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9b00c69a0 msgr2=0x7ff9b01e7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7ff9af400640 1 --2- 172.21.15.29:0/2430975076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9b00c69a0 0x7ff9b01e7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7ff9af400640 1 -- 172.21.15.29:0/2430975076 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9b01fa600 con 0x7ff9b00729b0 2024-04-17T19:48:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f5410a00640 1 -- 172.21.15.29:0/2224789403 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f540c1ead30 con 0x7f540c1ec290 2024-04-17T19:48:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f5400c00640 1 -- 172.21.15.29:0/2224789403 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f540c04ff10 con 0x7f540c1ec290 2024-04-17T19:48:15.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.861+0000 7f0575600640 1 -- 172.21.15.29:0/672165178 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057000db60 con 0x7f05780c4970 2024-04-17T19:48:15.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f0580000640 1 -- 172.21.15.29:0/672165178 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05781fa460 con 0x7f05780c4970 2024-04-17T19:48:15.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f0580000640 1 -- 172.21.15.29:0/672165178 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05780cd350 con 0x7f05780c4970 2024-04-17T19:48:15.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f0580000640 1 -- 172.21.15.29:0/672165178 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f05780c7440 con 0x7f05780c4970 2024-04-17T19:48:15.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7ff9af400640 1 --2- 172.21.15.29:0/2430975076 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9b00729b0 0x7ff9b01fa0c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff9a80b2040 tx=0x7ff9a801bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f1413400640 1 -- 172.21.15.29:0/4186490848 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1414073d70 msgr2=0x7f1414159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f1413400640 1 --2- 172.21.15.29:0/4186490848 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1414073d70 0x7f1414159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f1413400640 1 -- 172.21.15.29:0/4186490848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14141618e0 msgr2=0x7f141414d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f1413400640 1 --2- 172.21.15.29:0/4186490848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14141618e0 0x7f141414d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f1413400640 1 -- 172.21.15.29:0/4186490848 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1414159860 con 0x7f1414157f60 2024-04-17T19:48:15.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f1418e00640 1 --2- 172.21.15.29:0/4186490848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14141618e0 0x7f141414d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:15.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.865+0000 7f1413400640 1 --2- 172.21.15.29:0/4186490848 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1414157f60 0x7f14140cd370 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f140400c460 tx=0x7f140400c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.869+0000 7ff9a7400640 1 -- 172.21.15.29:0/2430975076 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a80b4040 con 0x7ff9b00729b0 2024-04-17T19:48:15.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.869+0000 7ff9a7400640 1 -- 172.21.15.29:0/2430975076 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9a80c1070 con 0x7ff9b00729b0 2024-04-17T19:48:15.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.869+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2430975076 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9b01f6b40 con 0x7ff9b00729b0 2024-04-17T19:48:15.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.869+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2430975076 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9b01f7100 con 0x7ff9b00729b0 2024-04-17T19:48:15.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.869+0000 7f140be00640 1 -- 172.21.15.29:0/4186490848 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14040052d0 con 0x7f1414157f60 2024-04-17T19:48:15.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.869+0000 7f140be00640 1 -- 172.21.15.29:0/4186490848 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1404005470 con 0x7f1414157f60 2024-04-17T19:48:15.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.869+0000 7f140be00640 1 -- 172.21.15.29:0/4186490848 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1404012870 con 0x7f1414157f60 2024-04-17T19:48:15.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.869+0000 7ff9a7400640 1 -- 172.21.15.29:0/2430975076 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a80b4040 con 0x7ff9b00729b0 2024-04-17T19:48:15.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.869+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2430975076 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff9b004ff10 con 0x7ff9b00729b0 2024-04-17T19:48:15.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.869+0000 7f141ae00640 1 -- 172.21.15.29:0/4186490848 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14141f6420 con 0x7f1414157f60 2024-04-17T19:48:15.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.869+0000 7f141ae00640 1 -- 172.21.15.29:0/4186490848 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14141f69c0 con 0x7f1414157f60 2024-04-17T19:48:15.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.869+0000 7f141ae00640 1 -- 172.21.15.29:0/4186490848 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1414069b90 con 0x7f1414157f60 2024-04-17T19:48:15.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.869+0000 7f5403400640 1 -- 172.21.15.29:0/2224789403 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f53f401d020 con 0x7f540c1ec290 2024-04-17T19:48:15.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.869+0000 7f0575600640 1 -- 172.21.15.29:0/672165178 <== mon.1 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0570014070 con 0x7f05780c4970 2024-04-17T19:48:15.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7ff9a7400640 1 -- 172.21.15.29:0/2430975076 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff9a80b7070 con 0x7ff9b00729b0 2024-04-17T19:48:15.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7f5403400640 1 --2- 172.21.15.29:0/2224789403 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f53e8068400 0x7f53e806a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7f0575600640 1 --2- 172.21.15.29:0/672165178 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0558068380 0x7f055806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7f0575600640 1 -- 172.21.15.29:0/672165178 <== mon.1 v2:172.21.15.37:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0570089900 con 0x7f05780c4970 2024-04-17T19:48:15.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7f0575600640 1 --2- 172.21.15.29:0/672165178 >> [v2:172.21.15.37:6824/255655587,v1:172.21.15.37:6825/255655587] conn(0x7f0558071ac0 0x7f0558073f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7f0575600640 1 -- 172.21.15.29:0/672165178 --> [v2:172.21.15.37:6824/255655587,v1:172.21.15.37:6825/255655587] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0558074620 con 0x7f0558071ac0 2024-04-17T19:48:15.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7f0575600640 1 -- 172.21.15.29:0/672165178 <== mon.1 v2:172.21.15.37:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0570089cf0 con 0x7f05780c4970 2024-04-17T19:48:15.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7f5403400640 1 -- 172.21.15.29:0/2224789403 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f53f4087400 con 0x7f540c1ec290 2024-04-17T19:48:15.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7f5403400640 1 --2- 172.21.15.29:0/2224789403 >> [v2:172.21.15.37:6800/811671695,v1:172.21.15.37:6801/811671695] conn(0x7f53e8071bc0 0x7f53e8074050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7f540b400640 1 --2- 172.21.15.29:0/2224789403 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f53e8068400 0x7f53e806a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7f5403400640 1 -- 172.21.15.29:0/2224789403 --> [v2:172.21.15.37:6800/811671695,v1:172.21.15.37:6801/811671695] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53f40100a0 con 0x7f53e8071bc0 2024-04-17T19:48:15.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7f5403400640 1 -- 172.21.15.29:0/2224789403 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f53f408e050 con 0x7f540c1ec290 2024-04-17T19:48:15.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7f057d600640 1 --2- 172.21.15.29:0/672165178 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0558068380 0x7f055806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7f540be00640 1 --2- 172.21.15.29:0/2224789403 >> [v2:172.21.15.37:6800/811671695,v1:172.21.15.37:6801/811671695] conn(0x7f53e8071bc0 0x7f53e8074050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7ff9a7400640 1 --2- 172.21.15.29:0/2430975076 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff9900685b0 0x7ff99006aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7f140be00640 1 -- 172.21.15.29:0/4186490848 <== mon.1 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1404012a10 con 0x7f1414157f60 2024-04-17T19:48:15.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7ff9a7400640 1 -- 172.21.15.29:0/2430975076 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff9a8124fa0 con 0x7ff9b00729b0 2024-04-17T19:48:15.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7f057d600640 1 --2- 172.21.15.29:0/672165178 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0558068380 0x7f055806a890 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f056c0aa000 tx=0x7f056c000fc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.873+0000 7f140be00640 1 --2- 172.21.15.29:0/4186490848 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f13f4068450 0x7f13f406a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7fa0cd600640 1 -- 172.21.15.29:0/4245952925 <== osd.3 v2:172.21.15.29:6816/155534919 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa0b00745a0 con 0x7fa0b0071a40 2024-04-17T19:48:15.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7ff9aea00640 1 --2- 172.21.15.29:0/2430975076 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff9900685b0 0x7ff99006aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7ff9a7400640 1 --2- 172.21.15.29:0/2430975076 >> [v2:172.21.15.37:6808/264607202,v1:172.21.15.37:6809/264607202] conn(0x7ff990071c80 0x7ff990074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7f140be00640 1 -- 172.21.15.29:0/4186490848 <== mon.1 v2:172.21.15.37:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f14040872a0 con 0x7f1414157f60 2024-04-17T19:48:15.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7ff9a7400640 1 -- 172.21.15.29:0/2430975076 --> [v2:172.21.15.37:6808/264607202,v1:172.21.15.37:6809/264607202] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff9900747e0 con 0x7ff990071c80 2024-04-17T19:48:15.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7ff9a7400640 1 -- 172.21.15.29:0/2430975076 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff9a8125350 con 0x7ff9b00729b0 2024-04-17T19:48:15.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7f540be00640 1 --2- 172.21.15.29:0/2224789403 >> [v2:172.21.15.37:6800/811671695,v1:172.21.15.37:6801/811671695] conn(0x7f53e8071bc0 0x7f53e8074050 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7f540b400640 1 --2- 172.21.15.29:0/2224789403 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f53e8068400 0x7f53e806a910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f53fc0af000 tx=0x7f53fc0046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7f140be00640 1 --2- 172.21.15.29:0/4186490848 >> [v2:172.21.15.37:6816/3883152709,v1:172.21.15.37:6817/3883152709] conn(0x7f13f4071c10 0x7f13f40740a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:15.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7f140be00640 1 -- 172.21.15.29:0/4186490848 --> [v2:172.21.15.37:6816/3883152709,v1:172.21.15.37:6817/3883152709] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1404026ea0 con 0x7f13f4071c10 2024-04-17T19:48:15.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7f140be00640 1 -- 172.21.15.29:0/4186490848 <== mon.1 v2:172.21.15.37:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1404087690 con 0x7f1414157f60 2024-04-17T19:48:15.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7f1413e00640 1 --2- 172.21.15.29:0/4186490848 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f13f4068450 0x7f13f406a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7f1418e00640 1 --2- 172.21.15.29:0/4186490848 >> [v2:172.21.15.37:6816/3883152709,v1:172.21.15.37:6817/3883152709] conn(0x7f13f4071c10 0x7f13f40740a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7f057cc00640 1 --2- 172.21.15.29:0/672165178 >> [v2:172.21.15.37:6824/255655587,v1:172.21.15.37:6825/255655587] conn(0x7f0558071ac0 0x7f0558073f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7ff9aea00640 1 --2- 172.21.15.29:0/2430975076 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff9900685b0 0x7ff99006aac0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0007fd0 tx=0x7ff9a0007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7f1413e00640 1 --2- 172.21.15.29:0/4186490848 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f13f4068450 0x7f13f406a960 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1400007fd0 tx=0x7f1400009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7ff9afe00640 1 --2- 172.21.15.29:0/2430975076 >> [v2:172.21.15.37:6808/264607202,v1:172.21.15.37:6809/264607202] conn(0x7ff990071c80 0x7ff990074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:15.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7f1418e00640 1 --2- 172.21.15.29:0/4186490848 >> [v2:172.21.15.37:6816/3883152709,v1:172.21.15.37:6817/3883152709] conn(0x7f13f4071c10 0x7f13f40740a0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7f057cc00640 1 --2- 172.21.15.29:0/672165178 >> [v2:172.21.15.37:6824/255655587,v1:172.21.15.37:6825/255655587] conn(0x7f0558071ac0 0x7f0558073f50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7ff9afe00640 1 --2- 172.21.15.29:0/2430975076 >> [v2:172.21.15.37:6808/264607202,v1:172.21.15.37:6809/264607202] conn(0x7ff990071c80 0x7ff990074110 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:15.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.877+0000 7f5403400640 1 -- 172.21.15.29:0/2224789403 <== osd.4 v2:172.21.15.37:6800/811671695 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f53f40100a0 con 0x7f53e8071bc0 2024-04-17T19:48:15.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.881+0000 7f140be00640 1 -- 172.21.15.29:0/4186490848 <== osd.7 v2:172.21.15.37:6816/3883152709 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1404026ea0 con 0x7f13f4071c10 2024-04-17T19:48:15.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.881+0000 7f0575600640 1 -- 172.21.15.29:0/672165178 <== osd.5 v2:172.21.15.37:6824/255655587 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0558074620 con 0x7f0558071ac0 2024-04-17T19:48:15.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.885+0000 7ff9a7400640 1 -- 172.21.15.29:0/2430975076 <== osd.6 v2:172.21.15.37:6808/264607202 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff9900747e0 con 0x7ff990071c80 2024-04-17T19:48:15.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.885+0000 7f5d50200640 1 -- 172.21.15.29:0/3980054984 --> [v2:172.21.15.29:6800/515534509,v1:172.21.15.29:6801/515534509] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5d400054b0 con 0x7f5d28071a70 2024-04-17T19:48:15.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.885+0000 7fe042600640 1 -- 172.21.15.29:0/1153649630 --> [v2:172.21.15.29:6824/843118748,v1:172.21.15.29:6825/843118748] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe018002de0 con 0x7fe01c071d40 2024-04-17T19:48:15.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.885+0000 7fe02f400640 1 -- 172.21.15.29:0/1153649630 <== osd.1 v2:172.21.15.29:6824/843118748 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe018002de0 con 0x7fe01c071d40 2024-04-17T19:48:15.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7f5d45600640 1 -- 172.21.15.29:0/3980054984 <== osd.0 v2:172.21.15.29:6800/515534509 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5d400054b0 con 0x7f5d28071a70 2024-04-17T19:48:15.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7f5d50200640 1 -- 172.21.15.29:0/3980054984 >> [v2:172.21.15.29:6800/515534509,v1:172.21.15.29:6801/515534509] conn(0x7f5d28071a70 msgr2=0x7f5d28073f00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7fe042600640 1 -- 172.21.15.29:0/1153649630 >> [v2:172.21.15.29:6824/843118748,v1:172.21.15.29:6825/843118748] conn(0x7fe01c071d40 msgr2=0x7fe01c0741d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7fe042600640 1 --2- 172.21.15.29:0/1153649630 >> [v2:172.21.15.29:6824/843118748,v1:172.21.15.29:6825/843118748] conn(0x7fe01c071d40 0x7fe01c0741d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7f5d50200640 1 --2- 172.21.15.29:0/3980054984 >> [v2:172.21.15.29:6800/515534509,v1:172.21.15.29:6801/515534509] conn(0x7f5d28071a70 0x7f5d28073f00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7fe042600640 1 -- 172.21.15.29:0/1153649630 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe01c068580 msgr2=0x7fe01c06aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7fe042600640 1 --2- 172.21.15.29:0/1153649630 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe01c068580 0x7fe01c06aa90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe030007db0 tx=0x7fe030008040 comp rx=0 tx=0).stop 2024-04-17T19:48:15.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7fe042600640 1 -- 172.21.15.29:0/1153649630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03c15f480 msgr2=0x7fe03c15d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7fe042600640 1 --2- 172.21.15.29:0/1153649630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe03c15f480 0x7fe03c15d050 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe03c0c7a30 tx=0x7fe03400cf00 comp rx=0 tx=0).stop 2024-04-17T19:48:15.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7fa755800640 1 -- 172.21.15.29:0/3406969078 --> [v2:172.21.15.29:6808/868032589,v1:172.21.15.29:6809/868032589] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa7500ca270 con 0x7fa730071b40 2024-04-17T19:48:15.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7f5d50200640 1 -- 172.21.15.29:0/3980054984 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5d280683b0 msgr2=0x7f5d2806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7f5d50200640 1 --2- 172.21.15.29:0/3980054984 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5d280683b0 0x7f5d2806a8c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5d400353b0 tx=0x7f5d34009190 comp rx=0 tx=0).stop 2024-04-17T19:48:15.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7f5d50200640 1 -- 172.21.15.29:0/3980054984 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d400060f0 msgr2=0x7f5d4004e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7f5d50200640 1 --2- 172.21.15.29:0/3980054984 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d400060f0 0x7f5d4004e770 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c01a1b0 tx=0x7f5d3c019320 comp rx=0 tx=0).stop 2024-04-17T19:48:15.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7fa747e00640 1 -- 172.21.15.29:0/3406969078 <== osd.2 v2:172.21.15.29:6808/868032589 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa7500ca270 con 0x7fa730071b40 2024-04-17T19:48:15.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7f5d4e200640 1 -- 172.21.15.29:0/3980054984 reap_dead start 2024-04-17T19:48:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7f5d50200640 1 -- 172.21.15.29:0/3980054984 shutdown_connections 2024-04-17T19:48:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7f5d50200640 1 -- 172.21.15.29:0/3980054984 >> 172.21.15.29:0/3980054984 conn(0x7f5d400503c0 msgr2=0x7f5d400dc350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7f5d50200640 1 -- 172.21.15.29:0/3980054984 shutdown_connections 2024-04-17T19:48:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7f5d50200640 1 -- 172.21.15.29:0/3980054984 wait complete. 2024-04-17T19:48:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7fa755800640 1 -- 172.21.15.29:0/3406969078 >> [v2:172.21.15.29:6808/868032589,v1:172.21.15.29:6809/868032589] conn(0x7fa730071b40 msgr2=0x7fa730073fd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7fa755800640 1 --2- 172.21.15.29:0/3406969078 >> [v2:172.21.15.29:6808/868032589,v1:172.21.15.29:6809/868032589] conn(0x7fa730071b40 0x7fa730073fd0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7fa755800640 1 -- 172.21.15.29:0/3406969078 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa730068380 msgr2=0x7fa73006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7fa755800640 1 --2- 172.21.15.29:0/3406969078 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa730068380 0x7fa73006a890 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fa748001570 tx=0x7fa748001a00 comp rx=0 tx=0).stop 2024-04-17T19:48:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7fa755800640 1 -- 172.21.15.29:0/3406969078 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa750160c80 msgr2=0x7fa7501fa330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7fa755800640 1 --2- 172.21.15.29:0/3406969078 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa750160c80 0x7fa7501fa330 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa750164830 tx=0x7fa73c00c820 comp rx=0 tx=0).stop 2024-04-17T19:48:15.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7fe03be00640 1 -- 172.21.15.29:0/1153649630 reap_dead start 2024-04-17T19:48:15.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7fe042600640 1 -- 172.21.15.29:0/1153649630 shutdown_connections 2024-04-17T19:48:15.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.889+0000 7fe042600640 1 -- 172.21.15.29:0/1153649630 >> 172.21.15.29:0/1153649630 conn(0x7fe03c06dd20 msgr2=0x7fe03c0c7a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.893+0000 7fa754e00640 1 -- 172.21.15.29:0/3406969078 reap_dead start 2024-04-17T19:48:15.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.893+0000 7fa755800640 1 -- 172.21.15.29:0/3406969078 shutdown_connections 2024-04-17T19:48:15.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.893+0000 7fa755800640 1 -- 172.21.15.29:0/3406969078 >> 172.21.15.29:0/3406969078 conn(0x7fa7500cdde0 msgr2=0x7fa7501579d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.893+0000 7fe042600640 1 -- 172.21.15.29:0/1153649630 shutdown_connections 2024-04-17T19:48:15.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.893+0000 7fe042600640 1 -- 172.21.15.29:0/1153649630 wait complete. 2024-04-17T19:48:15.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.893+0000 7fa755800640 1 -- 172.21.15.29:0/3406969078 shutdown_connections 2024-04-17T19:48:15.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.893+0000 7fa755800640 1 -- 172.21.15.29:0/3406969078 wait complete. 2024-04-17T19:48:15.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.901+0000 7f141ae00640 1 -- 172.21.15.29:0/4186490848 --> [v2:172.21.15.37:6816/3883152709,v1:172.21.15.37:6817/3883152709] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f14140c96b0 con 0x7f13f4071c10 2024-04-17T19:48:15.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.901+0000 7fa0d8000640 1 -- 172.21.15.29:0/4245952925 --> [v2:172.21.15.29:6816/155534919,v1:172.21.15.29:6817/155534919] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa0d01f1320 con 0x7fa0b0071a40 2024-04-17T19:48:15.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.901+0000 7f140be00640 1 -- 172.21.15.29:0/4186490848 <== osd.7 v2:172.21.15.37:6816/3883152709 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f14140c96b0 con 0x7f13f4071c10 2024-04-17T19:48:15.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7f141ae00640 1 -- 172.21.15.29:0/4186490848 >> [v2:172.21.15.37:6816/3883152709,v1:172.21.15.37:6817/3883152709] conn(0x7f13f4071c10 msgr2=0x7f13f40740a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7f141ae00640 1 --2- 172.21.15.29:0/4186490848 >> [v2:172.21.15.37:6816/3883152709,v1:172.21.15.37:6817/3883152709] conn(0x7f13f4071c10 0x7f13f40740a0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7fa0cd600640 1 -- 172.21.15.29:0/4245952925 <== osd.3 v2:172.21.15.29:6816/155534919 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa0d01f1320 con 0x7fa0b0071a40 2024-04-17T19:48:15.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7f5400c00640 1 -- 172.21.15.29:0/2224789403 --> [v2:172.21.15.37:6800/811671695,v1:172.21.15.37:6801/811671695] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f540c069580 con 0x7f53e8071bc0 2024-04-17T19:48:15.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7fa0d8000640 1 -- 172.21.15.29:0/4245952925 >> [v2:172.21.15.29:6816/155534919,v1:172.21.15.29:6817/155534919] conn(0x7fa0b0071a40 msgr2=0x7fa0b0073ed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7fa0d8000640 1 --2- 172.21.15.29:0/4245952925 >> [v2:172.21.15.29:6816/155534919,v1:172.21.15.29:6817/155534919] conn(0x7fa0b0071a40 0x7fa0b0073ed0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7fa0d8000640 1 -- 172.21.15.29:0/4245952925 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa0b0068380 msgr2=0x7fa0b006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7fa0d8000640 1 --2- 172.21.15.29:0/4245952925 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa0b0068380 0x7fa0b006a890 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa0c80aa000 tx=0x7fa0c8005360 comp rx=0 tx=0).stop 2024-04-17T19:48:15.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7fa0d8000640 1 -- 172.21.15.29:0/4245952925 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0d00cafd0 msgr2=0x7fa0d0149b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7fa0d8000640 1 --2- 172.21.15.29:0/4245952925 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0d00cafd0 0x7fa0d0149b70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa0d00ca3c0 tx=0x7fa0bc011860 comp rx=0 tx=0).stop 2024-04-17T19:48:15.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7fa0d6000640 1 -- 172.21.15.29:0/4245952925 reap_dead start 2024-04-17T19:48:15.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7f5403400640 1 -- 172.21.15.29:0/2224789403 <== osd.4 v2:172.21.15.37:6800/811671695 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f540c069580 con 0x7f53e8071bc0 2024-04-17T19:48:15.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7f141ae00640 1 -- 172.21.15.29:0/4186490848 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f13f4068450 msgr2=0x7f13f406a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7f141ae00640 1 --2- 172.21.15.29:0/4186490848 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f13f4068450 0x7f13f406a960 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1400007fd0 tx=0x7f1400009040 comp rx=0 tx=0).stop 2024-04-17T19:48:15.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7f141ae00640 1 -- 172.21.15.29:0/4186490848 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1414157f60 msgr2=0x7f14140cd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7f141ae00640 1 --2- 172.21.15.29:0/4186490848 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1414157f60 0x7f14140cd370 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f140400c460 tx=0x7f140400c920 comp rx=0 tx=0).stop 2024-04-17T19:48:15.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7f1418e00640 1 -- 172.21.15.29:0/4186490848 reap_dead start 2024-04-17T19:48:15.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7f141ae00640 1 -- 172.21.15.29:0/4186490848 shutdown_connections 2024-04-17T19:48:15.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7f141ae00640 1 -- 172.21.15.29:0/4186490848 >> 172.21.15.29:0/4186490848 conn(0x7f141406bf90 msgr2=0x7f14140c69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7f141ae00640 1 -- 172.21.15.29:0/4186490848 shutdown_connections 2024-04-17T19:48:15.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7f141ae00640 1 -- 172.21.15.29:0/4186490848 wait complete. 2024-04-17T19:48:15.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7fa0d8000640 1 -- 172.21.15.29:0/4245952925 shutdown_connections 2024-04-17T19:48:15.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.905+0000 7fa0d8000640 1 -- 172.21.15.29:0/4245952925 >> 172.21.15.29:0/4245952925 conn(0x7fa0d006bf90 msgr2=0x7fa0d01506f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.909+0000 7fa0d8000640 1 -- 172.21.15.29:0/4245952925 shutdown_connections 2024-04-17T19:48:15.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.909+0000 7fa0d8000640 1 -- 172.21.15.29:0/4245952925 wait complete. 2024-04-17T19:48:15.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.909+0000 7f5400c00640 1 -- 172.21.15.29:0/2224789403 >> [v2:172.21.15.37:6800/811671695,v1:172.21.15.37:6801/811671695] conn(0x7f53e8071bc0 msgr2=0x7f53e8074050 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.909+0000 7f5400c00640 1 --2- 172.21.15.29:0/2224789403 >> [v2:172.21.15.37:6800/811671695,v1:172.21.15.37:6801/811671695] conn(0x7f53e8071bc0 0x7f53e8074050 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.909+0000 7f5400c00640 1 -- 172.21.15.29:0/2224789403 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f53e8068400 msgr2=0x7f53e806a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.909+0000 7f5400c00640 1 --2- 172.21.15.29:0/2224789403 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f53e8068400 0x7f53e806a910 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f53fc0af000 tx=0x7f53fc0046d0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.909+0000 7f5400c00640 1 -- 172.21.15.29:0/2224789403 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f540c1ec290 msgr2=0x7f540c0735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.909+0000 7f5400c00640 1 --2- 172.21.15.29:0/2224789403 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f540c1ec290 0x7f540c0735c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f540c06f5c0 tx=0x7f53f4010d00 comp rx=0 tx=0).stop 2024-04-17T19:48:15.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.909+0000 7f540be00640 1 -- 172.21.15.29:0/2224789403 reap_dead start 2024-04-17T19:48:15.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.909+0000 7f5400c00640 1 -- 172.21.15.29:0/2224789403 shutdown_connections 2024-04-17T19:48:15.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.909+0000 7f5400c00640 1 -- 172.21.15.29:0/2224789403 >> 172.21.15.29:0/2224789403 conn(0x7f540c06ddd0 msgr2=0x7f540c14d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.909+0000 7f5400c00640 1 -- 172.21.15.29:0/2224789403 shutdown_connections 2024-04-17T19:48:15.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.909+0000 7f5400c00640 1 -- 172.21.15.29:0/2224789403 wait complete. 2024-04-17T19:48:15.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.909+0000 7f0580000640 1 -- 172.21.15.29:0/672165178 --> [v2:172.21.15.37:6824/255655587,v1:172.21.15.37:6825/255655587] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0578069b90 con 0x7f0558071ac0 2024-04-17T19:48:15.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.913+0000 7f0575600640 1 -- 172.21.15.29:0/672165178 <== osd.5 v2:172.21.15.37:6824/255655587 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0578069b90 con 0x7f0558071ac0 2024-04-17T19:48:15.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.913+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2430975076 --> [v2:172.21.15.37:6808/264607202,v1:172.21.15.37:6809/264607202] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff9b00cd4e0 con 0x7ff990071c80 2024-04-17T19:48:15.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.913+0000 7f056aa00640 1 -- 172.21.15.29:0/672165178 >> [v2:172.21.15.37:6824/255655587,v1:172.21.15.37:6825/255655587] conn(0x7f0558071ac0 msgr2=0x7f0558073f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.913+0000 7f056aa00640 1 --2- 172.21.15.29:0/672165178 >> [v2:172.21.15.37:6824/255655587,v1:172.21.15.37:6825/255655587] conn(0x7f0558071ac0 0x7f0558073f50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.913+0000 7f056aa00640 1 -- 172.21.15.29:0/672165178 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0558068380 msgr2=0x7f055806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.913+0000 7f056aa00640 1 --2- 172.21.15.29:0/672165178 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0558068380 0x7f055806a890 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f056c0aa000 tx=0x7f056c000fc0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.913+0000 7f056aa00640 1 -- 172.21.15.29:0/672165178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05780c4970 msgr2=0x7f05780c8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.913+0000 7f056aa00640 1 --2- 172.21.15.29:0/672165178 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05780c4970 0x7f05780c8010 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0578070cc0 tx=0x7f057000fe80 comp rx=0 tx=0).stop 2024-04-17T19:48:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.913+0000 7ff9a7400640 1 -- 172.21.15.29:0/2430975076 <== osd.6 v2:172.21.15.37:6808/264607202 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff9b00cd4e0 con 0x7ff990071c80 2024-04-17T19:48:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.913+0000 7f057e000640 1 -- 172.21.15.29:0/672165178 reap_dead start 2024-04-17T19:48:15.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.913+0000 7f056aa00640 1 -- 172.21.15.29:0/672165178 shutdown_connections 2024-04-17T19:48:15.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.913+0000 7f056aa00640 1 -- 172.21.15.29:0/672165178 >> 172.21.15.29:0/672165178 conn(0x7f05780cf770 msgr2=0x7f0578159620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.917+0000 7f056aa00640 1 -- 172.21.15.29:0/672165178 shutdown_connections 2024-04-17T19:48:15.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.917+0000 7f056aa00640 1 -- 172.21.15.29:0/672165178 wait complete. 2024-04-17T19:48:15.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.921+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2430975076 >> [v2:172.21.15.37:6808/264607202,v1:172.21.15.37:6809/264607202] conn(0x7ff990071c80 msgr2=0x7ff990074110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.921+0000 7ff9b4a00640 1 --2- 172.21.15.29:0/2430975076 >> [v2:172.21.15.37:6808/264607202,v1:172.21.15.37:6809/264607202] conn(0x7ff990071c80 0x7ff990074110 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.921+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2430975076 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff9900685b0 msgr2=0x7ff99006aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.921+0000 7ff9b4a00640 1 --2- 172.21.15.29:0/2430975076 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff9900685b0 0x7ff99006aac0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0007fd0 tx=0x7ff9a0007900 comp rx=0 tx=0).stop 2024-04-17T19:48:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.921+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2430975076 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9b00729b0 msgr2=0x7ff9b01fa0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.921+0000 7ff9b4a00640 1 --2- 172.21.15.29:0/2430975076 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff9b00729b0 0x7ff9b01fa0c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff9a80b2040 tx=0x7ff9a801bfa0 comp rx=0 tx=0).stop 2024-04-17T19:48:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.921+0000 7ff9afe00640 1 -- 172.21.15.29:0/2430975076 reap_dead start 2024-04-17T19:48:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.921+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2430975076 shutdown_connections 2024-04-17T19:48:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.921+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2430975076 >> 172.21.15.29:0/2430975076 conn(0x7ff9b00d18e0 msgr2=0x7ff9b00ce660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.925+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2430975076 shutdown_connections 2024-04-17T19:48:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:15.925+0000 7ff9b4a00640 1 -- 172.21.15.29:0/2430975076 wait complete. 2024-04-17T19:48:15.951 INFO:teuthology.orchestra.run.smithi029.stdout:55834574859 2024-04-17T19:48:15.951 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-17T19:48:15.952 INFO:teuthology.orchestra.run.smithi029.stdout:55834574860 2024-04-17T19:48:15.953 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-17T19:48:15.953 INFO:teuthology.orchestra.run.smithi029.stdout:55834574859 2024-04-17T19:48:15.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-17T19:48:15.954 INFO:teuthology.orchestra.run.smithi029.stdout:55834574860 2024-04-17T19:48:15.955 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-17T19:48:15.955 INFO:teuthology.orchestra.run.smithi029.stdout:55834574860 2024-04-17T19:48:15.955 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-17T19:48:15.956 INFO:teuthology.orchestra.run.smithi029.stdout:55834574859 2024-04-17T19:48:15.956 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-17T19:48:15.957 INFO:teuthology.orchestra.run.smithi029.stdout:55834574858 2024-04-17T19:48:15.957 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-17T19:48:15.958 INFO:teuthology.orchestra.run.smithi029.stdout:55834574860 2024-04-17T19:48:15.959 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-17T19:48:16.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.281+0000 7fe73b600640 1 Processor -- start 2024-04-17T19:48:16.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.281+0000 7fe73b600640 1 -- start start 2024-04-17T19:48:16.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.281+0000 7fe73b600640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe72c0e6f40 0x7fe72c0e9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.281+0000 7fe73b600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe72c047540 con 0x7fe72c0db2b0 2024-04-17T19:48:16.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.281+0000 7fe73b600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe72c0461c0 con 0x7fe72c04c8e0 2024-04-17T19:48:16.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.281+0000 7fe73b600640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fe72c0464a0 con 0x7fe72c0e6f40 2024-04-17T19:48:16.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.285+0000 7fe738c00640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe72c04c8e0 0x7fe72c0060f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:49852/0 (socket says 172.21.15.29:49852) 2024-04-17T19:48:16.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.285+0000 7fe738c00640 1 -- 172.21.15.29:0/1827240539 learned_addr learned my addr 172.21.15.29:0/1827240539 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:16.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.285+0000 7fe739600640 1 --2- 172.21.15.29:0/1827240539 >> v2:172.21.15.37:3300/0 conn(0x7fe72c0e6f40 0x7fe72c0e9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.285+0000 7fe739600640 1 -- 172.21.15.29:0/1827240539 >> v1:172.21.15.37:6789/0 conn(0x7fe72c04c8e0 legacy=0x7fe72c0060f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.285+0000 7fe739600640 1 -- 172.21.15.29:0/1827240539 >> v1:172.21.15.29:6789/0 conn(0x7fe72c0db2b0 legacy=0x7fe72c04bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:16.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.285+0000 7fe739600640 1 -- 172.21.15.29:0/1827240539 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe72c046780 con 0x7fe72c0e6f40 2024-04-17T19:48:16.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.285+0000 7f58fbe00640 1 Processor -- start 2024-04-17T19:48:16.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.289+0000 7f58fbe00640 1 -- start start 2024-04-17T19:48:16.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.289+0000 7fb7db200640 1 Processor -- start 2024-04-17T19:48:16.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.289+0000 7fb7db200640 1 -- start start 2024-04-17T19:48:16.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.289+0000 7f58fbe00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58fc0c2fc0 0x7f58fc0ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.289+0000 7f58fbe00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f58fc0c33d0 0x7f58fc0cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.289+0000 7f58fbe00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58fc160660 con 0x7f58fc15a310 2024-04-17T19:48:16.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.289+0000 7f58fbe00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f58fc0c2b30 con 0x7f58fc0c33d0 2024-04-17T19:48:16.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.289+0000 7f58fbe00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f58fc15a720 con 0x7f58fc0c2fc0 2024-04-17T19:48:16.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.289+0000 7fe739600640 1 --2- 172.21.15.29:0/1827240539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe72c0e6f40 0x7fe72c0e9360 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe7340503c0 tx=0x7fe7340997b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a436707ad34de88a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7f58fa000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58fc0c2fc0 0x7f58fc0ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7f58fb400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f58fc0c33d0 0x7f58fc0cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7f58fa000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58fc0c2fc0 0x7f58fc0ccea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36408/0 (socket says 172.21.15.29:36408) 2024-04-17T19:48:16.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7f58fa000640 1 -- 172.21.15.29:0/3064658390 learned_addr learned my addr 172.21.15.29:0/3064658390 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:16.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7f58faa00640 1 --1- 172.21.15.29:0/3064658390 >> v1:172.21.15.29:6789/0 conn(0x7f58fc15a310 0x7f58fc0c7b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38136/0 (socket says 172.21.15.29:38136) 2024-04-17T19:48:16.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.289+0000 7fe733400640 1 -- 172.21.15.29:0/1827240539 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73412e070 con 0x7fe72c0e6f40 2024-04-17T19:48:16.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fe733400640 1 -- 172.21.15.29:0/1827240539 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe734129600 con 0x7fe72c0e6f40 2024-04-17T19:48:16.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fe733400640 1 -- 172.21.15.29:0/1827240539 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73412e070 con 0x7fe72c0e6f40 2024-04-17T19:48:16.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7f58fb400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f58fc0c33d0 0x7f58fc0cd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46800/0 (socket says 172.21.15.29:46800) 2024-04-17T19:48:16.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7f58f9600640 1 -- 172.21.15.29:0/3064658390 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 420172541 0 0) 0x7f58fc160660 con 0x7f58fc15a310 2024-04-17T19:48:16.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7f58fb400640 1 -- 172.21.15.29:0/3064658390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58fc0c2fc0 msgr2=0x7f58fc0ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:16.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7f58fb400640 1 --2- 172.21.15.29:0/3064658390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58fc0c2fc0 0x7f58fc0ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7f58fb400640 1 -- 172.21.15.29:0/3064658390 >> v1:172.21.15.29:6789/0 conn(0x7f58fc15a310 legacy=0x7f58fc0c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7f58fb400640 1 -- 172.21.15.29:0/3064658390 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58fc0cda10 con 0x7f58fc0c33d0 2024-04-17T19:48:16.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fe73b600640 1 -- 172.21.15.29:0/1827240539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe72c0e6f40 msgr2=0x7fe72c0e9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fe73b600640 1 --2- 172.21.15.29:0/1827240539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe72c0e6f40 0x7fe72c0e9360 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe7340503c0 tx=0x7fe7340997b0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fb7db200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7dc0c7b10 0x7fb7dc16c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fb7db200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7dc16ccd0 0x7fb7dc173470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fb7db200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7dc0c8bd0 con 0x7fb7dc0c7f20 2024-04-17T19:48:16.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fb7db200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fb7dc16c300 con 0x7fb7dc16ccd0 2024-04-17T19:48:16.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fb7db200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb7dc0c9710 con 0x7fb7dc0c7b10 2024-04-17T19:48:16.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fb7da800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7dc16ccd0 0x7fb7dc173470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fb7d9e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7dc0c7b10 0x7fb7dc16c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fb7da800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7dc16ccd0 0x7fb7dc173470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52724/0 (socket says 172.21.15.29:52724) 2024-04-17T19:48:16.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fb7da800640 1 -- 172.21.15.29:0/1611281940 learned_addr learned my addr 172.21.15.29:0/1611281940 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:16.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fe73b600640 1 -- 172.21.15.29:0/1827240539 shutdown_connections 2024-04-17T19:48:16.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fe73b600640 1 --2- 172.21.15.29:0/1827240539 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe72c0e6f40 0x7fe72c0e9360 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fe73b600640 1 -- 172.21.15.29:0/1827240539 >> 172.21.15.29:0/1827240539 conn(0x7fe72c0503c0 msgr2=0x7fe72c0507d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:16.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fe73b600640 1 -- 172.21.15.29:0/1827240539 shutdown_connections 2024-04-17T19:48:16.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fb7d8a00640 1 -- 172.21.15.29:0/1611281940 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4026591520 0 0) 0x7fb7dc0c8bd0 con 0x7fb7dc0c7f20 2024-04-17T19:48:16.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fb7d8a00640 1 -- 172.21.15.29:0/1611281940 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7c0003530 con 0x7fb7dc0c7f20 2024-04-17T19:48:16.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fb7da800640 1 -- 172.21.15.29:0/1611281940 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7dc0c7b10 msgr2=0x7fb7dc16c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:16.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fe73b600640 1 -- 172.21.15.29:0/1827240539 wait complete. 2024-04-17T19:48:16.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7f165bc00640 1 Processor -- start 2024-04-17T19:48:16.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7f165bc00640 1 -- start start 2024-04-17T19:48:16.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fb7da800640 1 --2- 172.21.15.29:0/1611281940 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7dc0c7b10 0x7fb7dc16c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fb7da800640 1 -- 172.21.15.29:0/1611281940 >> v1:172.21.15.37:6789/0 conn(0x7fb7dc0c7f20 legacy=0x7fb7dc0cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.293+0000 7fb7da800640 1 -- 172.21.15.29:0/1611281940 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7dc0c8e70 con 0x7fb7dc16ccd0 2024-04-17T19:48:16.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.297+0000 7f165bc00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f1654071ce0 0x7f1654141de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.297+0000 7fe73b600640 1 Processor -- start 2024-04-17T19:48:16.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.297+0000 7fe73b600640 1 -- start start 2024-04-17T19:48:16.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.297+0000 7f165bc00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f16541597e0 0x7f1654163620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.297+0000 7f58fb400640 1 --2- 172.21.15.29:0/3064658390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58fc0c33d0 0x7f58fc0cd3e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009d90 tx=0x7f58f401f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44efcb8da27b8cb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.297+0000 7f58f9600640 1 -- 172.21.15.29:0/3064658390 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f40b4070 con 0x7f58fc0c33d0 2024-04-17T19:48:16.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.297+0000 7f58f9600640 1 -- 172.21.15.29:0/3064658390 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58f40af600 con 0x7f58fc0c33d0 2024-04-17T19:48:16.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.297+0000 7f58f9600640 1 -- 172.21.15.29:0/3064658390 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f40af9d0 con 0x7f58fc0c33d0 2024-04-17T19:48:16.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe73b600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe72c04c8e0 0x7fe72c17c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe73b600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe72c0db2b0 0x7fe72c0c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe73b600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe72c0e6f40 0x7fe72c0d4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe73b600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe72c034490 con 0x7fe72c04c8e0 2024-04-17T19:48:16.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe73b600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe72c0341b0 con 0x7fe72c0e6f40 2024-04-17T19:48:16.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe73b600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe72c006800 con 0x7fe72c0db2b0 2024-04-17T19:48:16.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.297+0000 7fb7da800640 1 --2- 172.21.15.29:0/1611281940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc16ccd0 0x7fb7dc173470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009a50 tx=0x7fb7d401cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3df23076d70607c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.297+0000 7f1659200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f1654071ce0 0x7f1654141de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.297+0000 7f1659200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f1654071ce0 0x7f1654141de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46806/0 (socket says 172.21.15.29:46806) 2024-04-17T19:48:16.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.297+0000 7f1659200640 1 -- 172.21.15.29:0/1913221847 learned_addr learned my addr 172.21.15.29:0/1913221847 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:16.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.297+0000 7f165bc00640 1 -- 172.21.15.29:0/1913221847 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1654142320 con 0x7f16541593d0 2024-04-17T19:48:16.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.297+0000 7f165bc00640 1 -- 172.21.15.29:0/1913221847 --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f16540c7bd0 con 0x7f16541597e0 2024-04-17T19:48:16.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.297+0000 7f165bc00640 1 -- 172.21.15.29:0/1913221847 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f165415c6d0 con 0x7f1654071ce0 2024-04-17T19:48:16.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.297+0000 7f1659c00640 1 --2- 172.21.15.29:0/1913221847 >> v2:172.21.15.29:3300/0 conn(0x7f16541597e0 0x7f1654163620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.297+0000 7f1653400640 1 -- 172.21.15.29:0/1913221847 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 897597751 0 0) 0x7f1654142320 con 0x7f16541593d0 2024-04-17T19:48:16.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.297+0000 7f1653400640 1 -- 172.21.15.29:0/1913221847 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f163800bd40 con 0x7f16541593d0 2024-04-17T19:48:16.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7f1659200640 1 -- 172.21.15.29:0/1913221847 >> v2:172.21.15.29:3300/0 conn(0x7f16541597e0 msgr2=0x7f1654163620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:16.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7f1659200640 1 --2- 172.21.15.29:0/1913221847 >> v2:172.21.15.29:3300/0 conn(0x7f16541597e0 0x7f1654163620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7f1659200640 1 -- 172.21.15.29:0/1913221847 >> v1:172.21.15.37:6789/0 conn(0x7f16541593d0 legacy=0x7f16540c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7f1659200640 1 -- 172.21.15.29:0/1913221847 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f165415c9b0 con 0x7f1654071ce0 2024-04-17T19:48:16.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fa7bae00640 1 Processor -- start 2024-04-17T19:48:16.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fa7bae00640 1 -- start start 2024-04-17T19:48:16.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe739600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe72c0e6f40 0x7fe72c0d4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe739600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe72c0e6f40 0x7fe72c0d4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46808/0 (socket says 172.21.15.29:46808) 2024-04-17T19:48:16.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe739600640 1 -- 172.21.15.29:0/3118686006 learned_addr learned my addr 172.21.15.29:0/3118686006 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:16.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe738c00640 1 --2- 172.21.15.29:0/3118686006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe72c04c8e0 0x7fe72c17c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe733e00640 1 --2- 172.21.15.29:0/3118686006 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe72c0db2b0 0x7fe72c0c87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fb7d8a00640 1 -- 172.21.15.29:0/1611281940 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d40a5a00 con 0x7fb7dc16ccd0 2024-04-17T19:48:16.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7f2142600640 1 Processor -- start 2024-04-17T19:48:16.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fb7d8a00640 1 -- 172.21.15.29:0/1611281940 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7d40a5ba0 con 0x7fb7dc16ccd0 2024-04-17T19:48:16.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fb7d8a00640 1 -- 172.21.15.29:0/1611281940 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7d40ad3f0 con 0x7fb7dc16ccd0 2024-04-17T19:48:16.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7f2142600640 1 -- start start 2024-04-17T19:48:16.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7f58fbe00640 1 -- 172.21.15.29:0/3064658390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58fc0c33d0 msgr2=0x7f58fc0cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7f58fbe00640 1 --2- 172.21.15.29:0/3064658390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58fc0c33d0 0x7f58fc0cd3e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009d90 tx=0x7f58f401f150 comp rx=0 tx=0).stop 2024-04-17T19:48:16.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7f58fbe00640 1 -- 172.21.15.29:0/3064658390 shutdown_connections 2024-04-17T19:48:16.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7f58fbe00640 1 --2- 172.21.15.29:0/3064658390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58fc0c33d0 0x7f58fc0cd3e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7f58fbe00640 1 --2- 172.21.15.29:0/3064658390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58fc0c2fc0 0x7f58fc0ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7f58fbe00640 1 -- 172.21.15.29:0/3064658390 >> 172.21.15.29:0/3064658390 conn(0x7f58fc06ddd0 msgr2=0x7f58fc06bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:16.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7f58fbe00640 1 -- 172.21.15.29:0/3064658390 shutdown_connections 2024-04-17T19:48:16.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe738c00640 1 -- 172.21.15.29:0/3118686006 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe72c0db2b0 msgr2=0x7fe72c0c87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe738c00640 1 --2- 172.21.15.29:0/3118686006 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe72c0db2b0 0x7fe72c0c87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe738c00640 1 -- 172.21.15.29:0/3118686006 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe72c0e6f40 msgr2=0x7fe72c0d4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe738c00640 1 --2- 172.21.15.29:0/3118686006 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe72c0e6f40 0x7fe72c0d4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe738c00640 1 -- 172.21.15.29:0/3118686006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe72c17cbc0 con 0x7fe72c04c8e0 2024-04-17T19:48:16.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe739600640 1 --2- 172.21.15.29:0/3118686006 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe72c0e6f40 0x7fe72c0d4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:48:16.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe738c00640 1 --2- 172.21.15.29:0/3118686006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe72c04c8e0 0x7fe72c17c680 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe72400c3c0 tx=0x7fe72400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe730c00640 1 -- 172.21.15.29:0/3118686006 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe724017070 con 0x7fe72c04c8e0 2024-04-17T19:48:16.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fb7db200640 1 -- 172.21.15.29:0/1611281940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc16ccd0 msgr2=0x7fb7dc173470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fb7db200640 1 --2- 172.21.15.29:0/1611281940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc16ccd0 0x7fb7dc173470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4009a50 tx=0x7fb7d401cd50 comp rx=0 tx=0).stop 2024-04-17T19:48:16.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe730c00640 1 -- 172.21.15.29:0/3118686006 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe724005cf0 con 0x7fe72c04c8e0 2024-04-17T19:48:16.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe730c00640 1 -- 172.21.15.29:0/3118686006 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7240072f0 con 0x7fe72c04c8e0 2024-04-17T19:48:16.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe73b600640 1 -- 172.21.15.29:0/3118686006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe72c0dae70 con 0x7fe72c04c8e0 2024-04-17T19:48:16.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fe73b600640 1 -- 172.21.15.29:0/3118686006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe72c0da5d0 con 0x7fe72c04c8e0 2024-04-17T19:48:16.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fb7db200640 1 -- 172.21.15.29:0/1611281940 shutdown_connections 2024-04-17T19:48:16.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fb7db200640 1 --2- 172.21.15.29:0/1611281940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc16ccd0 0x7fb7dc173470 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7fb7db200640 1 --2- 172.21.15.29:0/1611281940 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7dc0c7b10 0x7fb7dc16c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7fb7db200640 1 -- 172.21.15.29:0/1611281940 >> 172.21.15.29:0/1611281940 conn(0x7fb7dc06ddd0 msgr2=0x7fb7dc06bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:16.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fa7bae00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa7b40c86b0 0x7fa7b4161950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7fa7bae00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7b4157040 0x7fa7b416ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.301+0000 7f58fbe00640 1 -- 172.21.15.29:0/3064658390 wait complete. 2024-04-17T19:48:16.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7fa7bae00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7b40c8f40 con 0x7fa7b40ca0d0 2024-04-17T19:48:16.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7fa7b8e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7b4157040 0x7fa7b416ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7fa7b8e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7b4157040 0x7fa7b416ade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36418/0 (socket says 172.21.15.29:36418) 2024-04-17T19:48:16.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7fa7b8e00640 1 -- 172.21.15.29:0/3849927785 learned_addr learned my addr 172.21.15.29:0/3849927785 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:16.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7fa7bae00640 1 -- 172.21.15.29:0/3849927785 --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fa7b40c91e0 con 0x7fa7b40c86b0 2024-04-17T19:48:16.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7fa7bae00640 1 -- 172.21.15.29:0/3849927785 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa7b4163a50 con 0x7fa7b4157040 2024-04-17T19:48:16.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f2142600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f213c152330 0x7f213c15bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f2142600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f213c072ad0 0x7f213c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7fa7b8e00640 1 -- 172.21.15.29:0/3849927785 >> v2:172.21.15.29:3300/0 conn(0x7fa7b40c86b0 msgr2=0x7fa7b4161950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-17T19:48:16.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f327be00640 1 Processor -- start 2024-04-17T19:48:16.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f327be00640 1 -- start start 2024-04-17T19:48:16.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f1659200640 1 --2- 172.21.15.29:0/1913221847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1654071ce0 0x7f1654141de0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f16540cad90 tx=0x7f164002d530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9bdb4da8465867f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7fa7b3e00640 1 --2- 172.21.15.29:0/3849927785 >> v2:172.21.15.29:3300/0 conn(0x7fa7b40c86b0 0x7fa7b4161950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7fe73b600640 1 -- 172.21.15.29:0/3118686006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe72c0054b0 con 0x7fe72c04c8e0 2024-04-17T19:48:16.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7fb7db200640 1 -- 172.21.15.29:0/1611281940 shutdown_connections 2024-04-17T19:48:16.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f2142600640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f213c0cd900 0x7f213c0cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f2142600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f213c0c7700 con 0x7f213c072ad0 2024-04-17T19:48:16.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f2142600640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f213c0c7270 con 0x7f213c0cd900 2024-04-17T19:48:16.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f2142600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f213c15c4f0 con 0x7f213c152330 2024-04-17T19:48:16.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f213b400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f213c152330 0x7f213c15bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f213b400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f213c152330 0x7f213c15bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36420/0 (socket says 172.21.15.29:36420) 2024-04-17T19:48:16.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f213b400640 1 -- 172.21.15.29:0/1590244244 learned_addr learned my addr 172.21.15.29:0/1590244244 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:16.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f213be00640 1 --2- 172.21.15.29:0/1590244244 >> v2:172.21.15.37:3300/0 conn(0x7f213c0cd900 0x7f213c0cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f2f45a00640 1 Processor -- start 2024-04-17T19:48:16.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f2f45a00640 1 -- start start 2024-04-17T19:48:16.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f213aa00640 1 --2- 172.21.15.29:0/1590244244 >> v2:172.21.15.29:3300/0 conn(0x7f213c072ad0 0x7f213c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f58fbe00640 1 Processor -- start 2024-04-17T19:48:16.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7fe730c00640 1 -- 172.21.15.29:0/3118686006 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe72401a070 con 0x7fe72c04c8e0 2024-04-17T19:48:16.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7fe730c00640 1 --2- 172.21.15.29:0/3118686006 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe718068380 0x7fe71806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7fe730c00640 1 -- 172.21.15.29:0/3118686006 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe724085f40 con 0x7fe72c04c8e0 2024-04-17T19:48:16.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7fb7db200640 1 -- 172.21.15.29:0/1611281940 wait complete. 2024-04-17T19:48:16.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7fb7db200640 1 Processor -- start 2024-04-17T19:48:16.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7fb7db200640 1 -- start start 2024-04-17T19:48:16.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7fa7b8e00640 1 --2- 172.21.15.29:0/3849927785 >> v2:172.21.15.29:3300/0 conn(0x7fa7b40c86b0 0x7fa7b4161950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7fa7b8e00640 1 -- 172.21.15.29:0/3849927785 >> v1:172.21.15.29:6789/0 conn(0x7fa7b40ca0d0 legacy=0x7fa7b4156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7fa7b8e00640 1 -- 172.21.15.29:0/3849927785 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b4163cf0 con 0x7fa7b4157040 2024-04-17T19:48:16.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f213be00640 1 -- 172.21.15.29:0/1590244244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f213c152330 msgr2=0x7f213c15bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:16.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f213be00640 1 --2- 172.21.15.29:0/1590244244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f213c152330 0x7f213c15bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f213be00640 1 -- 172.21.15.29:0/1590244244 >> v2:172.21.15.29:3300/0 conn(0x7f213c072ad0 msgr2=0x7f213c072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:16.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f213be00640 1 --2- 172.21.15.29:0/1590244244 >> v2:172.21.15.29:3300/0 conn(0x7f213c072ad0 0x7f213c072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.305+0000 7f213be00640 1 -- 172.21.15.29:0/1590244244 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f213c0c60d0 con 0x7f213c0cd900 2024-04-17T19:48:16.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f2f45a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f40158a50 0x7f2f40151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f2f45a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f401523e0 0x7f2f400ca090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f2f45a00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f400c8d40 con 0x7f2f400729a0 2024-04-17T19:48:16.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f2f45a00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f2f4015bcb0 con 0x7f2f401523e0 2024-04-17T19:48:16.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f2f45a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2f400c9020 con 0x7f2f40158a50 2024-04-17T19:48:16.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f2f3ea00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2f400729a0 0x7f2f40071ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38156/0 (socket says 172.21.15.29:38156) 2024-04-17T19:48:16.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f2f3ea00640 1 -- 172.21.15.29:0/1679961713 learned_addr learned my addr 172.21.15.29:0/1679961713 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:16.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f2f3e000640 1 --2- 172.21.15.29:0/1679961713 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f40158a50 0x7f2f40151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f2f3f400640 1 --2- 172.21.15.29:0/1679961713 >> v2:172.21.15.29:3300/0 conn(0x7f2f401523e0 0x7f2f400ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f2f3f400640 1 -- 172.21.15.29:0/1679961713 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f40158a50 msgr2=0x7f2f40151ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:16.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f2f3f400640 1 --2- 172.21.15.29:0/1679961713 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f40158a50 0x7f2f40151ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f2f3f400640 1 -- 172.21.15.29:0/1679961713 >> v1:172.21.15.29:6789/0 conn(0x7f2f400729a0 legacy=0x7f2f40071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f2f3f400640 1 -- 172.21.15.29:0/1679961713 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f4015b660 con 0x7f2f401523e0 2024-04-17T19:48:16.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f327be00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3274158a50 0x7f32740ca090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f3279400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f32740729a0 0x7f3274152070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38168/0 (socket says 172.21.15.29:38168) 2024-04-17T19:48:16.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f3279400640 1 -- 172.21.15.29:0/1085407971 learned_addr learned my addr 172.21.15.29:0/1085407971 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:16.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f327be00640 1 -- 172.21.15.29:0/1085407971 --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32740c8a60 con 0x7f32740729a0 2024-04-17T19:48:16.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f327be00640 1 -- 172.21.15.29:0/1085407971 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32740c8d40 con 0x7f32740ca5d0 2024-04-17T19:48:16.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f327be00640 1 -- 172.21.15.29:0/1085407971 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f327415bcb0 con 0x7f3274158a50 2024-04-17T19:48:16.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f3278a00640 1 --2- 172.21.15.29:0/1085407971 >> v2:172.21.15.37:3300/0 conn(0x7f3274158a50 0x7f32740ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7fe733e00640 1 --2- 172.21.15.29:0/3118686006 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe718068380 0x7fe71806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f213aa00640 1 --2- 172.21.15.29:0/1590244244 >> v2:172.21.15.29:3300/0 conn(0x7f213c072ad0 0x7f213c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T19:48:16.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f1653400640 1 -- 172.21.15.29:0/1913221847 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16400c1070 con 0x7f1654071ce0 2024-04-17T19:48:16.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f1653400640 1 -- 172.21.15.29:0/1913221847 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16400bc600 con 0x7f1654071ce0 2024-04-17T19:48:16.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f1653400640 1 -- 172.21.15.29:0/1913221847 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16400bc9d0 con 0x7f1654071ce0 2024-04-17T19:48:16.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f58fbe00640 1 -- start start 2024-04-17T19:48:16.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f3273e00640 1 -- 172.21.15.29:0/1085407971 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2980522080 0 0) 0x7f32740c8d40 con 0x7f32740ca5d0 2024-04-17T19:48:16.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f3273e00640 1 -- 172.21.15.29:0/1085407971 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f325c003530 con 0x7f32740ca5d0 2024-04-17T19:48:16.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f3278a00640 1 -- 172.21.15.29:0/1085407971 >> v1:172.21.15.37:6789/0 conn(0x7f32740ca5d0 legacy=0x7f3274162950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.309+0000 7f3278a00640 1 -- 172.21.15.29:0/1085407971 >> v1:172.21.15.29:6789/0 conn(0x7f32740729a0 legacy=0x7f3274152070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f3278a00640 1 -- 172.21.15.29:0/1085407971 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32740c9020 con 0x7f3274158a50 2024-04-17T19:48:16.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f165bc00640 1 -- 172.21.15.29:0/1913221847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1654071ce0 msgr2=0x7f1654141de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f165bc00640 1 --2- 172.21.15.29:0/1913221847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1654071ce0 0x7f1654141de0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f16540cad90 tx=0x7f164002d530 comp rx=0 tx=0).stop 2024-04-17T19:48:16.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fe733e00640 1 --2- 172.21.15.29:0/3118686006 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe718068380 0x7fe71806a890 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe72c0c75d0 tx=0x7fe728009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f165bc00640 1 -- 172.21.15.29:0/1913221847 shutdown_connections 2024-04-17T19:48:16.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f165bc00640 1 --2- 172.21.15.29:0/1913221847 >> v2:172.21.15.29:3300/0 conn(0x7f16541597e0 0x7f1654163620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f165bc00640 1 --2- 172.21.15.29:0/1913221847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1654071ce0 0x7f1654141de0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f165bc00640 1 -- 172.21.15.29:0/1913221847 >> 172.21.15.29:0/1913221847 conn(0x7f165406bf90 msgr2=0x7f165406c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:16.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fa7b8e00640 1 --2- 172.21.15.29:0/3849927785 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7b4157040 0x7fa7b416ade0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4009870 tx=0x7fa7a401ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=892537b3b0454c9b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f165bc00640 1 -- 172.21.15.29:0/1913221847 shutdown_connections 2024-04-17T19:48:16.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f165bc00640 1 -- 172.21.15.29:0/1913221847 wait complete. 2024-04-17T19:48:16.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fa7b2a00640 1 -- 172.21.15.29:0/3849927785 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a40b5070 con 0x7fa7b4157040 2024-04-17T19:48:16.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fa7b2a00640 1 -- 172.21.15.29:0/3849927785 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7a40a8a50 con 0x7fa7b4157040 2024-04-17T19:48:16.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fa7b2a00640 1 -- 172.21.15.29:0/3849927785 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a40a8e20 con 0x7fa7b4157040 2024-04-17T19:48:16.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fb7db200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc0c7b10 0x7fb7dc168ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fb7db200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7dc0c7f20 0x7fb7dc20b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f2f3f400640 1 --2- 172.21.15.29:0/1679961713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f401523e0 0x7f2f400ca090 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2f38009d90 tx=0x7f2f3801c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3c8f5d1adfa07 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fb7db200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7dc16ccd0 0x7fb7dc167ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f213be00640 1 --2- 172.21.15.29:0/1590244244 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f213c0cd900 0x7f213c0cdd10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f213c06f190 tx=0x7f212c0230a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=380cb51f345c3666 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fb7db200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb7dc0cd570 con 0x7fb7dc0c7b10 2024-04-17T19:48:16.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fb7d9e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc0c7b10 0x7fb7dc168ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fb7d9400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7dc0c7f20 0x7fb7dc20b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fb7d9400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7dc0c7f20 0x7fb7dc20b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46832/0 (socket says 172.21.15.29:46832) 2024-04-17T19:48:16.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fb7d9400640 1 -- 172.21.15.29:0/1808747507 learned_addr learned my addr 172.21.15.29:0/1808747507 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:16.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f58fbe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58fc0c2fc0 0x7f58fc072900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f58fbe00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58fc0c33d0 0x7f58fc1e9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f58fbe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58fc15a310 0x7f58fc155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f58fbe00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc160e90 con 0x7f58fc0c2fc0 2024-04-17T19:48:16.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f58fbe00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc160bb0 con 0x7f58fc15a310 2024-04-17T19:48:16.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f58fbe00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f58fc161170 con 0x7f58fc0c33d0 2024-04-17T19:48:16.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f58fb400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58fc15a310 0x7f58fc155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f58fb400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58fc15a310 0x7f58fc155bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46834/0 (socket says 172.21.15.29:46834) 2024-04-17T19:48:16.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f58fb400640 1 -- 172.21.15.29:0/1417403344 learned_addr learned my addr 172.21.15.29:0/1417403344 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:16.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fa7bae00640 1 -- 172.21.15.29:0/3849927785 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7b4157040 msgr2=0x7fa7b416ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fa7bae00640 1 --2- 172.21.15.29:0/3849927785 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7b4157040 0x7fa7b416ade0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4009870 tx=0x7fa7a401ed90 comp rx=0 tx=0).stop 2024-04-17T19:48:16.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fb7db200640 1 -- 172.21.15.29:0/1808747507 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb7dc1633c0 con 0x7fb7dc0c7f20 2024-04-17T19:48:16.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fb7db200640 1 -- 172.21.15.29:0/1808747507 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb7dc0cd6f0 con 0x7fb7dc16ccd0 2024-04-17T19:48:16.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f58fa000640 1 --2- 172.21.15.29:0/1417403344 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58fc0c33d0 0x7f58fc1e9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fb7da800640 1 --2- 172.21.15.29:0/1808747507 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7dc16ccd0 0x7fb7dc167ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fa7bae00640 1 -- 172.21.15.29:0/3849927785 shutdown_connections 2024-04-17T19:48:16.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7f213a000640 1 -- 172.21.15.29:0/1590244244 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c0b9070 con 0x7f213c0cd900 2024-04-17T19:48:16.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.313+0000 7fa7bae00640 1 --2- 172.21.15.29:0/3849927785 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7b4157040 0x7fa7b416ade0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f213a000640 1 -- 172.21.15.29:0/1590244244 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f212c0acd40 con 0x7f213c0cd900 2024-04-17T19:48:16.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7fa7bae00640 1 --2- 172.21.15.29:0/3849927785 >> v2:172.21.15.29:3300/0 conn(0x7fa7b40c86b0 0x7fa7b4161950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7fa7bae00640 1 -- 172.21.15.29:0/3849927785 >> 172.21.15.29:0/3849927785 conn(0x7fa7b406bf90 msgr2=0x7fa7b406c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:16.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f213a000640 1 -- 172.21.15.29:0/1590244244 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f212c0b4660 con 0x7f213c0cd900 2024-04-17T19:48:16.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f58faa00640 1 --2- 172.21.15.29:0/1417403344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58fc0c2fc0 0x7f58fc072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f58fb400640 1 -- 172.21.15.29:0/1417403344 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58fc0c33d0 msgr2=0x7f58fc1e9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f58fb400640 1 --2- 172.21.15.29:0/1417403344 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58fc0c33d0 0x7f58fc1e9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f58fb400640 1 -- 172.21.15.29:0/1417403344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58fc0c2fc0 msgr2=0x7f58fc072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f58fb400640 1 --2- 172.21.15.29:0/1417403344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58fc0c2fc0 0x7f58fc072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f58fb400640 1 -- 172.21.15.29:0/1417403344 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58fc072e40 con 0x7f58fc15a310 2024-04-17T19:48:16.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f58faa00640 1 --2- 172.21.15.29:0/1417403344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58fc0c2fc0 0x7f58fc072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T19:48:16.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7fb7d9e00640 1 -- 172.21.15.29:0/1808747507 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7dc16ccd0 msgr2=0x7fb7dc167ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7fb7d9e00640 1 --2- 172.21.15.29:0/1808747507 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7dc16ccd0 0x7fb7dc167ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7fb7d9e00640 1 -- 172.21.15.29:0/1808747507 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7dc0c7f20 msgr2=0x7fb7dc20b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7fb7d9e00640 1 --2- 172.21.15.29:0/1808747507 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7dc0c7f20 0x7fb7dc20b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7fb7d9e00640 1 -- 172.21.15.29:0/1808747507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7dc169420 con 0x7fb7dc0c7b10 2024-04-17T19:48:16.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7fa7bae00640 1 -- 172.21.15.29:0/3849927785 shutdown_connections 2024-04-17T19:48:16.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7fa7bae00640 1 -- 172.21.15.29:0/3849927785 wait complete. 2024-04-17T19:48:16.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f165bc00640 1 Processor -- start 2024-04-17T19:48:16.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7fb7d9e00640 1 --2- 172.21.15.29:0/1808747507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc0c7b10 0x7fb7dc168ee0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb7c800d590 tx=0x7fb7c800da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2f3d600640 1 -- 172.21.15.29:0/1679961713 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f380b2070 con 0x7f2f401523e0 2024-04-17T19:48:16.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f3278a00640 1 --2- 172.21.15.29:0/1085407971 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3274158a50 0x7f32740ca090 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f326000a960 tx=0x7f326001d870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbd43e32e731e831 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2f3d600640 1 -- 172.21.15.29:0/1679961713 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f380a5830 con 0x7f2f401523e0 2024-04-17T19:48:16.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f58fb400640 1 --2- 172.21.15.29:0/1417403344 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58fc15a310 0x7f58fc155bd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009d90 tx=0x7f58f401bef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7fe730c00640 1 -- 172.21.15.29:0/3118686006 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe724055ad0 con 0x7fe72c04c8e0 2024-04-17T19:48:16.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f165bc00640 1 -- start start 2024-04-17T19:48:16.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2142600640 1 -- 172.21.15.29:0/1590244244 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f213c0cd900 msgr2=0x7f213c0cdd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7fb7d2000640 1 -- 172.21.15.29:0/1808747507 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8015070 con 0x7fb7dc0c7b10 2024-04-17T19:48:16.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7fb7d2000640 1 -- 172.21.15.29:0/1808747507 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7c8011870 con 0x7fb7dc0c7b10 2024-04-17T19:48:16.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f58f2a00640 1 -- 172.21.15.29:0/1417403344 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f40b4040 con 0x7f58fc15a310 2024-04-17T19:48:16.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f58fbe00640 1 -- 172.21.15.29:0/1417403344 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58fc1f6ae0 con 0x7f58fc15a310 2024-04-17T19:48:16.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2142600640 1 --2- 172.21.15.29:0/1590244244 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f213c0cd900 0x7f213c0cdd10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f213c06f190 tx=0x7f212c0230a0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f58fbe00640 1 -- 172.21.15.29:0/1417403344 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58fc1f70a0 con 0x7f58fc15a310 2024-04-17T19:48:16.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2f3d600640 1 -- 172.21.15.29:0/1679961713 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f380b2070 con 0x7f2f401523e0 2024-04-17T19:48:16.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2142600640 1 -- 172.21.15.29:0/1590244244 shutdown_connections 2024-04-17T19:48:16.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2142600640 1 --2- 172.21.15.29:0/1590244244 >> v2:172.21.15.29:3300/0 conn(0x7f213c072ad0 0x7f213c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2142600640 1 --2- 172.21.15.29:0/1590244244 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f213c0cd900 0x7f213c0cdd10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2142600640 1 --2- 172.21.15.29:0/1590244244 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f213c152330 0x7f213c15bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2142600640 1 -- 172.21.15.29:0/1590244244 >> 172.21.15.29:0/1590244244 conn(0x7f213c0700d0 msgr2=0x7f213c0704e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:16.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2142600640 1 -- 172.21.15.29:0/1590244244 shutdown_connections 2024-04-17T19:48:16.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2f45a00640 1 -- 172.21.15.29:0/1679961713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f401523e0 msgr2=0x7f2f400ca090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2f45a00640 1 --2- 172.21.15.29:0/1679961713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f401523e0 0x7f2f400ca090 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2f38009d90 tx=0x7f2f3801c990 comp rx=0 tx=0).stop 2024-04-17T19:48:16.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7fb7d2000640 1 -- 172.21.15.29:0/1808747507 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7c800b040 con 0x7fb7dc0c7b10 2024-04-17T19:48:16.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2142600640 1 -- 172.21.15.29:0/1590244244 wait complete. 2024-04-17T19:48:16.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f58f2a00640 1 -- 172.21.15.29:0/1417403344 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58f40c1070 con 0x7f58fc15a310 2024-04-17T19:48:16.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2f45a00640 1 -- 172.21.15.29:0/1679961713 shutdown_connections 2024-04-17T19:48:16.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2f45a00640 1 --2- 172.21.15.29:0/1679961713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f401523e0 0x7f2f400ca090 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2f45a00640 1 --2- 172.21.15.29:0/1679961713 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f40158a50 0x7f2f40151ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2f45a00640 1 -- 172.21.15.29:0/1679961713 >> 172.21.15.29:0/1679961713 conn(0x7f2f4006bf90 msgr2=0x7f2f4006c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:16.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2f45a00640 1 -- 172.21.15.29:0/1679961713 shutdown_connections 2024-04-17T19:48:16.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f3273e00640 1 -- 172.21.15.29:0/1085407971 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32600b3070 con 0x7f3274158a50 2024-04-17T19:48:16.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f3273e00640 1 -- 172.21.15.29:0/1085407971 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32600a6850 con 0x7f3274158a50 2024-04-17T19:48:16.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f327be00640 1 -- 172.21.15.29:0/1085407971 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3274158a50 msgr2=0x7f32740ca090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f327be00640 1 --2- 172.21.15.29:0/1085407971 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3274158a50 0x7f32740ca090 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f326000a960 tx=0x7f326001d870 comp rx=0 tx=0).stop 2024-04-17T19:48:16.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f3273e00640 1 -- 172.21.15.29:0/1085407971 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326000b040 con 0x7f3274158a50 2024-04-17T19:48:16.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f2f45a00640 1 -- 172.21.15.29:0/1679961713 wait complete. 2024-04-17T19:48:16.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f58f2a00640 1 -- 172.21.15.29:0/1417403344 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f40b4040 con 0x7f58fc15a310 2024-04-17T19:48:16.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7fb7db200640 1 -- 172.21.15.29:0/1808747507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7dc20b7a0 con 0x7fb7dc0c7b10 2024-04-17T19:48:16.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7fb7db200640 1 -- 172.21.15.29:0/1808747507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7dc15e8b0 con 0x7fb7dc0c7b10 2024-04-17T19:48:16.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7fa7bae00640 1 Processor -- start 2024-04-17T19:48:16.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f165bc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1654071ce0 0x7f165415b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f165bc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16541593d0 0x7f16540cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f165bc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16541597e0 0x7f1654151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f165bc00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f16540720f0 con 0x7f16541597e0 2024-04-17T19:48:16.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.317+0000 7f165bc00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1654159bf0 con 0x7f16541593d0 2024-04-17T19:48:16.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f165bc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1654072270 con 0x7f1654071ce0 2024-04-17T19:48:16.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7fa7bae00640 1 -- start start 2024-04-17T19:48:16.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f327be00640 1 -- 172.21.15.29:0/1085407971 shutdown_connections 2024-04-17T19:48:16.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f327be00640 1 --2- 172.21.15.29:0/1085407971 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3274158a50 0x7f32740ca090 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f327be00640 1 -- 172.21.15.29:0/1085407971 >> 172.21.15.29:0/1085407971 conn(0x7f327406bf90 msgr2=0x7f327406c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:16.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f58fbe00640 1 -- 172.21.15.29:0/1417403344 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58d40052a0 con 0x7f58fc15a310 2024-04-17T19:48:16.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f1659200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1654071ce0 0x7f165415b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f1659200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1654071ce0 0x7f165415b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36444/0 (socket says 172.21.15.29:36444) 2024-04-17T19:48:16.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f1659200640 1 -- 172.21.15.29:0/3911325835 learned_addr learned my addr 172.21.15.29:0/3911325835 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:16.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f1653e00640 1 --2- 172.21.15.29:0/3911325835 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16541593d0 0x7f16540cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f327be00640 1 -- 172.21.15.29:0/1085407971 shutdown_connections 2024-04-17T19:48:16.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7fb7db200640 1 -- 172.21.15.29:0/1808747507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7dc04ff10 con 0x7fb7dc0c7b10 2024-04-17T19:48:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f1659c00640 1 --2- 172.21.15.29:0/3911325835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16541597e0 0x7f1654151ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f1653e00640 1 -- 172.21.15.29:0/3911325835 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1654071ce0 msgr2=0x7f165415b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f1653e00640 1 --2- 172.21.15.29:0/3911325835 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1654071ce0 0x7f165415b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f1653e00640 1 -- 172.21.15.29:0/3911325835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16541597e0 msgr2=0x7f1654151ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f1653e00640 1 --2- 172.21.15.29:0/3911325835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16541597e0 0x7f1654151ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f1653e00640 1 -- 172.21.15.29:0/3911325835 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f165415beb0 con 0x7f16541593d0 2024-04-17T19:48:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f2f45a00640 1 Processor -- start 2024-04-17T19:48:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7fa7bae00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7b40c86b0 0x7fa7b41fa840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7fa7bae00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7b40ca0d0 0x7fa7b41ed360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7fa7bae00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7b4157040 0x7fa7b40cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7fa7bae00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b41612d0 con 0x7fa7b4157040 2024-04-17T19:48:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7fa7bae00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b4164290 con 0x7fa7b40ca0d0 2024-04-17T19:48:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7fa7bae00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa7b41647b0 con 0x7fa7b40c86b0 2024-04-17T19:48:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7fa7b3e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7b40c86b0 0x7fa7b41fa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f1653e00640 1 --2- 172.21.15.29:0/3911325835 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16541593d0 0x7f16540cb260 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f164c00c6b0 tx=0x7f164c00cb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7fa7b3e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7b40c86b0 0x7fa7b41fa840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36450/0 (socket says 172.21.15.29:36450) 2024-04-17T19:48:16.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7fa7b3e00640 1 -- 172.21.15.29:0/4034914937 learned_addr learned my addr 172.21.15.29:0/4034914937 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:16.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f2142600640 1 Processor -- start 2024-04-17T19:48:16.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f327be00640 1 -- 172.21.15.29:0/1085407971 wait complete. 2024-04-17T19:48:16.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f58f2a00640 1 -- 172.21.15.29:0/1417403344 <== mon.1 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f58f40ba470 con 0x7f58fc15a310 2024-04-17T19:48:16.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7fa7b3400640 1 --2- 172.21.15.29:0/4034914937 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7b40ca0d0 0x7fa7b41ed360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.321+0000 7f2f45a00640 1 -- start start 2024-04-17T19:48:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7fa7b8e00640 1 --2- 172.21.15.29:0/4034914937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7b4157040 0x7fa7b40cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7fa7b3e00640 1 -- 172.21.15.29:0/4034914937 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7b40ca0d0 msgr2=0x7fa7b41ed360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7fa7b3e00640 1 --2- 172.21.15.29:0/4034914937 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7b40ca0d0 0x7fa7b41ed360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7f2142600640 1 -- start start 2024-04-17T19:48:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7fa7b3e00640 1 -- 172.21.15.29:0/4034914937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7b4157040 msgr2=0x7fa7b40cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7fa7b3e00640 1 --2- 172.21.15.29:0/4034914937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7b4157040 0x7fa7b40cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7fa7b3e00640 1 -- 172.21.15.29:0/4034914937 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b41fad80 con 0x7fa7b40c86b0 2024-04-17T19:48:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7f327be00640 1 Processor -- start 2024-04-17T19:48:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7f327be00640 1 -- start start 2024-04-17T19:48:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7f58f2a00640 1 --2- 172.21.15.29:0/1417403344 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f58dc0685b0 0x7f58dc06aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7f58f2a00640 1 -- 172.21.15.29:0/1417403344 <== mon.1 v2:172.21.15.37:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f58f4125d30 con 0x7f58fc15a310 2024-04-17T19:48:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7fb7d2000640 1 -- 172.21.15.29:0/1808747507 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb7c8018070 con 0x7fb7dc0c7b10 2024-04-17T19:48:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7fa7b3e00640 1 --2- 172.21.15.29:0/4034914937 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7b40c86b0 0x7fa7b41fa840 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa7a000c820 tx=0x7fa7a000cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7fb7d2000640 1 --2- 172.21.15.29:0/1808747507 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb7bc0683b0 0x7fb7bc06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7fb7d2000640 1 -- 172.21.15.29:0/1808747507 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb7c8086c40 con 0x7fb7dc0c7b10 2024-04-17T19:48:16.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7fa7abe00640 1 -- 172.21.15.29:0/4034914937 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0015070 con 0x7fa7b40c86b0 2024-04-17T19:48:16.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7fa7abe00640 1 -- 172.21.15.29:0/4034914937 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7a0007a30 con 0x7fa7b40c86b0 2024-04-17T19:48:16.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7fa7bae00640 1 -- 172.21.15.29:0/4034914937 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7b41faf90 con 0x7fa7b40c86b0 2024-04-17T19:48:16.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7fa7bae00640 1 -- 172.21.15.29:0/4034914937 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7b41f7920 con 0x7fa7b40c86b0 2024-04-17T19:48:16.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7f58faa00640 1 --2- 172.21.15.29:0/1417403344 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f58dc0685b0 0x7f58dc06aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7fb7d9400640 1 --2- 172.21.15.29:0/1808747507 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb7bc0683b0 0x7fb7bc06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7fa7abe00640 1 -- 172.21.15.29:0/4034914937 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0007e20 con 0x7fa7b40c86b0 2024-04-17T19:48:16.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7f2142600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f213c072ad0 0x7f213c1e7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7f2f45a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f400729a0 0x7f2f401442a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7f2f45a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f401523e0 0x7f2f400cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7f2f45a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2f40158a50 0x7f2f400cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7f2f45a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2f400c86b0 con 0x7f2f401523e0 2024-04-17T19:48:16.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7f2f45a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4015f140 con 0x7f2f40158a50 2024-04-17T19:48:16.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7f2f45a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2f40144cb0 con 0x7f2f400729a0 2024-04-17T19:48:16.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.325+0000 7f2f3ea00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f400729a0 0x7f2f401442a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f2f3ea00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f400729a0 0x7f2f401442a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36452/0 (socket says 172.21.15.29:36452) 2024-04-17T19:48:16.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f2f3ea00640 1 -- 172.21.15.29:0/1821518041 learned_addr learned my addr 172.21.15.29:0/1821518041 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:16.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f2f3e000640 1 --2- 172.21.15.29:0/1821518041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f401523e0 0x7f2f400cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7fa7bae00640 1 -- 172.21.15.29:0/4034914937 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa78c0052a0 con 0x7fa7b40c86b0 2024-04-17T19:48:16.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f2142600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f213c0cd900 0x7f213c1ebec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f2f3f400640 1 --2- 172.21.15.29:0/1821518041 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2f40158a50 0x7f2f400cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f213b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f213c072ad0 0x7f213c1e7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f2142600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f213c152330 0x7f213c15bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f2142600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f213c0c6660 con 0x7f213c072ad0 2024-04-17T19:48:16.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f213b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f213c072ad0 0x7f213c1e7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52830/0 (socket says 172.21.15.29:52830) 2024-04-17T19:48:16.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f2142600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f213c0c6380 con 0x7f213c0cd900 2024-04-17T19:48:16.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f2142600640 1 -- 172.21.15.29:0/2195452433 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f213c0c6940 con 0x7f213c152330 2024-04-17T19:48:16.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f213b400640 1 -- 172.21.15.29:0/2195452433 learned_addr learned my addr 172.21.15.29:0/2195452433 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:16.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f213aa00640 1 --2- 172.21.15.29:0/2195452433 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f213c0cd900 0x7f213c1ebec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f2f3e000640 1 -- 172.21.15.29:0/1821518041 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f400729a0 msgr2=0x7f2f401442a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f2f3e000640 1 --2- 172.21.15.29:0/1821518041 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f400729a0 0x7f2f401442a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f2f3e000640 1 -- 172.21.15.29:0/1821518041 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2f40158a50 msgr2=0x7f2f400cd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f2f3e000640 1 --2- 172.21.15.29:0/1821518041 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2f40158a50 0x7f2f400cd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f2f3e000640 1 -- 172.21.15.29:0/1821518041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f401447e0 con 0x7f2f401523e0 2024-04-17T19:48:16.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f327be00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f32740729a0 0x7f327414cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7fb7d9400640 1 --2- 172.21.15.29:0/1808747507 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb7bc0683b0 0x7fb7bc06a8c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc08cd80 tx=0x7fb7cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f213b400640 1 -- 172.21.15.29:0/2195452433 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f213c152330 msgr2=0x7f213c15bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f213b400640 1 --2- 172.21.15.29:0/2195452433 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f213c152330 0x7f213c15bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f213b400640 1 -- 172.21.15.29:0/2195452433 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f213c0cd900 msgr2=0x7f213c1ebec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f327be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32740ca5d0 0x7f3274144120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f327be00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3274158a50 0x7f3274150b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f327be00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f32740c86b0 con 0x7f32740ca5d0 2024-04-17T19:48:16.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f327be00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3274158e60 con 0x7f32740729a0 2024-04-17T19:48:16.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f327be00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f327414a940 con 0x7f3274158a50 2024-04-17T19:48:16.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f58faa00640 1 --2- 172.21.15.29:0/1417403344 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f58dc0685b0 0x7f58dc06aac0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f58ec001040 tx=0x7f58ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f213b400640 1 --2- 172.21.15.29:0/2195452433 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f213c0cd900 0x7f213c1ebec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f3278a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32740ca5d0 0x7f3274144120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f3279400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f32740729a0 0x7f327414cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f3279400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f32740729a0 0x7f327414cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46860/0 (socket says 172.21.15.29:46860) 2024-04-17T19:48:16.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f3279400640 1 -- 172.21.15.29:0/754357375 learned_addr learned my addr 172.21.15.29:0/754357375 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:16.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f3279e00640 1 --2- 172.21.15.29:0/754357375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3274158a50 0x7f3274150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f213b400640 1 -- 172.21.15.29:0/2195452433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f213c1e8090 con 0x7f213c072ad0 2024-04-17T19:48:16.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f213aa00640 1 --2- 172.21.15.29:0/2195452433 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f213c0cd900 0x7f213c1ebec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:16.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f213b400640 1 --2- 172.21.15.29:0/2195452433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f213c072ad0 0x7f213c1e7b50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f213400b050 tx=0x7f213400b510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f2f3e000640 1 --2- 172.21.15.29:0/1821518041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f401523e0 0x7f2f400cc040 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2f4006cca0 tx=0x7f2f2c00c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f3279400640 1 -- 172.21.15.29:0/754357375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3274158a50 msgr2=0x7f3274150b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f3279400640 1 --2- 172.21.15.29:0/754357375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3274158a50 0x7f3274150b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f3279400640 1 -- 172.21.15.29:0/754357375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32740ca5d0 msgr2=0x7f3274144120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f3279400640 1 --2- 172.21.15.29:0/754357375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32740ca5d0 0x7f3274144120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f3279400640 1 -- 172.21.15.29:0/754357375 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f327414d2b0 con 0x7f32740729a0 2024-04-17T19:48:16.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f3278a00640 1 --2- 172.21.15.29:0/754357375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32740ca5d0 0x7f3274144120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:48:16.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7f3279400640 1 --2- 172.21.15.29:0/754357375 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f32740729a0 0x7f327414cd70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f326800b590 tx=0x7f326800ba50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.329+0000 7fa7abe00640 1 -- 172.21.15.29:0/4034914937 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa7a0018070 con 0x7fa7b40c86b0 2024-04-17T19:48:16.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.333+0000 7fa7abe00640 1 --2- 172.21.15.29:0/4034914937 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa7940683d0 0x7fa79406a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.333+0000 7fa7abe00640 1 -- 172.21.15.29:0/4034914937 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa7a0086f10 con 0x7fa7b40c86b0 2024-04-17T19:48:16.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.333+0000 7f2f36a00640 1 -- 172.21.15.29:0/1821518041 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c015070 con 0x7f2f401523e0 2024-04-17T19:48:16.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.333+0000 7fa7b3400640 1 --2- 172.21.15.29:0/4034914937 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa7940683d0 0x7fa79406a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.333+0000 7f2133400640 1 -- 172.21.15.29:0/2195452433 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2134013750 con 0x7f213c072ad0 2024-04-17T19:48:16.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.333+0000 7f2133400640 1 -- 172.21.15.29:0/2195452433 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21340138f0 con 0x7f213c072ad0 2024-04-17T19:48:16.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.333+0000 7f2133400640 1 -- 172.21.15.29:0/2195452433 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2134013ca0 con 0x7f213c072ad0 2024-04-17T19:48:16.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.333+0000 7f2f36a00640 1 -- 172.21.15.29:0/1821518041 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f2c011800 con 0x7f2f401523e0 2024-04-17T19:48:16.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.333+0000 7f2f36a00640 1 -- 172.21.15.29:0/1821518041 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c011bb0 con 0x7f2f401523e0 2024-04-17T19:48:16.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.333+0000 7f2f45a00640 1 -- 172.21.15.29:0/1821518041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f40157960 con 0x7f2f401523e0 2024-04-17T19:48:16.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.333+0000 7fb7d2000640 1 -- 172.21.15.29:0/1808747507 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb7dc20b7a0 con 0x7fb7dc0c7b10 2024-04-17T19:48:16.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.333+0000 7f2142600640 1 -- 172.21.15.29:0/2195452433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f213c1f66d0 con 0x7f213c072ad0 2024-04-17T19:48:16.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.333+0000 7f2142600640 1 -- 172.21.15.29:0/2195452433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f213c1f6c70 con 0x7f213c072ad0 2024-04-17T19:48:16.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.333+0000 7f2f45a00640 1 -- 172.21.15.29:0/1821518041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f40143d80 con 0x7f2f401523e0 2024-04-17T19:48:16.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.333+0000 7fa7b3400640 1 --2- 172.21.15.29:0/4034914937 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa7940683d0 0x7fa79406a8e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa7b406ca40 tx=0x7fa7ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.333+0000 7f2142600640 1 -- 172.21.15.29:0/2195452433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f213c069b90 con 0x7f213c072ad0 2024-04-17T19:48:16.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.333+0000 7f2f2be00640 1 -- 172.21.15.29:0/1821518041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f140052a0 con 0x7f2f401523e0 2024-04-17T19:48:16.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.337+0000 7f2f36a00640 1 -- 172.21.15.29:0/1821518041 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2f2c018070 con 0x7f2f401523e0 2024-04-17T19:48:16.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.337+0000 7f2f36a00640 1 --2- 172.21.15.29:0/1821518041 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2f200683d0 0x7f2f2006a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.337+0000 7f2133400640 1 -- 172.21.15.29:0/2195452433 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f213401d020 con 0x7f213c072ad0 2024-04-17T19:48:16.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.337+0000 7f2133400640 1 --2- 172.21.15.29:0/2195452433 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f211c068400 0x7f211c06a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.337+0000 7f2133400640 1 -- 172.21.15.29:0/2195452433 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2134087a90 con 0x7f213c072ad0 2024-04-17T19:48:16.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.337+0000 7f2f36a00640 1 -- 172.21.15.29:0/1821518041 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2f2c086650 con 0x7f2f401523e0 2024-04-17T19:48:16.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.337+0000 7f2f3ea00640 1 --2- 172.21.15.29:0/1821518041 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2f200683d0 0x7f2f2006a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.337+0000 7f2f3ea00640 1 --2- 172.21.15.29:0/1821518041 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2f200683d0 0x7f2f2006a8e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2f30000f70 tx=0x7f2f30007ef0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.341+0000 7f213aa00640 1 --2- 172.21.15.29:0/2195452433 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f211c068400 0x7f211c06a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.341+0000 7f2133400640 1 -- 172.21.15.29:0/2195452433 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2134024070 con 0x7f213c072ad0 2024-04-17T19:48:16.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.341+0000 7f213aa00640 1 --2- 172.21.15.29:0/2195452433 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f211c068400 0x7f211c06a910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f213c074850 tx=0x7f2128008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.345+0000 7f1650c00640 1 -- 172.21.15.29:0/3911325835 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f164c003b80 con 0x7f16541593d0 2024-04-17T19:48:16.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.345+0000 7f3271600640 1 -- 172.21.15.29:0/754357375 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3268007070 con 0x7f32740729a0 2024-04-17T19:48:16.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.345+0000 7f3271600640 1 -- 172.21.15.29:0/754357375 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3268015020 con 0x7f32740729a0 2024-04-17T19:48:16.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.345+0000 7f1650c00640 1 -- 172.21.15.29:0/3911325835 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f164c003d20 con 0x7f16541593d0 2024-04-17T19:48:16.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.345+0000 7f1650c00640 1 -- 172.21.15.29:0/3911325835 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f164c005310 con 0x7f16541593d0 2024-04-17T19:48:16.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.345+0000 7f58f2a00640 1 -- 172.21.15.29:0/1417403344 <== mon.1 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f58f4018cb0 con 0x7f58fc15a310 2024-04-17T19:48:16.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.345+0000 7f327be00640 1 -- 172.21.15.29:0/754357375 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3274149990 con 0x7f32740729a0 2024-04-17T19:48:16.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.345+0000 7f327be00640 1 -- 172.21.15.29:0/754357375 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f327414c570 con 0x7f32740729a0 2024-04-17T19:48:16.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.345+0000 7f165bc00640 1 -- 172.21.15.29:0/3911325835 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16540cbc70 con 0x7f16541593d0 2024-04-17T19:48:16.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.345+0000 7f165bc00640 1 -- 172.21.15.29:0/3911325835 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1654133a90 con 0x7f16541593d0 2024-04-17T19:48:16.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.345+0000 7f3271600640 1 -- 172.21.15.29:0/754357375 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f326800f850 con 0x7f32740729a0 2024-04-17T19:48:16.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.345+0000 7f2f36a00640 1 -- 172.21.15.29:0/1821518041 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2f2c056260 con 0x7f2f401523e0 2024-04-17T19:48:16.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.345+0000 7f165bc00640 1 -- 172.21.15.29:0/3911325835 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f165415a1f0 con 0x7f16541593d0 2024-04-17T19:48:16.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.345+0000 7f327be00640 1 -- 172.21.15.29:0/754357375 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32500052a0 con 0x7f32740729a0 2024-04-17T19:48:16.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.349+0000 7f3271600640 1 -- 172.21.15.29:0/754357375 <== mon.1 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f326800f9f0 con 0x7f32740729a0 2024-04-17T19:48:16.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.349+0000 7f3271600640 1 --2- 172.21.15.29:0/754357375 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3254068380 0x7f325406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.349+0000 7f3271600640 1 -- 172.21.15.29:0/754357375 <== mon.1 v2:172.21.15.37:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f32680875d0 con 0x7f32740729a0 2024-04-17T19:48:16.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.349+0000 7f3278a00640 1 --2- 172.21.15.29:0/754357375 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3254068380 0x7f325406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.353+0000 7f1650c00640 1 -- 172.21.15.29:0/3911325835 <== mon.1 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f164c0054b0 con 0x7f16541593d0 2024-04-17T19:48:16.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.353+0000 7f3278a00640 1 --2- 172.21.15.29:0/754357375 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3254068380 0x7f325406a890 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f32600b0000 tx=0x7f32600a8070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.353+0000 7f1650c00640 1 --2- 172.21.15.29:0/3911325835 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f16340683b0 0x7f163406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:16.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.353+0000 7f1650c00640 1 -- 172.21.15.29:0/3911325835 <== mon.1 v2:172.21.15.37:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f164c086e50 con 0x7f16541593d0 2024-04-17T19:48:16.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.353+0000 7f1659200640 1 --2- 172.21.15.29:0/3911325835 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f16340683b0 0x7f163406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:16.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.357+0000 7f1659200640 1 --2- 172.21.15.29:0/3911325835 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f16340683b0 0x7f163406a8c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1640000c10 tx=0x7f16400b7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:16.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.357+0000 7fa7abe00640 1 -- 172.21.15.29:0/4034914937 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa7a0057270 con 0x7fa7b40c86b0 2024-04-17T19:48:16.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.365+0000 7f1650c00640 1 -- 172.21.15.29:0/3911325835 <== mon.1 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f164c057020 con 0x7f16541593d0 2024-04-17T19:48:16.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.377+0000 7f3271600640 1 -- 172.21.15.29:0/754357375 <== mon.1 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f326801f030 con 0x7f32740729a0 2024-04-17T19:48:16.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.677+0000 7fe73b600640 1 -- 172.21.15.29:0/3118686006 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fe72c0c3b90 con 0x7fe72c04c8e0 2024-04-17T19:48:16.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.677+0000 7fe730c00640 1 -- 172.21.15.29:0/3118686006 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe724059dd0 con 0x7fe72c04c8e0 2024-04-17T19:48:16.669 INFO:teuthology.orchestra.run.smithi029.stdout:55834574857 2024-04-17T19:48:16.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.681+0000 7fe73b600640 1 -- 172.21.15.29:0/3118686006 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe718068380 msgr2=0x7fe71806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.681+0000 7fe73b600640 1 --2- 172.21.15.29:0/3118686006 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe718068380 0x7fe71806a890 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe72c0c75d0 tx=0x7fe728009060 comp rx=0 tx=0).stop 2024-04-17T19:48:16.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.681+0000 7fe73b600640 1 -- 172.21.15.29:0/3118686006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe72c04c8e0 msgr2=0x7fe72c17c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.681+0000 7fe73b600640 1 --2- 172.21.15.29:0/3118686006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe72c04c8e0 0x7fe72c17c680 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe72400c3c0 tx=0x7fe72400c880 comp rx=0 tx=0).stop 2024-04-17T19:48:16.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.681+0000 7fe73b600640 1 -- 172.21.15.29:0/3118686006 shutdown_connections 2024-04-17T19:48:16.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.681+0000 7fe73b600640 1 --2- 172.21.15.29:0/3118686006 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe718068380 0x7fe71806a890 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.681+0000 7fe73b600640 1 --2- 172.21.15.29:0/3118686006 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe72c0e6f40 0x7fe72c0d4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.681+0000 7fe73b600640 1 --2- 172.21.15.29:0/3118686006 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe72c0db2b0 0x7fe72c0c87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.681+0000 7fe73b600640 1 --2- 172.21.15.29:0/3118686006 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe72c04c8e0 0x7fe72c17c680 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.681+0000 7fe73b600640 1 -- 172.21.15.29:0/3118686006 >> 172.21.15.29:0/3118686006 conn(0x7fe72c0503c0 msgr2=0x7fe72c004950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:16.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.681+0000 7fe73b600640 1 -- 172.21.15.29:0/3118686006 shutdown_connections 2024-04-17T19:48:16.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.681+0000 7fe73b600640 1 -- 172.21.15.29:0/3118686006 wait complete. 2024-04-17T19:48:16.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.689+0000 7fb7db200640 1 -- 172.21.15.29:0/1808747507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fb7dc075b70 con 0x7fb7dc0c7b10 2024-04-17T19:48:16.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.689+0000 7fb7d2000640 1 -- 172.21.15.29:0/1808747507 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb7c8086090 con 0x7fb7dc0c7b10 2024-04-17T19:48:16.677 INFO:teuthology.orchestra.run.smithi029.stdout:55834574857 2024-04-17T19:48:16.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.693+0000 7fb7db200640 1 -- 172.21.15.29:0/1808747507 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb7bc0683b0 msgr2=0x7fb7bc06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.693+0000 7fb7db200640 1 --2- 172.21.15.29:0/1808747507 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb7bc0683b0 0x7fb7bc06a8c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc08cd80 tx=0x7fb7cc009040 comp rx=0 tx=0).stop 2024-04-17T19:48:16.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.693+0000 7fb7db200640 1 -- 172.21.15.29:0/1808747507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc0c7b10 msgr2=0x7fb7dc168ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.693+0000 7fb7db200640 1 --2- 172.21.15.29:0/1808747507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc0c7b10 0x7fb7dc168ee0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb7c800d590 tx=0x7fb7c800da50 comp rx=0 tx=0).stop 2024-04-17T19:48:16.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.693+0000 7fb7db200640 1 -- 172.21.15.29:0/1808747507 shutdown_connections 2024-04-17T19:48:16.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.693+0000 7fb7db200640 1 --2- 172.21.15.29:0/1808747507 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb7bc0683b0 0x7fb7bc06a8c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.693+0000 7fb7db200640 1 --2- 172.21.15.29:0/1808747507 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb7dc16ccd0 0x7fb7dc167ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.693+0000 7fb7db200640 1 --2- 172.21.15.29:0/1808747507 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb7dc0c7f20 0x7fb7dc20b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.693+0000 7fb7db200640 1 --2- 172.21.15.29:0/1808747507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7dc0c7b10 0x7fb7dc168ee0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.693+0000 7fb7db200640 1 -- 172.21.15.29:0/1808747507 >> 172.21.15.29:0/1808747507 conn(0x7fb7dc06ddd0 msgr2=0x7fb7dc06fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:16.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.693+0000 7fb7db200640 1 -- 172.21.15.29:0/1808747507 shutdown_connections 2024-04-17T19:48:16.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.693+0000 7fb7db200640 1 -- 172.21.15.29:0/1808747507 wait complete. 2024-04-17T19:48:16.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.693+0000 7f2142600640 1 -- 172.21.15.29:0/2195452433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f213c14c450 con 0x7f213c072ad0 2024-04-17T19:48:16.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.697+0000 7f58fbe00640 1 -- 172.21.15.29:0/1417403344 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f58d4005860 con 0x7f58fc15a310 2024-04-17T19:48:16.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.697+0000 7f2133400640 1 -- 172.21.15.29:0/2195452433 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2134057e80 con 0x7f213c072ad0 2024-04-17T19:48:16.686 INFO:teuthology.orchestra.run.smithi029.stdout:55834574857 2024-04-17T19:48:16.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.697+0000 7f58f2a00640 1 -- 172.21.15.29:0/1417403344 <== mon.1 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f58f40f5f10 con 0x7f58fc15a310 2024-04-17T19:48:16.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.701+0000 7f2f2be00640 1 -- 172.21.15.29:0/1821518041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f2f14005590 con 0x7f2f401523e0 2024-04-17T19:48:16.688 INFO:teuthology.orchestra.run.smithi029.stdout:55834574858 2024-04-17T19:48:16.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.701+0000 7f2f36a00640 1 -- 172.21.15.29:0/1821518041 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2f2c05a560 con 0x7f2f401523e0 2024-04-17T19:48:16.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.701+0000 7f2142600640 1 -- 172.21.15.29:0/2195452433 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f211c068400 msgr2=0x7f211c06a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.701+0000 7f2142600640 1 --2- 172.21.15.29:0/2195452433 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f211c068400 0x7f211c06a910 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f213c074850 tx=0x7f2128008040 comp rx=0 tx=0).stop 2024-04-17T19:48:16.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.701+0000 7f2142600640 1 -- 172.21.15.29:0/2195452433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f213c072ad0 msgr2=0x7f213c1e7b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.701+0000 7f2142600640 1 --2- 172.21.15.29:0/2195452433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f213c072ad0 0x7f213c1e7b50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f213400b050 tx=0x7f213400b510 comp rx=0 tx=0).stop 2024-04-17T19:48:16.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.701+0000 7f58fbe00640 1 -- 172.21.15.29:0/1417403344 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f58dc0685b0 msgr2=0x7f58dc06aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.701+0000 7f2142600640 1 -- 172.21.15.29:0/2195452433 shutdown_connections 2024-04-17T19:48:16.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.701+0000 7f2142600640 1 --2- 172.21.15.29:0/2195452433 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f211c068400 0x7f211c06a910 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.701+0000 7f2142600640 1 --2- 172.21.15.29:0/2195452433 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f213c152330 0x7f213c15bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.701+0000 7f2142600640 1 --2- 172.21.15.29:0/2195452433 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f213c0cd900 0x7f213c1ebec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.701+0000 7f2142600640 1 --2- 172.21.15.29:0/2195452433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f213c072ad0 0x7f213c1e7b50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.701+0000 7f2142600640 1 -- 172.21.15.29:0/2195452433 >> 172.21.15.29:0/2195452433 conn(0x7f213c0700d0 msgr2=0x7f213c0704e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:16.692 INFO:teuthology.orchestra.run.smithi029.stdout:55834574856 2024-04-17T19:48:16.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.705+0000 7f58fbe00640 1 --2- 172.21.15.29:0/1417403344 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f58dc0685b0 0x7f58dc06aac0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f58ec001040 tx=0x7f58ec009040 comp rx=0 tx=0).stop 2024-04-17T19:48:16.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.705+0000 7f58fbe00640 1 -- 172.21.15.29:0/1417403344 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58fc15a310 msgr2=0x7f58fc155bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.705+0000 7f58fbe00640 1 --2- 172.21.15.29:0/1417403344 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58fc15a310 0x7f58fc155bd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009d90 tx=0x7f58f401bef0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.705+0000 7f58fbe00640 1 -- 172.21.15.29:0/1417403344 shutdown_connections 2024-04-17T19:48:16.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.705+0000 7f58fbe00640 1 --2- 172.21.15.29:0/1417403344 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f58dc0685b0 0x7f58dc06aac0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.705+0000 7f58fbe00640 1 --2- 172.21.15.29:0/1417403344 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f58fc15a310 0x7f58fc155bd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.705+0000 7f58fbe00640 1 --2- 172.21.15.29:0/1417403344 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f58fc0c33d0 0x7f58fc1e9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.705+0000 7f58fbe00640 1 --2- 172.21.15.29:0/1417403344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58fc0c2fc0 0x7f58fc072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.705+0000 7f58fbe00640 1 -- 172.21.15.29:0/1417403344 >> 172.21.15.29:0/1417403344 conn(0x7f58fc06ddd0 msgr2=0x7f58fc15e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:16.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.705+0000 7f58fbe00640 1 -- 172.21.15.29:0/1417403344 shutdown_connections 2024-04-17T19:48:16.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.705+0000 7f2142600640 1 -- 172.21.15.29:0/2195452433 shutdown_connections 2024-04-17T19:48:16.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.705+0000 7f2142600640 1 -- 172.21.15.29:0/2195452433 wait complete. 2024-04-17T19:48:16.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.705+0000 7f58fbe00640 1 -- 172.21.15.29:0/1417403344 wait complete. 2024-04-17T19:48:16.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.709+0000 7f2f2be00640 1 -- 172.21.15.29:0/1821518041 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2f200683d0 msgr2=0x7f2f2006a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.709+0000 7f2f2be00640 1 --2- 172.21.15.29:0/1821518041 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2f200683d0 0x7f2f2006a8e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2f30000f70 tx=0x7f2f30007ef0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.709+0000 7f2f2be00640 1 -- 172.21.15.29:0/1821518041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f401523e0 msgr2=0x7f2f400cc040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.709+0000 7f2f2be00640 1 --2- 172.21.15.29:0/1821518041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f401523e0 0x7f2f400cc040 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2f4006cca0 tx=0x7f2f2c00c9d0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.709+0000 7f2f2be00640 1 -- 172.21.15.29:0/1821518041 shutdown_connections 2024-04-17T19:48:16.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.709+0000 7f2f2be00640 1 --2- 172.21.15.29:0/1821518041 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2f200683d0 0x7f2f2006a8e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.709+0000 7f2f2be00640 1 --2- 172.21.15.29:0/1821518041 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2f40158a50 0x7f2f400cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.709+0000 7f2f2be00640 1 --2- 172.21.15.29:0/1821518041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f401523e0 0x7f2f400cc040 secure :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2f4006cca0 tx=0x7f2f2c00c9d0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.709+0000 7f2f2be00640 1 --2- 172.21.15.29:0/1821518041 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f400729a0 0x7f2f401442a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.709+0000 7f2f2be00640 1 -- 172.21.15.29:0/1821518041 >> 172.21.15.29:0/1821518041 conn(0x7f2f4006bf90 msgr2=0x7f2f40149210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:16.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.709+0000 7f165bc00640 1 -- 172.21.15.29:0/3911325835 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f1654069b90 con 0x7f16541593d0 2024-04-17T19:48:16.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.709+0000 7fa7bae00640 1 -- 172.21.15.29:0/4034914937 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fa78c005860 con 0x7fa7b40c86b0 2024-04-17T19:48:16.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.709+0000 7f1650c00640 1 -- 172.21.15.29:0/3911325835 <== mon.1 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f164c056920 con 0x7f16541593d0 2024-04-17T19:48:16.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.709+0000 7f2f2be00640 1 -- 172.21.15.29:0/1821518041 shutdown_connections 2024-04-17T19:48:16.700 INFO:teuthology.orchestra.run.smithi029.stdout:55834574858 2024-04-17T19:48:16.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.713+0000 7fa7abe00640 1 -- 172.21.15.29:0/4034914937 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa7a0056b70 con 0x7fa7b40c86b0 2024-04-17T19:48:16.701 INFO:teuthology.orchestra.run.smithi029.stdout:55834574858 2024-04-17T19:48:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.713+0000 7fa7bae00640 1 -- 172.21.15.29:0/4034914937 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa7940683d0 msgr2=0x7fa79406a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.713+0000 7fa7bae00640 1 --2- 172.21.15.29:0/4034914937 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa7940683d0 0x7fa79406a8e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa7b406ca40 tx=0x7fa7ac009040 comp rx=0 tx=0).stop 2024-04-17T19:48:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.713+0000 7fa7bae00640 1 -- 172.21.15.29:0/4034914937 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7b40c86b0 msgr2=0x7fa7b41fa840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.713+0000 7fa7bae00640 1 --2- 172.21.15.29:0/4034914937 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7b40c86b0 0x7fa7b41fa840 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa7a000c820 tx=0x7fa7a000cce0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.713+0000 7fa7bae00640 1 -- 172.21.15.29:0/4034914937 shutdown_connections 2024-04-17T19:48:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.713+0000 7fa7bae00640 1 --2- 172.21.15.29:0/4034914937 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa7940683d0 0x7fa79406a8e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.713+0000 7fa7bae00640 1 --2- 172.21.15.29:0/4034914937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7b4157040 0x7fa7b40cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.713+0000 7fa7bae00640 1 --2- 172.21.15.29:0/4034914937 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa7b40ca0d0 0x7fa7b41ed360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.713+0000 7fa7bae00640 1 --2- 172.21.15.29:0/4034914937 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa7b40c86b0 0x7fa7b41fa840 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.713+0000 7fa7bae00640 1 -- 172.21.15.29:0/4034914937 >> 172.21.15.29:0/4034914937 conn(0x7fa7b406bf90 msgr2=0x7fa7b4162260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.713+0000 7fa7bae00640 1 -- 172.21.15.29:0/4034914937 shutdown_connections 2024-04-17T19:48:16.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.713+0000 7fa7bae00640 1 -- 172.21.15.29:0/4034914937 wait complete. 2024-04-17T19:48:16.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.713+0000 7f2f2be00640 1 -- 172.21.15.29:0/1821518041 wait complete. 2024-04-17T19:48:16.711 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.2 2024-04-17T19:48:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.721+0000 7f165bc00640 1 -- 172.21.15.29:0/3911325835 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f16340683b0 msgr2=0x7f163406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.721+0000 7f165bc00640 1 --2- 172.21.15.29:0/3911325835 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f16340683b0 0x7f163406a8c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1640000c10 tx=0x7f16400b7000 comp rx=0 tx=0).stop 2024-04-17T19:48:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.721+0000 7f165bc00640 1 -- 172.21.15.29:0/3911325835 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16541593d0 msgr2=0x7f16540cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.721+0000 7f165bc00640 1 --2- 172.21.15.29:0/3911325835 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16541593d0 0x7f16540cb260 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f164c00c6b0 tx=0x7f164c00cb70 comp rx=0 tx=0).stop 2024-04-17T19:48:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.721+0000 7f165bc00640 1 -- 172.21.15.29:0/3911325835 shutdown_connections 2024-04-17T19:48:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.721+0000 7f165bc00640 1 --2- 172.21.15.29:0/3911325835 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f16340683b0 0x7f163406a8c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.721+0000 7f165bc00640 1 --2- 172.21.15.29:0/3911325835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16541597e0 0x7f1654151ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.721+0000 7f165bc00640 1 --2- 172.21.15.29:0/3911325835 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16541593d0 0x7f16540cb260 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f164c00c6b0 tx=0x7f164c00cb70 comp rx=0 tx=0).stop 2024-04-17T19:48:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.721+0000 7f165bc00640 1 --2- 172.21.15.29:0/3911325835 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1654071ce0 0x7f165415b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.721+0000 7f165bc00640 1 -- 172.21.15.29:0/3911325835 >> 172.21.15.29:0/3911325835 conn(0x7f165406bf90 msgr2=0x7f1654072c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:16.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.721+0000 7f165bc00640 1 -- 172.21.15.29:0/3911325835 shutdown_connections 2024-04-17T19:48:16.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.721+0000 7f165bc00640 1 -- 172.21.15.29:0/3911325835 wait complete. 2024-04-17T19:48:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.729+0000 7f327be00640 1 -- 172.21.15.29:0/754357375 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f3250005860 con 0x7f32740729a0 2024-04-17T19:48:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.729+0000 7f3271600640 1 -- 172.21.15.29:0/754357375 <== mon.1 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f32680571f0 con 0x7f32740729a0 2024-04-17T19:48:16.720 INFO:teuthology.orchestra.run.smithi029.stdout:55834574858 2024-04-17T19:48:16.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.733+0000 7f327be00640 1 -- 172.21.15.29:0/754357375 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3254068380 msgr2=0x7f325406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.733+0000 7f327be00640 1 --2- 172.21.15.29:0/754357375 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3254068380 0x7f325406a890 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f32600b0000 tx=0x7f32600a8070 comp rx=0 tx=0).stop 2024-04-17T19:48:16.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.733+0000 7f327be00640 1 -- 172.21.15.29:0/754357375 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f32740729a0 msgr2=0x7f327414cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:16.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.733+0000 7f327be00640 1 --2- 172.21.15.29:0/754357375 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f32740729a0 0x7f327414cd70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f326800b590 tx=0x7f326800ba50 comp rx=0 tx=0).stop 2024-04-17T19:48:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.733+0000 7f327be00640 1 -- 172.21.15.29:0/754357375 shutdown_connections 2024-04-17T19:48:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.733+0000 7f327be00640 1 --2- 172.21.15.29:0/754357375 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3254068380 0x7f325406a890 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.733+0000 7f327be00640 1 --2- 172.21.15.29:0/754357375 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3274158a50 0x7f3274150b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.733+0000 7f327be00640 1 --2- 172.21.15.29:0/754357375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32740ca5d0 0x7f3274144120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.733+0000 7f327be00640 1 --2- 172.21.15.29:0/754357375 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f32740729a0 0x7f327414cd70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.733+0000 7f327be00640 1 -- 172.21.15.29:0/754357375 >> 172.21.15.29:0/754357375 conn(0x7f327406bf90 msgr2=0x7f327406c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.737+0000 7f327be00640 1 -- 172.21.15.29:0/754357375 shutdown_connections 2024-04-17T19:48:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:16.737+0000 7f327be00640 1 -- 172.21.15.29:0/754357375 wait complete. 2024-04-17T19:48:16.726 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.0 2024-04-17T19:48:16.733 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.4 2024-04-17T19:48:16.737 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574858 for osd.1 2024-04-17T19:48:16.741 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574858 for osd.7 2024-04-17T19:48:16.744 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574858 for osd.3 2024-04-17T19:48:16.745 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.6 2024-04-17T19:48:16.754 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574858 for osd.5 2024-04-17T19:48:17.711 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-17T19:48:17.727 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-17T19:48:17.734 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-17T19:48:17.738 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-17T19:48:17.741 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-17T19:48:17.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-17T19:48:17.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-17T19:48:17.755 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-17T19:48:18.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.017+0000 7f209f200640 1 Processor -- start 2024-04-17T19:48:18.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.021+0000 7f209f200640 1 -- start start 2024-04-17T19:48:18.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.021+0000 7f209f200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20a00cce60 0x7f20a00cd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.021+0000 7f209f200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f20a00c8d90 0x7f20a00c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.021+0000 7f209f200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20a00c8290 con 0x7f20a01562b0 2024-04-17T19:48:18.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.021+0000 7f209f200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f20a00c27a0 con 0x7f20a00c8d90 2024-04-17T19:48:18.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.021+0000 7f209d400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20a00cce60 0x7f20a00cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.021+0000 7f209f200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f20a015cec0 con 0x7f20a00cce60 2024-04-17T19:48:18.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.021+0000 7f209d400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20a00cce60 0x7f20a00cd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36478/0 (socket says 172.21.15.29:36478) 2024-04-17T19:48:18.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.021+0000 7f209d400640 1 -- 172.21.15.29:0/4158100491 learned_addr learned my addr 172.21.15.29:0/4158100491 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:18.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.025+0000 7f209e800640 1 --2- 172.21.15.29:0/4158100491 >> v2:172.21.15.29:3300/0 conn(0x7f20a00c8d90 0x7f20a00c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.025+0000 7f209d400640 1 -- 172.21.15.29:0/4158100491 >> v2:172.21.15.29:3300/0 conn(0x7f20a00c8d90 msgr2=0x7f20a00c91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:18.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.025+0000 7f209d400640 1 --2- 172.21.15.29:0/4158100491 >> v2:172.21.15.29:3300/0 conn(0x7f20a00c8d90 0x7f20a00c91a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.025+0000 7f209d400640 1 -- 172.21.15.29:0/4158100491 >> v1:172.21.15.29:6789/0 conn(0x7f20a01562b0 legacy=0x7f20a00c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:18.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.025+0000 7f209d400640 1 -- 172.21.15.29:0/4158100491 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20a00c2a60 con 0x7f20a00cce60 2024-04-17T19:48:18.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.037+0000 7f209d400640 1 --2- 172.21.15.29:0/4158100491 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20a00cce60 0x7f20a00cd270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2088009870 tx=0x7f208801eb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81dae5628caa06b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.037+0000 7f209ca00640 1 -- 172.21.15.29:0/4158100491 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20880b5070 con 0x7f20a00cce60 2024-04-17T19:48:18.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.037+0000 7f209ca00640 1 -- 172.21.15.29:0/4158100491 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20880a87f0 con 0x7f20a00cce60 2024-04-17T19:48:18.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.037+0000 7f209ca00640 1 -- 172.21.15.29:0/4158100491 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20880a8bc0 con 0x7f20a00cce60 2024-04-17T19:48:18.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.037+0000 7f209f200640 1 -- 172.21.15.29:0/4158100491 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20a00cce60 msgr2=0x7f20a00cd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.037+0000 7f209f200640 1 --2- 172.21.15.29:0/4158100491 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20a00cce60 0x7f20a00cd270 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2088009870 tx=0x7f208801eb30 comp rx=0 tx=0).stop 2024-04-17T19:48:18.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.041+0000 7f209f200640 1 -- 172.21.15.29:0/4158100491 shutdown_connections 2024-04-17T19:48:18.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.041+0000 7f209f200640 1 --2- 172.21.15.29:0/4158100491 >> v2:172.21.15.29:3300/0 conn(0x7f20a00c8d90 0x7f20a00c91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.041+0000 7f209f200640 1 --2- 172.21.15.29:0/4158100491 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20a00cce60 0x7f20a00cd270 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.041+0000 7f209f200640 1 -- 172.21.15.29:0/4158100491 >> 172.21.15.29:0/4158100491 conn(0x7f20a006ddd0 msgr2=0x7f20a006bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:18.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.041+0000 7f209f200640 1 -- 172.21.15.29:0/4158100491 shutdown_connections 2024-04-17T19:48:18.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.041+0000 7f209f200640 1 -- 172.21.15.29:0/4158100491 wait complete. 2024-04-17T19:48:18.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.049+0000 7f209f200640 1 Processor -- start 2024-04-17T19:48:18.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.049+0000 7f209f200640 1 -- start start 2024-04-17T19:48:18.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.049+0000 7f209f200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20a00c8d90 0x7f20a01505f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.049+0000 7f209f200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20a00cce60 0x7f20a02013e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.049+0000 7f209f200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20a01562b0 0x7f20a01f25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.049+0000 7f209f200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f20a00c3300 con 0x7f20a00cce60 2024-04-17T19:48:18.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.049+0000 7f209f200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f20a00c3020 con 0x7f20a01562b0 2024-04-17T19:48:18.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.049+0000 7f5e4a400640 1 Processor -- start 2024-04-17T19:48:18.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.049+0000 7f209f200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f20a01566c0 con 0x7f20a00c8d90 2024-04-17T19:48:18.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.049+0000 7f209de00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20a00c8d90 0x7f20a01505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.053+0000 7f5e4a400640 1 -- start start 2024-04-17T19:48:18.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.049+0000 7f209de00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20a00c8d90 0x7f20a01505f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36488/0 (socket says 172.21.15.29:36488) 2024-04-17T19:48:18.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.049+0000 7f209de00640 1 -- 172.21.15.29:0/427262431 learned_addr learned my addr 172.21.15.29:0/427262431 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:18.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.053+0000 7f209de00640 1 -- 172.21.15.29:0/427262431 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20a01562b0 msgr2=0x7f20a01f25f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T19:48:18.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.053+0000 7f209de00640 1 --2- 172.21.15.29:0/427262431 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20a01562b0 0x7f20a01f25f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.053+0000 7f209de00640 1 -- 172.21.15.29:0/427262431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20a00cce60 msgr2=0x7f20a02013e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T19:48:18.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.053+0000 7f209de00640 1 --2- 172.21.15.29:0/427262431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20a00cce60 0x7f20a02013e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.053+0000 7f209de00640 1 -- 172.21.15.29:0/427262431 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20a0201920 con 0x7f20a00c8d90 2024-04-17T19:48:18.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.053+0000 7f209de00640 1 --2- 172.21.15.29:0/427262431 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20a00c8d90 0x7f20a01505f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f209800ba50 tx=0x7f209800bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.053+0000 7f5e4a400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e4415bf20 0x7f5e440c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.057+0000 7f5e4a400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5e440cce60 0x7f5e440cd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.057+0000 7f5e43e00640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f5e441588e0 0x7f5e44152070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:49884/0 (socket says 172.21.15.29:49884) 2024-04-17T19:48:18.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.057+0000 7f5e43e00640 1 -- 172.21.15.29:0/427115673 learned_addr learned my addr 172.21.15.29:0/427115673 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:18.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.057+0000 7f5e4a400640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e4415b480 con 0x7f5e441588e0 2024-04-17T19:48:18.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.057+0000 7f5e4a400640 1 -- 172.21.15.29:0/427115673 --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f5e440c8dc0 con 0x7f5e440cce60 2024-04-17T19:48:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.057+0000 7f5e49000640 1 --2- 172.21.15.29:0/427115673 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e4415bf20 0x7f5e440c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.057+0000 7f5e4a400640 1 -- 172.21.15.29:0/427115673 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5e4415ba90 con 0x7f5e4415bf20 2024-04-17T19:48:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.057+0000 7f2096000640 1 -- 172.21.15.29:0/427262431 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2098019070 con 0x7f20a00c8d90 2024-04-17T19:48:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.057+0000 7f2096000640 1 -- 172.21.15.29:0/427262431 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f209800cc10 con 0x7f20a00c8d90 2024-04-17T19:48:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.057+0000 7f2096000640 1 -- 172.21.15.29:0/427262431 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2098014440 con 0x7f20a00c8d90 2024-04-17T19:48:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.057+0000 7f209f200640 1 -- 172.21.15.29:0/427262431 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20a00c7260 con 0x7f20a00c8d90 2024-04-17T19:48:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.057+0000 7f209f200640 1 -- 172.21.15.29:0/427262431 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f20a00c6f80 con 0x7f20a00c8d90 2024-04-17T19:48:18.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.057+0000 7f5e49a00640 1 --2- 172.21.15.29:0/427115673 >> v2:172.21.15.29:3300/0 conn(0x7f5e440cce60 0x7f5e440cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.061+0000 7f208f400640 1 -- 172.21.15.29:0/427262431 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20a004ff10 con 0x7f20a00c8d90 2024-04-17T19:48:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.061+0000 7f5e43400640 1 -- 172.21.15.29:0/427115673 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2675801384 0 0) 0x7f5e4415b480 con 0x7f5e441588e0 2024-04-17T19:48:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.061+0000 7f5e43400640 1 -- 172.21.15.29:0/427115673 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e28005740 con 0x7f5e441588e0 2024-04-17T19:48:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.061+0000 7f5e49a00640 1 -- 172.21.15.29:0/427115673 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e4415bf20 msgr2=0x7f5e440c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:18.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.061+0000 7f8e5d600640 1 Processor -- start 2024-04-17T19:48:18.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.061+0000 7f2096000640 1 -- 172.21.15.29:0/427262431 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2098007370 con 0x7f20a00c8d90 2024-04-17T19:48:18.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.061+0000 7f5e49a00640 1 --2- 172.21.15.29:0/427115673 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e4415bf20 0x7f5e440c7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.061+0000 7f5e49a00640 1 -- 172.21.15.29:0/427115673 >> v1:172.21.15.37:6789/0 conn(0x7f5e441588e0 legacy=0x7f5e44152070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.061+0000 7f5e49a00640 1 -- 172.21.15.29:0/427115673 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e4415c330 con 0x7f5e440cce60 2024-04-17T19:48:18.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.061+0000 7f5e49000640 1 --2- 172.21.15.29:0/427115673 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e4415bf20 0x7f5e440c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:18.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.061+0000 7f8e5d600640 1 -- start start 2024-04-17T19:48:18.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.061+0000 7f8e5d600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8e580ca0d0 0x7f8e58156930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.061+0000 7f8e5d600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e580ca4e0 0x7f8e58156e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.061+0000 7f2096000640 1 --2- 172.21.15.29:0/427262431 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f207c068400 0x7f207c06a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f2096000640 1 -- 172.21.15.29:0/427262431 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2098086940 con 0x7f20a00c8d90 2024-04-17T19:48:18.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f8e57400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e580ca4e0 0x7f8e58156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f8e57400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e580ca4e0 0x7f8e58156e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36506/0 (socket says 172.21.15.29:36506) 2024-04-17T19:48:18.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f8e57400640 1 -- 172.21.15.29:0/1866291538 learned_addr learned my addr 172.21.15.29:0/1866291538 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:18.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f8e56000640 1 --2- 172.21.15.29:0/1866291538 >> v2:172.21.15.29:3300/0 conn(0x7f8e580ca0d0 0x7f8e58156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f209d400640 1 --2- 172.21.15.29:0/427262431 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f207c068400 0x7f207c06a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f209d400640 1 --2- 172.21.15.29:0/427262431 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f207c068400 0x7f207c06a910 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f20880b2000 tx=0x7f20880aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f8e5d600640 1 -- 172.21.15.29:0/1866291538 --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e580c8f40 con 0x7f8e580c86b0 2024-04-17T19:48:18.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f8e5d600640 1 -- 172.21.15.29:0/1866291538 --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f8e58162060 con 0x7f8e580ca0d0 2024-04-17T19:48:18.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f8e5d600640 1 -- 172.21.15.29:0/1866291538 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8e580c91e0 con 0x7f8e580ca4e0 2024-04-17T19:48:18.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f9ca8c00640 1 Processor -- start 2024-04-17T19:48:18.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f9ca8c00640 1 -- start start 2024-04-17T19:48:18.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f5e49a00640 1 --2- 172.21.15.29:0/427115673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e440cce60 0x7f5e440cd270 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009c80 tx=0x7f5e3c01c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba0ea3d7462e4ec server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f5a0f400640 1 Processor -- start 2024-04-17T19:48:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f5a0f400640 1 -- start start 2024-04-17T19:48:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f8e57400640 1 -- 172.21.15.29:0/1866291538 >> v2:172.21.15.29:3300/0 conn(0x7f8e580ca0d0 msgr2=0x7f8e58156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f8e57400640 1 --2- 172.21.15.29:0/1866291538 >> v2:172.21.15.29:3300/0 conn(0x7f8e580ca0d0 0x7f8e58156930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f8e57400640 1 -- 172.21.15.29:0/1866291538 >> v1:172.21.15.29:6789/0 conn(0x7f8e580c86b0 legacy=0x7f8e58161950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f8e57400640 1 -- 172.21.15.29:0/1866291538 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e58163a50 con 0x7f8e580ca4e0 2024-04-17T19:48:18.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f5a0f400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5a081521e0 0x7f5a080c2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f5a0f400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5a08158bf0 0x7f5a080c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f5a0f400640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a080c40d0 con 0x7f5a080c99a0 2024-04-17T19:48:18.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f5a0f400640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f5a08158050 con 0x7f5a081521e0 2024-04-17T19:48:18.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f5a0f400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f5a081525f0 con 0x7f5a08158bf0 2024-04-17T19:48:18.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f5a0d400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5a08158bf0 0x7f5a080c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f5a0ca00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5a081521e0 0x7f5a080c2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f5a0ca00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5a081521e0 0x7f5a080c2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52876/0 (socket says 172.21.15.29:52876) 2024-04-17T19:48:18.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f5a0ca00640 1 -- 172.21.15.29:0/1569617840 learned_addr learned my addr 172.21.15.29:0/1569617840 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:18.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f5e43400640 1 -- 172.21.15.29:0/427115673 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c0b2070 con 0x7f5e440cce60 2024-04-17T19:48:18.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f5e43400640 1 -- 172.21.15.29:0/427115673 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e3c0a5820 con 0x7f5e440cce60 2024-04-17T19:48:18.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f5a0ca00640 1 -- 172.21.15.29:0/1569617840 >> v2:172.21.15.37:3300/0 conn(0x7f5a08158bf0 msgr2=0x7f5a080c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:18.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f5a0ca00640 1 --2- 172.21.15.29:0/1569617840 >> v2:172.21.15.37:3300/0 conn(0x7f5a08158bf0 0x7f5a080c9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f5a0ca00640 1 -- 172.21.15.29:0/1569617840 >> v1:172.21.15.29:6789/0 conn(0x7f5a080c99a0 legacy=0x7f5a081584e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-17T19:48:18.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f5a0ca00640 1 -- 172.21.15.29:0/1569617840 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0815c3d0 con 0x7f5a081521e0 2024-04-17T19:48:18.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f5e43400640 1 -- 172.21.15.29:0/427115673 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c0a5bf0 con 0x7f5e440cce60 2024-04-17T19:48:18.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f8e57400640 1 --2- 172.21.15.29:0/1866291538 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e580ca4e0 0x7f8e58156e70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8e48009e50 tx=0x7f8e4801f230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2594191725a6af7e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f9ca8c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9ca40c2460 0x7f9ca40c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f9ca8c00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ca415c3d0 con 0x7f9ca40c8d70 2024-04-17T19:48:18.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f9ca8c00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ca415b850 con 0x7f9ca40c9ee0 2024-04-17T19:48:18.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.065+0000 7f9ca8c00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f9ca4158050 con 0x7f9ca40c2460 2024-04-17T19:48:18.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca2800640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f9ca40c9ee0 0x7f9ca41584e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:49898/0 (socket says 172.21.15.29:49898) 2024-04-17T19:48:18.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca2800640 1 -- 172.21.15.29:0/2399162599 learned_addr learned my addr 172.21.15.29:0/2399162599 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca0a00640 1 -- 172.21.15.29:0/2399162599 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3936198646 0 0) 0x7f9ca415b850 con 0x7f9ca40c9ee0 2024-04-17T19:48:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca0a00640 1 -- 172.21.15.29:0/2399162599 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c88003530 con 0x7f9ca40c9ee0 2024-04-17T19:48:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca1400640 1 --2- 172.21.15.29:0/2399162599 >> v2:172.21.15.29:3300/0 conn(0x7f9ca40c2460 0x7f9ca40c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca0a00640 1 -- 172.21.15.29:0/2399162599 <== mon.1 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2810605663 0 0) 0x7f9c88003530 con 0x7f9ca40c9ee0 2024-04-17T19:48:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca0a00640 1 -- 172.21.15.29:0/2399162599 >> v2:172.21.15.29:3300/0 conn(0x7f9ca40c2460 msgr2=0x7f9ca40c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca0a00640 1 --2- 172.21.15.29:0/2399162599 >> v2:172.21.15.29:3300/0 conn(0x7f9ca40c2460 0x7f9ca40c99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca0a00640 1 -- 172.21.15.29:0/2399162599 >> v1:172.21.15.29:6789/0 conn(0x7f9ca40c8d70 legacy=0x7f9ca4152010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca0a00640 1 -- 172.21.15.29:0/2399162599 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca415bb30 con 0x7f9ca40c9ee0 2024-04-17T19:48:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca0a00640 1 -- 172.21.15.29:0/2399162599 <== mon.1 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f9c90001680 con 0x7f9ca40c9ee0 2024-04-17T19:48:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca0a00640 1 -- 172.21.15.29:0/2399162599 >> v1:172.21.15.37:6789/0 conn(0x7f9ca40c9ee0 legacy=0x7f9ca41584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca0a00640 1 --2- 172.21.15.29:0/2399162599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c88003a10 0x7f9c88003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca0a00640 1 --2- 172.21.15.29:0/2399162599 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c88004810 0x7f9c88004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca0a00640 1 --2- 172.21.15.29:0/2399162599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c88005280 0x7f9c8801c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca0a00640 1 -- 172.21.15.29:0/2399162599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca4158050 con 0x7f9c88005280 2024-04-17T19:48:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca0a00640 1 -- 172.21.15.29:0/2399162599 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9c8801d1b0 con 0x7f9c88003a10 2024-04-17T19:48:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca0a00640 1 -- 172.21.15.29:0/2399162599 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9c8801d3b0 con 0x7f9c88004810 2024-04-17T19:48:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca1400640 1 --2- 172.21.15.29:0/2399162599 >> v2:172.21.15.29:3300/0 conn(0x7f9ca40c2460 0x7f9ca40c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:48:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca1e00640 1 --2- 172.21.15.29:0/2399162599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c88003a10 0x7f9c88003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca1400640 1 --2- 172.21.15.29:0/2399162599 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c88004810 0x7f9c88004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.069+0000 7f9ca2800640 1 --2- 172.21.15.29:0/2399162599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c88005280 0x7f9c8801c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.073+0000 7f9ca2800640 1 -- 172.21.15.29:0/2399162599 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c88004810 msgr2=0x7f9c88004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.073+0000 7f9ca2800640 1 --2- 172.21.15.29:0/2399162599 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c88004810 0x7f9c88004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.073+0000 7f9ca2800640 1 -- 172.21.15.29:0/2399162599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c88003a10 msgr2=0x7f9c88003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.073+0000 7f9ca2800640 1 --2- 172.21.15.29:0/2399162599 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c88003a10 0x7f9c88003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.073+0000 7f9ca2800640 1 -- 172.21.15.29:0/2399162599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c88003530 con 0x7f9c88005280 2024-04-17T19:48:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.073+0000 7f9ca1400640 1 --2- 172.21.15.29:0/2399162599 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c88004810 0x7f9c88004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:48:18.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.073+0000 7f5e4a400640 1 -- 172.21.15.29:0/427115673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e440cce60 msgr2=0x7f5e440cd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.073+0000 7f5e4a400640 1 --2- 172.21.15.29:0/427115673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e440cce60 0x7f5e440cd270 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009c80 tx=0x7f5e3c01c980 comp rx=0 tx=0).stop 2024-04-17T19:48:18.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.073+0000 7f5e4a400640 1 -- 172.21.15.29:0/427115673 shutdown_connections 2024-04-17T19:48:18.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.073+0000 7f5e4a400640 1 --2- 172.21.15.29:0/427115673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e440cce60 0x7f5e440cd270 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.073+0000 7f5e4a400640 1 --2- 172.21.15.29:0/427115673 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e4415bf20 0x7f5e440c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.073+0000 7f5e4a400640 1 -- 172.21.15.29:0/427115673 >> 172.21.15.29:0/427115673 conn(0x7f5e4406ddd0 msgr2=0x7f5e4406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:18.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.073+0000 7f5e4a400640 1 -- 172.21.15.29:0/427115673 shutdown_connections 2024-04-17T19:48:18.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.073+0000 7f5e4a400640 1 -- 172.21.15.29:0/427115673 wait complete. 2024-04-17T19:48:18.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.073+0000 7f9ca2800640 1 -- 172.21.15.29:0/2399162599 reap_dead start 2024-04-17T19:48:18.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.073+0000 7f5a0ca00640 1 --2- 172.21.15.29:0/1569617840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a081521e0 0x7f5a080c2460 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5a0806e2c0 tx=0x7f59f401c300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b335112ff6a90a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.073+0000 7f9ca2800640 1 --2- 172.21.15.29:0/2399162599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c88005280 0x7f9c8801c9e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9c90006fa0 tx=0x7f9c9001cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61226eb0b29c8cf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f5a07400640 1 -- 172.21.15.29:0/1569617840 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f40b0070 con 0x7f5a081521e0 2024-04-17T19:48:18.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f5a07400640 1 -- 172.21.15.29:0/1569617840 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59f40ab600 con 0x7f5a081521e0 2024-04-17T19:48:18.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f5a07400640 1 -- 172.21.15.29:0/1569617840 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f40ab9d0 con 0x7f5a081521e0 2024-04-17T19:48:18.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f9ca0a00640 1 -- 172.21.15.29:0/2399162599 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c94009880 con 0x7f9c88005280 2024-04-17T19:48:18.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f9ca0a00640 1 -- 172.21.15.29:0/2399162599 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c900a9070 con 0x7f9c88005280 2024-04-17T19:48:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f9ca0a00640 1 -- 172.21.15.29:0/2399162599 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90006bf0 con 0x7f9c88005280 2024-04-17T19:48:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f8e55600640 1 -- 172.21.15.29:0/1866291538 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e480b4070 con 0x7f8e580ca4e0 2024-04-17T19:48:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f8e55600640 1 -- 172.21.15.29:0/1866291538 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e480af400 con 0x7f8e580ca4e0 2024-04-17T19:48:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f8e55600640 1 -- 172.21.15.29:0/1866291538 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e480af7d0 con 0x7f8e580ca4e0 2024-04-17T19:48:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f9ca8c00640 1 -- 172.21.15.29:0/2399162599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c88005280 msgr2=0x7f9c8801c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f9ca8c00640 1 --2- 172.21.15.29:0/2399162599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c88005280 0x7f9c8801c9e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9c90006fa0 tx=0x7f9c9001cc10 comp rx=0 tx=0).stop 2024-04-17T19:48:18.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f5a0f400640 1 -- 172.21.15.29:0/1569617840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a081521e0 msgr2=0x7f5a080c2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f5a0f400640 1 --2- 172.21.15.29:0/1569617840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a081521e0 0x7f5a080c2460 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5a0806e2c0 tx=0x7f59f401c300 comp rx=0 tx=0).stop 2024-04-17T19:48:18.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f1a49e00640 1 Processor -- start 2024-04-17T19:48:18.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f2096000640 1 -- 172.21.15.29:0/427262431 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2098056c10 con 0x7f20a00c8d90 2024-04-17T19:48:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f5e4a400640 1 Processor -- start 2024-04-17T19:48:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f9ca8c00640 1 -- 172.21.15.29:0/2399162599 shutdown_connections 2024-04-17T19:48:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f9ca8c00640 1 --2- 172.21.15.29:0/2399162599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c88005280 0x7f9c8801c9e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f9ca8c00640 1 -- 172.21.15.29:0/2399162599 >> 172.21.15.29:0/2399162599 conn(0x7f9ca406c950 msgr2=0x7f9ca406f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:18.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f5a0f400640 1 -- 172.21.15.29:0/1569617840 shutdown_connections 2024-04-17T19:48:18.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f5a0f400640 1 --2- 172.21.15.29:0/1569617840 >> v2:172.21.15.37:3300/0 conn(0x7f5a08158bf0 0x7f5a080c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f5a0f400640 1 --2- 172.21.15.29:0/1569617840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a081521e0 0x7f5a080c2460 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f5a0f400640 1 -- 172.21.15.29:0/1569617840 >> 172.21.15.29:0/1569617840 conn(0x7f5a0806c950 msgr2=0x7f5a0806f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:18.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f5a0f400640 1 -- 172.21.15.29:0/1569617840 shutdown_connections 2024-04-17T19:48:18.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f1a49e00640 1 -- start start 2024-04-17T19:48:18.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f5a0f400640 1 -- 172.21.15.29:0/1569617840 wait complete. 2024-04-17T19:48:18.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f8e5d600640 1 -- 172.21.15.29:0/1866291538 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e580ca4e0 msgr2=0x7f8e58156e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f8e5d600640 1 --2- 172.21.15.29:0/1866291538 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e580ca4e0 0x7f8e58156e70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8e48009e50 tx=0x7f8e4801f230 comp rx=0 tx=0).stop 2024-04-17T19:48:18.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f5e4a400640 1 -- start start 2024-04-17T19:48:18.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f8e5d600640 1 -- 172.21.15.29:0/1866291538 shutdown_connections 2024-04-17T19:48:18.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f8e5d600640 1 --2- 172.21.15.29:0/1866291538 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e580ca4e0 0x7f8e58156e70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f8e5d600640 1 --2- 172.21.15.29:0/1866291538 >> v2:172.21.15.29:3300/0 conn(0x7f8e580ca0d0 0x7f8e58156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f8e5d600640 1 -- 172.21.15.29:0/1866291538 >> 172.21.15.29:0/1866291538 conn(0x7f8e5806bf90 msgr2=0x7f8e5806c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:18.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f8e5d600640 1 -- 172.21.15.29:0/1866291538 shutdown_connections 2024-04-17T19:48:18.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.077+0000 7f8e5d600640 1 -- 172.21.15.29:0/1866291538 wait complete. 2024-04-17T19:48:18.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f5a0f400640 1 Processor -- start 2024-04-17T19:48:18.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f5a0f400640 1 -- start start 2024-04-17T19:48:18.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f1a49e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f1a4406c910 0x7f1a4406cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f1a49e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1a44164040 0x7f1a440ce560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f1a49e00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a44160380 con 0x7f1a440ce150 2024-04-17T19:48:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f1a49e00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f1a44160660 con 0x7f1a44164040 2024-04-17T19:48:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f1a49e00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f1a44160940 con 0x7f1a4406c910 2024-04-17T19:48:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f1a48a00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f1a4406c910 0x7f1a4406cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f1a48a00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f1a4406c910 0x7f1a4406cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46898/0 (socket says 172.21.15.29:46898) 2024-04-17T19:48:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f1a48a00640 1 -- 172.21.15.29:0/1935052866 learned_addr learned my addr 172.21.15.29:0/1935052866 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f1a49400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1a44164040 0x7f1a440ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f5a0f400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5a080c99a0 0x7f5a081514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f5a0f400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a081521e0 0x7f5a081f7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f5a0f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a08158bf0 0x7f5a08140110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f5a0f400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0815b4a0 con 0x7f5a08158bf0 2024-04-17T19:48:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f5a0f400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0815c0f0 con 0x7f5a081521e0 2024-04-17T19:48:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f5a0f400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5a08141600 con 0x7f5a080c99a0 2024-04-17T19:48:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f5a07e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a081521e0 0x7f5a081f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f5a07e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a081521e0 0x7f5a081f7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46908/0 (socket says 172.21.15.29:46908) 2024-04-17T19:48:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f5a07e00640 1 -- 172.21.15.29:0/176059016 learned_addr learned my addr 172.21.15.29:0/176059016 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f5a0d400640 1 --2- 172.21.15.29:0/176059016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a08158bf0 0x7f5a08140110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.081+0000 7f1a43400640 1 -- 172.21.15.29:0/1935052866 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 607155938 0 0) 0x7f1a44160380 con 0x7f1a440ce150 2024-04-17T19:48:18.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f9ca8c00640 1 -- 172.21.15.29:0/2399162599 shutdown_connections 2024-04-17T19:48:18.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f1a43400640 1 -- 172.21.15.29:0/1935052866 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a28003530 con 0x7f1a440ce150 2024-04-17T19:48:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5a0ca00640 1 --2- 172.21.15.29:0/176059016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5a080c99a0 0x7f5a081514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5a07e00640 1 -- 172.21.15.29:0/176059016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5a080c99a0 msgr2=0x7f5a081514d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5a07e00640 1 --2- 172.21.15.29:0/176059016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5a080c99a0 0x7f5a081514d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5a07e00640 1 -- 172.21.15.29:0/176059016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a08158bf0 msgr2=0x7f5a08140110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5a07e00640 1 --2- 172.21.15.29:0/176059016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a08158bf0 0x7f5a08140110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5a07e00640 1 -- 172.21.15.29:0/176059016 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0814f8e0 con 0x7f5a081521e0 2024-04-17T19:48:18.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5a0ca00640 1 --2- 172.21.15.29:0/176059016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5a080c99a0 0x7f5a081514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T19:48:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5e4a400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e440cce60 0x7f5e44143360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5e4a400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e441588e0 0x7f5e4414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5e4a400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e4415bf20 0x7f5e441427a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5e4a400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5e440c8820 con 0x7f5e441588e0 2024-04-17T19:48:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5e4a400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5e44158cf0 con 0x7f5e440cce60 2024-04-17T19:48:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5e4a400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5e44165030 con 0x7f5e4415bf20 2024-04-17T19:48:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5e49000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e440cce60 0x7f5e44143360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5e49000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e440cce60 0x7f5e44143360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46910/0 (socket says 172.21.15.29:46910) 2024-04-17T19:48:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5e49000640 1 -- 172.21.15.29:0/2529910808 learned_addr learned my addr 172.21.15.29:0/2529910808 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:18.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f8e5d600640 1 Processor -- start 2024-04-17T19:48:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5e49a00640 1 --2- 172.21.15.29:0/2529910808 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e4415bf20 0x7f5e441427a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5e43e00640 1 --2- 172.21.15.29:0/2529910808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e441588e0 0x7f5e4414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f8e5d600640 1 -- start start 2024-04-17T19:48:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5e43e00640 1 -- 172.21.15.29:0/2529910808 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e4415bf20 msgr2=0x7f5e441427a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5e43e00640 1 --2- 172.21.15.29:0/2529910808 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e4415bf20 0x7f5e441427a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5e43e00640 1 -- 172.21.15.29:0/2529910808 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e440cce60 msgr2=0x7f5e44143360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f1a48a00640 1 -- 172.21.15.29:0/1935052866 >> v2:172.21.15.29:3300/0 conn(0x7f1a44164040 msgr2=0x7f1a440ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f1a48a00640 1 --2- 172.21.15.29:0/1935052866 >> v2:172.21.15.29:3300/0 conn(0x7f1a44164040 0x7f1a440ce560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f1a48a00640 1 -- 172.21.15.29:0/1935052866 >> v1:172.21.15.37:6789/0 conn(0x7f1a440ce150 legacy=0x7f1a44163930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f1a48a00640 1 -- 172.21.15.29:0/1935052866 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a44160c20 con 0x7f1a4406c910 2024-04-17T19:48:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5e43e00640 1 --2- 172.21.15.29:0/2529910808 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e440cce60 0x7f5e44143360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5e43e00640 1 -- 172.21.15.29:0/2529910808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e441593d0 con 0x7f5e441588e0 2024-04-17T19:48:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f5e43e00640 1 --2- 172.21.15.29:0/2529910808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e441588e0 0x7f5e4414a6d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5e4415b650 tx=0x7f5e300018a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f8e5d600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e580c86b0 0x7f8e581f6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f8e5d600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e580ca0d0 0x7f8e58153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f8e5d600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e580ca4e0 0x7f8e58152270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f8e5d600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8e58164290 con 0x7f8e580ca4e0 2024-04-17T19:48:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f8e5d600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8e58163fb0 con 0x7f8e580ca0d0 2024-04-17T19:48:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f8e5d600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8e581612d0 con 0x7f8e580c86b0 2024-04-17T19:48:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.085+0000 7f9ca8c00640 1 -- 172.21.15.29:0/2399162599 wait complete. 2024-04-17T19:48:18.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f5e40c00640 1 -- 172.21.15.29:0/2529910808 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30014020 con 0x7f5e441588e0 2024-04-17T19:48:18.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f5e40c00640 1 -- 172.21.15.29:0/2529910808 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e3000f850 con 0x7f5e441588e0 2024-04-17T19:48:18.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f5e40c00640 1 -- 172.21.15.29:0/2529910808 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e30007050 con 0x7f5e441588e0 2024-04-17T19:48:18.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f5a07e00640 1 --2- 172.21.15.29:0/176059016 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a081521e0 0x7f5a081f7070 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5a0000ba50 tx=0x7f5a0000be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f5a04c00640 1 -- 172.21.15.29:0/176059016 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00019070 con 0x7f5a081521e0 2024-04-17T19:48:18.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f5a04c00640 1 -- 172.21.15.29:0/176059016 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a0000cb50 con 0x7f5a081521e0 2024-04-17T19:48:18.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f5a04c00640 1 -- 172.21.15.29:0/176059016 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00014440 con 0x7f5a081521e0 2024-04-17T19:48:18.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f5a0f400640 1 -- 172.21.15.29:0/176059016 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a08151a10 con 0x7f5a081521e0 2024-04-17T19:48:18.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f5a0f400640 1 -- 172.21.15.29:0/176059016 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a081f7b80 con 0x7f5a081521e0 2024-04-17T19:48:18.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f8e56000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e580ca0d0 0x7f8e58153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f8e56000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e580ca0d0 0x7f8e58153260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46926/0 (socket says 172.21.15.29:46926) 2024-04-17T19:48:18.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f8e56000640 1 -- 172.21.15.29:0/1241881343 learned_addr learned my addr 172.21.15.29:0/1241881343 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:18.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f9ca8c00640 1 Processor -- start 2024-04-17T19:48:18.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f9ca8c00640 1 -- start start 2024-04-17T19:48:18.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f5a0f400640 1 -- 172.21.15.29:0/176059016 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59e00052a0 con 0x7f5a081521e0 2024-04-17T19:48:18.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.089+0000 7f8e56a00640 1 --2- 172.21.15.29:0/1241881343 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e580c86b0 0x7f8e581f6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f8e57400640 1 --2- 172.21.15.29:0/1241881343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e580ca4e0 0x7f8e58152270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f8e56a00640 1 -- 172.21.15.29:0/1241881343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e580ca0d0 msgr2=0x7f8e58153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f8e56a00640 1 --2- 172.21.15.29:0/1241881343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e580ca0d0 0x7f8e58153260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f5e4a400640 1 -- 172.21.15.29:0/2529910808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e44145ce0 con 0x7f5e441588e0 2024-04-17T19:48:18.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f8e56a00640 1 -- 172.21.15.29:0/1241881343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e580ca4e0 msgr2=0x7f8e58152270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f8e56a00640 1 --2- 172.21.15.29:0/1241881343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e580ca4e0 0x7f8e58152270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f9ca8c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c88005280 0x7f9ca41515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f5e4a400640 1 -- 172.21.15.29:0/2529910808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e441f1380 con 0x7f5e441588e0 2024-04-17T19:48:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f9ca8c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ca415ade0 0x7f9ca4159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f9ca8c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ca414d8c0 0x7f9ca414f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f9ca8c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca415b4a0 con 0x7f9ca415ade0 2024-04-17T19:48:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f9ca2800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ca414d8c0 0x7f9ca414f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f9ca1e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c88005280 0x7f9ca41515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f9ca1e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c88005280 0x7f9ca41515b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46934/0 (socket says 172.21.15.29:46934) 2024-04-17T19:48:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f9ca1e00640 1 -- 172.21.15.29:0/2371804773 learned_addr learned my addr 172.21.15.29:0/2371804773 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f9ca8c00640 1 -- 172.21.15.29:0/2371804773 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca40c2870 con 0x7f9c88005280 2024-04-17T19:48:18.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f9ca8c00640 1 -- 172.21.15.29:0/2371804773 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9ca415b620 con 0x7f9ca414d8c0 2024-04-17T19:48:18.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f5a04c00640 1 -- 172.21.15.29:0/176059016 <== mon.1 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5a00007370 con 0x7f5a081521e0 2024-04-17T19:48:18.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f5a04c00640 1 --2- 172.21.15.29:0/176059016 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f59e8068400 0x7f59e806a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f0012000640 1 Processor -- start 2024-04-17T19:48:18.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f1a48a00640 1 --2- 172.21.15.29:0/1935052866 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a4406c910 0x7f1a4406cd20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1a44071630 tx=0x7f1a3001ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df37fc732c618958 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f8e56a00640 1 -- 172.21.15.29:0/1241881343 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e580c5a20 con 0x7f8e580c86b0 2024-04-17T19:48:18.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f8e57400640 1 --2- 172.21.15.29:0/1241881343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e580ca4e0 0x7f8e58152270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:48:18.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f8e56000640 1 --2- 172.21.15.29:0/1241881343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e580ca0d0 0x7f8e58153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:48:18.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f9ca1400640 1 --2- 172.21.15.29:0/2371804773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ca415ade0 0x7f9ca4159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f5a04c00640 1 -- 172.21.15.29:0/176059016 <== mon.1 v2:172.21.15.37:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5a00054510 con 0x7f5a081521e0 2024-04-17T19:48:18.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f5a0ca00640 1 --2- 172.21.15.29:0/176059016 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f59e8068400 0x7f59e806a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f9ca1e00640 1 -- 172.21.15.29:0/2371804773 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ca414d8c0 msgr2=0x7f9ca414f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f9ca1e00640 1 --2- 172.21.15.29:0/2371804773 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ca414d8c0 0x7f9ca414f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f9ca1e00640 1 -- 172.21.15.29:0/2371804773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ca415ade0 msgr2=0x7f9ca4159170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f9ca1e00640 1 --2- 172.21.15.29:0/2371804773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ca415ade0 0x7f9ca4159170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f9ca1e00640 1 -- 172.21.15.29:0/2371804773 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca41ef440 con 0x7f9c88005280 2024-04-17T19:48:18.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f5e4a400640 1 -- 172.21.15.29:0/2529910808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e4404ff10 con 0x7f5e441588e0 2024-04-17T19:48:18.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f0012000640 1 -- start start 2024-04-17T19:48:18.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f1a43400640 1 -- 172.21.15.29:0/1935052866 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a300b5070 con 0x7f1a4406c910 2024-04-17T19:48:18.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.093+0000 7f1a43400640 1 -- 172.21.15.29:0/1935052866 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a300a8a10 con 0x7f1a4406c910 2024-04-17T19:48:18.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f8e56a00640 1 --2- 172.21.15.29:0/1241881343 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e580c86b0 0x7f8e581f6880 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000c3f0 tx=0x7f8e5000c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f9ca1e00640 1 --2- 172.21.15.29:0/2371804773 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c88005280 0x7f9ca41515b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c031980 tx=0x7f9c9c031e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f1a43400640 1 -- 172.21.15.29:0/1935052866 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a300a8de0 con 0x7f1a4406c910 2024-04-17T19:48:18.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f8e4ea00640 1 -- 172.21.15.29:0/1241881343 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e50017070 con 0x7f8e580c86b0 2024-04-17T19:48:18.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f8e4ea00640 1 -- 172.21.15.29:0/1241881343 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e500052d0 con 0x7f8e580c86b0 2024-04-17T19:48:18.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f1a49e00640 1 -- 172.21.15.29:0/1935052866 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a4406c910 msgr2=0x7f1a4406cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f1a49e00640 1 --2- 172.21.15.29:0/1935052866 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a4406c910 0x7f1a4406cd20 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1a44071630 tx=0x7f1a3001ed50 comp rx=0 tx=0).stop 2024-04-17T19:48:18.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f1a49e00640 1 -- 172.21.15.29:0/1935052866 shutdown_connections 2024-04-17T19:48:18.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f1a49e00640 1 --2- 172.21.15.29:0/1935052866 >> v2:172.21.15.29:3300/0 conn(0x7f1a44164040 0x7f1a440ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f1a49e00640 1 --2- 172.21.15.29:0/1935052866 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a4406c910 0x7f1a4406cd20 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1a44071630 tx=0x7f1a3001ed50 comp rx=0 tx=0).stop 2024-04-17T19:48:18.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f1a49e00640 1 -- 172.21.15.29:0/1935052866 >> 172.21.15.29:0/1935052866 conn(0x7f1a44070880 msgr2=0x7f1a44070c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:18.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f8e4ea00640 1 -- 172.21.15.29:0/1241881343 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e50003860 con 0x7f8e580c86b0 2024-04-17T19:48:18.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f5a0ca00640 1 --2- 172.21.15.29:0/176059016 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f59e8068400 0x7f59e806a910 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f59f40ad000 tx=0x7f59f40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f8e5d600640 1 -- 172.21.15.29:0/1241881343 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e58160910 con 0x7f8e580c86b0 2024-04-17T19:48:18.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f8e5d600640 1 -- 172.21.15.29:0/1241881343 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e581fa070 con 0x7f8e580c86b0 2024-04-17T19:48:18.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f5e40c00640 1 -- 172.21.15.29:0/2529910808 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5e30017020 con 0x7f5e441588e0 2024-04-17T19:48:18.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f1a49e00640 1 -- 172.21.15.29:0/1935052866 shutdown_connections 2024-04-17T19:48:18.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f1a49e00640 1 -- 172.21.15.29:0/1935052866 wait complete. 2024-04-17T19:48:18.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f5e40c00640 1 --2- 172.21.15.29:0/2529910808 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5e240683b0 0x7f5e2406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f0012000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f000c0b4800 0x7f000c0cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f5e49000640 1 --2- 172.21.15.29:0/2529910808 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5e240683b0 0x7f5e2406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f0012000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f000c160970 0x7f000c16ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f0012000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f000c163b70 con 0x7f000c160560 2024-04-17T19:48:18.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f0012000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f000c164460 con 0x7f000c160970 2024-04-17T19:48:18.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f0012000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f000c0b6530 con 0x7f000c0b4800 2024-04-17T19:48:18.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.097+0000 7f5e40c00640 1 -- 172.21.15.29:0/2529910808 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5e300872c0 con 0x7f5e441588e0 2024-04-17T19:48:18.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.101+0000 7f000be00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f000c160970 0x7f000c16ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.101+0000 7f000be00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f000c160970 0x7f000c16ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52928/0 (socket says 172.21.15.29:52928) 2024-04-17T19:48:18.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.101+0000 7f000be00640 1 -- 172.21.15.29:0/4128135065 learned_addr learned my addr 172.21.15.29:0/4128135065 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:18.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.101+0000 7f000b400640 1 --2- 172.21.15.29:0/4128135065 >> v2:172.21.15.37:3300/0 conn(0x7f000c0b4800 0x7f000c0cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.101+0000 7f5e49000640 1 --2- 172.21.15.29:0/2529910808 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5e240683b0 0x7f5e2406a8c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5e3400ca80 tx=0x7f5e34008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.101+0000 7f8e5d600640 1 -- 172.21.15.29:0/1241881343 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e58069b90 con 0x7f8e580c86b0 2024-04-17T19:48:18.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.101+0000 7f1a49e00640 1 Processor -- start 2024-04-17T19:48:18.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.101+0000 7f8e4ea00640 1 -- 172.21.15.29:0/1241881343 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8e5001a070 con 0x7f8e580c86b0 2024-04-17T19:48:18.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.101+0000 7f000b400640 1 -- 172.21.15.29:0/4128135065 >> v2:172.21.15.29:3300/0 conn(0x7f000c160970 msgr2=0x7f000c16ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:18.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.101+0000 7f000b400640 1 --2- 172.21.15.29:0/4128135065 >> v2:172.21.15.29:3300/0 conn(0x7f000c160970 0x7f000c16ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.101+0000 7f000b400640 1 -- 172.21.15.29:0/4128135065 >> v1:172.21.15.37:6789/0 conn(0x7f000c160560 legacy=0x7f000c06bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:18.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.101+0000 7f000b400640 1 -- 172.21.15.29:0/4128135065 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f000c163e10 con 0x7f000c0b4800 2024-04-17T19:48:18.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.101+0000 7f1a49e00640 1 -- start start 2024-04-17T19:48:18.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f8e4ea00640 1 --2- 172.21.15.29:0/1241881343 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8e38068480 0x7f8e3806a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f8e4ea00640 1 -- 172.21.15.29:0/1241881343 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f8e50085f60 con 0x7f8e580c86b0 2024-04-17T19:48:18.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f8e56000640 1 --2- 172.21.15.29:0/1241881343 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8e38068480 0x7f8e3806a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f000b400640 1 --2- 172.21.15.29:0/4128135065 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f000c0b4800 0x7f000c0cd3d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f000c0b60d0 tx=0x7efff801ec10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c39197b008b841f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f1a49e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a440ce150 0x7f1a4415b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f1a49e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a44164040 0x7f1a441f9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f1a49e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1a441f7110 0x7f1a440cc400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f1a49e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1a440c9300 con 0x7f1a44164040 2024-04-17T19:48:18.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f1a49e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1a44073220 con 0x7f1a440ce150 2024-04-17T19:48:18.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f1a49e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1a441faa00 con 0x7f1a441f7110 2024-04-17T19:48:18.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f1a43e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a44164040 0x7f1a441f9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f1a43e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a44164040 0x7f1a441f9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52934/0 (socket says 172.21.15.29:52934) 2024-04-17T19:48:18.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f1a43e00640 1 -- 172.21.15.29:0/2124108723 learned_addr learned my addr 172.21.15.29:0/2124108723 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:18.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f1a49400640 1 --2- 172.21.15.29:0/2124108723 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1a441f7110 0x7f1a440cc400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f1a43e00640 1 -- 172.21.15.29:0/2124108723 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1a441f7110 msgr2=0x7f1a440cc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f1a43e00640 1 --2- 172.21.15.29:0/2124108723 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1a441f7110 0x7f1a440cc400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f1a43e00640 1 -- 172.21.15.29:0/2124108723 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a440ce150 msgr2=0x7f1a4415b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f1a43e00640 1 --2- 172.21.15.29:0/2124108723 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a440ce150 0x7f1a4415b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.105+0000 7f1a43e00640 1 -- 172.21.15.29:0/2124108723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a441fa530 con 0x7f1a44164040 2024-04-17T19:48:18.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.109+0000 7f1a43e00640 1 --2- 172.21.15.29:0/2124108723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a44164040 0x7f1a441f9ff0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1a3400c3f0 tx=0x7f1a3400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.109+0000 7f8e56000640 1 --2- 172.21.15.29:0/1241881343 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8e38068480 0x7f8e3806a990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8e581eab40 tx=0x7f8e44009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.109+0000 7f1a40c00640 1 -- 172.21.15.29:0/2124108723 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a34017070 con 0x7f1a44164040 2024-04-17T19:48:18.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.109+0000 7f5e40c00640 1 -- 172.21.15.29:0/2529910808 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5e3001e020 con 0x7f5e441588e0 2024-04-17T19:48:18.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.109+0000 7f1a40c00640 1 -- 172.21.15.29:0/2124108723 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a340052d0 con 0x7f1a44164040 2024-04-17T19:48:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.109+0000 7f1a40c00640 1 -- 172.21.15.29:0/2124108723 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a34003840 con 0x7f1a44164040 2024-04-17T19:48:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.109+0000 7f1a49e00640 1 -- 172.21.15.29:0/2124108723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a440c4460 con 0x7f1a44164040 2024-04-17T19:48:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.109+0000 7f1a49e00640 1 -- 172.21.15.29:0/2124108723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a440c4a20 con 0x7f1a44164040 2024-04-17T19:48:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.109+0000 7f1a49e00640 1 -- 172.21.15.29:0/2124108723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a4404ff10 con 0x7f1a44164040 2024-04-17T19:48:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.109+0000 7f9c9a000640 1 -- 172.21.15.29:0/2371804773 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c003b90 con 0x7f9c88005280 2024-04-17T19:48:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.109+0000 7f000a000640 1 -- 172.21.15.29:0/4128135065 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff80b5070 con 0x7f000c0b4800 2024-04-17T19:48:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.109+0000 7f000a000640 1 -- 172.21.15.29:0/4128135065 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efff80a88f0 con 0x7f000c0b4800 2024-04-17T19:48:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.109+0000 7f000a000640 1 -- 172.21.15.29:0/4128135065 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff80a8cc0 con 0x7f000c0b4800 2024-04-17T19:48:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.109+0000 7f9c9a000640 1 -- 172.21.15.29:0/2371804773 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c9c03a070 con 0x7f9c88005280 2024-04-17T19:48:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.109+0000 7f9c9a000640 1 -- 172.21.15.29:0/2371804773 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c9c02f7f0 con 0x7f9c88005280 2024-04-17T19:48:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.113+0000 7f9ca8c00640 1 -- 172.21.15.29:0/2371804773 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca41ef720 con 0x7f9c88005280 2024-04-17T19:48:18.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.109+0000 7f5a04c00640 1 -- 172.21.15.29:0/176059016 <== mon.1 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5a00056b70 con 0x7f5a081521e0 2024-04-17T19:48:18.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.113+0000 7f9ca8c00640 1 -- 172.21.15.29:0/2371804773 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca41efce0 con 0x7f9c88005280 2024-04-17T19:48:18.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.113+0000 7f1a40c00640 1 -- 172.21.15.29:0/2124108723 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1a3401a070 con 0x7f1a44164040 2024-04-17T19:48:18.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.113+0000 7f9ca8c00640 1 -- 172.21.15.29:0/2371804773 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ca4074b50 con 0x7f9c88005280 2024-04-17T19:48:18.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.113+0000 7f0012000640 1 -- 172.21.15.29:0/4128135065 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f000c0b4800 msgr2=0x7f000c0cd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.113+0000 7f0012000640 1 --2- 172.21.15.29:0/4128135065 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f000c0b4800 0x7f000c0cd3d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f000c0b60d0 tx=0x7efff801ec10 comp rx=0 tx=0).stop 2024-04-17T19:48:18.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.113+0000 7f0012000640 1 -- 172.21.15.29:0/4128135065 shutdown_connections 2024-04-17T19:48:18.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.113+0000 7f0012000640 1 --2- 172.21.15.29:0/4128135065 >> v2:172.21.15.29:3300/0 conn(0x7f000c160970 0x7f000c16ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.113+0000 7f0012000640 1 --2- 172.21.15.29:0/4128135065 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f000c0b4800 0x7f000c0cd3d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.113+0000 7f0012000640 1 -- 172.21.15.29:0/4128135065 >> 172.21.15.29:0/4128135065 conn(0x7f000c0d1440 msgr2=0x7f000c0d1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:18.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.117+0000 7f1a40c00640 1 --2- 172.21.15.29:0/2124108723 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1a24068380 0x7f1a2406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.117+0000 7f1a40c00640 1 -- 172.21.15.29:0/2124108723 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f1a34085fa0 con 0x7f1a44164040 2024-04-17T19:48:18.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.117+0000 7f0012000640 1 -- 172.21.15.29:0/4128135065 shutdown_connections 2024-04-17T19:48:18.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.117+0000 7f0012000640 1 -- 172.21.15.29:0/4128135065 wait complete. 2024-04-17T19:48:18.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.117+0000 7f1a48a00640 1 --2- 172.21.15.29:0/2124108723 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1a24068380 0x7f1a2406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.121+0000 7f0012000640 1 Processor -- start 2024-04-17T19:48:18.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.121+0000 7f9c9a000640 1 -- 172.21.15.29:0/2371804773 <== mon.1 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9c9c02f990 con 0x7f9c88005280 2024-04-17T19:48:18.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.121+0000 7f9c9a000640 1 --2- 172.21.15.29:0/2371804773 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9c84068380 0x7f9c8406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.121+0000 7f9c9a000640 1 -- 172.21.15.29:0/2371804773 <== mon.1 v2:172.21.15.37:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9c9c0a9990 con 0x7f9c88005280 2024-04-17T19:48:18.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.121+0000 7f8e4ea00640 1 -- 172.21.15.29:0/1241881343 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8e50005480 con 0x7f8e580c86b0 2024-04-17T19:48:18.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.121+0000 7f1a48a00640 1 --2- 172.21.15.29:0/2124108723 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1a24068380 0x7f1a2406a890 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1a300b2000 tx=0x7f1a30004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.121+0000 7f9ca1400640 1 --2- 172.21.15.29:0/2371804773 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9c84068380 0x7f9c8406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.125+0000 7f0012000640 1 -- start start 2024-04-17T19:48:18.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.125+0000 7f9ca1400640 1 --2- 172.21.15.29:0/2371804773 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9c84068380 0x7f9c8406a890 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4140b10 tx=0x7f9c94008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.125+0000 7f1a40c00640 1 -- 172.21.15.29:0/2124108723 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1a34005480 con 0x7f1a44164040 2024-04-17T19:48:18.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.125+0000 7f0012000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f000c0b4800 0x7f000c0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.125+0000 7f0012000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f000c160560 0x7f000c0ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.125+0000 7f0012000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f000c160970 0x7f000c0c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.125+0000 7f000aa00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f000c160560 0x7f000c0ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.125+0000 7f000aa00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f000c160560 0x7f000c0ca2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36580/0 (socket says 172.21.15.29:36580) 2024-04-17T19:48:18.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.125+0000 7f000aa00640 1 -- 172.21.15.29:0/1164539349 learned_addr learned my addr 172.21.15.29:0/1164539349 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:18.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.125+0000 7f000be00640 1 --2- 172.21.15.29:0/1164539349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f000c160970 0x7f000c0c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.129+0000 7f0012000640 1 -- 172.21.15.29:0/1164539349 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f000c0b6810 con 0x7f000c160970 2024-04-17T19:48:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.129+0000 7f0012000640 1 -- 172.21.15.29:0/1164539349 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f000c0b4c10 con 0x7f000c0b4800 2024-04-17T19:48:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.129+0000 7f0012000640 1 -- 172.21.15.29:0/1164539349 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f000c0cd910 con 0x7f000c160560 2024-04-17T19:48:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.129+0000 7f000b400640 1 --2- 172.21.15.29:0/1164539349 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f000c0b4800 0x7f000c0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.129+0000 7f000b400640 1 -- 172.21.15.29:0/1164539349 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f000c160560 msgr2=0x7f000c0ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.129+0000 7f000b400640 1 --2- 172.21.15.29:0/1164539349 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f000c160560 0x7f000c0ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.129+0000 7f000b400640 1 -- 172.21.15.29:0/1164539349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f000c160970 msgr2=0x7f000c0c8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.129+0000 7f000b400640 1 --2- 172.21.15.29:0/1164539349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f000c160970 0x7f000c0c8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.129+0000 7f000b400640 1 -- 172.21.15.29:0/1164539349 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f000c073500 con 0x7f000c0b4800 2024-04-17T19:48:18.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.129+0000 7f9c9a000640 1 -- 172.21.15.29:0/2371804773 <== mon.1 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9c9c079bf0 con 0x7f9c88005280 2024-04-17T19:48:18.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.133+0000 7f000b400640 1 --2- 172.21.15.29:0/1164539349 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f000c0b4800 0x7f000c0c9270 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7efff80af000 tx=0x7efff801b9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.133+0000 7f0003400640 1 -- 172.21.15.29:0/1164539349 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff80b5040 con 0x7f000c0b4800 2024-04-17T19:48:18.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.137+0000 7f0003400640 1 -- 172.21.15.29:0/1164539349 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efff80b0440 con 0x7f000c0b4800 2024-04-17T19:48:18.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.137+0000 7f0003400640 1 -- 172.21.15.29:0/1164539349 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efff80b0810 con 0x7f000c0b4800 2024-04-17T19:48:18.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.137+0000 7f0012000640 1 -- 172.21.15.29:0/1164539349 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f000c15f190 con 0x7f000c0b4800 2024-04-17T19:48:18.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.137+0000 7f0012000640 1 -- 172.21.15.29:0/1164539349 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f000c0c9dd0 con 0x7f000c0b4800 2024-04-17T19:48:18.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.141+0000 7f0003400640 1 -- 172.21.15.29:0/1164539349 <== mon.1 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efff80c2030 con 0x7f000c0b4800 2024-04-17T19:48:18.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.141+0000 7f0003400640 1 --2- 172.21.15.29:0/1164539349 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7effec068380 0x7effec06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.141+0000 7f0003400640 1 -- 172.21.15.29:0/1164539349 <== mon.1 v2:172.21.15.37:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7efff8123640 con 0x7f000c0b4800 2024-04-17T19:48:18.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.141+0000 7f000aa00640 1 --2- 172.21.15.29:0/1164539349 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7effec068380 0x7effec06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.141+0000 7f000aa00640 1 --2- 172.21.15.29:0/1164539349 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7effec068380 0x7effec06a890 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f000c15e830 tx=0x7efffc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.145+0000 7f0012000640 1 -- 172.21.15.29:0/1164539349 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f000c069b90 con 0x7f000c0b4800 2024-04-17T19:48:18.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.157+0000 7f0003400640 1 -- 172.21.15.29:0/1164539349 <== mon.1 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7efff8018770 con 0x7f000c0b4800 2024-04-17T19:48:18.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.161+0000 7efc6de00640 1 Processor -- start 2024-04-17T19:48:18.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.161+0000 7efc6de00640 1 -- start start 2024-04-17T19:48:18.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.165+0000 7efc6de00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7efc680c2460 0x7efc681584e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.165+0000 7efc6de00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efc68158a20 0x7efc68162830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.165+0000 7efc6de00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc680c40d0 con 0x7efc681521e0 2024-04-17T19:48:18.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.165+0000 7efc6de00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7efc68158050 con 0x7efc68158a20 2024-04-17T19:48:18.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.165+0000 7efc66000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7efc680c2460 0x7efc681584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.169+0000 7efc6de00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7efc681525f0 con 0x7efc680c2460 2024-04-17T19:48:18.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.169+0000 7efc66000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7efc680c2460 0x7efc681584e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46966/0 (socket says 172.21.15.29:46966) 2024-04-17T19:48:18.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.169+0000 7efc66000640 1 -- 172.21.15.29:0/280929227 learned_addr learned my addr 172.21.15.29:0/280929227 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:18.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.169+0000 7efc67400640 1 --2- 172.21.15.29:0/280929227 >> v2:172.21.15.29:3300/0 conn(0x7efc68158a20 0x7efc68162830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.169+0000 7efc66000640 1 -- 172.21.15.29:0/280929227 >> v2:172.21.15.29:3300/0 conn(0x7efc68158a20 msgr2=0x7efc68162830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:18.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.169+0000 7efc66000640 1 --2- 172.21.15.29:0/280929227 >> v2:172.21.15.29:3300/0 conn(0x7efc68158a20 0x7efc68162830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.169+0000 7efc66000640 1 -- 172.21.15.29:0/280929227 >> v1:172.21.15.37:6789/0 conn(0x7efc681521e0 legacy=0x7efc680c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:18.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.169+0000 7efc66000640 1 -- 172.21.15.29:0/280929227 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc6815c3d0 con 0x7efc680c2460 2024-04-17T19:48:18.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.181+0000 7efc66000640 1 --2- 172.21.15.29:0/280929227 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc680c2460 0x7efc681584e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efc6806e410 tx=0x7efc6001c7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0ba5515cf6ca04 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.185+0000 7efc65600640 1 -- 172.21.15.29:0/280929227 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc600a5660 con 0x7efc680c2460 2024-04-17T19:48:18.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.185+0000 7efc65600640 1 -- 172.21.15.29:0/280929227 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc600a5800 con 0x7efc680c2460 2024-04-17T19:48:18.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.185+0000 7efc65600640 1 -- 172.21.15.29:0/280929227 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc600a5bd0 con 0x7efc680c2460 2024-04-17T19:48:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.189+0000 7efc6de00640 1 -- 172.21.15.29:0/280929227 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc680c2460 msgr2=0x7efc681584e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.189+0000 7efc6de00640 1 --2- 172.21.15.29:0/280929227 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc680c2460 0x7efc681584e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efc6806e410 tx=0x7efc6001c7a0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.189+0000 7efc6de00640 1 -- 172.21.15.29:0/280929227 shutdown_connections 2024-04-17T19:48:18.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.189+0000 7efc6de00640 1 --2- 172.21.15.29:0/280929227 >> v2:172.21.15.29:3300/0 conn(0x7efc68158a20 0x7efc68162830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.189+0000 7efc6de00640 1 --2- 172.21.15.29:0/280929227 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc680c2460 0x7efc681584e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.189+0000 7efc6de00640 1 -- 172.21.15.29:0/280929227 >> 172.21.15.29:0/280929227 conn(0x7efc6806c950 msgr2=0x7efc6806f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.189+0000 7efc6de00640 1 -- 172.21.15.29:0/280929227 shutdown_connections 2024-04-17T19:48:18.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.189+0000 7efc6de00640 1 -- 172.21.15.29:0/280929227 wait complete. 2024-04-17T19:48:18.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.201+0000 7efc6de00640 1 Processor -- start 2024-04-17T19:48:18.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.201+0000 7efc6de00640 1 -- start start 2024-04-17T19:48:18.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.201+0000 7efc6de00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc680c2460 0x7efc6814c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.201+0000 7efc6de00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc681521e0 0x7efc680c5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.201+0000 7efc6de00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc68158a20 0x7efc68148a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.201+0000 7efc66a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc680c2460 0x7efc6814c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.201+0000 7efc66000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc681521e0 0x7efc680c5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.201+0000 7efc66a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc680c2460 0x7efc6814c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:46968/0 (socket says 172.21.15.29:46968) 2024-04-17T19:48:18.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.201+0000 7efc66000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc681521e0 0x7efc680c5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:36596/0 (socket says 172.21.15.29:36596) 2024-04-17T19:48:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.209+0000 7efc66a00640 1 -- 172.21.15.29:0/539345014 learned_addr learned my addr 172.21.15.29:0/539345014 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.209+0000 7efc6de00640 1 -- 172.21.15.29:0/539345014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efc68158e30 con 0x7efc68158a20 2024-04-17T19:48:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.209+0000 7efc6de00640 1 -- 172.21.15.29:0/539345014 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7efc6815c0f0 con 0x7efc680c2460 2024-04-17T19:48:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.209+0000 7efc6de00640 1 -- 172.21.15.29:0/539345014 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7efc680c2870 con 0x7efc681521e0 2024-04-17T19:48:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.209+0000 7efc67400640 1 --2- 172.21.15.29:0/539345014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc68158a20 0x7efc68148a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.209+0000 7efc67400640 1 -- 172.21.15.29:0/539345014 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc681521e0 msgr2=0x7efc680c5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.209+0000 7efc67400640 1 --2- 172.21.15.29:0/539345014 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc681521e0 0x7efc680c5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.209+0000 7efc67400640 1 -- 172.21.15.29:0/539345014 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc680c2460 msgr2=0x7efc6814c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.209+0000 7efc67400640 1 --2- 172.21.15.29:0/539345014 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc680c2460 0x7efc6814c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.209+0000 7efc67400640 1 -- 172.21.15.29:0/539345014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc680c6090 con 0x7efc68158a20 2024-04-17T19:48:18.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.209+0000 7efc67400640 1 --2- 172.21.15.29:0/539345014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc68158a20 0x7efc68148a90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efc5400e950 tx=0x7efc5400ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.209+0000 7efc5f400640 1 -- 172.21.15.29:0/539345014 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc5400dd40 con 0x7efc68158a20 2024-04-17T19:48:18.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.209+0000 7efc5f400640 1 -- 172.21.15.29:0/539345014 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc54010440 con 0x7efc68158a20 2024-04-17T19:48:18.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.209+0000 7efc5f400640 1 -- 172.21.15.29:0/539345014 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc54010830 con 0x7efc68158a20 2024-04-17T19:48:18.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.209+0000 7efc6de00640 1 -- 172.21.15.29:0/539345014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc6814e390 con 0x7efc68158a20 2024-04-17T19:48:18.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.209+0000 7efc66000640 1 --2- 172.21.15.29:0/539345014 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc681521e0 0x7efc680c5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T19:48:18.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.209+0000 7efc6de00640 1 -- 172.21.15.29:0/539345014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc6814e950 con 0x7efc68158a20 2024-04-17T19:48:18.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.213+0000 7efc66a00640 1 --2- 172.21.15.29:0/539345014 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc680c2460 0x7efc6814c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T19:48:18.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.213+0000 7efc6de00640 1 -- 172.21.15.29:0/539345014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc6814c9d0 con 0x7efc68158a20 2024-04-17T19:48:18.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.213+0000 7efc5f400640 1 -- 172.21.15.29:0/539345014 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efc540109d0 con 0x7efc68158a20 2024-04-17T19:48:18.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.217+0000 7efc5f400640 1 --2- 172.21.15.29:0/539345014 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efc480683b0 0x7efc4806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.217+0000 7efc66a00640 1 --2- 172.21.15.29:0/539345014 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efc480683b0 0x7efc4806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.217+0000 7efc66a00640 1 --2- 172.21.15.29:0/539345014 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efc480683b0 0x7efc4806a8c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efc58006fd0 tx=0x7efc58008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.207 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:48:18.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.225+0000 7efc5f400640 1 -- 172.21.15.29:0/539345014 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7efc54085740 con 0x7efc68158a20 2024-04-17T19:48:18.215 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:48:18.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.241+0000 7efc5f400640 1 -- 172.21.15.29:0/539345014 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7efc54057630 con 0x7efc68158a20 2024-04-17T19:48:18.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.429+0000 7f208f400640 1 -- 172.21.15.29:0/427262431 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f20a00c94d0 con 0x7f20a00c8d90 2024-04-17T19:48:18.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.433+0000 7f2096000640 1 -- 172.21.15.29:0/427262431 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2098056510 con 0x7f20a00c8d90 2024-04-17T19:48:18.420 INFO:teuthology.orchestra.run.smithi029.stdout:55834574859 2024-04-17T19:48:18.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.437+0000 7f208f400640 1 -- 172.21.15.29:0/427262431 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f207c068400 msgr2=0x7f207c06a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.437+0000 7f208f400640 1 --2- 172.21.15.29:0/427262431 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f207c068400 0x7f207c06a910 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f20880b2000 tx=0x7f20880aa070 comp rx=0 tx=0).stop 2024-04-17T19:48:18.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.437+0000 7f208f400640 1 -- 172.21.15.29:0/427262431 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20a00c8d90 msgr2=0x7f20a01505f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.437+0000 7f208f400640 1 --2- 172.21.15.29:0/427262431 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20a00c8d90 0x7f20a01505f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f209800ba50 tx=0x7f209800bf10 comp rx=0 tx=0).stop 2024-04-17T19:48:18.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.437+0000 7f208f400640 1 -- 172.21.15.29:0/427262431 shutdown_connections 2024-04-17T19:48:18.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.437+0000 7f208f400640 1 --2- 172.21.15.29:0/427262431 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f207c068400 0x7f207c06a910 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.437+0000 7f208f400640 1 --2- 172.21.15.29:0/427262431 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20a01562b0 0x7f20a01f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.437+0000 7f208f400640 1 --2- 172.21.15.29:0/427262431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20a00cce60 0x7f20a02013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.437+0000 7f208f400640 1 --2- 172.21.15.29:0/427262431 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20a00c8d90 0x7f20a01505f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.437+0000 7f208f400640 1 -- 172.21.15.29:0/427262431 >> 172.21.15.29:0/427262431 conn(0x7f20a006ddd0 msgr2=0x7f20a00cc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:18.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.441+0000 7f208f400640 1 -- 172.21.15.29:0/427262431 shutdown_connections 2024-04-17T19:48:18.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.441+0000 7f208f400640 1 -- 172.21.15.29:0/427262431 wait complete. 2024-04-17T19:48:18.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.469+0000 7f5a0f400640 1 -- 172.21.15.29:0/176059016 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f59e0005860 con 0x7f5a081521e0 2024-04-17T19:48:18.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.469+0000 7f8e5d600640 1 -- 172.21.15.29:0/1241881343 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f8e58075950 con 0x7f8e580c86b0 2024-04-17T19:48:18.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.473+0000 7f8e4ea00640 1 -- 172.21.15.29:0/1241881343 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8e50055b70 con 0x7f8e580c86b0 2024-04-17T19:48:18.460 INFO:teuthology.orchestra.run.smithi029.stdout:55834574859 2024-04-17T19:48:18.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.473+0000 7f5a04c00640 1 -- 172.21.15.29:0/176059016 <== mon.1 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5a00056470 con 0x7f5a081521e0 2024-04-17T19:48:18.463 INFO:teuthology.orchestra.run.smithi029.stdout:55834574860 2024-04-17T19:48:18.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.477+0000 7f8e5d600640 1 -- 172.21.15.29:0/1241881343 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8e38068480 msgr2=0x7f8e3806a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.477+0000 7f8e5d600640 1 --2- 172.21.15.29:0/1241881343 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8e38068480 0x7f8e3806a990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8e581eab40 tx=0x7f8e44009190 comp rx=0 tx=0).stop 2024-04-17T19:48:18.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.477+0000 7f8e5d600640 1 -- 172.21.15.29:0/1241881343 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e580c86b0 msgr2=0x7f8e581f6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.477+0000 7f8e5d600640 1 --2- 172.21.15.29:0/1241881343 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e580c86b0 0x7f8e581f6880 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000c3f0 tx=0x7f8e5000c8b0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f8e5d600640 1 -- 172.21.15.29:0/1241881343 shutdown_connections 2024-04-17T19:48:18.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f8e5d600640 1 --2- 172.21.15.29:0/1241881343 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8e38068480 0x7f8e3806a990 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f8e5d600640 1 --2- 172.21.15.29:0/1241881343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e580ca4e0 0x7f8e58152270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f8e5d600640 1 --2- 172.21.15.29:0/1241881343 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e580ca0d0 0x7f8e58153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f8e5d600640 1 --2- 172.21.15.29:0/1241881343 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e580c86b0 0x7f8e581f6880 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.468 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f8e5d600640 1 -- 172.21.15.29:0/1241881343 >> 172.21.15.29:0/1241881343 conn(0x7f8e5806bf90 msgr2=0x7f8e581621e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:18.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f5a0f400640 1 -- 172.21.15.29:0/176059016 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f59e8068400 msgr2=0x7f59e806a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f5a0f400640 1 --2- 172.21.15.29:0/176059016 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f59e8068400 0x7f59e806a910 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f59f40ad000 tx=0x7f59f40a6000 comp rx=0 tx=0).stop 2024-04-17T19:48:18.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f5a0f400640 1 -- 172.21.15.29:0/176059016 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a081521e0 msgr2=0x7f5a081f7070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f5a0f400640 1 --2- 172.21.15.29:0/176059016 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a081521e0 0x7f5a081f7070 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5a0000ba50 tx=0x7f5a0000be50 comp rx=0 tx=0).stop 2024-04-17T19:48:18.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f5a0f400640 1 -- 172.21.15.29:0/176059016 shutdown_connections 2024-04-17T19:48:18.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f5a0f400640 1 --2- 172.21.15.29:0/176059016 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f59e8068400 0x7f59e806a910 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f5a0f400640 1 --2- 172.21.15.29:0/176059016 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a08158bf0 0x7f5a08140110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f5a0f400640 1 --2- 172.21.15.29:0/176059016 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5a081521e0 0x7f5a081f7070 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5a0000ba50 tx=0x7f5a0000be50 comp rx=0 tx=0).stop 2024-04-17T19:48:18.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f5a0f400640 1 --2- 172.21.15.29:0/176059016 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5a080c99a0 0x7f5a081514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f5a0f400640 1 -- 172.21.15.29:0/176059016 >> 172.21.15.29:0/176059016 conn(0x7f5a0806c950 msgr2=0x7f5a0806f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:18.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f8e5d600640 1 -- 172.21.15.29:0/1241881343 shutdown_connections 2024-04-17T19:48:18.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f5a0f400640 1 -- 172.21.15.29:0/176059016 shutdown_connections 2024-04-17T19:48:18.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f5a0f400640 1 -- 172.21.15.29:0/176059016 wait complete. 2024-04-17T19:48:18.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.481+0000 7f8e5d600640 1 -- 172.21.15.29:0/1241881343 wait complete. 2024-04-17T19:48:18.474 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.2 2024-04-17T19:48:18.474 DEBUG:teuthology.parallel:result is None 2024-04-17T19:48:18.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.485+0000 7f1a49e00640 1 -- 172.21.15.29:0/2124108723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f1a4406d4e0 con 0x7f1a44164040 2024-04-17T19:48:18.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.489+0000 7f1a40c00640 1 -- 172.21.15.29:0/2124108723 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1a34055bd0 con 0x7f1a44164040 2024-04-17T19:48:18.479 INFO:teuthology.orchestra.run.smithi029.stdout:55834574860 2024-04-17T19:48:18.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.497+0000 7f1a49e00640 1 -- 172.21.15.29:0/2124108723 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1a24068380 msgr2=0x7f1a2406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.497+0000 7f1a49e00640 1 --2- 172.21.15.29:0/2124108723 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1a24068380 0x7f1a2406a890 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1a300b2000 tx=0x7f1a30004740 comp rx=0 tx=0).stop 2024-04-17T19:48:18.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.497+0000 7f1a49e00640 1 -- 172.21.15.29:0/2124108723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a44164040 msgr2=0x7f1a441f9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.497+0000 7f1a49e00640 1 --2- 172.21.15.29:0/2124108723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a44164040 0x7f1a441f9ff0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1a3400c3f0 tx=0x7f1a3400c8b0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.497+0000 7f1a49e00640 1 -- 172.21.15.29:0/2124108723 shutdown_connections 2024-04-17T19:48:18.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.497+0000 7f1a49e00640 1 --2- 172.21.15.29:0/2124108723 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1a24068380 0x7f1a2406a890 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.497+0000 7f1a49e00640 1 --2- 172.21.15.29:0/2124108723 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1a441f7110 0x7f1a440cc400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.497+0000 7f1a49e00640 1 --2- 172.21.15.29:0/2124108723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a44164040 0x7f1a441f9ff0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.497+0000 7f1a49e00640 1 --2- 172.21.15.29:0/2124108723 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1a440ce150 0x7f1a4415b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.497+0000 7f1a49e00640 1 -- 172.21.15.29:0/2124108723 >> 172.21.15.29:0/2124108723 conn(0x7f1a44070880 msgr2=0x7f1a44073860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:18.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.497+0000 7f1a49e00640 1 -- 172.21.15.29:0/2124108723 shutdown_connections 2024-04-17T19:48:18.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.497+0000 7f1a49e00640 1 -- 172.21.15.29:0/2124108723 wait complete. 2024-04-17T19:48:18.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.513+0000 7f5e4a400640 1 -- 172.21.15.29:0/2529910808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f5e440c2260 con 0x7f5e441588e0 2024-04-17T19:48:18.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.513+0000 7f5e40c00640 1 -- 172.21.15.29:0/2529910808 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5e441593d0 con 0x7f5e441588e0 2024-04-17T19:48:18.501 INFO:teuthology.orchestra.run.smithi029.stdout:55834574860 2024-04-17T19:48:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.517+0000 7f5e4a400640 1 -- 172.21.15.29:0/2529910808 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5e240683b0 msgr2=0x7f5e2406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.517+0000 7f5e4a400640 1 --2- 172.21.15.29:0/2529910808 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5e240683b0 0x7f5e2406a8c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5e3400ca80 tx=0x7f5e34008040 comp rx=0 tx=0).stop 2024-04-17T19:48:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.517+0000 7f5e4a400640 1 -- 172.21.15.29:0/2529910808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e441588e0 msgr2=0x7f5e4414a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.517+0000 7f5e4a400640 1 --2- 172.21.15.29:0/2529910808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e441588e0 0x7f5e4414a6d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5e4415b650 tx=0x7f5e300018a0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.517+0000 7f5e4a400640 1 -- 172.21.15.29:0/2529910808 shutdown_connections 2024-04-17T19:48:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.517+0000 7f5e4a400640 1 --2- 172.21.15.29:0/2529910808 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5e240683b0 0x7f5e2406a8c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.517+0000 7f5e4a400640 1 --2- 172.21.15.29:0/2529910808 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5e4415bf20 0x7f5e441427a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.517+0000 7f5e4a400640 1 --2- 172.21.15.29:0/2529910808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e441588e0 0x7f5e4414a6d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.517+0000 7f5e4a400640 1 --2- 172.21.15.29:0/2529910808 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5e440cce60 0x7f5e44143360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.517+0000 7f5e4a400640 1 -- 172.21.15.29:0/2529910808 >> 172.21.15.29:0/2529910808 conn(0x7f5e4406ddd0 msgr2=0x7f5e440726e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.517+0000 7f5e4a400640 1 -- 172.21.15.29:0/2529910808 shutdown_connections 2024-04-17T19:48:18.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.517+0000 7f5e4a400640 1 -- 172.21.15.29:0/2529910808 wait complete. 2024-04-17T19:48:18.510 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.7 2024-04-17T19:48:18.510 DEBUG:teuthology.parallel:result is None 2024-04-17T19:48:18.511 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.0 2024-04-17T19:48:18.511 DEBUG:teuthology.parallel:result is None 2024-04-17T19:48:18.523 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.5 2024-04-17T19:48:18.523 DEBUG:teuthology.parallel:result is None 2024-04-17T19:48:18.532 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.1 2024-04-17T19:48:18.532 DEBUG:teuthology.parallel:result is None 2024-04-17T19:48:18.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.549+0000 7f0012000640 1 -- 172.21.15.29:0/1164539349 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f000c0693f0 con 0x7f000c0b4800 2024-04-17T19:48:18.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.549+0000 7f0003400640 1 -- 172.21.15.29:0/1164539349 <== mon.1 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efff80f38b0 con 0x7f000c0b4800 2024-04-17T19:48:18.538 INFO:teuthology.orchestra.run.smithi029.stdout:55834574860 2024-04-17T19:48:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.553+0000 7f0012000640 1 -- 172.21.15.29:0/1164539349 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7effec068380 msgr2=0x7effec06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.553+0000 7f0012000640 1 --2- 172.21.15.29:0/1164539349 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7effec068380 0x7effec06a890 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f000c15e830 tx=0x7efffc008040 comp rx=0 tx=0).stop 2024-04-17T19:48:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.553+0000 7f0012000640 1 -- 172.21.15.29:0/1164539349 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f000c0b4800 msgr2=0x7f000c0c9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.553+0000 7f0012000640 1 --2- 172.21.15.29:0/1164539349 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f000c0b4800 0x7f000c0c9270 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7efff80af000 tx=0x7efff801b9b0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.553+0000 7f0012000640 1 -- 172.21.15.29:0/1164539349 shutdown_connections 2024-04-17T19:48:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.553+0000 7f0012000640 1 --2- 172.21.15.29:0/1164539349 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7effec068380 0x7effec06a890 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.553+0000 7f0012000640 1 --2- 172.21.15.29:0/1164539349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f000c160970 0x7f000c0c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.553+0000 7f0012000640 1 --2- 172.21.15.29:0/1164539349 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f000c160560 0x7f000c0ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.553+0000 7f0012000640 1 --2- 172.21.15.29:0/1164539349 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f000c0b4800 0x7f000c0c9270 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.553+0000 7f0012000640 1 -- 172.21.15.29:0/1164539349 >> 172.21.15.29:0/1164539349 conn(0x7f000c0d1440 msgr2=0x7f000c156a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:18.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.553+0000 7f0012000640 1 -- 172.21.15.29:0/1164539349 shutdown_connections 2024-04-17T19:48:18.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.553+0000 7f0012000640 1 -- 172.21.15.29:0/1164539349 wait complete. 2024-04-17T19:48:18.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.553+0000 7f9ca8c00640 1 -- 172.21.15.29:0/2371804773 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f9ca4072140 con 0x7f9c88005280 2024-04-17T19:48:18.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.553+0000 7f9c9a000640 1 -- 172.21.15.29:0/2371804773 <== mon.1 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9c9c0794f0 con 0x7f9c88005280 2024-04-17T19:48:18.542 INFO:teuthology.orchestra.run.smithi029.stdout:55834574859 2024-04-17T19:48:18.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.557+0000 7f9ca8c00640 1 -- 172.21.15.29:0/2371804773 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9c84068380 msgr2=0x7f9c8406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.557+0000 7f9ca8c00640 1 --2- 172.21.15.29:0/2371804773 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9c84068380 0x7f9c8406a890 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4140b10 tx=0x7f9c94008040 comp rx=0 tx=0).stop 2024-04-17T19:48:18.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.557+0000 7f9ca8c00640 1 -- 172.21.15.29:0/2371804773 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c88005280 msgr2=0x7f9ca41515b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.557+0000 7f9ca8c00640 1 --2- 172.21.15.29:0/2371804773 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c88005280 0x7f9ca41515b0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c031980 tx=0x7f9c9c031e40 comp rx=0 tx=0).stop 2024-04-17T19:48:18.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.557+0000 7f9ca8c00640 1 -- 172.21.15.29:0/2371804773 shutdown_connections 2024-04-17T19:48:18.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.557+0000 7f9ca8c00640 1 --2- 172.21.15.29:0/2371804773 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9c84068380 0x7f9c8406a890 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.557+0000 7f9ca8c00640 1 --2- 172.21.15.29:0/2371804773 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ca414d8c0 0x7f9ca414f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.557+0000 7f9ca8c00640 1 --2- 172.21.15.29:0/2371804773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ca415ade0 0x7f9ca4159170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.557+0000 7f9ca8c00640 1 --2- 172.21.15.29:0/2371804773 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c88005280 0x7f9ca41515b0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.557+0000 7f9ca8c00640 1 -- 172.21.15.29:0/2371804773 >> 172.21.15.29:0/2371804773 conn(0x7f9ca406c950 msgr2=0x7f9ca40c2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:18.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.557+0000 7f9ca8c00640 1 -- 172.21.15.29:0/2371804773 shutdown_connections 2024-04-17T19:48:18.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.557+0000 7f9ca8c00640 1 -- 172.21.15.29:0/2371804773 wait complete. 2024-04-17T19:48:18.571 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.3 2024-04-17T19:48:18.572 DEBUG:teuthology.parallel:result is None 2024-04-17T19:48:18.574 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.4 2024-04-17T19:48:18.574 DEBUG:teuthology.parallel:result is None 2024-04-17T19:48:18.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.621+0000 7efc6de00640 1 -- 172.21.15.29:0/539345014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7efc68074b50 con 0x7efc68158a20 2024-04-17T19:48:18.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.625+0000 7efc5f400640 1 -- 172.21.15.29:0/539345014 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efc54056f30 con 0x7efc68158a20 2024-04-17T19:48:18.612 INFO:teuthology.orchestra.run.smithi029.stdout:55834574858 2024-04-17T19:48:18.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.625+0000 7efc6de00640 1 -- 172.21.15.29:0/539345014 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efc480683b0 msgr2=0x7efc4806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.625+0000 7efc6de00640 1 --2- 172.21.15.29:0/539345014 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efc480683b0 0x7efc4806a8c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efc58006fd0 tx=0x7efc58008040 comp rx=0 tx=0).stop 2024-04-17T19:48:18.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.625+0000 7efc6de00640 1 -- 172.21.15.29:0/539345014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc68158a20 msgr2=0x7efc68148a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.625+0000 7efc6de00640 1 --2- 172.21.15.29:0/539345014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc68158a20 0x7efc68148a90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7efc5400e950 tx=0x7efc5400ee10 comp rx=0 tx=0).stop 2024-04-17T19:48:18.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.625+0000 7efc6de00640 1 -- 172.21.15.29:0/539345014 shutdown_connections 2024-04-17T19:48:18.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.625+0000 7efc6de00640 1 --2- 172.21.15.29:0/539345014 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efc480683b0 0x7efc4806a8c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.625+0000 7efc6de00640 1 --2- 172.21.15.29:0/539345014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc68158a20 0x7efc68148a90 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.625+0000 7efc6de00640 1 --2- 172.21.15.29:0/539345014 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc681521e0 0x7efc680c5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.625+0000 7efc6de00640 1 --2- 172.21.15.29:0/539345014 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc680c2460 0x7efc6814c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.625+0000 7efc6de00640 1 -- 172.21.15.29:0/539345014 >> 172.21.15.29:0/539345014 conn(0x7efc6806c950 msgr2=0x7efc681575b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:18.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.625+0000 7efc6de00640 1 -- 172.21.15.29:0/539345014 shutdown_connections 2024-04-17T19:48:18.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.625+0000 7efc6de00640 1 -- 172.21.15.29:0/539345014 wait complete. 2024-04-17T19:48:18.634 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.6 2024-04-17T19:48:18.634 DEBUG:teuthology.parallel:result is None 2024-04-17T19:48:18.634 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-17T19:48:18.634 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T19:48:18.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f1929c00640 1 Processor -- start 2024-04-17T19:48:18.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f1929c00640 1 -- start start 2024-04-17T19:48:18.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f1929c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1924159c90 0x7f19240ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f1929c00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f19240ca7f0 0x7f19240c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f1929c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19240c74b0 0x7f192406c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f1929c00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f192415a0a0 con 0x7f1924159c90 2024-04-17T19:48:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f1929c00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f192415abe0 con 0x7f19240ca7f0 2024-04-17T19:48:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f1929c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f192415a340 con 0x7f19240c74b0 2024-04-17T19:48:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f1922a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1924159c90 0x7f19240ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f1922a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1924159c90 0x7f19240ca2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58136/0 (socket says 172.21.15.29:58136) 2024-04-17T19:48:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f1922a00640 1 -- 172.21.15.29:0/159574030 learned_addr learned my addr 172.21.15.29:0/159574030 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f191be00640 1 --2- 172.21.15.29:0/159574030 >> v2:172.21.15.37:3300/0 conn(0x7f19240ca7f0 0x7f19240c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f1923400640 1 --2- 172.21.15.29:0/159574030 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19240c74b0 0x7f192406c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f1922a00640 1 -- 172.21.15.29:0/159574030 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19240c74b0 msgr2=0x7f192406c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f1922a00640 1 --2- 172.21.15.29:0/159574030 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19240c74b0 0x7f192406c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f1922a00640 1 -- 172.21.15.29:0/159574030 >> v2:172.21.15.37:3300/0 conn(0x7f19240ca7f0 msgr2=0x7f19240c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f1922a00640 1 --2- 172.21.15.29:0/159574030 >> v2:172.21.15.37:3300/0 conn(0x7f19240ca7f0 0x7f19240c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f1922a00640 1 -- 172.21.15.29:0/159574030 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f192415a620 con 0x7f1924159c90 2024-04-17T19:48:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.841+0000 7f191be00640 1 --2- 172.21.15.29:0/159574030 >> v2:172.21.15.37:3300/0 conn(0x7f19240ca7f0 0x7f19240c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:18.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.845+0000 7f1922a00640 1 --2- 172.21.15.29:0/159574030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1924159c90 0x7f19240ca2b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f19240cd310 tx=0x7f191401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e5cf879ba40c7e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.845+0000 7f1922000640 1 -- 172.21.15.29:0/159574030 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19140ab5f0 con 0x7f1924159c90 2024-04-17T19:48:18.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.845+0000 7f1922000640 1 -- 172.21.15.29:0/159574030 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19140b0070 con 0x7f1924159c90 2024-04-17T19:48:18.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.845+0000 7f1922000640 1 -- 172.21.15.29:0/159574030 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19140ab9b0 con 0x7f1924159c90 2024-04-17T19:48:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.845+0000 7f1929c00640 1 -- 172.21.15.29:0/159574030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1924159c90 msgr2=0x7f19240ca2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.845+0000 7f1929c00640 1 --2- 172.21.15.29:0/159574030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1924159c90 0x7f19240ca2b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f19240cd310 tx=0x7f191401c520 comp rx=0 tx=0).stop 2024-04-17T19:48:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.845+0000 7f1929c00640 1 -- 172.21.15.29:0/159574030 shutdown_connections 2024-04-17T19:48:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.845+0000 7f1929c00640 1 --2- 172.21.15.29:0/159574030 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19240c74b0 0x7f192406c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.845+0000 7f1929c00640 1 --2- 172.21.15.29:0/159574030 >> v2:172.21.15.37:3300/0 conn(0x7f19240ca7f0 0x7f19240c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.845+0000 7f1929c00640 1 --2- 172.21.15.29:0/159574030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1924159c90 0x7f19240ca2b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.845+0000 7f1929c00640 1 -- 172.21.15.29:0/159574030 >> 172.21.15.29:0/159574030 conn(0x7f192406fde0 msgr2=0x7f19240701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.845+0000 7f1929c00640 1 -- 172.21.15.29:0/159574030 shutdown_connections 2024-04-17T19:48:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.845+0000 7f1929c00640 1 -- 172.21.15.29:0/159574030 wait complete. 2024-04-17T19:48:18.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f1929c00640 1 Processor -- start 2024-04-17T19:48:18.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f1929c00640 1 -- start start 2024-04-17T19:48:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f1929c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19240c74b0 0x7f192414df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f1929c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19240ca7f0 0x7f192414b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f1929c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1924159c90 0x7f192414bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f1929c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f192415da90 con 0x7f19240ca7f0 2024-04-17T19:48:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f1929c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f192406cfb0 con 0x7f1924159c90 2024-04-17T19:48:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f1929c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f192415dc10 con 0x7f19240c74b0 2024-04-17T19:48:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f191be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19240ca7f0 0x7f192414b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f191be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19240ca7f0 0x7f192414b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58142/0 (socket says 172.21.15.29:58142) 2024-04-17T19:48:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f191be00640 1 -- 172.21.15.29:0/3905151703 learned_addr learned my addr 172.21.15.29:0/3905151703 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f1923400640 1 --2- 172.21.15.29:0/3905151703 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1924159c90 0x7f192414bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f1922a00640 1 --2- 172.21.15.29:0/3905151703 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19240c74b0 0x7f192414df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f191be00640 1 -- 172.21.15.29:0/3905151703 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19240c74b0 msgr2=0x7f192414df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f191be00640 1 --2- 172.21.15.29:0/3905151703 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19240c74b0 0x7f192414df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f191be00640 1 -- 172.21.15.29:0/3905151703 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1924159c90 msgr2=0x7f192414bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:18.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f191be00640 1 --2- 172.21.15.29:0/3905151703 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1924159c90 0x7f192414bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:18.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f191be00640 1 -- 172.21.15.29:0/3905151703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f192414e4a0 con 0x7f19240ca7f0 2024-04-17T19:48:18.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f1923400640 1 --2- 172.21.15.29:0/3905151703 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1924159c90 0x7f192414bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:18.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f191be00640 1 --2- 172.21.15.29:0/3905151703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19240ca7f0 0x7f192414b740 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f190c00d800 tx=0x7f190c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.849+0000 7f191aa00640 1 -- 172.21.15.29:0/3905151703 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190c00b6c0 con 0x7f19240ca7f0 2024-04-17T19:48:18.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.853+0000 7f191aa00640 1 -- 172.21.15.29:0/3905151703 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f190c019070 con 0x7f19240ca7f0 2024-04-17T19:48:18.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.853+0000 7f1929c00640 1 -- 172.21.15.29:0/3905151703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f192414f180 con 0x7f19240ca7f0 2024-04-17T19:48:18.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.853+0000 7f191aa00640 1 -- 172.21.15.29:0/3905151703 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190c00ba80 con 0x7f19240ca7f0 2024-04-17T19:48:18.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.853+0000 7f1929c00640 1 -- 172.21.15.29:0/3905151703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19240c4a90 con 0x7f19240ca7f0 2024-04-17T19:48:18.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.853+0000 7f1929c00640 1 -- 172.21.15.29:0/3905151703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1924069b90 con 0x7f19240ca7f0 2024-04-17T19:48:18.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.853+0000 7f191aa00640 1 -- 172.21.15.29:0/3905151703 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f190c00bc20 con 0x7f19240ca7f0 2024-04-17T19:48:18.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.853+0000 7f191aa00640 1 --2- 172.21.15.29:0/3905151703 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1900068380 0x7f190006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:18.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.853+0000 7f191aa00640 1 -- 172.21.15.29:0/3905151703 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f190c087120 con 0x7f19240ca7f0 2024-04-17T19:48:18.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.853+0000 7f1922a00640 1 --2- 172.21.15.29:0/3905151703 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1900068380 0x7f190006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:18.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.857+0000 7f1922a00640 1 --2- 172.21.15.29:0/3905151703 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1900068380 0x7f190006a890 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f19140af000 tx=0x7f19140a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:18.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:18.857+0000 7f191aa00640 1 -- 172.21.15.29:0/3905151703 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f190c057390 con 0x7f19240ca7f0 2024-04-17T19:48:19.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.025+0000 7f1929c00640 1 -- 172.21.15.29:0/3905151703 --> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f19241f6be0 con 0x7f1900068380 2024-04-17T19:48:19.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.049+0000 7f191aa00640 1 -- 172.21.15.29:0/3905151703 <== mgr.4105 v2:172.21.15.37:6832/1588862286 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332572 (secure 0 0 0) 0x7f19241f6be0 con 0x7f1900068380 2024-04-17T19:48:19.040 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:19.042 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-04-17T19:48:19.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.053+0000 7f1929c00640 1 -- 172.21.15.29:0/3905151703 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1900068380 msgr2=0x7f190006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:19.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.053+0000 7f1929c00640 1 --2- 172.21.15.29:0/3905151703 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1900068380 0x7f190006a890 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f19140af000 tx=0x7f19140a7000 comp rx=0 tx=0).stop 2024-04-17T19:48:19.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.053+0000 7f1929c00640 1 -- 172.21.15.29:0/3905151703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19240ca7f0 msgr2=0x7f192414b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:19.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.053+0000 7f1929c00640 1 --2- 172.21.15.29:0/3905151703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19240ca7f0 0x7f192414b740 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f190c00d800 tx=0x7f190c00dcc0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.053+0000 7f1929c00640 1 -- 172.21.15.29:0/3905151703 shutdown_connections 2024-04-17T19:48:19.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.053+0000 7f1929c00640 1 --2- 172.21.15.29:0/3905151703 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1900068380 0x7f190006a890 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.053+0000 7f1929c00640 1 --2- 172.21.15.29:0/3905151703 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1924159c90 0x7f192414bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.053+0000 7f1929c00640 1 --2- 172.21.15.29:0/3905151703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19240ca7f0 0x7f192414b740 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.053+0000 7f1929c00640 1 --2- 172.21.15.29:0/3905151703 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f19240c74b0 0x7f192414df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.053+0000 7f1929c00640 1 -- 172.21.15.29:0/3905151703 >> 172.21.15.29:0/3905151703 conn(0x7f192406fde0 msgr2=0x7f192414ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:19.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.053+0000 7f1929c00640 1 -- 172.21.15.29:0/3905151703 shutdown_connections 2024-04-17T19:48:19.044 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.053+0000 7f1929c00640 1 -- 172.21.15.29:0/3905151703 wait complete. 2024-04-17T19:48:19.062 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-04-17T19:48:18.637552+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596414,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":246,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":193,"ondisk_log_size":193,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240168,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754734552,"statfs":{"total":773094113280,"available":772848181248,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1547,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006087"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842100+0000","last_change":"2024-04-17T19:47:48.885235+0000","last_active":"2024-04-17T19:47:55.842100+0000","last_peered":"2024-04-17T19:47:55.842100+0000","last_clean":"2024-04-17T19:47:55.842100+0000","last_became_active":"2024-04-17T19:47:48.884844+0000","last_became_peered":"2024-04-17T19:47:48.884844+0000","last_unstale":"2024-04-17T19:47:55.842100+0000","last_undegraded":"2024-04-17T19:47:55.842100+0000","last_fullsized":"2024-04-17T19:47:55.842100+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:01:52.854763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924264+0000","last_change":"2024-04-17T19:47:49.872314+0000","last_active":"2024-04-17T19:47:54.924264+0000","last_peered":"2024-04-17T19:47:54.924264+0000","last_clean":"2024-04-17T19:47:54.924264+0000","last_became_active":"2024-04-17T19:47:49.871990+0000","last_became_peered":"2024-04-17T19:47:49.871990+0000","last_unstale":"2024-04-17T19:47:54.924264+0000","last_undegraded":"2024-04-17T19:47:54.924264+0000","last_fullsized":"2024-04-17T19:47:54.924264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:12:09.196610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923798+0000","last_change":"2024-04-17T19:47:53.929840+0000","last_active":"2024-04-17T19:47:54.923798+0000","last_peered":"2024-04-17T19:47:54.923798+0000","last_clean":"2024-04-17T19:47:54.923798+0000","last_became_active":"2024-04-17T19:47:53.928272+0000","last_became_peered":"2024-04-17T19:47:53.928272+0000","last_unstale":"2024-04-17T19:47:54.923798+0000","last_undegraded":"2024-04-17T19:47:54.923798+0000","last_fullsized":"2024-04-17T19:47:54.923798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:27:29.734327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924636+0000","last_change":"2024-04-17T19:47:48.862315+0000","last_active":"2024-04-17T19:47:54.924636+0000","last_peered":"2024-04-17T19:47:54.924636+0000","last_clean":"2024-04-17T19:47:54.924636+0000","last_became_active":"2024-04-17T19:47:48.860408+0000","last_became_peered":"2024-04-17T19:47:48.860408+0000","last_unstale":"2024-04-17T19:47:54.924636+0000","last_undegraded":"2024-04-17T19:47:54.924636+0000","last_fullsized":"2024-04-17T19:47:54.924636+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:10:26.336063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841873+0000","last_change":"2024-04-17T19:47:49.902423+0000","last_active":"2024-04-17T19:47:55.841873+0000","last_peered":"2024-04-17T19:47:55.841873+0000","last_clean":"2024-04-17T19:47:55.841873+0000","last_became_active":"2024-04-17T19:47:49.902096+0000","last_became_peered":"2024-04-17T19:47:49.902096+0000","last_unstale":"2024-04-17T19:47:55.841873+0000","last_undegraded":"2024-04-17T19:47:55.841873+0000","last_fullsized":"2024-04-17T19:47:55.841873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:40.805247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955520+0000","last_change":"2024-04-17T19:47:53.956851+0000","last_active":"2024-04-17T19:47:56.955520+0000","last_peered":"2024-04-17T19:47:56.955520+0000","last_clean":"2024-04-17T19:47:56.955520+0000","last_became_active":"2024-04-17T19:47:53.954197+0000","last_became_peered":"2024-04-17T19:47:53.954197+0000","last_unstale":"2024-04-17T19:47:56.955520+0000","last_undegraded":"2024-04-17T19:47:56.955520+0000","last_fullsized":"2024-04-17T19:47:56.955520+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:16:18.379831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049047+0000","last_change":"2024-04-17T19:47:48.893969+0000","last_active":"2024-04-17T19:47:56.049047+0000","last_peered":"2024-04-17T19:47:56.049047+0000","last_clean":"2024-04-17T19:47:56.049047+0000","last_became_active":"2024-04-17T19:47:48.893229+0000","last_became_peered":"2024-04-17T19:47:48.893229+0000","last_unstale":"2024-04-17T19:47:56.049047+0000","last_undegraded":"2024-04-17T19:47:56.049047+0000","last_fullsized":"2024-04-17T19:47:56.049047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:38:42.472634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832673+0000","last_change":"2024-04-17T19:47:49.897987+0000","last_active":"2024-04-17T19:47:56.832673+0000","last_peered":"2024-04-17T19:47:56.832673+0000","last_clean":"2024-04-17T19:47:56.832673+0000","last_became_active":"2024-04-17T19:47:49.897096+0000","last_became_peered":"2024-04-17T19:47:49.897096+0000","last_unstale":"2024-04-17T19:47:56.832673+0000","last_undegraded":"2024-04-17T19:47:56.832673+0000","last_fullsized":"2024-04-17T19:47:56.832673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:36:36.856454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955695+0000","last_change":"2024-04-17T19:47:53.930784+0000","last_active":"2024-04-17T19:47:56.955695+0000","last_peered":"2024-04-17T19:47:56.955695+0000","last_clean":"2024-04-17T19:47:56.955695+0000","last_became_active":"2024-04-17T19:47:53.928281+0000","last_became_peered":"2024-04-17T19:47:53.928281+0000","last_unstale":"2024-04-17T19:47:56.955695+0000","last_undegraded":"2024-04-17T19:47:56.955695+0000","last_fullsized":"2024-04-17T19:47:56.955695+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:15:55.064622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048308+0000","last_change":"2024-04-17T19:47:48.893184+0000","last_active":"2024-04-17T19:47:56.048308+0000","last_peered":"2024-04-17T19:47:56.048308+0000","last_clean":"2024-04-17T19:47:56.048308+0000","last_became_active":"2024-04-17T19:47:48.892375+0000","last_became_peered":"2024-04-17T19:47:48.892375+0000","last_unstale":"2024-04-17T19:47:56.048308+0000","last_undegraded":"2024-04-17T19:47:56.048308+0000","last_fullsized":"2024-04-17T19:47:56.048308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:56:01.211666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832916+0000","last_change":"2024-04-17T19:47:49.896040+0000","last_active":"2024-04-17T19:47:56.832916+0000","last_peered":"2024-04-17T19:47:56.832916+0000","last_clean":"2024-04-17T19:47:56.832916+0000","last_became_active":"2024-04-17T19:47:49.894779+0000","last_became_peered":"2024-04-17T19:47:49.894779+0000","last_unstale":"2024-04-17T19:47:56.832916+0000","last_undegraded":"2024-04-17T19:47:56.832916+0000","last_fullsized":"2024-04-17T19:47:56.832916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:30:17.930203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955688+0000","last_change":"2024-04-17T19:47:53.930402+0000","last_active":"2024-04-17T19:47:56.955688+0000","last_peered":"2024-04-17T19:47:56.955688+0000","last_clean":"2024-04-17T19:47:56.955688+0000","last_became_active":"2024-04-17T19:47:53.929166+0000","last_became_peered":"2024-04-17T19:47:53.929166+0000","last_unstale":"2024-04-17T19:47:56.955688+0000","last_undegraded":"2024-04-17T19:47:56.955688+0000","last_fullsized":"2024-04-17T19:47:56.955688+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:34:08.282697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927108+0000","last_change":"2024-04-17T19:47:49.900142+0000","last_active":"2024-04-17T19:47:54.927108+0000","last_peered":"2024-04-17T19:47:54.927108+0000","last_clean":"2024-04-17T19:47:54.927108+0000","last_became_active":"2024-04-17T19:47:49.899365+0000","last_became_peered":"2024-04-17T19:47:49.899365+0000","last_unstale":"2024-04-17T19:47:54.927108+0000","last_undegraded":"2024-04-17T19:47:54.927108+0000","last_fullsized":"2024-04-17T19:47:54.927108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:34:49.616539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"24'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.758013+0000","last_change":"2024-04-17T19:47:48.891513+0000","last_active":"2024-04-17T19:48:15.758013+0000","last_peered":"2024-04-17T19:48:15.758013+0000","last_clean":"2024-04-17T19:48:15.758013+0000","last_became_active":"2024-04-17T19:47:48.886561+0000","last_became_peered":"2024-04-17T19:47:48.886561+0000","last_unstale":"2024-04-17T19:48:15.758013+0000","last_undegraded":"2024-04-17T19:48:15.758013+0000","last_fullsized":"2024-04-17T19:48:15.758013+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:59:14.039700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.957430+0000","last_change":"2024-04-17T19:47:53.925058+0000","last_active":"2024-04-17T19:47:56.957430+0000","last_peered":"2024-04-17T19:47:56.957430+0000","last_clean":"2024-04-17T19:47:56.957430+0000","last_became_active":"2024-04-17T19:47:53.923721+0000","last_became_peered":"2024-04-17T19:47:53.923721+0000","last_unstale":"2024-04-17T19:47:56.957430+0000","last_undegraded":"2024-04-17T19:47:56.957430+0000","last_fullsized":"2024-04-17T19:47:56.957430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:22:42.106657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924842+0000","last_change":"2024-04-17T19:47:49.852183+0000","last_active":"2024-04-17T19:47:54.924842+0000","last_peered":"2024-04-17T19:47:54.924842+0000","last_clean":"2024-04-17T19:47:54.924842+0000","last_became_active":"2024-04-17T19:47:49.851765+0000","last_became_peered":"2024-04-17T19:47:49.851765+0000","last_unstale":"2024-04-17T19:47:54.924842+0000","last_undegraded":"2024-04-17T19:47:54.924842+0000","last_fullsized":"2024-04-17T19:47:54.924842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:02:03.101317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051463+0000","last_change":"2024-04-17T19:47:48.862865+0000","last_active":"2024-04-17T19:47:55.051463+0000","last_peered":"2024-04-17T19:47:55.051463+0000","last_clean":"2024-04-17T19:47:55.051463+0000","last_became_active":"2024-04-17T19:47:48.862524+0000","last_became_peered":"2024-04-17T19:47:48.862524+0000","last_unstale":"2024-04-17T19:47:55.051463+0000","last_undegraded":"2024-04-17T19:47:55.051463+0000","last_fullsized":"2024-04-17T19:47:55.051463+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:01:21.080029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956615+0000","last_change":"2024-04-17T19:47:53.943674+0000","last_active":"2024-04-17T19:47:56.956615+0000","last_peered":"2024-04-17T19:47:56.956615+0000","last_clean":"2024-04-17T19:47:56.956615+0000","last_became_active":"2024-04-17T19:47:53.942549+0000","last_became_peered":"2024-04-17T19:47:53.942549+0000","last_unstale":"2024-04-17T19:47:56.956615+0000","last_undegraded":"2024-04-17T19:47:56.956615+0000","last_fullsized":"2024-04-17T19:47:56.956615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:00:42.300738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924513+0000","last_change":"2024-04-17T19:47:49.862312+0000","last_active":"2024-04-17T19:47:54.924513+0000","last_peered":"2024-04-17T19:47:54.924513+0000","last_clean":"2024-04-17T19:47:54.924513+0000","last_became_active":"2024-04-17T19:47:49.860664+0000","last_became_peered":"2024-04-17T19:47:49.860664+0000","last_unstale":"2024-04-17T19:47:54.924513+0000","last_undegraded":"2024-04-17T19:47:54.924513+0000","last_fullsized":"2024-04-17T19:47:54.924513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:07:47.676973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"24'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.777109+0000","last_change":"2024-04-17T19:47:48.870638+0000","last_active":"2024-04-17T19:48:15.777109+0000","last_peered":"2024-04-17T19:48:15.777109+0000","last_clean":"2024-04-17T19:48:15.777109+0000","last_became_active":"2024-04-17T19:47:48.870353+0000","last_became_peered":"2024-04-17T19:47:48.870353+0000","last_unstale":"2024-04-17T19:48:15.777109+0000","last_undegraded":"2024-04-17T19:48:15.777109+0000","last_fullsized":"2024-04-17T19:48:15.777109+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:12:19.927824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.052170+0000","last_change":"2024-04-17T19:47:53.929285+0000","last_active":"2024-04-17T19:47:55.052170+0000","last_peered":"2024-04-17T19:47:55.052170+0000","last_clean":"2024-04-17T19:47:55.052170+0000","last_became_active":"2024-04-17T19:47:53.928189+0000","last_became_peered":"2024-04-17T19:47:53.928189+0000","last_unstale":"2024-04-17T19:47:55.052170+0000","last_undegraded":"2024-04-17T19:47:55.052170+0000","last_fullsized":"2024-04-17T19:47:55.052170+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:53:26.112634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956382+0000","last_change":"2024-04-17T19:47:49.888118+0000","last_active":"2024-04-17T19:47:56.956382+0000","last_peered":"2024-04-17T19:47:56.956382+0000","last_clean":"2024-04-17T19:47:56.956382+0000","last_became_active":"2024-04-17T19:47:49.884037+0000","last_became_peered":"2024-04-17T19:47:49.884037+0000","last_unstale":"2024-04-17T19:47:56.956382+0000","last_undegraded":"2024-04-17T19:47:56.956382+0000","last_fullsized":"2024-04-17T19:47:56.956382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:44:27.180380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"24'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.759737+0000","last_change":"2024-04-17T19:47:48.883686+0000","last_active":"2024-04-17T19:48:15.759737+0000","last_peered":"2024-04-17T19:48:15.759737+0000","last_clean":"2024-04-17T19:48:15.759737+0000","last_became_active":"2024-04-17T19:47:48.882375+0000","last_became_peered":"2024-04-17T19:47:48.882375+0000","last_unstale":"2024-04-17T19:48:15.759737+0000","last_undegraded":"2024-04-17T19:48:15.759737+0000","last_fullsized":"2024-04-17T19:48:15.759737+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:38:31.225374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.957175+0000","last_change":"2024-04-17T19:47:53.935396+0000","last_active":"2024-04-17T19:47:56.957175+0000","last_peered":"2024-04-17T19:47:56.957175+0000","last_clean":"2024-04-17T19:47:56.957175+0000","last_became_active":"2024-04-17T19:47:53.934094+0000","last_became_peered":"2024-04-17T19:47:53.934094+0000","last_unstale":"2024-04-17T19:47:56.957175+0000","last_undegraded":"2024-04-17T19:47:56.957175+0000","last_fullsized":"2024-04-17T19:47:56.957175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:58.334357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842041+0000","last_change":"2024-04-17T19:47:49.870297+0000","last_active":"2024-04-17T19:47:55.842041+0000","last_peered":"2024-04-17T19:47:55.842041+0000","last_clean":"2024-04-17T19:47:55.842041+0000","last_became_active":"2024-04-17T19:47:49.870113+0000","last_became_peered":"2024-04-17T19:47:49.870113+0000","last_unstale":"2024-04-17T19:47:55.842041+0000","last_undegraded":"2024-04-17T19:47:55.842041+0000","last_fullsized":"2024-04-17T19:47:55.842041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:50:54.517273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"24'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.766655+0000","last_change":"2024-04-17T19:47:48.895905+0000","last_active":"2024-04-17T19:48:15.766655+0000","last_peered":"2024-04-17T19:48:15.766655+0000","last_clean":"2024-04-17T19:48:15.766655+0000","last_became_active":"2024-04-17T19:47:48.895263+0000","last_became_peered":"2024-04-17T19:47:48.895263+0000","last_unstale":"2024-04-17T19:48:15.766655+0000","last_undegraded":"2024-04-17T19:48:15.766655+0000","last_fullsized":"2024-04-17T19:48:15.766655+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:07:23.179031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956013+0000","last_change":"2024-04-17T19:47:53.943286+0000","last_active":"2024-04-17T19:47:56.956013+0000","last_peered":"2024-04-17T19:47:56.956013+0000","last_clean":"2024-04-17T19:47:56.956013+0000","last_became_active":"2024-04-17T19:47:53.942325+0000","last_became_peered":"2024-04-17T19:47:53.942325+0000","last_unstale":"2024-04-17T19:47:56.956013+0000","last_undegraded":"2024-04-17T19:47:56.956013+0000","last_fullsized":"2024-04-17T19:47:56.956013+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:47:18.853991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924723+0000","last_change":"2024-04-17T19:47:49.872925+0000","last_active":"2024-04-17T19:47:54.924723+0000","last_peered":"2024-04-17T19:47:54.924723+0000","last_clean":"2024-04-17T19:47:54.924723+0000","last_became_active":"2024-04-17T19:47:49.872668+0000","last_became_peered":"2024-04-17T19:47:49.872668+0000","last_unstale":"2024-04-17T19:47:54.924723+0000","last_undegraded":"2024-04-17T19:47:54.924723+0000","last_fullsized":"2024-04-17T19:47:54.924723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:16:21.209634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051807+0000","last_change":"2024-04-17T19:47:48.896669+0000","last_active":"2024-04-17T19:47:55.051807+0000","last_peered":"2024-04-17T19:47:55.051807+0000","last_clean":"2024-04-17T19:47:55.051807+0000","last_became_active":"2024-04-17T19:47:48.896397+0000","last_became_peered":"2024-04-17T19:47:48.896397+0000","last_unstale":"2024-04-17T19:47:55.051807+0000","last_undegraded":"2024-04-17T19:47:55.051807+0000","last_fullsized":"2024-04-17T19:47:55.051807+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:07:23.179031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923702+0000","last_change":"2024-04-17T19:47:53.906261+0000","last_active":"2024-04-17T19:47:54.923702+0000","last_peered":"2024-04-17T19:47:54.923702+0000","last_clean":"2024-04-17T19:47:54.923702+0000","last_became_active":"2024-04-17T19:47:53.903702+0000","last_became_peered":"2024-04-17T19:47:53.903702+0000","last_unstale":"2024-04-17T19:47:54.923702+0000","last_undegraded":"2024-04-17T19:47:54.923702+0000","last_fullsized":"2024-04-17T19:47:54.923702+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:35:53.892917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051477+0000","last_change":"2024-04-17T19:47:49.870272+0000","last_active":"2024-04-17T19:47:55.051477+0000","last_peered":"2024-04-17T19:47:55.051477+0000","last_clean":"2024-04-17T19:47:55.051477+0000","last_became_active":"2024-04-17T19:47:49.869914+0000","last_became_peered":"2024-04-17T19:47:49.869914+0000","last_unstale":"2024-04-17T19:47:55.051477+0000","last_undegraded":"2024-04-17T19:47:55.051477+0000","last_fullsized":"2024-04-17T19:47:55.051477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:44:30.553286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049411+0000","last_change":"2024-04-17T19:47:48.892675+0000","last_active":"2024-04-17T19:47:56.049411+0000","last_peered":"2024-04-17T19:47:56.049411+0000","last_clean":"2024-04-17T19:47:56.049411+0000","last_became_active":"2024-04-17T19:47:48.892057+0000","last_became_peered":"2024-04-17T19:47:48.892057+0000","last_unstale":"2024-04-17T19:47:56.049411+0000","last_undegraded":"2024-04-17T19:47:56.049411+0000","last_fullsized":"2024-04-17T19:47:56.049411+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:18:35.735350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841981+0000","last_change":"2024-04-17T19:47:53.925135+0000","last_active":"2024-04-17T19:47:55.841981+0000","last_peered":"2024-04-17T19:47:55.841981+0000","last_clean":"2024-04-17T19:47:55.841981+0000","last_became_active":"2024-04-17T19:47:53.924617+0000","last_became_peered":"2024-04-17T19:47:53.924617+0000","last_unstale":"2024-04-17T19:47:55.841981+0000","last_undegraded":"2024-04-17T19:47:55.841981+0000","last_fullsized":"2024-04-17T19:47:55.841981+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:37:02.082648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.833132+0000","last_change":"2024-04-17T19:47:49.895815+0000","last_active":"2024-04-17T19:47:56.833132+0000","last_peered":"2024-04-17T19:47:56.833132+0000","last_clean":"2024-04-17T19:47:56.833132+0000","last_became_active":"2024-04-17T19:47:49.894978+0000","last_became_peered":"2024-04-17T19:47:49.894978+0000","last_unstale":"2024-04-17T19:47:56.833132+0000","last_undegraded":"2024-04-17T19:47:56.833132+0000","last_fullsized":"2024-04-17T19:47:56.833132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:32:12.184723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051589+0000","last_change":"2024-04-17T19:47:48.859247+0000","last_active":"2024-04-17T19:47:55.051589+0000","last_peered":"2024-04-17T19:47:55.051589+0000","last_clean":"2024-04-17T19:47:55.051589+0000","last_became_active":"2024-04-17T19:47:48.858756+0000","last_became_peered":"2024-04-17T19:47:48.858756+0000","last_unstale":"2024-04-17T19:47:55.051589+0000","last_undegraded":"2024-04-17T19:47:55.051589+0000","last_fullsized":"2024-04-17T19:47:55.051589+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:19:56.987957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051118+0000","last_change":"2024-04-17T19:47:53.929952+0000","last_active":"2024-04-17T19:47:55.051118+0000","last_peered":"2024-04-17T19:47:55.051118+0000","last_clean":"2024-04-17T19:47:55.051118+0000","last_became_active":"2024-04-17T19:47:53.929528+0000","last_became_peered":"2024-04-17T19:47:53.929528+0000","last_unstale":"2024-04-17T19:47:55.051118+0000","last_undegraded":"2024-04-17T19:47:55.051118+0000","last_fullsized":"2024-04-17T19:47:55.051118+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:11:38.041301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956659+0000","last_change":"2024-04-17T19:47:49.874150+0000","last_active":"2024-04-17T19:47:56.956659+0000","last_peered":"2024-04-17T19:47:56.956659+0000","last_clean":"2024-04-17T19:47:56.956659+0000","last_became_active":"2024-04-17T19:47:49.872859+0000","last_became_peered":"2024-04-17T19:47:49.872859+0000","last_unstale":"2024-04-17T19:47:56.956659+0000","last_undegraded":"2024-04-17T19:47:56.956659+0000","last_fullsized":"2024-04-17T19:47:56.956659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:57:54.666279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832613+0000","last_change":"2024-04-17T19:47:48.876672+0000","last_active":"2024-04-17T19:47:56.832613+0000","last_peered":"2024-04-17T19:47:56.832613+0000","last_clean":"2024-04-17T19:47:56.832613+0000","last_became_active":"2024-04-17T19:47:48.875263+0000","last_became_peered":"2024-04-17T19:47:48.875263+0000","last_unstale":"2024-04-17T19:47:56.832613+0000","last_undegraded":"2024-04-17T19:47:56.832613+0000","last_fullsized":"2024-04-17T19:47:56.832613+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:45:46.633540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051584+0000","last_change":"2024-04-17T19:47:53.932330+0000","last_active":"2024-04-17T19:47:55.051584+0000","last_peered":"2024-04-17T19:47:55.051584+0000","last_clean":"2024-04-17T19:47:55.051584+0000","last_became_active":"2024-04-17T19:47:53.931873+0000","last_became_peered":"2024-04-17T19:47:53.931873+0000","last_unstale":"2024-04-17T19:47:55.051584+0000","last_undegraded":"2024-04-17T19:47:55.051584+0000","last_fullsized":"2024-04-17T19:47:55.051584+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:05:42.346878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051044+0000","last_change":"2024-04-17T19:47:49.865020+0000","last_active":"2024-04-17T19:47:55.051044+0000","last_peered":"2024-04-17T19:47:55.051044+0000","last_clean":"2024-04-17T19:47:55.051044+0000","last_became_active":"2024-04-17T19:47:49.864764+0000","last_became_peered":"2024-04-17T19:47:49.864764+0000","last_unstale":"2024-04-17T19:47:55.051044+0000","last_undegraded":"2024-04-17T19:47:55.051044+0000","last_fullsized":"2024-04-17T19:47:55.051044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:45:36.325765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926669+0000","last_change":"2024-04-17T19:47:48.846570+0000","last_active":"2024-04-17T19:47:54.926669+0000","last_peered":"2024-04-17T19:47:54.926669+0000","last_clean":"2024-04-17T19:47:54.926669+0000","last_became_active":"2024-04-17T19:47:48.846175+0000","last_became_peered":"2024-04-17T19:47:48.846175+0000","last_unstale":"2024-04-17T19:47:54.926669+0000","last_undegraded":"2024-04-17T19:47:54.926669+0000","last_fullsized":"2024-04-17T19:47:54.926669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:20:22.737016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.052346+0000","last_change":"2024-04-17T19:47:53.904590+0000","last_active":"2024-04-17T19:47:55.052346+0000","last_peered":"2024-04-17T19:47:55.052346+0000","last_clean":"2024-04-17T19:47:55.052346+0000","last_became_active":"2024-04-17T19:47:53.904026+0000","last_became_peered":"2024-04-17T19:47:53.904026+0000","last_unstale":"2024-04-17T19:47:55.052346+0000","last_undegraded":"2024-04-17T19:47:55.052346+0000","last_fullsized":"2024-04-17T19:47:55.052346+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:16:22.837570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841469+0000","last_change":"2024-04-17T19:47:49.860010+0000","last_active":"2024-04-17T19:47:55.841469+0000","last_peered":"2024-04-17T19:47:55.841469+0000","last_clean":"2024-04-17T19:47:55.841469+0000","last_became_active":"2024-04-17T19:47:49.859593+0000","last_became_peered":"2024-04-17T19:47:49.859593+0000","last_unstale":"2024-04-17T19:47:55.841469+0000","last_undegraded":"2024-04-17T19:47:55.841469+0000","last_fullsized":"2024-04-17T19:47:55.841469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:33:24.594397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.047779+0000","last_change":"2024-04-17T19:47:48.883126+0000","last_active":"2024-04-17T19:47:56.047779+0000","last_peered":"2024-04-17T19:47:56.047779+0000","last_clean":"2024-04-17T19:47:56.047779+0000","last_became_active":"2024-04-17T19:47:48.882173+0000","last_became_peered":"2024-04-17T19:47:48.882173+0000","last_unstale":"2024-04-17T19:47:56.047779+0000","last_undegraded":"2024-04-17T19:47:56.047779+0000","last_fullsized":"2024-04-17T19:47:56.047779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:11:59.395316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841972+0000","last_change":"2024-04-17T19:47:53.931258+0000","last_active":"2024-04-17T19:47:55.841972+0000","last_peered":"2024-04-17T19:47:55.841972+0000","last_clean":"2024-04-17T19:47:55.841972+0000","last_became_active":"2024-04-17T19:47:53.930773+0000","last_became_peered":"2024-04-17T19:47:53.930773+0000","last_unstale":"2024-04-17T19:47:55.841972+0000","last_undegraded":"2024-04-17T19:47:55.841972+0000","last_fullsized":"2024-04-17T19:47:55.841972+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:10:47.583714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924520+0000","last_change":"2024-04-17T19:47:49.909404+0000","last_active":"2024-04-17T19:47:54.924520+0000","last_peered":"2024-04-17T19:47:54.924520+0000","last_clean":"2024-04-17T19:47:54.924520+0000","last_became_active":"2024-04-17T19:47:49.909154+0000","last_became_peered":"2024-04-17T19:47:49.909154+0000","last_unstale":"2024-04-17T19:47:54.924520+0000","last_undegraded":"2024-04-17T19:47:54.924520+0000","last_fullsized":"2024-04-17T19:47:54.924520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:14:39.682888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"24'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.771615+0000","last_change":"2024-04-17T19:47:48.878592+0000","last_active":"2024-04-17T19:48:15.771615+0000","last_peered":"2024-04-17T19:48:15.771615+0000","last_clean":"2024-04-17T19:48:15.771615+0000","last_became_active":"2024-04-17T19:47:48.877617+0000","last_became_peered":"2024-04-17T19:47:48.877617+0000","last_unstale":"2024-04-17T19:48:15.771615+0000","last_undegraded":"2024-04-17T19:48:15.771615+0000","last_fullsized":"2024-04-17T19:48:15.771615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:30:17.013176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048011+0000","last_change":"2024-04-17T19:47:38.277798+0000","last_active":"2024-04-17T19:47:56.048011+0000","last_peered":"2024-04-17T19:47:56.048011+0000","last_clean":"2024-04-17T19:47:56.048011+0000","last_became_active":"2024-04-17T19:47:38.274697+0000","last_became_peered":"2024-04-17T19:47:38.274697+0000","last_unstale":"2024-04-17T19:47:56.048011+0000","last_undegraded":"2024-04-17T19:47:56.048011+0000","last_fullsized":"2024-04-17T19:47:56.048011+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:36.897071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:36.897071+0000","last_clean_scrub_stamp":"2024-04-17T19:47:36.897071+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:31:35.366600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929077+0000","last_change":"2024-04-17T19:47:53.953415+0000","last_active":"2024-04-17T19:47:54.929077+0000","last_peered":"2024-04-17T19:47:54.929077+0000","last_clean":"2024-04-17T19:47:54.929077+0000","last_became_active":"2024-04-17T19:47:53.952644+0000","last_became_peered":"2024-04-17T19:47:53.952644+0000","last_unstale":"2024-04-17T19:47:54.929077+0000","last_undegraded":"2024-04-17T19:47:54.929077+0000","last_fullsized":"2024-04-17T19:47:54.929077+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:00:25.923909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048406+0000","last_change":"2024-04-17T19:47:49.901572+0000","last_active":"2024-04-17T19:47:56.048406+0000","last_peered":"2024-04-17T19:47:56.048406+0000","last_clean":"2024-04-17T19:47:56.048406+0000","last_became_active":"2024-04-17T19:47:49.900707+0000","last_became_peered":"2024-04-17T19:47:49.900707+0000","last_unstale":"2024-04-17T19:47:56.048406+0000","last_undegraded":"2024-04-17T19:47:56.048406+0000","last_fullsized":"2024-04-17T19:47:56.048406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:58:02.897564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926753+0000","last_change":"2024-04-17T19:47:48.874721+0000","last_active":"2024-04-17T19:47:54.926753+0000","last_peered":"2024-04-17T19:47:54.926753+0000","last_clean":"2024-04-17T19:47:54.926753+0000","last_became_active":"2024-04-17T19:47:48.874364+0000","last_became_peered":"2024-04-17T19:47:48.874364+0000","last_unstale":"2024-04-17T19:47:54.926753+0000","last_undegraded":"2024-04-17T19:47:54.926753+0000","last_fullsized":"2024-04-17T19:47:54.926753+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:47:08.856753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927738+0000","last_change":"2024-04-17T19:47:53.942791+0000","last_active":"2024-04-17T19:47:54.927738+0000","last_peered":"2024-04-17T19:47:54.927738+0000","last_clean":"2024-04-17T19:47:54.927738+0000","last_became_active":"2024-04-17T19:47:53.941569+0000","last_became_peered":"2024-04-17T19:47:53.941569+0000","last_unstale":"2024-04-17T19:47:54.927738+0000","last_undegraded":"2024-04-17T19:47:54.927738+0000","last_fullsized":"2024-04-17T19:47:54.927738+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:23:35.452056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832644+0000","last_change":"2024-04-17T19:47:49.895046+0000","last_active":"2024-04-17T19:47:56.832644+0000","last_peered":"2024-04-17T19:47:56.832644+0000","last_clean":"2024-04-17T19:47:56.832644+0000","last_became_active":"2024-04-17T19:47:49.894250+0000","last_became_peered":"2024-04-17T19:47:49.894250+0000","last_unstale":"2024-04-17T19:47:56.832644+0000","last_undegraded":"2024-04-17T19:47:56.832644+0000","last_fullsized":"2024-04-17T19:47:56.832644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:30:17.210135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051734+0000","last_change":"2024-04-17T19:47:48.861802+0000","last_active":"2024-04-17T19:47:55.051734+0000","last_peered":"2024-04-17T19:47:55.051734+0000","last_clean":"2024-04-17T19:47:55.051734+0000","last_became_active":"2024-04-17T19:47:48.860418+0000","last_became_peered":"2024-04-17T19:47:48.860418+0000","last_unstale":"2024-04-17T19:47:55.051734+0000","last_undegraded":"2024-04-17T19:47:55.051734+0000","last_fullsized":"2024-04-17T19:47:55.051734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:54:41.949746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842288+0000","last_change":"2024-04-17T19:47:53.917732+0000","last_active":"2024-04-17T19:47:55.842288+0000","last_peered":"2024-04-17T19:47:55.842288+0000","last_clean":"2024-04-17T19:47:55.842288+0000","last_became_active":"2024-04-17T19:47:53.917160+0000","last_became_peered":"2024-04-17T19:47:53.917160+0000","last_unstale":"2024-04-17T19:47:55.842288+0000","last_undegraded":"2024-04-17T19:47:55.842288+0000","last_fullsized":"2024-04-17T19:47:55.842288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:19:20.394166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049773+0000","last_change":"2024-04-17T19:47:49.893077+0000","last_active":"2024-04-17T19:47:56.049773+0000","last_peered":"2024-04-17T19:47:56.049773+0000","last_clean":"2024-04-17T19:47:56.049773+0000","last_became_active":"2024-04-17T19:47:49.892280+0000","last_became_peered":"2024-04-17T19:47:49.892280+0000","last_unstale":"2024-04-17T19:47:56.049773+0000","last_undegraded":"2024-04-17T19:47:56.049773+0000","last_fullsized":"2024-04-17T19:47:56.049773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:58:24.218821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926706+0000","last_change":"2024-04-17T19:47:48.849027+0000","last_active":"2024-04-17T19:47:54.926706+0000","last_peered":"2024-04-17T19:47:54.926706+0000","last_clean":"2024-04-17T19:47:54.926706+0000","last_became_active":"2024-04-17T19:47:48.848485+0000","last_became_peered":"2024-04-17T19:47:48.848485+0000","last_unstale":"2024-04-17T19:47:54.926706+0000","last_undegraded":"2024-04-17T19:47:54.926706+0000","last_fullsized":"2024-04-17T19:47:54.926706+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:58:14.548891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.831438+0000","last_change":"2024-04-17T19:47:53.946933+0000","last_active":"2024-04-17T19:47:56.831438+0000","last_peered":"2024-04-17T19:47:56.831438+0000","last_clean":"2024-04-17T19:47:56.831438+0000","last_became_active":"2024-04-17T19:47:53.943030+0000","last_became_peered":"2024-04-17T19:47:53.943030+0000","last_unstale":"2024-04-17T19:47:56.831438+0000","last_undegraded":"2024-04-17T19:47:56.831438+0000","last_fullsized":"2024-04-17T19:47:56.831438+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:37:03.489250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049281+0000","last_change":"2024-04-17T19:47:49.893137+0000","last_active":"2024-04-17T19:47:56.049281+0000","last_peered":"2024-04-17T19:47:56.049281+0000","last_clean":"2024-04-17T19:47:56.049281+0000","last_became_active":"2024-04-17T19:47:49.892379+0000","last_became_peered":"2024-04-17T19:47:49.892379+0000","last_unstale":"2024-04-17T19:47:56.049281+0000","last_undegraded":"2024-04-17T19:47:56.049281+0000","last_fullsized":"2024-04-17T19:47:56.049281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:55:08.562395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.954554+0000","last_change":"2024-04-17T19:47:48.861154+0000","last_active":"2024-04-17T19:47:56.954554+0000","last_peered":"2024-04-17T19:47:56.954554+0000","last_clean":"2024-04-17T19:47:56.954554+0000","last_became_active":"2024-04-17T19:47:48.859268+0000","last_became_peered":"2024-04-17T19:47:48.859268+0000","last_unstale":"2024-04-17T19:47:56.954554+0000","last_undegraded":"2024-04-17T19:47:56.954554+0000","last_fullsized":"2024-04-17T19:47:56.954554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:04:19.867115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955200+0000","last_change":"2024-04-17T19:47:53.950580+0000","last_active":"2024-04-17T19:47:56.955200+0000","last_peered":"2024-04-17T19:47:56.955200+0000","last_clean":"2024-04-17T19:47:56.955200+0000","last_became_active":"2024-04-17T19:47:53.943798+0000","last_became_peered":"2024-04-17T19:47:53.943798+0000","last_unstale":"2024-04-17T19:47:56.955200+0000","last_undegraded":"2024-04-17T19:47:56.955200+0000","last_fullsized":"2024-04-17T19:47:56.955200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:52:14.763799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048020+0000","last_change":"2024-04-17T19:47:49.893774+0000","last_active":"2024-04-17T19:47:56.048020+0000","last_peered":"2024-04-17T19:47:56.048020+0000","last_clean":"2024-04-17T19:47:56.048020+0000","last_became_active":"2024-04-17T19:47:49.892780+0000","last_became_peered":"2024-04-17T19:47:49.892780+0000","last_unstale":"2024-04-17T19:47:56.048020+0000","last_undegraded":"2024-04-17T19:47:56.048020+0000","last_fullsized":"2024-04-17T19:47:56.048020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:50:47.658650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"24'10","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.776912+0000","last_change":"2024-04-17T19:47:48.852970+0000","last_active":"2024-04-17T19:48:15.776912+0000","last_peered":"2024-04-17T19:48:15.776912+0000","last_clean":"2024-04-17T19:48:15.776912+0000","last_became_active":"2024-04-17T19:47:48.852676+0000","last_became_peered":"2024-04-17T19:47:48.852676+0000","last_unstale":"2024-04-17T19:48:15.776912+0000","last_undegraded":"2024-04-17T19:48:15.776912+0000","last_fullsized":"2024-04-17T19:48:15.776912+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:32:14.529596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841883+0000","last_change":"2024-04-17T19:47:53.915397+0000","last_active":"2024-04-17T19:47:55.841883+0000","last_peered":"2024-04-17T19:47:55.841883+0000","last_clean":"2024-04-17T19:47:55.841883+0000","last_became_active":"2024-04-17T19:47:53.914913+0000","last_became_peered":"2024-04-17T19:47:53.914913+0000","last_unstale":"2024-04-17T19:47:55.841883+0000","last_undegraded":"2024-04-17T19:47:55.841883+0000","last_fullsized":"2024-04-17T19:47:55.841883+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:02:01.175158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927357+0000","last_change":"2024-04-17T19:47:49.880484+0000","last_active":"2024-04-17T19:47:54.927357+0000","last_peered":"2024-04-17T19:47:54.927357+0000","last_clean":"2024-04-17T19:47:54.927357+0000","last_became_active":"2024-04-17T19:47:49.879625+0000","last_became_peered":"2024-04-17T19:47:49.879625+0000","last_unstale":"2024-04-17T19:47:54.927357+0000","last_undegraded":"2024-04-17T19:47:54.927357+0000","last_fullsized":"2024-04-17T19:47:54.927357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:50:00.611796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927417+0000","last_change":"2024-04-17T19:47:48.899160+0000","last_active":"2024-04-17T19:47:54.927417+0000","last_peered":"2024-04-17T19:47:54.927417+0000","last_clean":"2024-04-17T19:47:54.927417+0000","last_became_active":"2024-04-17T19:47:48.889831+0000","last_became_peered":"2024-04-17T19:47:48.889831+0000","last_unstale":"2024-04-17T19:47:54.927417+0000","last_undegraded":"2024-04-17T19:47:54.927417+0000","last_fullsized":"2024-04-17T19:47:54.927417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:55:20.167510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924391+0000","last_change":"2024-04-17T19:47:53.921093+0000","last_active":"2024-04-17T19:47:54.924391+0000","last_peered":"2024-04-17T19:47:54.924391+0000","last_clean":"2024-04-17T19:47:54.924391+0000","last_became_active":"2024-04-17T19:47:53.920479+0000","last_became_peered":"2024-04-17T19:47:53.920479+0000","last_unstale":"2024-04-17T19:47:54.924391+0000","last_undegraded":"2024-04-17T19:47:54.924391+0000","last_fullsized":"2024-04-17T19:47:54.924391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:08:09.569820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048418+0000","last_change":"2024-04-17T19:47:49.895275+0000","last_active":"2024-04-17T19:47:56.048418+0000","last_peered":"2024-04-17T19:47:56.048418+0000","last_clean":"2024-04-17T19:47:56.048418+0000","last_became_active":"2024-04-17T19:47:49.894534+0000","last_became_peered":"2024-04-17T19:47:49.894534+0000","last_unstale":"2024-04-17T19:47:56.048418+0000","last_undegraded":"2024-04-17T19:47:56.048418+0000","last_fullsized":"2024-04-17T19:47:56.048418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:31:40.818807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"24'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.775373+0000","last_change":"2024-04-17T19:47:48.881715+0000","last_active":"2024-04-17T19:48:15.775373+0000","last_peered":"2024-04-17T19:48:15.775373+0000","last_clean":"2024-04-17T19:48:15.775373+0000","last_became_active":"2024-04-17T19:47:48.876203+0000","last_became_peered":"2024-04-17T19:47:48.876203+0000","last_unstale":"2024-04-17T19:48:15.775373+0000","last_undegraded":"2024-04-17T19:48:15.775373+0000","last_fullsized":"2024-04-17T19:48:15.775373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:03:16.656528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924597+0000","last_change":"2024-04-17T19:47:53.905149+0000","last_active":"2024-04-17T19:47:54.924597+0000","last_peered":"2024-04-17T19:47:54.924597+0000","last_clean":"2024-04-17T19:47:54.924597+0000","last_became_active":"2024-04-17T19:47:53.904718+0000","last_became_peered":"2024-04-17T19:47:53.904718+0000","last_unstale":"2024-04-17T19:47:54.924597+0000","last_undegraded":"2024-04-17T19:47:54.924597+0000","last_fullsized":"2024-04-17T19:47:54.924597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:09:31.908752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841674+0000","last_change":"2024-04-17T19:47:49.875819+0000","last_active":"2024-04-17T19:47:55.841674+0000","last_peered":"2024-04-17T19:47:55.841674+0000","last_clean":"2024-04-17T19:47:55.841674+0000","last_became_active":"2024-04-17T19:47:49.872075+0000","last_became_peered":"2024-04-17T19:47:49.872075+0000","last_unstale":"2024-04-17T19:47:55.841674+0000","last_undegraded":"2024-04-17T19:47:55.841674+0000","last_fullsized":"2024-04-17T19:47:55.841674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:06:48.823135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"24'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.770580+0000","last_change":"2024-04-17T19:47:48.867633+0000","last_active":"2024-04-17T19:48:15.770580+0000","last_peered":"2024-04-17T19:48:15.770580+0000","last_clean":"2024-04-17T19:48:15.770580+0000","last_became_active":"2024-04-17T19:47:48.867150+0000","last_became_peered":"2024-04-17T19:47:48.867150+0000","last_unstale":"2024-04-17T19:48:15.770580+0000","last_undegraded":"2024-04-17T19:48:15.770580+0000","last_fullsized":"2024-04-17T19:48:15.770580+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:33:34.275816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832910+0000","last_change":"2024-04-17T19:47:53.925785+0000","last_active":"2024-04-17T19:47:56.832910+0000","last_peered":"2024-04-17T19:47:56.832910+0000","last_clean":"2024-04-17T19:47:56.832910+0000","last_became_active":"2024-04-17T19:47:53.924006+0000","last_became_peered":"2024-04-17T19:47:53.924006+0000","last_unstale":"2024-04-17T19:47:56.832910+0000","last_undegraded":"2024-04-17T19:47:56.832910+0000","last_fullsized":"2024-04-17T19:47:56.832910+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:46:31.965271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048194+0000","last_change":"2024-04-17T19:47:49.869829+0000","last_active":"2024-04-17T19:47:56.048194+0000","last_peered":"2024-04-17T19:47:56.048194+0000","last_clean":"2024-04-17T19:47:56.048194+0000","last_became_active":"2024-04-17T19:47:49.868933+0000","last_became_peered":"2024-04-17T19:47:49.868933+0000","last_unstale":"2024-04-17T19:47:56.048194+0000","last_undegraded":"2024-04-17T19:47:56.048194+0000","last_fullsized":"2024-04-17T19:47:56.048194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:36:08.866159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'11","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.773643+0000","last_change":"2024-04-17T19:47:48.887461+0000","last_active":"2024-04-17T19:48:15.773643+0000","last_peered":"2024-04-17T19:48:15.773643+0000","last_clean":"2024-04-17T19:48:15.773643+0000","last_became_active":"2024-04-17T19:47:48.886689+0000","last_became_peered":"2024-04-17T19:47:48.886689+0000","last_unstale":"2024-04-17T19:48:15.773643+0000","last_undegraded":"2024-04-17T19:48:15.773643+0000","last_fullsized":"2024-04-17T19:48:15.773643+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:55:35.262995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051405+0000","last_change":"2024-04-17T19:47:53.941607+0000","last_active":"2024-04-17T19:47:55.051405+0000","last_peered":"2024-04-17T19:47:55.051405+0000","last_clean":"2024-04-17T19:47:55.051405+0000","last_became_active":"2024-04-17T19:47:53.941050+0000","last_became_peered":"2024-04-17T19:47:53.941050+0000","last_unstale":"2024-04-17T19:47:55.051405+0000","last_undegraded":"2024-04-17T19:47:55.051405+0000","last_fullsized":"2024-04-17T19:47:55.051405+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:26:08.964058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.957037+0000","last_change":"2024-04-17T19:47:49.879524+0000","last_active":"2024-04-17T19:47:56.957037+0000","last_peered":"2024-04-17T19:47:56.957037+0000","last_clean":"2024-04-17T19:47:56.957037+0000","last_became_active":"2024-04-17T19:47:49.877563+0000","last_became_peered":"2024-04-17T19:47:49.877563+0000","last_unstale":"2024-04-17T19:47:56.957037+0000","last_undegraded":"2024-04-17T19:47:56.957037+0000","last_fullsized":"2024-04-17T19:47:56.957037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:34:58.539940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"24'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051731+0000","last_change":"2024-04-17T19:47:48.870100+0000","last_active":"2024-04-17T19:47:55.051731+0000","last_peered":"2024-04-17T19:47:55.051731+0000","last_clean":"2024-04-17T19:47:55.051731+0000","last_became_active":"2024-04-17T19:47:48.869797+0000","last_became_peered":"2024-04-17T19:47:48.869797+0000","last_unstale":"2024-04-17T19:47:55.051731+0000","last_undegraded":"2024-04-17T19:47:55.051731+0000","last_fullsized":"2024-04-17T19:47:55.051731+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:51:21.463081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956297+0000","last_change":"2024-04-17T19:47:53.944568+0000","last_active":"2024-04-17T19:47:56.956297+0000","last_peered":"2024-04-17T19:47:56.956297+0000","last_clean":"2024-04-17T19:47:56.956297+0000","last_became_active":"2024-04-17T19:47:53.943151+0000","last_became_peered":"2024-04-17T19:47:53.943151+0000","last_unstale":"2024-04-17T19:47:56.956297+0000","last_undegraded":"2024-04-17T19:47:56.956297+0000","last_fullsized":"2024-04-17T19:47:56.956297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:19:00.150450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842029+0000","last_change":"2024-04-17T19:47:49.902401+0000","last_active":"2024-04-17T19:47:55.842029+0000","last_peered":"2024-04-17T19:47:55.842029+0000","last_clean":"2024-04-17T19:47:55.842029+0000","last_became_active":"2024-04-17T19:47:49.902166+0000","last_became_peered":"2024-04-17T19:47:49.902166+0000","last_unstale":"2024-04-17T19:47:55.842029+0000","last_undegraded":"2024-04-17T19:47:55.842029+0000","last_fullsized":"2024-04-17T19:47:55.842029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:54:13.129748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"24'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.768195+0000","last_change":"2024-04-17T19:47:48.900800+0000","last_active":"2024-04-17T19:48:15.768195+0000","last_peered":"2024-04-17T19:48:15.768195+0000","last_clean":"2024-04-17T19:48:15.768195+0000","last_became_active":"2024-04-17T19:47:48.900188+0000","last_became_peered":"2024-04-17T19:47:48.900188+0000","last_unstale":"2024-04-17T19:48:15.768195+0000","last_undegraded":"2024-04-17T19:48:15.768195+0000","last_fullsized":"2024-04-17T19:48:15.768195+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:05:04.305871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923962+0000","last_change":"2024-04-17T19:47:53.928721+0000","last_active":"2024-04-17T19:47:54.923962+0000","last_peered":"2024-04-17T19:47:54.923962+0000","last_clean":"2024-04-17T19:47:54.923962+0000","last_became_active":"2024-04-17T19:47:53.928311+0000","last_became_peered":"2024-04-17T19:47:53.928311+0000","last_unstale":"2024-04-17T19:47:54.923962+0000","last_undegraded":"2024-04-17T19:47:54.923962+0000","last_fullsized":"2024-04-17T19:47:54.923962+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:35:28.870803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832516+0000","last_change":"2024-04-17T19:47:49.905421+0000","last_active":"2024-04-17T19:47:56.832516+0000","last_peered":"2024-04-17T19:47:56.832516+0000","last_clean":"2024-04-17T19:47:56.832516+0000","last_became_active":"2024-04-17T19:47:49.904646+0000","last_became_peered":"2024-04-17T19:47:49.904646+0000","last_unstale":"2024-04-17T19:47:56.832516+0000","last_undegraded":"2024-04-17T19:47:56.832516+0000","last_fullsized":"2024-04-17T19:47:56.832516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:02:23.936080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"24'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.770374+0000","last_change":"2024-04-17T19:47:48.895356+0000","last_active":"2024-04-17T19:48:15.770374+0000","last_peered":"2024-04-17T19:48:15.770374+0000","last_clean":"2024-04-17T19:48:15.770374+0000","last_became_active":"2024-04-17T19:47:48.889225+0000","last_became_peered":"2024-04-17T19:47:48.889225+0000","last_unstale":"2024-04-17T19:48:15.770374+0000","last_undegraded":"2024-04-17T19:48:15.770374+0000","last_fullsized":"2024-04-17T19:48:15.770374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:00:22.587917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923872+0000","last_change":"2024-04-17T19:47:53.940206+0000","last_active":"2024-04-17T19:47:54.923872+0000","last_peered":"2024-04-17T19:47:54.923872+0000","last_clean":"2024-04-17T19:47:54.923872+0000","last_became_active":"2024-04-17T19:47:53.939843+0000","last_became_peered":"2024-04-17T19:47:53.939843+0000","last_unstale":"2024-04-17T19:47:54.923872+0000","last_undegraded":"2024-04-17T19:47:54.923872+0000","last_fullsized":"2024-04-17T19:47:54.923872+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:18:55.562573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841764+0000","last_change":"2024-04-17T19:47:53.920258+0000","last_active":"2024-04-17T19:47:55.841764+0000","last_peered":"2024-04-17T19:47:55.841764+0000","last_clean":"2024-04-17T19:47:55.841764+0000","last_became_active":"2024-04-17T19:47:53.918932+0000","last_became_peered":"2024-04-17T19:47:53.918932+0000","last_unstale":"2024-04-17T19:47:55.841764+0000","last_undegraded":"2024-04-17T19:47:55.841764+0000","last_fullsized":"2024-04-17T19:47:55.841764+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:45:55.456803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.050900+0000","last_change":"2024-04-17T19:47:49.861692+0000","last_active":"2024-04-17T19:47:55.050900+0000","last_peered":"2024-04-17T19:47:55.050900+0000","last_clean":"2024-04-17T19:47:55.050900+0000","last_became_active":"2024-04-17T19:47:49.861333+0000","last_became_peered":"2024-04-17T19:47:49.861333+0000","last_unstale":"2024-04-17T19:47:55.050900+0000","last_undegraded":"2024-04-17T19:47:55.050900+0000","last_fullsized":"2024-04-17T19:47:55.050900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:47:13.317679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832300+0000","last_change":"2024-04-17T19:47:48.873355+0000","last_active":"2024-04-17T19:47:56.832300+0000","last_peered":"2024-04-17T19:47:56.832300+0000","last_clean":"2024-04-17T19:47:56.832300+0000","last_became_active":"2024-04-17T19:47:48.872427+0000","last_became_peered":"2024-04-17T19:47:48.872427+0000","last_unstale":"2024-04-17T19:47:56.832300+0000","last_undegraded":"2024-04-17T19:47:56.832300+0000","last_fullsized":"2024-04-17T19:47:56.832300+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:32:41.792605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929095+0000","last_change":"2024-04-17T19:47:53.952602+0000","last_active":"2024-04-17T19:47:54.929095+0000","last_peered":"2024-04-17T19:47:54.929095+0000","last_clean":"2024-04-17T19:47:54.929095+0000","last_became_active":"2024-04-17T19:47:53.951750+0000","last_became_peered":"2024-04-17T19:47:53.951750+0000","last_unstale":"2024-04-17T19:47:54.929095+0000","last_undegraded":"2024-04-17T19:47:54.929095+0000","last_fullsized":"2024-04-17T19:47:54.929095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:13:08.485238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923863+0000","last_change":"2024-04-17T19:47:49.862071+0000","last_active":"2024-04-17T19:47:54.923863+0000","last_peered":"2024-04-17T19:47:54.923863+0000","last_clean":"2024-04-17T19:47:54.923863+0000","last_became_active":"2024-04-17T19:47:49.860809+0000","last_became_peered":"2024-04-17T19:47:49.860809+0000","last_unstale":"2024-04-17T19:47:54.923863+0000","last_undegraded":"2024-04-17T19:47:54.923863+0000","last_fullsized":"2024-04-17T19:47:54.923863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:42:34.451839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924173+0000","last_change":"2024-04-17T19:47:48.867855+0000","last_active":"2024-04-17T19:47:54.924173+0000","last_peered":"2024-04-17T19:47:54.924173+0000","last_clean":"2024-04-17T19:47:54.924173+0000","last_became_active":"2024-04-17T19:47:48.867532+0000","last_became_peered":"2024-04-17T19:47:48.867532+0000","last_unstale":"2024-04-17T19:47:54.924173+0000","last_undegraded":"2024-04-17T19:47:54.924173+0000","last_fullsized":"2024-04-17T19:47:54.924173+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:21:26.903245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927964+0000","last_change":"2024-04-17T19:47:48.894946+0000","last_active":"2024-04-17T19:47:54.927964+0000","last_peered":"2024-04-17T19:47:54.927964+0000","last_clean":"2024-04-17T19:47:54.927964+0000","last_became_active":"2024-04-17T19:47:48.893407+0000","last_became_peered":"2024-04-17T19:47:48.893407+0000","last_unstale":"2024-04-17T19:47:54.927964+0000","last_undegraded":"2024-04-17T19:47:54.927964+0000","last_fullsized":"2024-04-17T19:47:54.927964+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:09:56.284638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842374+0000","last_change":"2024-04-17T19:47:49.859985+0000","last_active":"2024-04-17T19:47:55.842374+0000","last_peered":"2024-04-17T19:47:55.842374+0000","last_clean":"2024-04-17T19:47:55.842374+0000","last_became_active":"2024-04-17T19:47:49.859635+0000","last_became_peered":"2024-04-17T19:47:49.859635+0000","last_unstale":"2024-04-17T19:47:55.842374+0000","last_undegraded":"2024-04-17T19:47:55.842374+0000","last_fullsized":"2024-04-17T19:47:55.842374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:04:14.834218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926673+0000","last_change":"2024-04-17T19:47:53.925444+0000","last_active":"2024-04-17T19:47:54.926673+0000","last_peered":"2024-04-17T19:47:54.926673+0000","last_clean":"2024-04-17T19:47:54.926673+0000","last_became_active":"2024-04-17T19:47:53.924805+0000","last_became_peered":"2024-04-17T19:47:53.924805+0000","last_unstale":"2024-04-17T19:47:54.926673+0000","last_undegraded":"2024-04-17T19:47:54.926673+0000","last_fullsized":"2024-04-17T19:47:54.926673+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:19:24.022368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"24'5","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.775537+0000","last_change":"2024-04-17T19:47:48.867176+0000","last_active":"2024-04-17T19:48:15.775537+0000","last_peered":"2024-04-17T19:48:15.775537+0000","last_clean":"2024-04-17T19:48:15.775537+0000","last_became_active":"2024-04-17T19:47:48.866817+0000","last_became_peered":"2024-04-17T19:47:48.866817+0000","last_unstale":"2024-04-17T19:48:15.775537+0000","last_undegraded":"2024-04-17T19:48:15.775537+0000","last_fullsized":"2024-04-17T19:48:15.775537+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:44:30.842583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955878+0000","last_change":"2024-04-17T19:47:49.858924+0000","last_active":"2024-04-17T19:47:56.955878+0000","last_peered":"2024-04-17T19:47:56.955878+0000","last_clean":"2024-04-17T19:47:56.955878+0000","last_became_active":"2024-04-17T19:47:49.857820+0000","last_became_peered":"2024-04-17T19:47:49.857820+0000","last_unstale":"2024-04-17T19:47:56.955878+0000","last_undegraded":"2024-04-17T19:47:56.955878+0000","last_fullsized":"2024-04-17T19:47:56.955878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:09:28.706054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924516+0000","last_change":"2024-04-17T19:47:53.933337+0000","last_active":"2024-04-17T19:47:54.924516+0000","last_peered":"2024-04-17T19:47:54.924516+0000","last_clean":"2024-04-17T19:47:54.924516+0000","last_became_active":"2024-04-17T19:47:53.931623+0000","last_became_peered":"2024-04-17T19:47:53.931623+0000","last_unstale":"2024-04-17T19:47:54.924516+0000","last_undegraded":"2024-04-17T19:47:54.924516+0000","last_fullsized":"2024-04-17T19:47:54.924516+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:57:37.600957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927056+0000","last_change":"2024-04-17T19:47:49.881093+0000","last_active":"2024-04-17T19:47:54.927056+0000","last_peered":"2024-04-17T19:47:54.927056+0000","last_clean":"2024-04-17T19:47:54.927056+0000","last_became_active":"2024-04-17T19:47:49.880210+0000","last_became_peered":"2024-04-17T19:47:49.880210+0000","last_unstale":"2024-04-17T19:47:54.927056+0000","last_undegraded":"2024-04-17T19:47:54.927056+0000","last_fullsized":"2024-04-17T19:47:54.927056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:12:15.522651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926335+0000","last_change":"2024-04-17T19:47:49.886936+0000","last_active":"2024-04-17T19:47:54.926335+0000","last_peered":"2024-04-17T19:47:54.926335+0000","last_clean":"2024-04-17T19:47:54.926335+0000","last_became_active":"2024-04-17T19:47:49.886580+0000","last_became_peered":"2024-04-17T19:47:49.886580+0000","last_unstale":"2024-04-17T19:47:54.926335+0000","last_undegraded":"2024-04-17T19:47:54.926335+0000","last_fullsized":"2024-04-17T19:47:54.926335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:00:44.599463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832155+0000","last_change":"2024-04-17T19:47:49.905445+0000","last_active":"2024-04-17T19:47:56.832155+0000","last_peered":"2024-04-17T19:47:56.832155+0000","last_clean":"2024-04-17T19:47:56.832155+0000","last_became_active":"2024-04-17T19:47:49.896510+0000","last_became_peered":"2024-04-17T19:47:49.896510+0000","last_unstale":"2024-04-17T19:47:56.832155+0000","last_undegraded":"2024-04-17T19:47:56.832155+0000","last_fullsized":"2024-04-17T19:47:56.832155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:23:27.401861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842320+0000","last_change":"2024-04-17T19:47:49.850490+0000","last_active":"2024-04-17T19:47:55.842320+0000","last_peered":"2024-04-17T19:47:55.842320+0000","last_clean":"2024-04-17T19:47:55.842320+0000","last_became_active":"2024-04-17T19:47:49.849344+0000","last_became_peered":"2024-04-17T19:47:49.849344+0000","last_unstale":"2024-04-17T19:47:55.842320+0000","last_undegraded":"2024-04-17T19:47:55.842320+0000","last_fullsized":"2024-04-17T19:47:55.842320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:44:32.231058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924753+0000","last_change":"2024-04-17T19:47:49.857737+0000","last_active":"2024-04-17T19:47:54.924753+0000","last_peered":"2024-04-17T19:47:54.924753+0000","last_clean":"2024-04-17T19:47:54.924753+0000","last_became_active":"2024-04-17T19:47:49.857476+0000","last_became_peered":"2024-04-17T19:47:49.857476+0000","last_unstale":"2024-04-17T19:47:54.924753+0000","last_undegraded":"2024-04-17T19:47:54.924753+0000","last_fullsized":"2024-04-17T19:47:54.924753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:50:13.070628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051526+0000","last_change":"2024-04-17T19:47:49.864985+0000","last_active":"2024-04-17T19:47:55.051526+0000","last_peered":"2024-04-17T19:47:55.051526+0000","last_clean":"2024-04-17T19:47:55.051526+0000","last_became_active":"2024-04-17T19:47:49.864329+0000","last_became_peered":"2024-04-17T19:47:49.864329+0000","last_unstale":"2024-04-17T19:47:55.051526+0000","last_undegraded":"2024-04-17T19:47:55.051526+0000","last_fullsized":"2024-04-17T19:47:55.051526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:43:57.448269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929577+0000","last_change":"2024-04-17T19:47:49.881785+0000","last_active":"2024-04-17T19:47:54.929577+0000","last_peered":"2024-04-17T19:47:54.929577+0000","last_clean":"2024-04-17T19:47:54.929577+0000","last_became_active":"2024-04-17T19:47:49.880939+0000","last_became_peered":"2024-04-17T19:47:49.880939+0000","last_unstale":"2024-04-17T19:47:54.929577+0000","last_undegraded":"2024-04-17T19:47:54.929577+0000","last_fullsized":"2024-04-17T19:47:54.929577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:49:07.194643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924461+0000","last_change":"2024-04-17T19:47:49.884944+0000","last_active":"2024-04-17T19:47:54.924461+0000","last_peered":"2024-04-17T19:47:54.924461+0000","last_clean":"2024-04-17T19:47:54.924461+0000","last_became_active":"2024-04-17T19:47:49.884559+0000","last_became_peered":"2024-04-17T19:47:49.884559+0000","last_unstale":"2024-04-17T19:47:54.924461+0000","last_undegraded":"2024-04-17T19:47:54.924461+0000","last_fullsized":"2024-04-17T19:47:54.924461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:00.485472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.831772+0000","last_change":"2024-04-17T19:47:49.891230+0000","last_active":"2024-04-17T19:47:56.831772+0000","last_peered":"2024-04-17T19:47:56.831772+0000","last_clean":"2024-04-17T19:47:56.831772+0000","last_became_active":"2024-04-17T19:47:49.886001+0000","last_became_peered":"2024-04-17T19:47:49.886001+0000","last_unstale":"2024-04-17T19:47:56.831772+0000","last_undegraded":"2024-04-17T19:47:56.831772+0000","last_fullsized":"2024-04-17T19:47:56.831772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:00.485472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923958+0000","last_change":"2024-04-17T19:47:49.859382+0000","last_active":"2024-04-17T19:47:54.923958+0000","last_peered":"2024-04-17T19:47:54.923958+0000","last_clean":"2024-04-17T19:47:54.923958+0000","last_became_active":"2024-04-17T19:47:49.859170+0000","last_became_peered":"2024-04-17T19:47:49.859170+0000","last_unstale":"2024-04-17T19:47:54.923958+0000","last_undegraded":"2024-04-17T19:47:54.923958+0000","last_fullsized":"2024-04-17T19:47:54.923958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:58:51.111053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832143+0000","last_change":"2024-04-17T19:47:49.909188+0000","last_active":"2024-04-17T19:47:56.832143+0000","last_peered":"2024-04-17T19:47:56.832143+0000","last_clean":"2024-04-17T19:47:56.832143+0000","last_became_active":"2024-04-17T19:47:49.908432+0000","last_became_peered":"2024-04-17T19:47:49.908432+0000","last_unstale":"2024-04-17T19:47:56.832143+0000","last_undegraded":"2024-04-17T19:47:56.832143+0000","last_fullsized":"2024-04-17T19:47:56.832143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:16:21.209634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842710+0000","last_change":"2024-04-17T19:47:49.876406+0000","last_active":"2024-04-17T19:47:55.842710+0000","last_peered":"2024-04-17T19:47:55.842710+0000","last_clean":"2024-04-17T19:47:55.842710+0000","last_became_active":"2024-04-17T19:47:49.876132+0000","last_became_peered":"2024-04-17T19:47:49.876132+0000","last_unstale":"2024-04-17T19:47:55.842710+0000","last_undegraded":"2024-04-17T19:47:55.842710+0000","last_fullsized":"2024-04-17T19:47:55.842710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:51:20.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.928888+0000","last_change":"2024-04-17T19:47:49.896244+0000","last_active":"2024-04-17T19:47:54.928888+0000","last_peered":"2024-04-17T19:47:54.928888+0000","last_clean":"2024-04-17T19:47:54.928888+0000","last_became_active":"2024-04-17T19:47:49.895380+0000","last_became_peered":"2024-04-17T19:47:49.895380+0000","last_unstale":"2024-04-17T19:47:54.928888+0000","last_undegraded":"2024-04-17T19:47:54.928888+0000","last_fullsized":"2024-04-17T19:47:54.928888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:04:43.091307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049138+0000","last_change":"2024-04-17T19:47:49.870551+0000","last_active":"2024-04-17T19:47:56.049138+0000","last_peered":"2024-04-17T19:47:56.049138+0000","last_clean":"2024-04-17T19:47:56.049138+0000","last_became_active":"2024-04-17T19:47:49.869315+0000","last_became_peered":"2024-04-17T19:47:49.869315+0000","last_unstale":"2024-04-17T19:47:56.049138+0000","last_undegraded":"2024-04-17T19:47:56.049138+0000","last_fullsized":"2024-04-17T19:47:56.049138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:06.341123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926618+0000","last_change":"2024-04-17T19:47:49.880060+0000","last_active":"2024-04-17T19:47:54.926618+0000","last_peered":"2024-04-17T19:47:54.926618+0000","last_clean":"2024-04-17T19:47:54.926618+0000","last_became_active":"2024-04-17T19:47:49.879711+0000","last_became_peered":"2024-04-17T19:47:49.879711+0000","last_unstale":"2024-04-17T19:47:54.926618+0000","last_undegraded":"2024-04-17T19:47:54.926618+0000","last_fullsized":"2024-04-17T19:47:54.926618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:41:25.628870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.833444+0000","last_change":"2024-04-17T19:47:49.895452+0000","last_active":"2024-04-17T19:47:56.833444+0000","last_peered":"2024-04-17T19:47:56.833444+0000","last_clean":"2024-04-17T19:47:56.833444+0000","last_became_active":"2024-04-17T19:47:49.894684+0000","last_became_peered":"2024-04-17T19:47:49.894684+0000","last_unstale":"2024-04-17T19:47:56.833444+0000","last_undegraded":"2024-04-17T19:47:56.833444+0000","last_fullsized":"2024-04-17T19:47:56.833444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:15:55.224238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.831437+0000","last_change":"2024-04-17T19:47:49.886902+0000","last_active":"2024-04-17T19:47:56.831437+0000","last_peered":"2024-04-17T19:47:56.831437+0000","last_clean":"2024-04-17T19:47:56.831437+0000","last_became_active":"2024-04-17T19:47:49.885451+0000","last_became_peered":"2024-04-17T19:47:49.885451+0000","last_unstale":"2024-04-17T19:47:56.831437+0000","last_undegraded":"2024-04-17T19:47:56.831437+0000","last_fullsized":"2024-04-17T19:47:56.831437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:12:09.196610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.050989+0000","last_change":"2024-04-17T19:47:49.858908+0000","last_active":"2024-04-17T19:47:55.050989+0000","last_peered":"2024-04-17T19:47:55.050989+0000","last_clean":"2024-04-17T19:47:55.050989+0000","last_became_active":"2024-04-17T19:47:49.858457+0000","last_became_peered":"2024-04-17T19:47:49.858457+0000","last_unstale":"2024-04-17T19:47:55.050989+0000","last_undegraded":"2024-04-17T19:47:55.050989+0000","last_fullsized":"2024-04-17T19:47:55.050989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:09.467859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841618+0000","last_change":"2024-04-17T19:47:49.856371+0000","last_active":"2024-04-17T19:47:55.841618+0000","last_peered":"2024-04-17T19:47:55.841618+0000","last_clean":"2024-04-17T19:47:55.841618+0000","last_became_active":"2024-04-17T19:47:49.856062+0000","last_became_peered":"2024-04-17T19:47:49.856062+0000","last_unstale":"2024-04-17T19:47:55.841618+0000","last_undegraded":"2024-04-17T19:47:55.841618+0000","last_fullsized":"2024-04-17T19:47:55.841618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:39:54.502619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924769+0000","last_change":"2024-04-17T19:47:49.873350+0000","last_active":"2024-04-17T19:47:54.924769+0000","last_peered":"2024-04-17T19:47:54.924769+0000","last_clean":"2024-04-17T19:47:54.924769+0000","last_became_active":"2024-04-17T19:47:49.873019+0000","last_became_peered":"2024-04-17T19:47:49.873019+0000","last_unstale":"2024-04-17T19:47:54.924769+0000","last_undegraded":"2024-04-17T19:47:54.924769+0000","last_fullsized":"2024-04-17T19:47:54.924769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:06:58.641727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956704+0000","last_change":"2024-04-17T19:47:49.891551+0000","last_active":"2024-04-17T19:47:56.956704+0000","last_peered":"2024-04-17T19:47:56.956704+0000","last_clean":"2024-04-17T19:47:56.956704+0000","last_became_active":"2024-04-17T19:47:49.890132+0000","last_became_peered":"2024-04-17T19:47:49.890132+0000","last_unstale":"2024-04-17T19:47:56.956704+0000","last_undegraded":"2024-04-17T19:47:56.956704+0000","last_fullsized":"2024-04-17T19:47:56.956704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:26:22.971489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.928586+0000","last_change":"2024-04-17T19:47:49.903854+0000","last_active":"2024-04-17T19:47:54.928586+0000","last_peered":"2024-04-17T19:47:54.928586+0000","last_clean":"2024-04-17T19:47:54.928586+0000","last_became_active":"2024-04-17T19:47:49.903178+0000","last_became_peered":"2024-04-17T19:47:49.903178+0000","last_unstale":"2024-04-17T19:47:54.928586+0000","last_undegraded":"2024-04-17T19:47:54.928586+0000","last_fullsized":"2024-04-17T19:47:54.928586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:24:52.699532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929394+0000","last_change":"2024-04-17T19:47:49.900848+0000","last_active":"2024-04-17T19:47:54.929394+0000","last_peered":"2024-04-17T19:47:54.929394+0000","last_clean":"2024-04-17T19:47:54.929394+0000","last_became_active":"2024-04-17T19:47:49.900052+0000","last_became_peered":"2024-04-17T19:47:49.900052+0000","last_unstale":"2024-04-17T19:47:54.929394+0000","last_undegraded":"2024-04-17T19:47:54.929394+0000","last_fullsized":"2024-04-17T19:47:54.929394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:28:41.115173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049711+0000","last_change":"2024-04-17T19:47:49.906531+0000","last_active":"2024-04-17T19:47:56.049711+0000","last_peered":"2024-04-17T19:47:56.049711+0000","last_clean":"2024-04-17T19:47:56.049711+0000","last_became_active":"2024-04-17T19:47:49.905217+0000","last_became_peered":"2024-04-17T19:47:49.905217+0000","last_unstale":"2024-04-17T19:47:56.049711+0000","last_undegraded":"2024-04-17T19:47:56.049711+0000","last_fullsized":"2024-04-17T19:47:56.049711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:22.078473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923726+0000","last_change":"2024-04-17T19:47:49.864577+0000","last_active":"2024-04-17T19:47:54.923726+0000","last_peered":"2024-04-17T19:47:54.923726+0000","last_clean":"2024-04-17T19:47:54.923726+0000","last_became_active":"2024-04-17T19:47:49.864251+0000","last_became_peered":"2024-04-17T19:47:49.864251+0000","last_unstale":"2024-04-17T19:47:54.923726+0000","last_undegraded":"2024-04-17T19:47:54.923726+0000","last_fullsized":"2024-04-17T19:47:54.923726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:32:10.329543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.047770+0000","last_change":"2024-04-17T19:47:49.869245+0000","last_active":"2024-04-17T19:47:56.047770+0000","last_peered":"2024-04-17T19:47:56.047770+0000","last_clean":"2024-04-17T19:47:56.047770+0000","last_became_active":"2024-04-17T19:47:49.868156+0000","last_became_peered":"2024-04-17T19:47:49.868156+0000","last_unstale":"2024-04-17T19:47:56.047770+0000","last_undegraded":"2024-04-17T19:47:56.047770+0000","last_fullsized":"2024-04-17T19:47:56.047770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:58:39.814933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.831810+0000","last_change":"2024-04-17T19:47:49.907573+0000","last_active":"2024-04-17T19:47:56.831810+0000","last_peered":"2024-04-17T19:47:56.831810+0000","last_clean":"2024-04-17T19:47:56.831810+0000","last_became_active":"2024-04-17T19:47:49.897308+0000","last_became_peered":"2024-04-17T19:47:49.897308+0000","last_unstale":"2024-04-17T19:47:56.831810+0000","last_undegraded":"2024-04-17T19:47:56.831810+0000","last_fullsized":"2024-04-17T19:47:56.831810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:02:03.101317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.047866+0000","last_change":"2024-04-17T19:47:49.876052+0000","last_active":"2024-04-17T19:47:56.047866+0000","last_peered":"2024-04-17T19:47:56.047866+0000","last_clean":"2024-04-17T19:47:56.047866+0000","last_became_active":"2024-04-17T19:47:49.875008+0000","last_became_peered":"2024-04-17T19:47:49.875008+0000","last_unstale":"2024-04-17T19:47:56.047866+0000","last_undegraded":"2024-04-17T19:47:56.047866+0000","last_fullsized":"2024-04-17T19:47:56.047866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:36:14.486150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832276+0000","last_change":"2024-04-17T19:47:49.906840+0000","last_active":"2024-04-17T19:47:56.832276+0000","last_peered":"2024-04-17T19:47:56.832276+0000","last_clean":"2024-04-17T19:47:56.832276+0000","last_became_active":"2024-04-17T19:47:49.905703+0000","last_became_peered":"2024-04-17T19:47:49.905703+0000","last_unstale":"2024-04-17T19:47:56.832276+0000","last_undegraded":"2024-04-17T19:47:56.832276+0000","last_fullsized":"2024-04-17T19:47:56.832276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:28:26.686472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841783+0000","last_change":"2024-04-17T19:47:49.869425+0000","last_active":"2024-04-17T19:47:55.841783+0000","last_peered":"2024-04-17T19:47:55.841783+0000","last_clean":"2024-04-17T19:47:55.841783+0000","last_became_active":"2024-04-17T19:47:49.868758+0000","last_became_peered":"2024-04-17T19:47:49.868758+0000","last_unstale":"2024-04-17T19:47:55.841783+0000","last_undegraded":"2024-04-17T19:47:55.841783+0000","last_fullsized":"2024-04-17T19:47:55.841783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:57:27.214150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051778+0000","last_change":"2024-04-17T19:47:49.856309+0000","last_active":"2024-04-17T19:47:55.051778+0000","last_peered":"2024-04-17T19:47:55.051778+0000","last_clean":"2024-04-17T19:47:55.051778+0000","last_became_active":"2024-04-17T19:47:49.856026+0000","last_became_peered":"2024-04-17T19:47:49.856026+0000","last_unstale":"2024-04-17T19:47:55.051778+0000","last_undegraded":"2024-04-17T19:47:55.051778+0000","last_fullsized":"2024-04-17T19:47:55.051778+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:51:57.963637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049403+0000","last_change":"2024-04-17T19:47:49.890343+0000","last_active":"2024-04-17T19:47:56.049403+0000","last_peered":"2024-04-17T19:47:56.049403+0000","last_clean":"2024-04-17T19:47:56.049403+0000","last_became_active":"2024-04-17T19:47:49.880566+0000","last_became_peered":"2024-04-17T19:47:49.880566+0000","last_unstale":"2024-04-17T19:47:56.049403+0000","last_undegraded":"2024-04-17T19:47:56.049403+0000","last_fullsized":"2024-04-17T19:47:56.049403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:19:29.359024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6046,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":12092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T19:48:19.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T19:48:19.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.265+0000 7fce57800640 1 Processor -- start 2024-04-17T19:48:19.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.265+0000 7fce57800640 1 -- start start 2024-04-17T19:48:19.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.269+0000 7fce57800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce50159e60 0x7fce500cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:19.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.269+0000 7fce57800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fce50153d20 0x7fce50071940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:19.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.269+0000 7fce57800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce500c4930 con 0x7fce50071e80 2024-04-17T19:48:19.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.269+0000 7fce57800640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fce500d0320 con 0x7fce50153d20 2024-04-17T19:48:19.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.269+0000 7fce57800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fce500cf620 con 0x7fce50159e60 2024-04-17T19:48:19.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.269+0000 7fce4fe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fce50153d20 0x7fce50071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:19.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.269+0000 7fce4fe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fce50153d20 0x7fce50071940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58158/0 (socket says 172.21.15.29:58158) 2024-04-17T19:48:19.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.269+0000 7fce4fe00640 1 -- 172.21.15.29:0/1910823034 learned_addr learned my addr 172.21.15.29:0/1910823034 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:19.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.269+0000 7fce54e00640 1 --2- 172.21.15.29:0/1910823034 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce50159e60 0x7fce500cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:19.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.269+0000 7fce4fe00640 1 -- 172.21.15.29:0/1910823034 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce50159e60 msgr2=0x7fce500cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:19.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.269+0000 7fce4fe00640 1 --2- 172.21.15.29:0/1910823034 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce50159e60 0x7fce500cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.269+0000 7fce4fe00640 1 -- 172.21.15.29:0/1910823034 >> v1:172.21.15.37:6789/0 conn(0x7fce50071e80 legacy=0x7fce500c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:19.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.269+0000 7fce4fe00640 1 -- 172.21.15.29:0/1910823034 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce500c5190 con 0x7fce50153d20 2024-04-17T19:48:19.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.273+0000 7fce4fe00640 1 --2- 172.21.15.29:0/1910823034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce50153d20 0x7fce50071940 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009870 tx=0x7fce3c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a43e2c7b4abe8237 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:19.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.273+0000 7fce4f400640 1 -- 172.21.15.29:0/1910823034 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c0b0070 con 0x7fce50153d20 2024-04-17T19:48:19.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.273+0000 7fce4f400640 1 -- 172.21.15.29:0/1910823034 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce3c0ab600 con 0x7fce50153d20 2024-04-17T19:48:19.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.273+0000 7fce4f400640 1 -- 172.21.15.29:0/1910823034 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c0ab9d0 con 0x7fce50153d20 2024-04-17T19:48:19.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.273+0000 7fce57800640 1 -- 172.21.15.29:0/1910823034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce50153d20 msgr2=0x7fce50071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:19.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.273+0000 7fce57800640 1 --2- 172.21.15.29:0/1910823034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce50153d20 0x7fce50071940 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009870 tx=0x7fce3c01c560 comp rx=0 tx=0).stop 2024-04-17T19:48:19.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.273+0000 7fce57800640 1 -- 172.21.15.29:0/1910823034 shutdown_connections 2024-04-17T19:48:19.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.273+0000 7fce57800640 1 --2- 172.21.15.29:0/1910823034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce50153d20 0x7fce50071940 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.273+0000 7fce57800640 1 --2- 172.21.15.29:0/1910823034 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce50159e60 0x7fce500cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.273+0000 7fce57800640 1 -- 172.21.15.29:0/1910823034 >> 172.21.15.29:0/1910823034 conn(0x7fce50067eb0 msgr2=0x7fce50070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:19.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.273+0000 7fce57800640 1 -- 172.21.15.29:0/1910823034 shutdown_connections 2024-04-17T19:48:19.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.273+0000 7fce57800640 1 -- 172.21.15.29:0/1910823034 wait complete. 2024-04-17T19:48:19.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce57800640 1 Processor -- start 2024-04-17T19:48:19.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce57800640 1 -- start start 2024-04-17T19:48:19.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce57800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce50071e80 0x7fce5015b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:19.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce57800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce50153d20 0x7fce5015c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:19.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce57800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce50159e60 0x7fce50148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:19.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce57800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fce500cbd10 con 0x7fce50159e60 2024-04-17T19:48:19.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce57800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fce500cbb90 con 0x7fce50153d20 2024-04-17T19:48:19.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce57800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fce500c4510 con 0x7fce50071e80 2024-04-17T19:48:19.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce55800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce50159e60 0x7fce50148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:19.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce55800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce50159e60 0x7fce50148d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58160/0 (socket says 172.21.15.29:58160) 2024-04-17T19:48:19.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce55800640 1 -- 172.21.15.29:0/1995401228 learned_addr learned my addr 172.21.15.29:0/1995401228 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:19.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce54e00640 1 --2- 172.21.15.29:0/1995401228 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce50071e80 0x7fce5015b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:19.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce4fe00640 1 --2- 172.21.15.29:0/1995401228 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce50153d20 0x7fce5015c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:19.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce55800640 1 -- 172.21.15.29:0/1995401228 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce50071e80 msgr2=0x7fce5015b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:19.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce55800640 1 --2- 172.21.15.29:0/1995401228 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce50071e80 0x7fce5015b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce55800640 1 -- 172.21.15.29:0/1995401228 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce50153d20 msgr2=0x7fce5015c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:19.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce55800640 1 --2- 172.21.15.29:0/1995401228 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce50153d20 0x7fce5015c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce55800640 1 -- 172.21.15.29:0/1995401228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5015c9b0 con 0x7fce50159e60 2024-04-17T19:48:19.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce55800640 1 --2- 172.21.15.29:0/1995401228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce50159e60 0x7fce50148d50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fce4800c730 tx=0x7fce4800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:19.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce4cc00640 1 -- 172.21.15.29:0/1995401228 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce48003b80 con 0x7fce50159e60 2024-04-17T19:48:19.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce4cc00640 1 -- 172.21.15.29:0/1995401228 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce48003d20 con 0x7fce50159e60 2024-04-17T19:48:19.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce57800640 1 -- 172.21.15.29:0/1995401228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce5014db70 con 0x7fce50159e60 2024-04-17T19:48:19.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce4cc00640 1 -- 172.21.15.29:0/1995401228 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4800aa30 con 0x7fce50159e60 2024-04-17T19:48:19.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.277+0000 7fce57800640 1 -- 172.21.15.29:0/1995401228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce50152800 con 0x7fce50159e60 2024-04-17T19:48:19.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.281+0000 7fce57800640 1 -- 172.21.15.29:0/1995401228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce50069b90 con 0x7fce50159e60 2024-04-17T19:48:19.269 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.281+0000 7fce4cc00640 1 -- 172.21.15.29:0/1995401228 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fce4800abd0 con 0x7fce50159e60 2024-04-17T19:48:19.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.281+0000 7fce4cc00640 1 --2- 172.21.15.29:0/1995401228 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fce30068380 0x7fce3006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:19.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.281+0000 7fce4cc00640 1 -- 172.21.15.29:0/1995401228 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fce48086a80 con 0x7fce50159e60 2024-04-17T19:48:19.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.281+0000 7fce54e00640 1 --2- 172.21.15.29:0/1995401228 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fce30068380 0x7fce3006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:19.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.281+0000 7fce54e00640 1 --2- 172.21.15.29:0/1995401228 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fce30068380 0x7fce3006a890 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fce44009ac0 tx=0x7fce44009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:19.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.285+0000 7fce4cc00640 1 -- 172.21.15.29:0/1995401228 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fce48088e20 con 0x7fce50159e60 2024-04-17T19:48:19.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.449+0000 7fce57800640 1 -- 172.21.15.29:0/1995401228 --> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fce500cce20 con 0x7fce30068380 2024-04-17T19:48:19.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.473+0000 7fce4cc00640 1 -- 172.21.15.29:0/1995401228 <== mgr.4105 v2:172.21.15.37:6832/1588862286 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332572 (secure 0 0 0) 0x7fce500cce20 con 0x7fce30068380 2024-04-17T19:48:19.463 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:19.465 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-04-17T19:48:19.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.477+0000 7fce57800640 1 -- 172.21.15.29:0/1995401228 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fce30068380 msgr2=0x7fce3006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:19.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.477+0000 7fce57800640 1 --2- 172.21.15.29:0/1995401228 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fce30068380 0x7fce3006a890 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fce44009ac0 tx=0x7fce44009190 comp rx=0 tx=0).stop 2024-04-17T19:48:19.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.477+0000 7fce57800640 1 -- 172.21.15.29:0/1995401228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce50159e60 msgr2=0x7fce50148d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:19.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.477+0000 7fce57800640 1 --2- 172.21.15.29:0/1995401228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce50159e60 0x7fce50148d50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fce4800c730 tx=0x7fce4800cbf0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.477+0000 7fce57800640 1 -- 172.21.15.29:0/1995401228 shutdown_connections 2024-04-17T19:48:19.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.477+0000 7fce57800640 1 --2- 172.21.15.29:0/1995401228 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fce30068380 0x7fce3006a890 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.477+0000 7fce57800640 1 --2- 172.21.15.29:0/1995401228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce50159e60 0x7fce50148d50 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.477+0000 7fce57800640 1 --2- 172.21.15.29:0/1995401228 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce50153d20 0x7fce5015c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.477+0000 7fce57800640 1 --2- 172.21.15.29:0/1995401228 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce50071e80 0x7fce5015b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.477+0000 7fce57800640 1 -- 172.21.15.29:0/1995401228 >> 172.21.15.29:0/1995401228 conn(0x7fce50067eb0 msgr2=0x7fce50159030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:19.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.477+0000 7fce57800640 1 -- 172.21.15.29:0/1995401228 shutdown_connections 2024-04-17T19:48:19.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.477+0000 7fce57800640 1 -- 172.21.15.29:0/1995401228 wait complete. 2024-04-17T19:48:19.485 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-04-17T19:48:18.637552+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596414,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":246,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":193,"ondisk_log_size":193,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240168,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754734552,"statfs":{"total":773094113280,"available":772848181248,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1547,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":12,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006087"},"pg_stats":[{"pgid":"2.1f","version":"23'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842100+0000","last_change":"2024-04-17T19:47:48.885235+0000","last_active":"2024-04-17T19:47:55.842100+0000","last_peered":"2024-04-17T19:47:55.842100+0000","last_clean":"2024-04-17T19:47:55.842100+0000","last_became_active":"2024-04-17T19:47:48.884844+0000","last_became_peered":"2024-04-17T19:47:48.884844+0000","last_unstale":"2024-04-17T19:47:55.842100+0000","last_undegraded":"2024-04-17T19:47:55.842100+0000","last_fullsized":"2024-04-17T19:47:55.842100+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:01:52.854763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924264+0000","last_change":"2024-04-17T19:47:49.872314+0000","last_active":"2024-04-17T19:47:54.924264+0000","last_peered":"2024-04-17T19:47:54.924264+0000","last_clean":"2024-04-17T19:47:54.924264+0000","last_became_active":"2024-04-17T19:47:49.871990+0000","last_became_peered":"2024-04-17T19:47:49.871990+0000","last_unstale":"2024-04-17T19:47:54.924264+0000","last_undegraded":"2024-04-17T19:47:54.924264+0000","last_fullsized":"2024-04-17T19:47:54.924264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:12:09.196610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923798+0000","last_change":"2024-04-17T19:47:53.929840+0000","last_active":"2024-04-17T19:47:54.923798+0000","last_peered":"2024-04-17T19:47:54.923798+0000","last_clean":"2024-04-17T19:47:54.923798+0000","last_became_active":"2024-04-17T19:47:53.928272+0000","last_became_peered":"2024-04-17T19:47:53.928272+0000","last_unstale":"2024-04-17T19:47:54.923798+0000","last_undegraded":"2024-04-17T19:47:54.923798+0000","last_fullsized":"2024-04-17T19:47:54.923798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:27:29.734327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924636+0000","last_change":"2024-04-17T19:47:48.862315+0000","last_active":"2024-04-17T19:47:54.924636+0000","last_peered":"2024-04-17T19:47:54.924636+0000","last_clean":"2024-04-17T19:47:54.924636+0000","last_became_active":"2024-04-17T19:47:48.860408+0000","last_became_peered":"2024-04-17T19:47:48.860408+0000","last_unstale":"2024-04-17T19:47:54.924636+0000","last_undegraded":"2024-04-17T19:47:54.924636+0000","last_fullsized":"2024-04-17T19:47:54.924636+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:10:26.336063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841873+0000","last_change":"2024-04-17T19:47:49.902423+0000","last_active":"2024-04-17T19:47:55.841873+0000","last_peered":"2024-04-17T19:47:55.841873+0000","last_clean":"2024-04-17T19:47:55.841873+0000","last_became_active":"2024-04-17T19:47:49.902096+0000","last_became_peered":"2024-04-17T19:47:49.902096+0000","last_unstale":"2024-04-17T19:47:55.841873+0000","last_undegraded":"2024-04-17T19:47:55.841873+0000","last_fullsized":"2024-04-17T19:47:55.841873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:40.805247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955520+0000","last_change":"2024-04-17T19:47:53.956851+0000","last_active":"2024-04-17T19:47:56.955520+0000","last_peered":"2024-04-17T19:47:56.955520+0000","last_clean":"2024-04-17T19:47:56.955520+0000","last_became_active":"2024-04-17T19:47:53.954197+0000","last_became_peered":"2024-04-17T19:47:53.954197+0000","last_unstale":"2024-04-17T19:47:56.955520+0000","last_undegraded":"2024-04-17T19:47:56.955520+0000","last_fullsized":"2024-04-17T19:47:56.955520+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:16:18.379831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049047+0000","last_change":"2024-04-17T19:47:48.893969+0000","last_active":"2024-04-17T19:47:56.049047+0000","last_peered":"2024-04-17T19:47:56.049047+0000","last_clean":"2024-04-17T19:47:56.049047+0000","last_became_active":"2024-04-17T19:47:48.893229+0000","last_became_peered":"2024-04-17T19:47:48.893229+0000","last_unstale":"2024-04-17T19:47:56.049047+0000","last_undegraded":"2024-04-17T19:47:56.049047+0000","last_fullsized":"2024-04-17T19:47:56.049047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:38:42.472634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832673+0000","last_change":"2024-04-17T19:47:49.897987+0000","last_active":"2024-04-17T19:47:56.832673+0000","last_peered":"2024-04-17T19:47:56.832673+0000","last_clean":"2024-04-17T19:47:56.832673+0000","last_became_active":"2024-04-17T19:47:49.897096+0000","last_became_peered":"2024-04-17T19:47:49.897096+0000","last_unstale":"2024-04-17T19:47:56.832673+0000","last_undegraded":"2024-04-17T19:47:56.832673+0000","last_fullsized":"2024-04-17T19:47:56.832673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:36:36.856454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955695+0000","last_change":"2024-04-17T19:47:53.930784+0000","last_active":"2024-04-17T19:47:56.955695+0000","last_peered":"2024-04-17T19:47:56.955695+0000","last_clean":"2024-04-17T19:47:56.955695+0000","last_became_active":"2024-04-17T19:47:53.928281+0000","last_became_peered":"2024-04-17T19:47:53.928281+0000","last_unstale":"2024-04-17T19:47:56.955695+0000","last_undegraded":"2024-04-17T19:47:56.955695+0000","last_fullsized":"2024-04-17T19:47:56.955695+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:15:55.064622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048308+0000","last_change":"2024-04-17T19:47:48.893184+0000","last_active":"2024-04-17T19:47:56.048308+0000","last_peered":"2024-04-17T19:47:56.048308+0000","last_clean":"2024-04-17T19:47:56.048308+0000","last_became_active":"2024-04-17T19:47:48.892375+0000","last_became_peered":"2024-04-17T19:47:48.892375+0000","last_unstale":"2024-04-17T19:47:56.048308+0000","last_undegraded":"2024-04-17T19:47:56.048308+0000","last_fullsized":"2024-04-17T19:47:56.048308+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:56:01.211666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832916+0000","last_change":"2024-04-17T19:47:49.896040+0000","last_active":"2024-04-17T19:47:56.832916+0000","last_peered":"2024-04-17T19:47:56.832916+0000","last_clean":"2024-04-17T19:47:56.832916+0000","last_became_active":"2024-04-17T19:47:49.894779+0000","last_became_peered":"2024-04-17T19:47:49.894779+0000","last_unstale":"2024-04-17T19:47:56.832916+0000","last_undegraded":"2024-04-17T19:47:56.832916+0000","last_fullsized":"2024-04-17T19:47:56.832916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:30:17.930203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955688+0000","last_change":"2024-04-17T19:47:53.930402+0000","last_active":"2024-04-17T19:47:56.955688+0000","last_peered":"2024-04-17T19:47:56.955688+0000","last_clean":"2024-04-17T19:47:56.955688+0000","last_became_active":"2024-04-17T19:47:53.929166+0000","last_became_peered":"2024-04-17T19:47:53.929166+0000","last_unstale":"2024-04-17T19:47:56.955688+0000","last_undegraded":"2024-04-17T19:47:56.955688+0000","last_fullsized":"2024-04-17T19:47:56.955688+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:34:08.282697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927108+0000","last_change":"2024-04-17T19:47:49.900142+0000","last_active":"2024-04-17T19:47:54.927108+0000","last_peered":"2024-04-17T19:47:54.927108+0000","last_clean":"2024-04-17T19:47:54.927108+0000","last_became_active":"2024-04-17T19:47:49.899365+0000","last_became_peered":"2024-04-17T19:47:49.899365+0000","last_unstale":"2024-04-17T19:47:54.927108+0000","last_undegraded":"2024-04-17T19:47:54.927108+0000","last_fullsized":"2024-04-17T19:47:54.927108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:34:49.616539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"24'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.758013+0000","last_change":"2024-04-17T19:47:48.891513+0000","last_active":"2024-04-17T19:48:15.758013+0000","last_peered":"2024-04-17T19:48:15.758013+0000","last_clean":"2024-04-17T19:48:15.758013+0000","last_became_active":"2024-04-17T19:47:48.886561+0000","last_became_peered":"2024-04-17T19:47:48.886561+0000","last_unstale":"2024-04-17T19:48:15.758013+0000","last_undegraded":"2024-04-17T19:48:15.758013+0000","last_fullsized":"2024-04-17T19:48:15.758013+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:59:14.039700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.957430+0000","last_change":"2024-04-17T19:47:53.925058+0000","last_active":"2024-04-17T19:47:56.957430+0000","last_peered":"2024-04-17T19:47:56.957430+0000","last_clean":"2024-04-17T19:47:56.957430+0000","last_became_active":"2024-04-17T19:47:53.923721+0000","last_became_peered":"2024-04-17T19:47:53.923721+0000","last_unstale":"2024-04-17T19:47:56.957430+0000","last_undegraded":"2024-04-17T19:47:56.957430+0000","last_fullsized":"2024-04-17T19:47:56.957430+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:22:42.106657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924842+0000","last_change":"2024-04-17T19:47:49.852183+0000","last_active":"2024-04-17T19:47:54.924842+0000","last_peered":"2024-04-17T19:47:54.924842+0000","last_clean":"2024-04-17T19:47:54.924842+0000","last_became_active":"2024-04-17T19:47:49.851765+0000","last_became_peered":"2024-04-17T19:47:49.851765+0000","last_unstale":"2024-04-17T19:47:54.924842+0000","last_undegraded":"2024-04-17T19:47:54.924842+0000","last_fullsized":"2024-04-17T19:47:54.924842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:02:03.101317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051463+0000","last_change":"2024-04-17T19:47:48.862865+0000","last_active":"2024-04-17T19:47:55.051463+0000","last_peered":"2024-04-17T19:47:55.051463+0000","last_clean":"2024-04-17T19:47:55.051463+0000","last_became_active":"2024-04-17T19:47:48.862524+0000","last_became_peered":"2024-04-17T19:47:48.862524+0000","last_unstale":"2024-04-17T19:47:55.051463+0000","last_undegraded":"2024-04-17T19:47:55.051463+0000","last_fullsized":"2024-04-17T19:47:55.051463+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:01:21.080029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956615+0000","last_change":"2024-04-17T19:47:53.943674+0000","last_active":"2024-04-17T19:47:56.956615+0000","last_peered":"2024-04-17T19:47:56.956615+0000","last_clean":"2024-04-17T19:47:56.956615+0000","last_became_active":"2024-04-17T19:47:53.942549+0000","last_became_peered":"2024-04-17T19:47:53.942549+0000","last_unstale":"2024-04-17T19:47:56.956615+0000","last_undegraded":"2024-04-17T19:47:56.956615+0000","last_fullsized":"2024-04-17T19:47:56.956615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:00:42.300738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924513+0000","last_change":"2024-04-17T19:47:49.862312+0000","last_active":"2024-04-17T19:47:54.924513+0000","last_peered":"2024-04-17T19:47:54.924513+0000","last_clean":"2024-04-17T19:47:54.924513+0000","last_became_active":"2024-04-17T19:47:49.860664+0000","last_became_peered":"2024-04-17T19:47:49.860664+0000","last_unstale":"2024-04-17T19:47:54.924513+0000","last_undegraded":"2024-04-17T19:47:54.924513+0000","last_fullsized":"2024-04-17T19:47:54.924513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:07:47.676973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"24'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.777109+0000","last_change":"2024-04-17T19:47:48.870638+0000","last_active":"2024-04-17T19:48:15.777109+0000","last_peered":"2024-04-17T19:48:15.777109+0000","last_clean":"2024-04-17T19:48:15.777109+0000","last_became_active":"2024-04-17T19:47:48.870353+0000","last_became_peered":"2024-04-17T19:47:48.870353+0000","last_unstale":"2024-04-17T19:48:15.777109+0000","last_undegraded":"2024-04-17T19:48:15.777109+0000","last_fullsized":"2024-04-17T19:48:15.777109+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:12:19.927824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.052170+0000","last_change":"2024-04-17T19:47:53.929285+0000","last_active":"2024-04-17T19:47:55.052170+0000","last_peered":"2024-04-17T19:47:55.052170+0000","last_clean":"2024-04-17T19:47:55.052170+0000","last_became_active":"2024-04-17T19:47:53.928189+0000","last_became_peered":"2024-04-17T19:47:53.928189+0000","last_unstale":"2024-04-17T19:47:55.052170+0000","last_undegraded":"2024-04-17T19:47:55.052170+0000","last_fullsized":"2024-04-17T19:47:55.052170+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:53:26.112634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956382+0000","last_change":"2024-04-17T19:47:49.888118+0000","last_active":"2024-04-17T19:47:56.956382+0000","last_peered":"2024-04-17T19:47:56.956382+0000","last_clean":"2024-04-17T19:47:56.956382+0000","last_became_active":"2024-04-17T19:47:49.884037+0000","last_became_peered":"2024-04-17T19:47:49.884037+0000","last_unstale":"2024-04-17T19:47:56.956382+0000","last_undegraded":"2024-04-17T19:47:56.956382+0000","last_fullsized":"2024-04-17T19:47:56.956382+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:44:27.180380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"24'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.759737+0000","last_change":"2024-04-17T19:47:48.883686+0000","last_active":"2024-04-17T19:48:15.759737+0000","last_peered":"2024-04-17T19:48:15.759737+0000","last_clean":"2024-04-17T19:48:15.759737+0000","last_became_active":"2024-04-17T19:47:48.882375+0000","last_became_peered":"2024-04-17T19:47:48.882375+0000","last_unstale":"2024-04-17T19:48:15.759737+0000","last_undegraded":"2024-04-17T19:48:15.759737+0000","last_fullsized":"2024-04-17T19:48:15.759737+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:38:31.225374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.957175+0000","last_change":"2024-04-17T19:47:53.935396+0000","last_active":"2024-04-17T19:47:56.957175+0000","last_peered":"2024-04-17T19:47:56.957175+0000","last_clean":"2024-04-17T19:47:56.957175+0000","last_became_active":"2024-04-17T19:47:53.934094+0000","last_became_peered":"2024-04-17T19:47:53.934094+0000","last_unstale":"2024-04-17T19:47:56.957175+0000","last_undegraded":"2024-04-17T19:47:56.957175+0000","last_fullsized":"2024-04-17T19:47:56.957175+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:58.334357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842041+0000","last_change":"2024-04-17T19:47:49.870297+0000","last_active":"2024-04-17T19:47:55.842041+0000","last_peered":"2024-04-17T19:47:55.842041+0000","last_clean":"2024-04-17T19:47:55.842041+0000","last_became_active":"2024-04-17T19:47:49.870113+0000","last_became_peered":"2024-04-17T19:47:49.870113+0000","last_unstale":"2024-04-17T19:47:55.842041+0000","last_undegraded":"2024-04-17T19:47:55.842041+0000","last_fullsized":"2024-04-17T19:47:55.842041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:50:54.517273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"24'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.766655+0000","last_change":"2024-04-17T19:47:48.895905+0000","last_active":"2024-04-17T19:48:15.766655+0000","last_peered":"2024-04-17T19:48:15.766655+0000","last_clean":"2024-04-17T19:48:15.766655+0000","last_became_active":"2024-04-17T19:47:48.895263+0000","last_became_peered":"2024-04-17T19:47:48.895263+0000","last_unstale":"2024-04-17T19:48:15.766655+0000","last_undegraded":"2024-04-17T19:48:15.766655+0000","last_fullsized":"2024-04-17T19:48:15.766655+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:07:23.179031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956013+0000","last_change":"2024-04-17T19:47:53.943286+0000","last_active":"2024-04-17T19:47:56.956013+0000","last_peered":"2024-04-17T19:47:56.956013+0000","last_clean":"2024-04-17T19:47:56.956013+0000","last_became_active":"2024-04-17T19:47:53.942325+0000","last_became_peered":"2024-04-17T19:47:53.942325+0000","last_unstale":"2024-04-17T19:47:56.956013+0000","last_undegraded":"2024-04-17T19:47:56.956013+0000","last_fullsized":"2024-04-17T19:47:56.956013+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:47:18.853991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924723+0000","last_change":"2024-04-17T19:47:49.872925+0000","last_active":"2024-04-17T19:47:54.924723+0000","last_peered":"2024-04-17T19:47:54.924723+0000","last_clean":"2024-04-17T19:47:54.924723+0000","last_became_active":"2024-04-17T19:47:49.872668+0000","last_became_peered":"2024-04-17T19:47:49.872668+0000","last_unstale":"2024-04-17T19:47:54.924723+0000","last_undegraded":"2024-04-17T19:47:54.924723+0000","last_fullsized":"2024-04-17T19:47:54.924723+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:16:21.209634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051807+0000","last_change":"2024-04-17T19:47:48.896669+0000","last_active":"2024-04-17T19:47:55.051807+0000","last_peered":"2024-04-17T19:47:55.051807+0000","last_clean":"2024-04-17T19:47:55.051807+0000","last_became_active":"2024-04-17T19:47:48.896397+0000","last_became_peered":"2024-04-17T19:47:48.896397+0000","last_unstale":"2024-04-17T19:47:55.051807+0000","last_undegraded":"2024-04-17T19:47:55.051807+0000","last_fullsized":"2024-04-17T19:47:55.051807+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:07:23.179031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923702+0000","last_change":"2024-04-17T19:47:53.906261+0000","last_active":"2024-04-17T19:47:54.923702+0000","last_peered":"2024-04-17T19:47:54.923702+0000","last_clean":"2024-04-17T19:47:54.923702+0000","last_became_active":"2024-04-17T19:47:53.903702+0000","last_became_peered":"2024-04-17T19:47:53.903702+0000","last_unstale":"2024-04-17T19:47:54.923702+0000","last_undegraded":"2024-04-17T19:47:54.923702+0000","last_fullsized":"2024-04-17T19:47:54.923702+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:35:53.892917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051477+0000","last_change":"2024-04-17T19:47:49.870272+0000","last_active":"2024-04-17T19:47:55.051477+0000","last_peered":"2024-04-17T19:47:55.051477+0000","last_clean":"2024-04-17T19:47:55.051477+0000","last_became_active":"2024-04-17T19:47:49.869914+0000","last_became_peered":"2024-04-17T19:47:49.869914+0000","last_unstale":"2024-04-17T19:47:55.051477+0000","last_undegraded":"2024-04-17T19:47:55.051477+0000","last_fullsized":"2024-04-17T19:47:55.051477+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:44:30.553286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049411+0000","last_change":"2024-04-17T19:47:48.892675+0000","last_active":"2024-04-17T19:47:56.049411+0000","last_peered":"2024-04-17T19:47:56.049411+0000","last_clean":"2024-04-17T19:47:56.049411+0000","last_became_active":"2024-04-17T19:47:48.892057+0000","last_became_peered":"2024-04-17T19:47:48.892057+0000","last_unstale":"2024-04-17T19:47:56.049411+0000","last_undegraded":"2024-04-17T19:47:56.049411+0000","last_fullsized":"2024-04-17T19:47:56.049411+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:18:35.735350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841981+0000","last_change":"2024-04-17T19:47:53.925135+0000","last_active":"2024-04-17T19:47:55.841981+0000","last_peered":"2024-04-17T19:47:55.841981+0000","last_clean":"2024-04-17T19:47:55.841981+0000","last_became_active":"2024-04-17T19:47:53.924617+0000","last_became_peered":"2024-04-17T19:47:53.924617+0000","last_unstale":"2024-04-17T19:47:55.841981+0000","last_undegraded":"2024-04-17T19:47:55.841981+0000","last_fullsized":"2024-04-17T19:47:55.841981+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:37:02.082648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.833132+0000","last_change":"2024-04-17T19:47:49.895815+0000","last_active":"2024-04-17T19:47:56.833132+0000","last_peered":"2024-04-17T19:47:56.833132+0000","last_clean":"2024-04-17T19:47:56.833132+0000","last_became_active":"2024-04-17T19:47:49.894978+0000","last_became_peered":"2024-04-17T19:47:49.894978+0000","last_unstale":"2024-04-17T19:47:56.833132+0000","last_undegraded":"2024-04-17T19:47:56.833132+0000","last_fullsized":"2024-04-17T19:47:56.833132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:32:12.184723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051589+0000","last_change":"2024-04-17T19:47:48.859247+0000","last_active":"2024-04-17T19:47:55.051589+0000","last_peered":"2024-04-17T19:47:55.051589+0000","last_clean":"2024-04-17T19:47:55.051589+0000","last_became_active":"2024-04-17T19:47:48.858756+0000","last_became_peered":"2024-04-17T19:47:48.858756+0000","last_unstale":"2024-04-17T19:47:55.051589+0000","last_undegraded":"2024-04-17T19:47:55.051589+0000","last_fullsized":"2024-04-17T19:47:55.051589+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:19:56.987957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051118+0000","last_change":"2024-04-17T19:47:53.929952+0000","last_active":"2024-04-17T19:47:55.051118+0000","last_peered":"2024-04-17T19:47:55.051118+0000","last_clean":"2024-04-17T19:47:55.051118+0000","last_became_active":"2024-04-17T19:47:53.929528+0000","last_became_peered":"2024-04-17T19:47:53.929528+0000","last_unstale":"2024-04-17T19:47:55.051118+0000","last_undegraded":"2024-04-17T19:47:55.051118+0000","last_fullsized":"2024-04-17T19:47:55.051118+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:11:38.041301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956659+0000","last_change":"2024-04-17T19:47:49.874150+0000","last_active":"2024-04-17T19:47:56.956659+0000","last_peered":"2024-04-17T19:47:56.956659+0000","last_clean":"2024-04-17T19:47:56.956659+0000","last_became_active":"2024-04-17T19:47:49.872859+0000","last_became_peered":"2024-04-17T19:47:49.872859+0000","last_unstale":"2024-04-17T19:47:56.956659+0000","last_undegraded":"2024-04-17T19:47:56.956659+0000","last_fullsized":"2024-04-17T19:47:56.956659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:57:54.666279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832613+0000","last_change":"2024-04-17T19:47:48.876672+0000","last_active":"2024-04-17T19:47:56.832613+0000","last_peered":"2024-04-17T19:47:56.832613+0000","last_clean":"2024-04-17T19:47:56.832613+0000","last_became_active":"2024-04-17T19:47:48.875263+0000","last_became_peered":"2024-04-17T19:47:48.875263+0000","last_unstale":"2024-04-17T19:47:56.832613+0000","last_undegraded":"2024-04-17T19:47:56.832613+0000","last_fullsized":"2024-04-17T19:47:56.832613+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:45:46.633540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051584+0000","last_change":"2024-04-17T19:47:53.932330+0000","last_active":"2024-04-17T19:47:55.051584+0000","last_peered":"2024-04-17T19:47:55.051584+0000","last_clean":"2024-04-17T19:47:55.051584+0000","last_became_active":"2024-04-17T19:47:53.931873+0000","last_became_peered":"2024-04-17T19:47:53.931873+0000","last_unstale":"2024-04-17T19:47:55.051584+0000","last_undegraded":"2024-04-17T19:47:55.051584+0000","last_fullsized":"2024-04-17T19:47:55.051584+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:05:42.346878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051044+0000","last_change":"2024-04-17T19:47:49.865020+0000","last_active":"2024-04-17T19:47:55.051044+0000","last_peered":"2024-04-17T19:47:55.051044+0000","last_clean":"2024-04-17T19:47:55.051044+0000","last_became_active":"2024-04-17T19:47:49.864764+0000","last_became_peered":"2024-04-17T19:47:49.864764+0000","last_unstale":"2024-04-17T19:47:55.051044+0000","last_undegraded":"2024-04-17T19:47:55.051044+0000","last_fullsized":"2024-04-17T19:47:55.051044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:45:36.325765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926669+0000","last_change":"2024-04-17T19:47:48.846570+0000","last_active":"2024-04-17T19:47:54.926669+0000","last_peered":"2024-04-17T19:47:54.926669+0000","last_clean":"2024-04-17T19:47:54.926669+0000","last_became_active":"2024-04-17T19:47:48.846175+0000","last_became_peered":"2024-04-17T19:47:48.846175+0000","last_unstale":"2024-04-17T19:47:54.926669+0000","last_undegraded":"2024-04-17T19:47:54.926669+0000","last_fullsized":"2024-04-17T19:47:54.926669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:20:22.737016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.052346+0000","last_change":"2024-04-17T19:47:53.904590+0000","last_active":"2024-04-17T19:47:55.052346+0000","last_peered":"2024-04-17T19:47:55.052346+0000","last_clean":"2024-04-17T19:47:55.052346+0000","last_became_active":"2024-04-17T19:47:53.904026+0000","last_became_peered":"2024-04-17T19:47:53.904026+0000","last_unstale":"2024-04-17T19:47:55.052346+0000","last_undegraded":"2024-04-17T19:47:55.052346+0000","last_fullsized":"2024-04-17T19:47:55.052346+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:16:22.837570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841469+0000","last_change":"2024-04-17T19:47:49.860010+0000","last_active":"2024-04-17T19:47:55.841469+0000","last_peered":"2024-04-17T19:47:55.841469+0000","last_clean":"2024-04-17T19:47:55.841469+0000","last_became_active":"2024-04-17T19:47:49.859593+0000","last_became_peered":"2024-04-17T19:47:49.859593+0000","last_unstale":"2024-04-17T19:47:55.841469+0000","last_undegraded":"2024-04-17T19:47:55.841469+0000","last_fullsized":"2024-04-17T19:47:55.841469+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:33:24.594397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.047779+0000","last_change":"2024-04-17T19:47:48.883126+0000","last_active":"2024-04-17T19:47:56.047779+0000","last_peered":"2024-04-17T19:47:56.047779+0000","last_clean":"2024-04-17T19:47:56.047779+0000","last_became_active":"2024-04-17T19:47:48.882173+0000","last_became_peered":"2024-04-17T19:47:48.882173+0000","last_unstale":"2024-04-17T19:47:56.047779+0000","last_undegraded":"2024-04-17T19:47:56.047779+0000","last_fullsized":"2024-04-17T19:47:56.047779+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:11:59.395316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841972+0000","last_change":"2024-04-17T19:47:53.931258+0000","last_active":"2024-04-17T19:47:55.841972+0000","last_peered":"2024-04-17T19:47:55.841972+0000","last_clean":"2024-04-17T19:47:55.841972+0000","last_became_active":"2024-04-17T19:47:53.930773+0000","last_became_peered":"2024-04-17T19:47:53.930773+0000","last_unstale":"2024-04-17T19:47:55.841972+0000","last_undegraded":"2024-04-17T19:47:55.841972+0000","last_fullsized":"2024-04-17T19:47:55.841972+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:10:47.583714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924520+0000","last_change":"2024-04-17T19:47:49.909404+0000","last_active":"2024-04-17T19:47:54.924520+0000","last_peered":"2024-04-17T19:47:54.924520+0000","last_clean":"2024-04-17T19:47:54.924520+0000","last_became_active":"2024-04-17T19:47:49.909154+0000","last_became_peered":"2024-04-17T19:47:49.909154+0000","last_unstale":"2024-04-17T19:47:54.924520+0000","last_undegraded":"2024-04-17T19:47:54.924520+0000","last_fullsized":"2024-04-17T19:47:54.924520+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:14:39.682888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"24'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.771615+0000","last_change":"2024-04-17T19:47:48.878592+0000","last_active":"2024-04-17T19:48:15.771615+0000","last_peered":"2024-04-17T19:48:15.771615+0000","last_clean":"2024-04-17T19:48:15.771615+0000","last_became_active":"2024-04-17T19:47:48.877617+0000","last_became_peered":"2024-04-17T19:47:48.877617+0000","last_unstale":"2024-04-17T19:48:15.771615+0000","last_undegraded":"2024-04-17T19:48:15.771615+0000","last_fullsized":"2024-04-17T19:48:15.771615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:30:17.013176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048011+0000","last_change":"2024-04-17T19:47:38.277798+0000","last_active":"2024-04-17T19:47:56.048011+0000","last_peered":"2024-04-17T19:47:56.048011+0000","last_clean":"2024-04-17T19:47:56.048011+0000","last_became_active":"2024-04-17T19:47:38.274697+0000","last_became_peered":"2024-04-17T19:47:38.274697+0000","last_unstale":"2024-04-17T19:47:56.048011+0000","last_undegraded":"2024-04-17T19:47:56.048011+0000","last_fullsized":"2024-04-17T19:47:56.048011+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:36.897071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:36.897071+0000","last_clean_scrub_stamp":"2024-04-17T19:47:36.897071+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:31:35.366600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929077+0000","last_change":"2024-04-17T19:47:53.953415+0000","last_active":"2024-04-17T19:47:54.929077+0000","last_peered":"2024-04-17T19:47:54.929077+0000","last_clean":"2024-04-17T19:47:54.929077+0000","last_became_active":"2024-04-17T19:47:53.952644+0000","last_became_peered":"2024-04-17T19:47:53.952644+0000","last_unstale":"2024-04-17T19:47:54.929077+0000","last_undegraded":"2024-04-17T19:47:54.929077+0000","last_fullsized":"2024-04-17T19:47:54.929077+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:00:25.923909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048406+0000","last_change":"2024-04-17T19:47:49.901572+0000","last_active":"2024-04-17T19:47:56.048406+0000","last_peered":"2024-04-17T19:47:56.048406+0000","last_clean":"2024-04-17T19:47:56.048406+0000","last_became_active":"2024-04-17T19:47:49.900707+0000","last_became_peered":"2024-04-17T19:47:49.900707+0000","last_unstale":"2024-04-17T19:47:56.048406+0000","last_undegraded":"2024-04-17T19:47:56.048406+0000","last_fullsized":"2024-04-17T19:47:56.048406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:58:02.897564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926753+0000","last_change":"2024-04-17T19:47:48.874721+0000","last_active":"2024-04-17T19:47:54.926753+0000","last_peered":"2024-04-17T19:47:54.926753+0000","last_clean":"2024-04-17T19:47:54.926753+0000","last_became_active":"2024-04-17T19:47:48.874364+0000","last_became_peered":"2024-04-17T19:47:48.874364+0000","last_unstale":"2024-04-17T19:47:54.926753+0000","last_undegraded":"2024-04-17T19:47:54.926753+0000","last_fullsized":"2024-04-17T19:47:54.926753+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:47:08.856753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927738+0000","last_change":"2024-04-17T19:47:53.942791+0000","last_active":"2024-04-17T19:47:54.927738+0000","last_peered":"2024-04-17T19:47:54.927738+0000","last_clean":"2024-04-17T19:47:54.927738+0000","last_became_active":"2024-04-17T19:47:53.941569+0000","last_became_peered":"2024-04-17T19:47:53.941569+0000","last_unstale":"2024-04-17T19:47:54.927738+0000","last_undegraded":"2024-04-17T19:47:54.927738+0000","last_fullsized":"2024-04-17T19:47:54.927738+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:23:35.452056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832644+0000","last_change":"2024-04-17T19:47:49.895046+0000","last_active":"2024-04-17T19:47:56.832644+0000","last_peered":"2024-04-17T19:47:56.832644+0000","last_clean":"2024-04-17T19:47:56.832644+0000","last_became_active":"2024-04-17T19:47:49.894250+0000","last_became_peered":"2024-04-17T19:47:49.894250+0000","last_unstale":"2024-04-17T19:47:56.832644+0000","last_undegraded":"2024-04-17T19:47:56.832644+0000","last_fullsized":"2024-04-17T19:47:56.832644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:30:17.210135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051734+0000","last_change":"2024-04-17T19:47:48.861802+0000","last_active":"2024-04-17T19:47:55.051734+0000","last_peered":"2024-04-17T19:47:55.051734+0000","last_clean":"2024-04-17T19:47:55.051734+0000","last_became_active":"2024-04-17T19:47:48.860418+0000","last_became_peered":"2024-04-17T19:47:48.860418+0000","last_unstale":"2024-04-17T19:47:55.051734+0000","last_undegraded":"2024-04-17T19:47:55.051734+0000","last_fullsized":"2024-04-17T19:47:55.051734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:54:41.949746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842288+0000","last_change":"2024-04-17T19:47:53.917732+0000","last_active":"2024-04-17T19:47:55.842288+0000","last_peered":"2024-04-17T19:47:55.842288+0000","last_clean":"2024-04-17T19:47:55.842288+0000","last_became_active":"2024-04-17T19:47:53.917160+0000","last_became_peered":"2024-04-17T19:47:53.917160+0000","last_unstale":"2024-04-17T19:47:55.842288+0000","last_undegraded":"2024-04-17T19:47:55.842288+0000","last_fullsized":"2024-04-17T19:47:55.842288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:19:20.394166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049773+0000","last_change":"2024-04-17T19:47:49.893077+0000","last_active":"2024-04-17T19:47:56.049773+0000","last_peered":"2024-04-17T19:47:56.049773+0000","last_clean":"2024-04-17T19:47:56.049773+0000","last_became_active":"2024-04-17T19:47:49.892280+0000","last_became_peered":"2024-04-17T19:47:49.892280+0000","last_unstale":"2024-04-17T19:47:56.049773+0000","last_undegraded":"2024-04-17T19:47:56.049773+0000","last_fullsized":"2024-04-17T19:47:56.049773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:58:24.218821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926706+0000","last_change":"2024-04-17T19:47:48.849027+0000","last_active":"2024-04-17T19:47:54.926706+0000","last_peered":"2024-04-17T19:47:54.926706+0000","last_clean":"2024-04-17T19:47:54.926706+0000","last_became_active":"2024-04-17T19:47:48.848485+0000","last_became_peered":"2024-04-17T19:47:48.848485+0000","last_unstale":"2024-04-17T19:47:54.926706+0000","last_undegraded":"2024-04-17T19:47:54.926706+0000","last_fullsized":"2024-04-17T19:47:54.926706+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:58:14.548891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.831438+0000","last_change":"2024-04-17T19:47:53.946933+0000","last_active":"2024-04-17T19:47:56.831438+0000","last_peered":"2024-04-17T19:47:56.831438+0000","last_clean":"2024-04-17T19:47:56.831438+0000","last_became_active":"2024-04-17T19:47:53.943030+0000","last_became_peered":"2024-04-17T19:47:53.943030+0000","last_unstale":"2024-04-17T19:47:56.831438+0000","last_undegraded":"2024-04-17T19:47:56.831438+0000","last_fullsized":"2024-04-17T19:47:56.831438+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:37:03.489250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049281+0000","last_change":"2024-04-17T19:47:49.893137+0000","last_active":"2024-04-17T19:47:56.049281+0000","last_peered":"2024-04-17T19:47:56.049281+0000","last_clean":"2024-04-17T19:47:56.049281+0000","last_became_active":"2024-04-17T19:47:49.892379+0000","last_became_peered":"2024-04-17T19:47:49.892379+0000","last_unstale":"2024-04-17T19:47:56.049281+0000","last_undegraded":"2024-04-17T19:47:56.049281+0000","last_fullsized":"2024-04-17T19:47:56.049281+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:55:08.562395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.954554+0000","last_change":"2024-04-17T19:47:48.861154+0000","last_active":"2024-04-17T19:47:56.954554+0000","last_peered":"2024-04-17T19:47:56.954554+0000","last_clean":"2024-04-17T19:47:56.954554+0000","last_became_active":"2024-04-17T19:47:48.859268+0000","last_became_peered":"2024-04-17T19:47:48.859268+0000","last_unstale":"2024-04-17T19:47:56.954554+0000","last_undegraded":"2024-04-17T19:47:56.954554+0000","last_fullsized":"2024-04-17T19:47:56.954554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:04:19.867115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955200+0000","last_change":"2024-04-17T19:47:53.950580+0000","last_active":"2024-04-17T19:47:56.955200+0000","last_peered":"2024-04-17T19:47:56.955200+0000","last_clean":"2024-04-17T19:47:56.955200+0000","last_became_active":"2024-04-17T19:47:53.943798+0000","last_became_peered":"2024-04-17T19:47:53.943798+0000","last_unstale":"2024-04-17T19:47:56.955200+0000","last_undegraded":"2024-04-17T19:47:56.955200+0000","last_fullsized":"2024-04-17T19:47:56.955200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:52:14.763799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048020+0000","last_change":"2024-04-17T19:47:49.893774+0000","last_active":"2024-04-17T19:47:56.048020+0000","last_peered":"2024-04-17T19:47:56.048020+0000","last_clean":"2024-04-17T19:47:56.048020+0000","last_became_active":"2024-04-17T19:47:49.892780+0000","last_became_peered":"2024-04-17T19:47:49.892780+0000","last_unstale":"2024-04-17T19:47:56.048020+0000","last_undegraded":"2024-04-17T19:47:56.048020+0000","last_fullsized":"2024-04-17T19:47:56.048020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:50:47.658650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"24'10","reported_seq":32,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.776912+0000","last_change":"2024-04-17T19:47:48.852970+0000","last_active":"2024-04-17T19:48:15.776912+0000","last_peered":"2024-04-17T19:48:15.776912+0000","last_clean":"2024-04-17T19:48:15.776912+0000","last_became_active":"2024-04-17T19:47:48.852676+0000","last_became_peered":"2024-04-17T19:47:48.852676+0000","last_unstale":"2024-04-17T19:48:15.776912+0000","last_undegraded":"2024-04-17T19:48:15.776912+0000","last_fullsized":"2024-04-17T19:48:15.776912+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:32:14.529596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841883+0000","last_change":"2024-04-17T19:47:53.915397+0000","last_active":"2024-04-17T19:47:55.841883+0000","last_peered":"2024-04-17T19:47:55.841883+0000","last_clean":"2024-04-17T19:47:55.841883+0000","last_became_active":"2024-04-17T19:47:53.914913+0000","last_became_peered":"2024-04-17T19:47:53.914913+0000","last_unstale":"2024-04-17T19:47:55.841883+0000","last_undegraded":"2024-04-17T19:47:55.841883+0000","last_fullsized":"2024-04-17T19:47:55.841883+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:02:01.175158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927357+0000","last_change":"2024-04-17T19:47:49.880484+0000","last_active":"2024-04-17T19:47:54.927357+0000","last_peered":"2024-04-17T19:47:54.927357+0000","last_clean":"2024-04-17T19:47:54.927357+0000","last_became_active":"2024-04-17T19:47:49.879625+0000","last_became_peered":"2024-04-17T19:47:49.879625+0000","last_unstale":"2024-04-17T19:47:54.927357+0000","last_undegraded":"2024-04-17T19:47:54.927357+0000","last_fullsized":"2024-04-17T19:47:54.927357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:50:00.611796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927417+0000","last_change":"2024-04-17T19:47:48.899160+0000","last_active":"2024-04-17T19:47:54.927417+0000","last_peered":"2024-04-17T19:47:54.927417+0000","last_clean":"2024-04-17T19:47:54.927417+0000","last_became_active":"2024-04-17T19:47:48.889831+0000","last_became_peered":"2024-04-17T19:47:48.889831+0000","last_unstale":"2024-04-17T19:47:54.927417+0000","last_undegraded":"2024-04-17T19:47:54.927417+0000","last_fullsized":"2024-04-17T19:47:54.927417+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:55:20.167510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924391+0000","last_change":"2024-04-17T19:47:53.921093+0000","last_active":"2024-04-17T19:47:54.924391+0000","last_peered":"2024-04-17T19:47:54.924391+0000","last_clean":"2024-04-17T19:47:54.924391+0000","last_became_active":"2024-04-17T19:47:53.920479+0000","last_became_peered":"2024-04-17T19:47:53.920479+0000","last_unstale":"2024-04-17T19:47:54.924391+0000","last_undegraded":"2024-04-17T19:47:54.924391+0000","last_fullsized":"2024-04-17T19:47:54.924391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:08:09.569820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048418+0000","last_change":"2024-04-17T19:47:49.895275+0000","last_active":"2024-04-17T19:47:56.048418+0000","last_peered":"2024-04-17T19:47:56.048418+0000","last_clean":"2024-04-17T19:47:56.048418+0000","last_became_active":"2024-04-17T19:47:49.894534+0000","last_became_peered":"2024-04-17T19:47:49.894534+0000","last_unstale":"2024-04-17T19:47:56.048418+0000","last_undegraded":"2024-04-17T19:47:56.048418+0000","last_fullsized":"2024-04-17T19:47:56.048418+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:31:40.818807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"24'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.775373+0000","last_change":"2024-04-17T19:47:48.881715+0000","last_active":"2024-04-17T19:48:15.775373+0000","last_peered":"2024-04-17T19:48:15.775373+0000","last_clean":"2024-04-17T19:48:15.775373+0000","last_became_active":"2024-04-17T19:47:48.876203+0000","last_became_peered":"2024-04-17T19:47:48.876203+0000","last_unstale":"2024-04-17T19:48:15.775373+0000","last_undegraded":"2024-04-17T19:48:15.775373+0000","last_fullsized":"2024-04-17T19:48:15.775373+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:03:16.656528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924597+0000","last_change":"2024-04-17T19:47:53.905149+0000","last_active":"2024-04-17T19:47:54.924597+0000","last_peered":"2024-04-17T19:47:54.924597+0000","last_clean":"2024-04-17T19:47:54.924597+0000","last_became_active":"2024-04-17T19:47:53.904718+0000","last_became_peered":"2024-04-17T19:47:53.904718+0000","last_unstale":"2024-04-17T19:47:54.924597+0000","last_undegraded":"2024-04-17T19:47:54.924597+0000","last_fullsized":"2024-04-17T19:47:54.924597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:09:31.908752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841674+0000","last_change":"2024-04-17T19:47:49.875819+0000","last_active":"2024-04-17T19:47:55.841674+0000","last_peered":"2024-04-17T19:47:55.841674+0000","last_clean":"2024-04-17T19:47:55.841674+0000","last_became_active":"2024-04-17T19:47:49.872075+0000","last_became_peered":"2024-04-17T19:47:49.872075+0000","last_unstale":"2024-04-17T19:47:55.841674+0000","last_undegraded":"2024-04-17T19:47:55.841674+0000","last_fullsized":"2024-04-17T19:47:55.841674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:06:48.823135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"24'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.770580+0000","last_change":"2024-04-17T19:47:48.867633+0000","last_active":"2024-04-17T19:48:15.770580+0000","last_peered":"2024-04-17T19:48:15.770580+0000","last_clean":"2024-04-17T19:48:15.770580+0000","last_became_active":"2024-04-17T19:47:48.867150+0000","last_became_peered":"2024-04-17T19:47:48.867150+0000","last_unstale":"2024-04-17T19:48:15.770580+0000","last_undegraded":"2024-04-17T19:48:15.770580+0000","last_fullsized":"2024-04-17T19:48:15.770580+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:33:34.275816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832910+0000","last_change":"2024-04-17T19:47:53.925785+0000","last_active":"2024-04-17T19:47:56.832910+0000","last_peered":"2024-04-17T19:47:56.832910+0000","last_clean":"2024-04-17T19:47:56.832910+0000","last_became_active":"2024-04-17T19:47:53.924006+0000","last_became_peered":"2024-04-17T19:47:53.924006+0000","last_unstale":"2024-04-17T19:47:56.832910+0000","last_undegraded":"2024-04-17T19:47:56.832910+0000","last_fullsized":"2024-04-17T19:47:56.832910+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:46:31.965271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.048194+0000","last_change":"2024-04-17T19:47:49.869829+0000","last_active":"2024-04-17T19:47:56.048194+0000","last_peered":"2024-04-17T19:47:56.048194+0000","last_clean":"2024-04-17T19:47:56.048194+0000","last_became_active":"2024-04-17T19:47:49.868933+0000","last_became_peered":"2024-04-17T19:47:49.868933+0000","last_unstale":"2024-04-17T19:47:56.048194+0000","last_undegraded":"2024-04-17T19:47:56.048194+0000","last_fullsized":"2024-04-17T19:47:56.048194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:36:08.866159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'11","reported_seq":33,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.773643+0000","last_change":"2024-04-17T19:47:48.887461+0000","last_active":"2024-04-17T19:48:15.773643+0000","last_peered":"2024-04-17T19:48:15.773643+0000","last_clean":"2024-04-17T19:48:15.773643+0000","last_became_active":"2024-04-17T19:47:48.886689+0000","last_became_peered":"2024-04-17T19:47:48.886689+0000","last_unstale":"2024-04-17T19:48:15.773643+0000","last_undegraded":"2024-04-17T19:48:15.773643+0000","last_fullsized":"2024-04-17T19:48:15.773643+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:55:35.262995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051405+0000","last_change":"2024-04-17T19:47:53.941607+0000","last_active":"2024-04-17T19:47:55.051405+0000","last_peered":"2024-04-17T19:47:55.051405+0000","last_clean":"2024-04-17T19:47:55.051405+0000","last_became_active":"2024-04-17T19:47:53.941050+0000","last_became_peered":"2024-04-17T19:47:53.941050+0000","last_unstale":"2024-04-17T19:47:55.051405+0000","last_undegraded":"2024-04-17T19:47:55.051405+0000","last_fullsized":"2024-04-17T19:47:55.051405+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:26:08.964058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.957037+0000","last_change":"2024-04-17T19:47:49.879524+0000","last_active":"2024-04-17T19:47:56.957037+0000","last_peered":"2024-04-17T19:47:56.957037+0000","last_clean":"2024-04-17T19:47:56.957037+0000","last_became_active":"2024-04-17T19:47:49.877563+0000","last_became_peered":"2024-04-17T19:47:49.877563+0000","last_unstale":"2024-04-17T19:47:56.957037+0000","last_undegraded":"2024-04-17T19:47:56.957037+0000","last_fullsized":"2024-04-17T19:47:56.957037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:34:58.539940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"24'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051731+0000","last_change":"2024-04-17T19:47:48.870100+0000","last_active":"2024-04-17T19:47:55.051731+0000","last_peered":"2024-04-17T19:47:55.051731+0000","last_clean":"2024-04-17T19:47:55.051731+0000","last_became_active":"2024-04-17T19:47:48.869797+0000","last_became_peered":"2024-04-17T19:47:48.869797+0000","last_unstale":"2024-04-17T19:47:55.051731+0000","last_undegraded":"2024-04-17T19:47:55.051731+0000","last_fullsized":"2024-04-17T19:47:55.051731+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:51:21.463081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956297+0000","last_change":"2024-04-17T19:47:53.944568+0000","last_active":"2024-04-17T19:47:56.956297+0000","last_peered":"2024-04-17T19:47:56.956297+0000","last_clean":"2024-04-17T19:47:56.956297+0000","last_became_active":"2024-04-17T19:47:53.943151+0000","last_became_peered":"2024-04-17T19:47:53.943151+0000","last_unstale":"2024-04-17T19:47:56.956297+0000","last_undegraded":"2024-04-17T19:47:56.956297+0000","last_fullsized":"2024-04-17T19:47:56.956297+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:19:00.150450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842029+0000","last_change":"2024-04-17T19:47:49.902401+0000","last_active":"2024-04-17T19:47:55.842029+0000","last_peered":"2024-04-17T19:47:55.842029+0000","last_clean":"2024-04-17T19:47:55.842029+0000","last_became_active":"2024-04-17T19:47:49.902166+0000","last_became_peered":"2024-04-17T19:47:49.902166+0000","last_unstale":"2024-04-17T19:47:55.842029+0000","last_undegraded":"2024-04-17T19:47:55.842029+0000","last_fullsized":"2024-04-17T19:47:55.842029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:54:13.129748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"24'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.768195+0000","last_change":"2024-04-17T19:47:48.900800+0000","last_active":"2024-04-17T19:48:15.768195+0000","last_peered":"2024-04-17T19:48:15.768195+0000","last_clean":"2024-04-17T19:48:15.768195+0000","last_became_active":"2024-04-17T19:47:48.900188+0000","last_became_peered":"2024-04-17T19:47:48.900188+0000","last_unstale":"2024-04-17T19:48:15.768195+0000","last_undegraded":"2024-04-17T19:48:15.768195+0000","last_fullsized":"2024-04-17T19:48:15.768195+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:05:04.305871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923962+0000","last_change":"2024-04-17T19:47:53.928721+0000","last_active":"2024-04-17T19:47:54.923962+0000","last_peered":"2024-04-17T19:47:54.923962+0000","last_clean":"2024-04-17T19:47:54.923962+0000","last_became_active":"2024-04-17T19:47:53.928311+0000","last_became_peered":"2024-04-17T19:47:53.928311+0000","last_unstale":"2024-04-17T19:47:54.923962+0000","last_undegraded":"2024-04-17T19:47:54.923962+0000","last_fullsized":"2024-04-17T19:47:54.923962+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:35:28.870803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832516+0000","last_change":"2024-04-17T19:47:49.905421+0000","last_active":"2024-04-17T19:47:56.832516+0000","last_peered":"2024-04-17T19:47:56.832516+0000","last_clean":"2024-04-17T19:47:56.832516+0000","last_became_active":"2024-04-17T19:47:49.904646+0000","last_became_peered":"2024-04-17T19:47:49.904646+0000","last_unstale":"2024-04-17T19:47:56.832516+0000","last_undegraded":"2024-04-17T19:47:56.832516+0000","last_fullsized":"2024-04-17T19:47:56.832516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:02:23.936080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"24'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.770374+0000","last_change":"2024-04-17T19:47:48.895356+0000","last_active":"2024-04-17T19:48:15.770374+0000","last_peered":"2024-04-17T19:48:15.770374+0000","last_clean":"2024-04-17T19:48:15.770374+0000","last_became_active":"2024-04-17T19:47:48.889225+0000","last_became_peered":"2024-04-17T19:47:48.889225+0000","last_unstale":"2024-04-17T19:48:15.770374+0000","last_undegraded":"2024-04-17T19:48:15.770374+0000","last_fullsized":"2024-04-17T19:48:15.770374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:00:22.587917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923872+0000","last_change":"2024-04-17T19:47:53.940206+0000","last_active":"2024-04-17T19:47:54.923872+0000","last_peered":"2024-04-17T19:47:54.923872+0000","last_clean":"2024-04-17T19:47:54.923872+0000","last_became_active":"2024-04-17T19:47:53.939843+0000","last_became_peered":"2024-04-17T19:47:53.939843+0000","last_unstale":"2024-04-17T19:47:54.923872+0000","last_undegraded":"2024-04-17T19:47:54.923872+0000","last_fullsized":"2024-04-17T19:47:54.923872+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:18:55.562573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841764+0000","last_change":"2024-04-17T19:47:53.920258+0000","last_active":"2024-04-17T19:47:55.841764+0000","last_peered":"2024-04-17T19:47:55.841764+0000","last_clean":"2024-04-17T19:47:55.841764+0000","last_became_active":"2024-04-17T19:47:53.918932+0000","last_became_peered":"2024-04-17T19:47:53.918932+0000","last_unstale":"2024-04-17T19:47:55.841764+0000","last_undegraded":"2024-04-17T19:47:55.841764+0000","last_fullsized":"2024-04-17T19:47:55.841764+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:45:55.456803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.050900+0000","last_change":"2024-04-17T19:47:49.861692+0000","last_active":"2024-04-17T19:47:55.050900+0000","last_peered":"2024-04-17T19:47:55.050900+0000","last_clean":"2024-04-17T19:47:55.050900+0000","last_became_active":"2024-04-17T19:47:49.861333+0000","last_became_peered":"2024-04-17T19:47:49.861333+0000","last_unstale":"2024-04-17T19:47:55.050900+0000","last_undegraded":"2024-04-17T19:47:55.050900+0000","last_fullsized":"2024-04-17T19:47:55.050900+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:47:13.317679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"24'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832300+0000","last_change":"2024-04-17T19:47:48.873355+0000","last_active":"2024-04-17T19:47:56.832300+0000","last_peered":"2024-04-17T19:47:56.832300+0000","last_clean":"2024-04-17T19:47:56.832300+0000","last_became_active":"2024-04-17T19:47:48.872427+0000","last_became_peered":"2024-04-17T19:47:48.872427+0000","last_unstale":"2024-04-17T19:47:56.832300+0000","last_undegraded":"2024-04-17T19:47:56.832300+0000","last_fullsized":"2024-04-17T19:47:56.832300+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:32:41.792605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929095+0000","last_change":"2024-04-17T19:47:53.952602+0000","last_active":"2024-04-17T19:47:54.929095+0000","last_peered":"2024-04-17T19:47:54.929095+0000","last_clean":"2024-04-17T19:47:54.929095+0000","last_became_active":"2024-04-17T19:47:53.951750+0000","last_became_peered":"2024-04-17T19:47:53.951750+0000","last_unstale":"2024-04-17T19:47:54.929095+0000","last_undegraded":"2024-04-17T19:47:54.929095+0000","last_fullsized":"2024-04-17T19:47:54.929095+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:13:08.485238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923863+0000","last_change":"2024-04-17T19:47:49.862071+0000","last_active":"2024-04-17T19:47:54.923863+0000","last_peered":"2024-04-17T19:47:54.923863+0000","last_clean":"2024-04-17T19:47:54.923863+0000","last_became_active":"2024-04-17T19:47:49.860809+0000","last_became_peered":"2024-04-17T19:47:49.860809+0000","last_unstale":"2024-04-17T19:47:54.923863+0000","last_undegraded":"2024-04-17T19:47:54.923863+0000","last_fullsized":"2024-04-17T19:47:54.923863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:42:34.451839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924173+0000","last_change":"2024-04-17T19:47:48.867855+0000","last_active":"2024-04-17T19:47:54.924173+0000","last_peered":"2024-04-17T19:47:54.924173+0000","last_clean":"2024-04-17T19:47:54.924173+0000","last_became_active":"2024-04-17T19:47:48.867532+0000","last_became_peered":"2024-04-17T19:47:48.867532+0000","last_unstale":"2024-04-17T19:47:54.924173+0000","last_undegraded":"2024-04-17T19:47:54.924173+0000","last_fullsized":"2024-04-17T19:47:54.924173+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:21:26.903245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"23'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927964+0000","last_change":"2024-04-17T19:47:48.894946+0000","last_active":"2024-04-17T19:47:54.927964+0000","last_peered":"2024-04-17T19:47:54.927964+0000","last_clean":"2024-04-17T19:47:54.927964+0000","last_became_active":"2024-04-17T19:47:48.893407+0000","last_became_peered":"2024-04-17T19:47:48.893407+0000","last_unstale":"2024-04-17T19:47:54.927964+0000","last_undegraded":"2024-04-17T19:47:54.927964+0000","last_fullsized":"2024-04-17T19:47:54.927964+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:09:56.284638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842374+0000","last_change":"2024-04-17T19:47:49.859985+0000","last_active":"2024-04-17T19:47:55.842374+0000","last_peered":"2024-04-17T19:47:55.842374+0000","last_clean":"2024-04-17T19:47:55.842374+0000","last_became_active":"2024-04-17T19:47:49.859635+0000","last_became_peered":"2024-04-17T19:47:49.859635+0000","last_unstale":"2024-04-17T19:47:55.842374+0000","last_undegraded":"2024-04-17T19:47:55.842374+0000","last_fullsized":"2024-04-17T19:47:55.842374+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:04:14.834218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926673+0000","last_change":"2024-04-17T19:47:53.925444+0000","last_active":"2024-04-17T19:47:54.926673+0000","last_peered":"2024-04-17T19:47:54.926673+0000","last_clean":"2024-04-17T19:47:54.926673+0000","last_became_active":"2024-04-17T19:47:53.924805+0000","last_became_peered":"2024-04-17T19:47:53.924805+0000","last_unstale":"2024-04-17T19:47:54.926673+0000","last_undegraded":"2024-04-17T19:47:54.926673+0000","last_fullsized":"2024-04-17T19:47:54.926673+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:19:24.022368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"24'5","reported_seq":27,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:48:15.775537+0000","last_change":"2024-04-17T19:47:48.867176+0000","last_active":"2024-04-17T19:48:15.775537+0000","last_peered":"2024-04-17T19:48:15.775537+0000","last_clean":"2024-04-17T19:48:15.775537+0000","last_became_active":"2024-04-17T19:47:48.866817+0000","last_became_peered":"2024-04-17T19:47:48.866817+0000","last_unstale":"2024-04-17T19:48:15.775537+0000","last_undegraded":"2024-04-17T19:48:15.775537+0000","last_fullsized":"2024-04-17T19:48:15.775537+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:44:30.842583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.955878+0000","last_change":"2024-04-17T19:47:49.858924+0000","last_active":"2024-04-17T19:47:56.955878+0000","last_peered":"2024-04-17T19:47:56.955878+0000","last_clean":"2024-04-17T19:47:56.955878+0000","last_became_active":"2024-04-17T19:47:49.857820+0000","last_became_peered":"2024-04-17T19:47:49.857820+0000","last_unstale":"2024-04-17T19:47:56.955878+0000","last_undegraded":"2024-04-17T19:47:56.955878+0000","last_fullsized":"2024-04-17T19:47:56.955878+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:09:28.706054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924516+0000","last_change":"2024-04-17T19:47:53.933337+0000","last_active":"2024-04-17T19:47:54.924516+0000","last_peered":"2024-04-17T19:47:54.924516+0000","last_clean":"2024-04-17T19:47:54.924516+0000","last_became_active":"2024-04-17T19:47:53.931623+0000","last_became_peered":"2024-04-17T19:47:53.931623+0000","last_unstale":"2024-04-17T19:47:54.924516+0000","last_undegraded":"2024-04-17T19:47:54.924516+0000","last_fullsized":"2024-04-17T19:47:54.924516+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:57:37.600957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.927056+0000","last_change":"2024-04-17T19:47:49.881093+0000","last_active":"2024-04-17T19:47:54.927056+0000","last_peered":"2024-04-17T19:47:54.927056+0000","last_clean":"2024-04-17T19:47:54.927056+0000","last_became_active":"2024-04-17T19:47:49.880210+0000","last_became_peered":"2024-04-17T19:47:49.880210+0000","last_unstale":"2024-04-17T19:47:54.927056+0000","last_undegraded":"2024-04-17T19:47:54.927056+0000","last_fullsized":"2024-04-17T19:47:54.927056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:12:15.522651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926335+0000","last_change":"2024-04-17T19:47:49.886936+0000","last_active":"2024-04-17T19:47:54.926335+0000","last_peered":"2024-04-17T19:47:54.926335+0000","last_clean":"2024-04-17T19:47:54.926335+0000","last_became_active":"2024-04-17T19:47:49.886580+0000","last_became_peered":"2024-04-17T19:47:49.886580+0000","last_unstale":"2024-04-17T19:47:54.926335+0000","last_undegraded":"2024-04-17T19:47:54.926335+0000","last_fullsized":"2024-04-17T19:47:54.926335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:00:44.599463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832155+0000","last_change":"2024-04-17T19:47:49.905445+0000","last_active":"2024-04-17T19:47:56.832155+0000","last_peered":"2024-04-17T19:47:56.832155+0000","last_clean":"2024-04-17T19:47:56.832155+0000","last_became_active":"2024-04-17T19:47:49.896510+0000","last_became_peered":"2024-04-17T19:47:49.896510+0000","last_unstale":"2024-04-17T19:47:56.832155+0000","last_undegraded":"2024-04-17T19:47:56.832155+0000","last_fullsized":"2024-04-17T19:47:56.832155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:23:27.401861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842320+0000","last_change":"2024-04-17T19:47:49.850490+0000","last_active":"2024-04-17T19:47:55.842320+0000","last_peered":"2024-04-17T19:47:55.842320+0000","last_clean":"2024-04-17T19:47:55.842320+0000","last_became_active":"2024-04-17T19:47:49.849344+0000","last_became_peered":"2024-04-17T19:47:49.849344+0000","last_unstale":"2024-04-17T19:47:55.842320+0000","last_undegraded":"2024-04-17T19:47:55.842320+0000","last_fullsized":"2024-04-17T19:47:55.842320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:44:32.231058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924753+0000","last_change":"2024-04-17T19:47:49.857737+0000","last_active":"2024-04-17T19:47:54.924753+0000","last_peered":"2024-04-17T19:47:54.924753+0000","last_clean":"2024-04-17T19:47:54.924753+0000","last_became_active":"2024-04-17T19:47:49.857476+0000","last_became_peered":"2024-04-17T19:47:49.857476+0000","last_unstale":"2024-04-17T19:47:54.924753+0000","last_undegraded":"2024-04-17T19:47:54.924753+0000","last_fullsized":"2024-04-17T19:47:54.924753+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:50:13.070628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051526+0000","last_change":"2024-04-17T19:47:49.864985+0000","last_active":"2024-04-17T19:47:55.051526+0000","last_peered":"2024-04-17T19:47:55.051526+0000","last_clean":"2024-04-17T19:47:55.051526+0000","last_became_active":"2024-04-17T19:47:49.864329+0000","last_became_peered":"2024-04-17T19:47:49.864329+0000","last_unstale":"2024-04-17T19:47:55.051526+0000","last_undegraded":"2024-04-17T19:47:55.051526+0000","last_fullsized":"2024-04-17T19:47:55.051526+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:43:57.448269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929577+0000","last_change":"2024-04-17T19:47:49.881785+0000","last_active":"2024-04-17T19:47:54.929577+0000","last_peered":"2024-04-17T19:47:54.929577+0000","last_clean":"2024-04-17T19:47:54.929577+0000","last_became_active":"2024-04-17T19:47:49.880939+0000","last_became_peered":"2024-04-17T19:47:49.880939+0000","last_unstale":"2024-04-17T19:47:54.929577+0000","last_undegraded":"2024-04-17T19:47:54.929577+0000","last_fullsized":"2024-04-17T19:47:54.929577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:49:07.194643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924461+0000","last_change":"2024-04-17T19:47:49.884944+0000","last_active":"2024-04-17T19:47:54.924461+0000","last_peered":"2024-04-17T19:47:54.924461+0000","last_clean":"2024-04-17T19:47:54.924461+0000","last_became_active":"2024-04-17T19:47:49.884559+0000","last_became_peered":"2024-04-17T19:47:49.884559+0000","last_unstale":"2024-04-17T19:47:54.924461+0000","last_undegraded":"2024-04-17T19:47:54.924461+0000","last_fullsized":"2024-04-17T19:47:54.924461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:00.485472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.831772+0000","last_change":"2024-04-17T19:47:49.891230+0000","last_active":"2024-04-17T19:47:56.831772+0000","last_peered":"2024-04-17T19:47:56.831772+0000","last_clean":"2024-04-17T19:47:56.831772+0000","last_became_active":"2024-04-17T19:47:49.886001+0000","last_became_peered":"2024-04-17T19:47:49.886001+0000","last_unstale":"2024-04-17T19:47:56.831772+0000","last_undegraded":"2024-04-17T19:47:56.831772+0000","last_fullsized":"2024-04-17T19:47:56.831772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:00.485472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923958+0000","last_change":"2024-04-17T19:47:49.859382+0000","last_active":"2024-04-17T19:47:54.923958+0000","last_peered":"2024-04-17T19:47:54.923958+0000","last_clean":"2024-04-17T19:47:54.923958+0000","last_became_active":"2024-04-17T19:47:49.859170+0000","last_became_peered":"2024-04-17T19:47:49.859170+0000","last_unstale":"2024-04-17T19:47:54.923958+0000","last_undegraded":"2024-04-17T19:47:54.923958+0000","last_fullsized":"2024-04-17T19:47:54.923958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:58:51.111053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832143+0000","last_change":"2024-04-17T19:47:49.909188+0000","last_active":"2024-04-17T19:47:56.832143+0000","last_peered":"2024-04-17T19:47:56.832143+0000","last_clean":"2024-04-17T19:47:56.832143+0000","last_became_active":"2024-04-17T19:47:49.908432+0000","last_became_peered":"2024-04-17T19:47:49.908432+0000","last_unstale":"2024-04-17T19:47:56.832143+0000","last_undegraded":"2024-04-17T19:47:56.832143+0000","last_fullsized":"2024-04-17T19:47:56.832143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:16:21.209634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.842710+0000","last_change":"2024-04-17T19:47:49.876406+0000","last_active":"2024-04-17T19:47:55.842710+0000","last_peered":"2024-04-17T19:47:55.842710+0000","last_clean":"2024-04-17T19:47:55.842710+0000","last_became_active":"2024-04-17T19:47:49.876132+0000","last_became_peered":"2024-04-17T19:47:49.876132+0000","last_unstale":"2024-04-17T19:47:55.842710+0000","last_undegraded":"2024-04-17T19:47:55.842710+0000","last_fullsized":"2024-04-17T19:47:55.842710+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:51:20.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.928888+0000","last_change":"2024-04-17T19:47:49.896244+0000","last_active":"2024-04-17T19:47:54.928888+0000","last_peered":"2024-04-17T19:47:54.928888+0000","last_clean":"2024-04-17T19:47:54.928888+0000","last_became_active":"2024-04-17T19:47:49.895380+0000","last_became_peered":"2024-04-17T19:47:49.895380+0000","last_unstale":"2024-04-17T19:47:54.928888+0000","last_undegraded":"2024-04-17T19:47:54.928888+0000","last_fullsized":"2024-04-17T19:47:54.928888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:04:43.091307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049138+0000","last_change":"2024-04-17T19:47:49.870551+0000","last_active":"2024-04-17T19:47:56.049138+0000","last_peered":"2024-04-17T19:47:56.049138+0000","last_clean":"2024-04-17T19:47:56.049138+0000","last_became_active":"2024-04-17T19:47:49.869315+0000","last_became_peered":"2024-04-17T19:47:49.869315+0000","last_unstale":"2024-04-17T19:47:56.049138+0000","last_undegraded":"2024-04-17T19:47:56.049138+0000","last_fullsized":"2024-04-17T19:47:56.049138+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:06.341123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.926618+0000","last_change":"2024-04-17T19:47:49.880060+0000","last_active":"2024-04-17T19:47:54.926618+0000","last_peered":"2024-04-17T19:47:54.926618+0000","last_clean":"2024-04-17T19:47:54.926618+0000","last_became_active":"2024-04-17T19:47:49.879711+0000","last_became_peered":"2024-04-17T19:47:49.879711+0000","last_unstale":"2024-04-17T19:47:54.926618+0000","last_undegraded":"2024-04-17T19:47:54.926618+0000","last_fullsized":"2024-04-17T19:47:54.926618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:41:25.628870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.833444+0000","last_change":"2024-04-17T19:47:49.895452+0000","last_active":"2024-04-17T19:47:56.833444+0000","last_peered":"2024-04-17T19:47:56.833444+0000","last_clean":"2024-04-17T19:47:56.833444+0000","last_became_active":"2024-04-17T19:47:49.894684+0000","last_became_peered":"2024-04-17T19:47:49.894684+0000","last_unstale":"2024-04-17T19:47:56.833444+0000","last_undegraded":"2024-04-17T19:47:56.833444+0000","last_fullsized":"2024-04-17T19:47:56.833444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:15:55.224238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.831437+0000","last_change":"2024-04-17T19:47:49.886902+0000","last_active":"2024-04-17T19:47:56.831437+0000","last_peered":"2024-04-17T19:47:56.831437+0000","last_clean":"2024-04-17T19:47:56.831437+0000","last_became_active":"2024-04-17T19:47:49.885451+0000","last_became_peered":"2024-04-17T19:47:49.885451+0000","last_unstale":"2024-04-17T19:47:56.831437+0000","last_undegraded":"2024-04-17T19:47:56.831437+0000","last_fullsized":"2024-04-17T19:47:56.831437+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:12:09.196610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.050989+0000","last_change":"2024-04-17T19:47:49.858908+0000","last_active":"2024-04-17T19:47:55.050989+0000","last_peered":"2024-04-17T19:47:55.050989+0000","last_clean":"2024-04-17T19:47:55.050989+0000","last_became_active":"2024-04-17T19:47:49.858457+0000","last_became_peered":"2024-04-17T19:47:49.858457+0000","last_unstale":"2024-04-17T19:47:55.050989+0000","last_undegraded":"2024-04-17T19:47:55.050989+0000","last_fullsized":"2024-04-17T19:47:55.050989+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:09.467859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841618+0000","last_change":"2024-04-17T19:47:49.856371+0000","last_active":"2024-04-17T19:47:55.841618+0000","last_peered":"2024-04-17T19:47:55.841618+0000","last_clean":"2024-04-17T19:47:55.841618+0000","last_became_active":"2024-04-17T19:47:49.856062+0000","last_became_peered":"2024-04-17T19:47:49.856062+0000","last_unstale":"2024-04-17T19:47:55.841618+0000","last_undegraded":"2024-04-17T19:47:55.841618+0000","last_fullsized":"2024-04-17T19:47:55.841618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:39:54.502619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.924769+0000","last_change":"2024-04-17T19:47:49.873350+0000","last_active":"2024-04-17T19:47:54.924769+0000","last_peered":"2024-04-17T19:47:54.924769+0000","last_clean":"2024-04-17T19:47:54.924769+0000","last_became_active":"2024-04-17T19:47:49.873019+0000","last_became_peered":"2024-04-17T19:47:49.873019+0000","last_unstale":"2024-04-17T19:47:54.924769+0000","last_undegraded":"2024-04-17T19:47:54.924769+0000","last_fullsized":"2024-04-17T19:47:54.924769+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:06:58.641727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.956704+0000","last_change":"2024-04-17T19:47:49.891551+0000","last_active":"2024-04-17T19:47:56.956704+0000","last_peered":"2024-04-17T19:47:56.956704+0000","last_clean":"2024-04-17T19:47:56.956704+0000","last_became_active":"2024-04-17T19:47:49.890132+0000","last_became_peered":"2024-04-17T19:47:49.890132+0000","last_unstale":"2024-04-17T19:47:56.956704+0000","last_undegraded":"2024-04-17T19:47:56.956704+0000","last_fullsized":"2024-04-17T19:47:56.956704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:26:22.971489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.928586+0000","last_change":"2024-04-17T19:47:49.903854+0000","last_active":"2024-04-17T19:47:54.928586+0000","last_peered":"2024-04-17T19:47:54.928586+0000","last_clean":"2024-04-17T19:47:54.928586+0000","last_became_active":"2024-04-17T19:47:49.903178+0000","last_became_peered":"2024-04-17T19:47:49.903178+0000","last_unstale":"2024-04-17T19:47:54.928586+0000","last_undegraded":"2024-04-17T19:47:54.928586+0000","last_fullsized":"2024-04-17T19:47:54.928586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:24:52.699532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.929394+0000","last_change":"2024-04-17T19:47:49.900848+0000","last_active":"2024-04-17T19:47:54.929394+0000","last_peered":"2024-04-17T19:47:54.929394+0000","last_clean":"2024-04-17T19:47:54.929394+0000","last_became_active":"2024-04-17T19:47:49.900052+0000","last_became_peered":"2024-04-17T19:47:49.900052+0000","last_unstale":"2024-04-17T19:47:54.929394+0000","last_undegraded":"2024-04-17T19:47:54.929394+0000","last_fullsized":"2024-04-17T19:47:54.929394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:28:41.115173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049711+0000","last_change":"2024-04-17T19:47:49.906531+0000","last_active":"2024-04-17T19:47:56.049711+0000","last_peered":"2024-04-17T19:47:56.049711+0000","last_clean":"2024-04-17T19:47:56.049711+0000","last_became_active":"2024-04-17T19:47:49.905217+0000","last_became_peered":"2024-04-17T19:47:49.905217+0000","last_unstale":"2024-04-17T19:47:56.049711+0000","last_undegraded":"2024-04-17T19:47:56.049711+0000","last_fullsized":"2024-04-17T19:47:56.049711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:22.078473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:54.923726+0000","last_change":"2024-04-17T19:47:49.864577+0000","last_active":"2024-04-17T19:47:54.923726+0000","last_peered":"2024-04-17T19:47:54.923726+0000","last_clean":"2024-04-17T19:47:54.923726+0000","last_became_active":"2024-04-17T19:47:49.864251+0000","last_became_peered":"2024-04-17T19:47:49.864251+0000","last_unstale":"2024-04-17T19:47:54.923726+0000","last_undegraded":"2024-04-17T19:47:54.923726+0000","last_fullsized":"2024-04-17T19:47:54.923726+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:32:10.329543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.047770+0000","last_change":"2024-04-17T19:47:49.869245+0000","last_active":"2024-04-17T19:47:56.047770+0000","last_peered":"2024-04-17T19:47:56.047770+0000","last_clean":"2024-04-17T19:47:56.047770+0000","last_became_active":"2024-04-17T19:47:49.868156+0000","last_became_peered":"2024-04-17T19:47:49.868156+0000","last_unstale":"2024-04-17T19:47:56.047770+0000","last_undegraded":"2024-04-17T19:47:56.047770+0000","last_fullsized":"2024-04-17T19:47:56.047770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:58:39.814933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.831810+0000","last_change":"2024-04-17T19:47:49.907573+0000","last_active":"2024-04-17T19:47:56.831810+0000","last_peered":"2024-04-17T19:47:56.831810+0000","last_clean":"2024-04-17T19:47:56.831810+0000","last_became_active":"2024-04-17T19:47:49.897308+0000","last_became_peered":"2024-04-17T19:47:49.897308+0000","last_unstale":"2024-04-17T19:47:56.831810+0000","last_undegraded":"2024-04-17T19:47:56.831810+0000","last_fullsized":"2024-04-17T19:47:56.831810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:02:03.101317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.047866+0000","last_change":"2024-04-17T19:47:49.876052+0000","last_active":"2024-04-17T19:47:56.047866+0000","last_peered":"2024-04-17T19:47:56.047866+0000","last_clean":"2024-04-17T19:47:56.047866+0000","last_became_active":"2024-04-17T19:47:49.875008+0000","last_became_peered":"2024-04-17T19:47:49.875008+0000","last_unstale":"2024-04-17T19:47:56.047866+0000","last_undegraded":"2024-04-17T19:47:56.047866+0000","last_fullsized":"2024-04-17T19:47:56.047866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:36:14.486150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.832276+0000","last_change":"2024-04-17T19:47:49.906840+0000","last_active":"2024-04-17T19:47:56.832276+0000","last_peered":"2024-04-17T19:47:56.832276+0000","last_clean":"2024-04-17T19:47:56.832276+0000","last_became_active":"2024-04-17T19:47:49.905703+0000","last_became_peered":"2024-04-17T19:47:49.905703+0000","last_unstale":"2024-04-17T19:47:56.832276+0000","last_undegraded":"2024-04-17T19:47:56.832276+0000","last_fullsized":"2024-04-17T19:47:56.832276+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:28:26.686472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.841783+0000","last_change":"2024-04-17T19:47:49.869425+0000","last_active":"2024-04-17T19:47:55.841783+0000","last_peered":"2024-04-17T19:47:55.841783+0000","last_clean":"2024-04-17T19:47:55.841783+0000","last_became_active":"2024-04-17T19:47:49.868758+0000","last_became_peered":"2024-04-17T19:47:49.868758+0000","last_unstale":"2024-04-17T19:47:55.841783+0000","last_undegraded":"2024-04-17T19:47:55.841783+0000","last_fullsized":"2024-04-17T19:47:55.841783+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:57:27.214150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:55.051778+0000","last_change":"2024-04-17T19:47:49.856309+0000","last_active":"2024-04-17T19:47:55.051778+0000","last_peered":"2024-04-17T19:47:55.051778+0000","last_clean":"2024-04-17T19:47:55.051778+0000","last_became_active":"2024-04-17T19:47:49.856026+0000","last_became_peered":"2024-04-17T19:47:49.856026+0000","last_unstale":"2024-04-17T19:47:55.051778+0000","last_undegraded":"2024-04-17T19:47:55.051778+0000","last_fullsized":"2024-04-17T19:47:55.051778+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:51:57.963637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-17T19:47:56.049403+0000","last_change":"2024-04-17T19:47:49.890343+0000","last_active":"2024-04-17T19:47:56.049403+0000","last_peered":"2024-04-17T19:47:56.049403+0000","last_clean":"2024-04-17T19:47:56.049403+0000","last_became_active":"2024-04-17T19:47:49.880566+0000","last_became_peered":"2024-04-17T19:47:49.880566+0000","last_unstale":"2024-04-17T19:47:56.049403+0000","last_undegraded":"2024-04-17T19:47:56.049403+0000","last_fullsized":"2024-04-17T19:47:56.049403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:19:29.359024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6046,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":12092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574860,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574860,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32024,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339816,"statfs":{"total":96636764160,"available":96603971584,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T19:48:19.490 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-17T19:48:19.490 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-17T19:48:19.490 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-17T19:48:19.490 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-17T19:48:19.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.689+0000 7f53f8e00640 1 Processor -- start 2024-04-17T19:48:19.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.689+0000 7f53f8e00640 1 -- start start 2024-04-17T19:48:19.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.689+0000 7f53f8e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f53f4159e60 0x7f53f40cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:19.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.689+0000 7f53f8e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f53f4153d20 0x7f53f4071940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:19.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.689+0000 7f53f8e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53f4071e80 0x7f53f40c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:19.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.689+0000 7f53f8e00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f53f40d0320 con 0x7f53f4153d20 2024-04-17T19:48:19.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.689+0000 7f53f8e00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f53f40c4930 con 0x7f53f4159e60 2024-04-17T19:48:19.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.689+0000 7f53f8e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f53f40cf620 con 0x7f53f4071e80 2024-04-17T19:48:19.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.689+0000 7f53ebe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f53f4153d20 0x7f53f4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:19.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.693+0000 7f53ebe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f53f4153d20 0x7f53f4071940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58170/0 (socket says 172.21.15.29:58170) 2024-04-17T19:48:19.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.693+0000 7f53ebe00640 1 -- 172.21.15.29:0/1743500749 learned_addr learned my addr 172.21.15.29:0/1743500749 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:19.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.693+0000 7f53f1e00640 1 --2- 172.21.15.29:0/1743500749 >> v2:172.21.15.37:3300/0 conn(0x7f53f4159e60 0x7f53f40cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:19.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.693+0000 7f53f2800640 1 --2- 172.21.15.29:0/1743500749 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53f4071e80 0x7f53f40c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:19.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.693+0000 7f53ebe00640 1 -- 172.21.15.29:0/1743500749 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53f4071e80 msgr2=0x7f53f40c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:19.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.693+0000 7f53ebe00640 1 --2- 172.21.15.29:0/1743500749 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53f4071e80 0x7f53f40c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.693+0000 7f53ebe00640 1 -- 172.21.15.29:0/1743500749 >> v2:172.21.15.37:3300/0 conn(0x7f53f4159e60 msgr2=0x7f53f40cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:19.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.693+0000 7f53ebe00640 1 --2- 172.21.15.29:0/1743500749 >> v2:172.21.15.37:3300/0 conn(0x7f53f4159e60 0x7f53f40cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.693+0000 7f53ebe00640 1 -- 172.21.15.29:0/1743500749 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f40c5190 con 0x7f53f4153d20 2024-04-17T19:48:19.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.693+0000 7f53f2800640 1 --2- 172.21.15.29:0/1743500749 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53f4071e80 0x7f53f40c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:19.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.693+0000 7f53ebe00640 1 --2- 172.21.15.29:0/1743500749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f4153d20 0x7f53f4071940 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f53f40733a0 tx=0x7f53dc01c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebe33d6d782b465 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:19.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.693+0000 7f53f1400640 1 -- 172.21.15.29:0/1743500749 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53dc0b0070 con 0x7f53f4153d20 2024-04-17T19:48:19.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.693+0000 7f53f1400640 1 -- 172.21.15.29:0/1743500749 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53dc0ab600 con 0x7f53f4153d20 2024-04-17T19:48:19.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.693+0000 7f53f1400640 1 -- 172.21.15.29:0/1743500749 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53dc0ab9d0 con 0x7f53f4153d20 2024-04-17T19:48:19.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.697+0000 7f53f8e00640 1 -- 172.21.15.29:0/1743500749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f4153d20 msgr2=0x7f53f4071940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:19.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.697+0000 7f53f8e00640 1 --2- 172.21.15.29:0/1743500749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f4153d20 0x7f53f4071940 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f53f40733a0 tx=0x7f53dc01c260 comp rx=0 tx=0).stop 2024-04-17T19:48:19.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.697+0000 7f53f8e00640 1 -- 172.21.15.29:0/1743500749 shutdown_connections 2024-04-17T19:48:19.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.697+0000 7f53f8e00640 1 --2- 172.21.15.29:0/1743500749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f4153d20 0x7f53f4071940 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.697+0000 7f53f8e00640 1 --2- 172.21.15.29:0/1743500749 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53f4071e80 0x7f53f40c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.697+0000 7f53f8e00640 1 --2- 172.21.15.29:0/1743500749 >> v2:172.21.15.37:3300/0 conn(0x7f53f4159e60 0x7f53f40cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.697+0000 7f53f8e00640 1 -- 172.21.15.29:0/1743500749 >> 172.21.15.29:0/1743500749 conn(0x7f53f4067eb0 msgr2=0x7f53f4070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:19.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.697+0000 7f53f8e00640 1 -- 172.21.15.29:0/1743500749 shutdown_connections 2024-04-17T19:48:19.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.697+0000 7f53f8e00640 1 -- 172.21.15.29:0/1743500749 wait complete. 2024-04-17T19:48:19.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.697+0000 7f53f8e00640 1 Processor -- start 2024-04-17T19:48:19.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.697+0000 7f53f8e00640 1 -- start start 2024-04-17T19:48:19.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53f8e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53f4071e80 0x7f53f406da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:19.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53f8e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f4153d20 0x7f53f41f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:19.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53f8e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53f4159e60 0x7f53f40cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:19.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53f8e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f53f40cbd10 con 0x7f53f4153d20 2024-04-17T19:48:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53f8e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f53f40cbb90 con 0x7f53f4071e80 2024-04-17T19:48:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53f8e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f53f40c4510 con 0x7f53f4159e60 2024-04-17T19:48:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53ebe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f4153d20 0x7f53f41f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53ebe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f4153d20 0x7f53f41f5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58180/0 (socket says 172.21.15.29:58180) 2024-04-17T19:48:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53ebe00640 1 -- 172.21.15.29:0/307316695 learned_addr learned my addr 172.21.15.29:0/307316695 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53f1e00640 1 --2- 172.21.15.29:0/307316695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53f4071e80 0x7f53f406da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53f2800640 1 --2- 172.21.15.29:0/307316695 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53f4159e60 0x7f53f40cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53ebe00640 1 -- 172.21.15.29:0/307316695 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53f4159e60 msgr2=0x7f53f40cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53ebe00640 1 --2- 172.21.15.29:0/307316695 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53f4159e60 0x7f53f40cd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53ebe00640 1 -- 172.21.15.29:0/307316695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53f4071e80 msgr2=0x7f53f406da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53ebe00640 1 --2- 172.21.15.29:0/307316695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53f4071e80 0x7f53f406da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53ebe00640 1 -- 172.21.15.29:0/307316695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f40cb6a0 con 0x7f53f4153d20 2024-04-17T19:48:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53f1e00640 1 --2- 172.21.15.29:0/307316695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53f4071e80 0x7f53f406da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53ebe00640 1 --2- 172.21.15.29:0/307316695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f4153d20 0x7f53f41f5d00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f53dc01a910 tx=0x7f53dc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53ea000640 1 -- 172.21.15.29:0/307316695 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53dc0ba070 con 0x7f53f4153d20 2024-04-17T19:48:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53ea000640 1 -- 172.21.15.29:0/307316695 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53dc0b4620 con 0x7f53f4153d20 2024-04-17T19:48:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53ea000640 1 -- 172.21.15.29:0/307316695 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53dc0abe60 con 0x7f53f4153d20 2024-04-17T19:48:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53f8e00640 1 -- 172.21.15.29:0/307316695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53f41f6240 con 0x7f53f4153d20 2024-04-17T19:48:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53f8e00640 1 -- 172.21.15.29:0/307316695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53f40b3b50 con 0x7f53f4153d20 2024-04-17T19:48:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53f8e00640 1 -- 172.21.15.29:0/307316695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53f4069b90 con 0x7f53f4153d20 2024-04-17T19:48:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.701+0000 7f53ea000640 1 -- 172.21.15.29:0/307316695 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f53dc0bd070 con 0x7f53f4153d20 2024-04-17T19:48:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.705+0000 7f53ea000640 1 --2- 172.21.15.29:0/307316695 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f53d00683b0 0x7f53d006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.705+0000 7f53ea000640 1 -- 172.21.15.29:0/307316695 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f53dc11fd40 con 0x7f53f4153d20 2024-04-17T19:48:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.705+0000 7f53f1e00640 1 --2- 172.21.15.29:0/307316695 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f53d00683b0 0x7f53d006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.705+0000 7f53f1e00640 1 --2- 172.21.15.29:0/307316695 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f53d00683b0 0x7f53d006a8c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f53e4009b10 tx=0x7f53e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:19.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.709+0000 7f53ea000640 1 -- 172.21.15.29:0/307316695 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f53dc016110 con 0x7f53f4153d20 2024-04-17T19:48:19.802 INFO:tasks.ceph.mds.c.smithi029.stderr:2024-04-17T19:48:19.813+0000 7ff3b0200640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-17T19:48:19.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.909+0000 7f53f8e00640 1 -- 172.21.15.29:0/307316695 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f53f40cce20 con 0x7f53f4153d20 2024-04-17T19:48:19.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.909+0000 7f53ea000640 1 -- 172.21.15.29:0/307316695 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7f53dc0eff20 con 0x7f53f4153d20 2024-04-17T19:48:19.899 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:19.899 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-04-17T19:48:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.913+0000 7f53f8e00640 1 -- 172.21.15.29:0/307316695 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f53d00683b0 msgr2=0x7f53d006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.913+0000 7f53f8e00640 1 --2- 172.21.15.29:0/307316695 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f53d00683b0 0x7f53d006a8c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f53e4009b10 tx=0x7f53e4008040 comp rx=0 tx=0).stop 2024-04-17T19:48:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.913+0000 7f53f8e00640 1 -- 172.21.15.29:0/307316695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f4153d20 msgr2=0x7f53f41f5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.913+0000 7f53f8e00640 1 --2- 172.21.15.29:0/307316695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f4153d20 0x7f53f41f5d00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f53dc01a910 tx=0x7f53dc019000 comp rx=0 tx=0).stop 2024-04-17T19:48:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.913+0000 7f53f8e00640 1 -- 172.21.15.29:0/307316695 shutdown_connections 2024-04-17T19:48:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.913+0000 7f53f8e00640 1 --2- 172.21.15.29:0/307316695 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f53d00683b0 0x7f53d006a8c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.913+0000 7f53f8e00640 1 --2- 172.21.15.29:0/307316695 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53f4159e60 0x7f53f40cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.913+0000 7f53f8e00640 1 --2- 172.21.15.29:0/307316695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53f4153d20 0x7f53f41f5d00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.913+0000 7f53f8e00640 1 --2- 172.21.15.29:0/307316695 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53f4071e80 0x7f53f406da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:19.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.913+0000 7f53f8e00640 1 -- 172.21.15.29:0/307316695 >> 172.21.15.29:0/307316695 conn(0x7f53f4067eb0 msgr2=0x7f53f4070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:19.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.913+0000 7f53f8e00640 1 -- 172.21.15.29:0/307316695 shutdown_connections 2024-04-17T19:48:19.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:19.913+0000 7f53f8e00640 1 -- 172.21.15.29:0/307316695 wait complete. 2024-04-17T19:48:22.790 INFO:tasks.ceph.mds.c.smithi029.stderr:2024-04-17T19:48:22.801+0000 7ff3b3400640 -1 quiesce.mds.0 failed to decode message of type 1285 v1: End of buffer [buffer:2] 2024-04-17T19:48:22.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-17T19:48:23.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.085+0000 7f73ee600640 1 Processor -- start 2024-04-17T19:48:23.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.085+0000 7f73ee600640 1 -- start start 2024-04-17T19:48:23.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.085+0000 7f73ee600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f73e8159c90 0x7f73e80ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.085+0000 7f73ee600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73e806c660 0x7f73e806ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.085+0000 7f73ee600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73e815a0a0 con 0x7f73e80ca7f0 2024-04-17T19:48:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.085+0000 7f73ee600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f73e815abe0 con 0x7f73e8159c90 2024-04-17T19:48:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.085+0000 7f73ee600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f73e815a340 con 0x7f73e806c660 2024-04-17T19:48:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.085+0000 7f73e7400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f73e8159c90 0x7f73e80ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.085+0000 7f73e7400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f73e8159c90 0x7f73e80ca2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58186/0 (socket says 172.21.15.29:58186) 2024-04-17T19:48:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.085+0000 7f73e7400640 1 -- 172.21.15.29:0/2831798184 learned_addr learned my addr 172.21.15.29:0/2831798184 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.085+0000 7f73e7e00640 1 --2- 172.21.15.29:0/2831798184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73e806c660 0x7f73e806ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.085+0000 7f73e7400640 1 -- 172.21.15.29:0/2831798184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73e806c660 msgr2=0x7f73e806ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.085+0000 7f73e7400640 1 --2- 172.21.15.29:0/2831798184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73e806c660 0x7f73e806ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.085+0000 7f73e7400640 1 -- 172.21.15.29:0/2831798184 >> v1:172.21.15.37:6789/0 conn(0x7f73e80ca7f0 legacy=0x7f73e80c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.085+0000 7f73e7400640 1 -- 172.21.15.29:0/2831798184 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e815a620 con 0x7f73e8159c90 2024-04-17T19:48:23.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.089+0000 7f73e7400640 1 --2- 172.21.15.29:0/2831798184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73e8159c90 0x7f73e80ca2b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009870 tx=0x7f73d801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=509ff9cc2b12f244 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:23.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.089+0000 7f73e6a00640 1 -- 172.21.15.29:0/2831798184 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d80a7070 con 0x7f73e8159c90 2024-04-17T19:48:23.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.089+0000 7f73e6a00640 1 -- 172.21.15.29:0/2831798184 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d80b0070 con 0x7f73e8159c90 2024-04-17T19:48:23.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.089+0000 7f73e6a00640 1 -- 172.21.15.29:0/2831798184 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d80ab890 con 0x7f73e8159c90 2024-04-17T19:48:23.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.089+0000 7f73ee600640 1 -- 172.21.15.29:0/2831798184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73e8159c90 msgr2=0x7f73e80ca2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:23.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.089+0000 7f73ee600640 1 --2- 172.21.15.29:0/2831798184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73e8159c90 0x7f73e80ca2b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009870 tx=0x7f73d801c560 comp rx=0 tx=0).stop 2024-04-17T19:48:23.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.089+0000 7f73ee600640 1 -- 172.21.15.29:0/2831798184 shutdown_connections 2024-04-17T19:48:23.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.089+0000 7f73ee600640 1 --2- 172.21.15.29:0/2831798184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73e806c660 0x7f73e806ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:23.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.089+0000 7f73ee600640 1 --2- 172.21.15.29:0/2831798184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73e8159c90 0x7f73e80ca2b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:23.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.089+0000 7f73ee600640 1 -- 172.21.15.29:0/2831798184 >> 172.21.15.29:0/2831798184 conn(0x7f73e806fde0 msgr2=0x7f73e80701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:23.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.089+0000 7f73ee600640 1 -- 172.21.15.29:0/2831798184 shutdown_connections 2024-04-17T19:48:23.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.089+0000 7f73ee600640 1 -- 172.21.15.29:0/2831798184 wait complete. 2024-04-17T19:48:23.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.093+0000 7f73ee600640 1 Processor -- start 2024-04-17T19:48:23.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.093+0000 7f73ee600640 1 -- start start 2024-04-17T19:48:23.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.093+0000 7f73ee600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73e806c660 0x7f73e80c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:23.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.093+0000 7f73ee600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73e80ca7f0 0x7f73e8153a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:23.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.093+0000 7f73ee600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73e8159c90 0x7f73e8146bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:23.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.093+0000 7f73ee600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f73e80c7680 con 0x7f73e806c660 2024-04-17T19:48:23.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.093+0000 7f73ee600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f73e806cfb0 con 0x7f73e80ca7f0 2024-04-17T19:48:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.093+0000 7f73ee600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f73e80c7800 con 0x7f73e8159c90 2024-04-17T19:48:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.093+0000 7f73e7400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73e806c660 0x7f73e80c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.093+0000 7f73e7400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73e806c660 0x7f73e80c6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58188/0 (socket says 172.21.15.29:58188) 2024-04-17T19:48:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.093+0000 7f73e7400640 1 -- 172.21.15.29:0/2031120143 learned_addr learned my addr 172.21.15.29:0/2031120143 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.093+0000 7f73dfe00640 1 --2- 172.21.15.29:0/2031120143 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73e80ca7f0 0x7f73e8153a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.093+0000 7f73e7e00640 1 --2- 172.21.15.29:0/2031120143 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73e8159c90 0x7f73e8146bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.093+0000 7f73e7400640 1 -- 172.21.15.29:0/2031120143 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73e8159c90 msgr2=0x7f73e8146bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.093+0000 7f73e7400640 1 --2- 172.21.15.29:0/2031120143 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73e8159c90 0x7f73e8146bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.093+0000 7f73e7400640 1 -- 172.21.15.29:0/2031120143 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73e80ca7f0 msgr2=0x7f73e8153a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.093+0000 7f73e7400640 1 --2- 172.21.15.29:0/2031120143 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73e80ca7f0 0x7f73e8153a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.093+0000 7f73e7400640 1 -- 172.21.15.29:0/2031120143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e8152c50 con 0x7f73e806c660 2024-04-17T19:48:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.097+0000 7f73e7400640 1 --2- 172.21.15.29:0/2031120143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73e806c660 0x7f73e80c6180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f73d801a190 tx=0x7f73d801ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.097+0000 7f73df400640 1 -- 172.21.15.29:0/2031120143 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d80be5f0 con 0x7f73e806c660 2024-04-17T19:48:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.097+0000 7f73df400640 1 -- 172.21.15.29:0/2031120143 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d80be7b0 con 0x7f73e806c660 2024-04-17T19:48:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.097+0000 7f73df400640 1 -- 172.21.15.29:0/2031120143 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d80beba0 con 0x7f73e806c660 2024-04-17T19:48:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.097+0000 7f73ee600640 1 -- 172.21.15.29:0/2031120143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73e8153220 con 0x7f73e806c660 2024-04-17T19:48:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.097+0000 7f73ee600640 1 -- 172.21.15.29:0/2031120143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73e80c5980 con 0x7f73e806c660 2024-04-17T19:48:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.097+0000 7f73ee600640 1 -- 172.21.15.29:0/2031120143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73e8069b90 con 0x7f73e806c660 2024-04-17T19:48:23.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.097+0000 7f73df400640 1 -- 172.21.15.29:0/2031120143 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f73d80b43f0 con 0x7f73e806c660 2024-04-17T19:48:23.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.097+0000 7f73df400640 1 --2- 172.21.15.29:0/2031120143 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f73c40683b0 0x7f73c406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:23.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.097+0000 7f73df400640 1 -- 172.21.15.29:0/2031120143 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f73d8120fe0 con 0x7f73e806c660 2024-04-17T19:48:23.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.097+0000 7f73dfe00640 1 --2- 172.21.15.29:0/2031120143 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f73c40683b0 0x7f73c406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:23.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.101+0000 7f73dfe00640 1 --2- 172.21.15.29:0/2031120143 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f73c40683b0 0x7f73c406a8c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f73e80c6a20 tx=0x7f73d0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:23.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.109+0000 7f73df400640 1 -- 172.21.15.29:0/2031120143 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f73d8016150 con 0x7f73e806c660 2024-04-17T19:48:23.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.309+0000 7f73ee600640 1 -- 172.21.15.29:0/2031120143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f73e815e090 con 0x7f73e806c660 2024-04-17T19:48:23.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.309+0000 7f73df400640 1 -- 172.21.15.29:0/2031120143 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7f73d80f0ba0 con 0x7f73e806c660 2024-04-17T19:48:23.299 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:23.299 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-04-17T19:48:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.313+0000 7f73ee600640 1 -- 172.21.15.29:0/2031120143 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f73c40683b0 msgr2=0x7f73c406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.313+0000 7f73ee600640 1 --2- 172.21.15.29:0/2031120143 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f73c40683b0 0x7f73c406a8c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f73e80c6a20 tx=0x7f73d0009040 comp rx=0 tx=0).stop 2024-04-17T19:48:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.313+0000 7f73ee600640 1 -- 172.21.15.29:0/2031120143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73e806c660 msgr2=0x7f73e80c6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.313+0000 7f73ee600640 1 --2- 172.21.15.29:0/2031120143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73e806c660 0x7f73e80c6180 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f73d801a190 tx=0x7f73d801ada0 comp rx=0 tx=0).stop 2024-04-17T19:48:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.313+0000 7f73ee600640 1 -- 172.21.15.29:0/2031120143 shutdown_connections 2024-04-17T19:48:23.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.313+0000 7f73ee600640 1 --2- 172.21.15.29:0/2031120143 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f73c40683b0 0x7f73c406a8c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:23.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.313+0000 7f73ee600640 1 --2- 172.21.15.29:0/2031120143 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f73e8159c90 0x7f73e8146bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:23.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.313+0000 7f73ee600640 1 --2- 172.21.15.29:0/2031120143 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f73e80ca7f0 0x7f73e8153a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:23.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.313+0000 7f73ee600640 1 --2- 172.21.15.29:0/2031120143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73e806c660 0x7f73e80c6180 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:23.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.313+0000 7f73ee600640 1 -- 172.21.15.29:0/2031120143 >> 172.21.15.29:0/2031120143 conn(0x7f73e806fde0 msgr2=0x7f73e80cb530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:23.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.313+0000 7f73ee600640 1 -- 172.21.15.29:0/2031120143 shutdown_connections 2024-04-17T19:48:23.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:23.313+0000 7f73ee600640 1 -- 172.21.15.29:0/2031120143 wait complete. 2024-04-17T19:48:24.803 INFO:tasks.ceph.mds.c.smithi029.stderr:2024-04-17T19:48:24.813+0000 7ff3b0200640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-17T19:48:26.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-17T19:48:26.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.485+0000 7f5523a00640 1 Processor -- start 2024-04-17T19:48:26.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.485+0000 7f5523a00640 1 -- start start 2024-04-17T19:48:26.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.485+0000 7f5523a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f551c0c7a20 0x7f551c0710d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:26.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.485+0000 7f5523a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f551c15db00 0x7f551c15df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:26.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.489+0000 7f5523a00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f551c0c8ae0 con 0x7f551c0c7e30 2024-04-17T19:48:26.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.489+0000 7f5523a00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f551c15a150 con 0x7f551c15db00 2024-04-17T19:48:26.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.489+0000 7f5523a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f551c0c9660 con 0x7f551c0c7a20 2024-04-17T19:48:26.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.489+0000 7f5521a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f551c15db00 0x7f551c15df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:26.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.489+0000 7f551be00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f551c0c7e30 0x7f551c15a5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39818/0 (socket says 172.21.15.29:39818) 2024-04-17T19:48:26.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.489+0000 7f551be00640 1 -- 172.21.15.29:0/2875033990 learned_addr learned my addr 172.21.15.29:0/2875033990 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:26.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.489+0000 7f5521000640 1 --2- 172.21.15.29:0/2875033990 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f551c0c7a20 0x7f551c0710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:26.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.489+0000 7f551b400640 1 -- 172.21.15.29:0/2875033990 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3668487602 0 0) 0x7f551c0c8ae0 con 0x7f551c0c7e30 2024-04-17T19:48:26.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.489+0000 7f551b400640 1 -- 172.21.15.29:0/2875033990 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5504003530 con 0x7f551c0c7e30 2024-04-17T19:48:26.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.489+0000 7f5521a00640 1 -- 172.21.15.29:0/2875033990 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f551c0c7a20 msgr2=0x7f551c0710d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:26.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.489+0000 7f5521a00640 1 --2- 172.21.15.29:0/2875033990 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f551c0c7a20 0x7f551c0710d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:26.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.489+0000 7f5521a00640 1 -- 172.21.15.29:0/2875033990 >> v1:172.21.15.29:6789/0 conn(0x7f551c0c7e30 legacy=0x7f551c15a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:26.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.489+0000 7f5521a00640 1 -- 172.21.15.29:0/2875033990 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551c0c8dc0 con 0x7f551c15db00 2024-04-17T19:48:26.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.489+0000 7f5521a00640 1 --2- 172.21.15.29:0/2875033990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c15db00 0x7f551c15df10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5514009d90 tx=0x7f551401f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74f1c47385a3083 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:26.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.493+0000 7f551b400640 1 -- 172.21.15.29:0/2875033990 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55140b4070 con 0x7f551c15db00 2024-04-17T19:48:26.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.493+0000 7f551b400640 1 -- 172.21.15.29:0/2875033990 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55140af600 con 0x7f551c15db00 2024-04-17T19:48:26.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.493+0000 7f551b400640 1 -- 172.21.15.29:0/2875033990 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55140af9d0 con 0x7f551c15db00 2024-04-17T19:48:26.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.493+0000 7f5523a00640 1 -- 172.21.15.29:0/2875033990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c15db00 msgr2=0x7f551c15df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:26.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.493+0000 7f5523a00640 1 --2- 172.21.15.29:0/2875033990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c15db00 0x7f551c15df10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5514009d90 tx=0x7f551401f150 comp rx=0 tx=0).stop 2024-04-17T19:48:26.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.493+0000 7f5523a00640 1 -- 172.21.15.29:0/2875033990 shutdown_connections 2024-04-17T19:48:26.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.493+0000 7f5523a00640 1 --2- 172.21.15.29:0/2875033990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c15db00 0x7f551c15df10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:26.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.493+0000 7f5523a00640 1 --2- 172.21.15.29:0/2875033990 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f551c0c7a20 0x7f551c0710d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:26.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.493+0000 7f5523a00640 1 -- 172.21.15.29:0/2875033990 >> 172.21.15.29:0/2875033990 conn(0x7f551c072e50 msgr2=0x7f551c073260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:26.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.493+0000 7f5523a00640 1 -- 172.21.15.29:0/2875033990 shutdown_connections 2024-04-17T19:48:26.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.493+0000 7f5523a00640 1 -- 172.21.15.29:0/2875033990 wait complete. 2024-04-17T19:48:26.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.493+0000 7f5523a00640 1 Processor -- start 2024-04-17T19:48:26.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.493+0000 7f5523a00640 1 -- start start 2024-04-17T19:48:26.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5523a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f551c0c7a20 0x7f551c0c6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5523a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f551c0c7e30 0x7f551c0c4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5523a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c15db00 0x7f551c0c5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5523a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f551c06bfe0 con 0x7f551c15db00 2024-04-17T19:48:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5523a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f551c0c9380 con 0x7f551c0c7e30 2024-04-17T19:48:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5523a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f551c15acf0 con 0x7f551c0c7a20 2024-04-17T19:48:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5521a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c15db00 0x7f551c0c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5521a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c15db00 0x7f551c0c5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58196/0 (socket says 172.21.15.29:58196) 2024-04-17T19:48:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5521a00640 1 -- 172.21.15.29:0/3821594722 learned_addr learned my addr 172.21.15.29:0/3821594722 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5521000640 1 --2- 172.21.15.29:0/3821594722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f551c0c7a20 0x7f551c0c6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f551be00640 1 --2- 172.21.15.29:0/3821594722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f551c0c7e30 0x7f551c0c4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5521a00640 1 -- 172.21.15.29:0/3821594722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f551c0c7a20 msgr2=0x7f551c0c6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5521a00640 1 --2- 172.21.15.29:0/3821594722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f551c0c7a20 0x7f551c0c6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:26.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5521a00640 1 -- 172.21.15.29:0/3821594722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f551c0c7e30 msgr2=0x7f551c0c4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:26.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5521a00640 1 --2- 172.21.15.29:0/3821594722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f551c0c7e30 0x7f551c0c4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:26.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5521a00640 1 -- 172.21.15.29:0/3821594722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551c0c5b80 con 0x7f551c15db00 2024-04-17T19:48:26.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f551be00640 1 --2- 172.21.15.29:0/3821594722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f551c0c7e30 0x7f551c0c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:26.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5521a00640 1 --2- 172.21.15.29:0/3821594722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c15db00 0x7f551c0c5640 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f55140b2040 tx=0x7f551401bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:26.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5518c00640 1 -- 172.21.15.29:0/3821594722 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55140b4040 con 0x7f551c15db00 2024-04-17T19:48:26.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5518c00640 1 -- 172.21.15.29:0/3821594722 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55140c1070 con 0x7f551c15db00 2024-04-17T19:48:26.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5523a00640 1 -- 172.21.15.29:0/3821594722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f551c1e2ea0 con 0x7f551c15db00 2024-04-17T19:48:26.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5518c00640 1 -- 172.21.15.29:0/3821594722 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5514009450 con 0x7f551c15db00 2024-04-17T19:48:26.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5523a00640 1 -- 172.21.15.29:0/3821594722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f551c1f60a0 con 0x7f551c15db00 2024-04-17T19:48:26.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.497+0000 7f5523a00640 1 -- 172.21.15.29:0/3821594722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f551c0c7630 con 0x7f551c15db00 2024-04-17T19:48:26.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.501+0000 7f5518c00640 1 -- 172.21.15.29:0/3821594722 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f55140ba6b0 con 0x7f551c15db00 2024-04-17T19:48:26.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.501+0000 7f5518c00640 1 --2- 172.21.15.29:0/3821594722 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54fc068380 0x7f54fc06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:26.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.501+0000 7f5518c00640 1 -- 172.21.15.29:0/3821594722 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5514125fa0 con 0x7f551c15db00 2024-04-17T19:48:26.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.501+0000 7f5521000640 1 --2- 172.21.15.29:0/3821594722 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54fc068380 0x7f54fc06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:26.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.501+0000 7f5521000640 1 --2- 172.21.15.29:0/3821594722 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54fc068380 0x7f54fc06a890 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5510009a50 tx=0x7f5510008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:26.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.505+0000 7f5518c00640 1 -- 172.21.15.29:0/3821594722 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5514018cb0 con 0x7f551c15db00 2024-04-17T19:48:26.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.709+0000 7f5523a00640 1 -- 172.21.15.29:0/3821594722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f551c069b90 con 0x7f551c15db00 2024-04-17T19:48:26.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.709+0000 7f5518c00640 1 -- 172.21.15.29:0/3821594722 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7f55140f6270 con 0x7f551c15db00 2024-04-17T19:48:26.697 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:26.697 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-04-17T19:48:26.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.709+0000 7f5523a00640 1 -- 172.21.15.29:0/3821594722 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54fc068380 msgr2=0x7f54fc06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:26.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.709+0000 7f5523a00640 1 --2- 172.21.15.29:0/3821594722 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54fc068380 0x7f54fc06a890 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f5510009a50 tx=0x7f5510008040 comp rx=0 tx=0).stop 2024-04-17T19:48:26.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.709+0000 7f5523a00640 1 -- 172.21.15.29:0/3821594722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c15db00 msgr2=0x7f551c0c5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:26.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.709+0000 7f5523a00640 1 --2- 172.21.15.29:0/3821594722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c15db00 0x7f551c0c5640 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f55140b2040 tx=0x7f551401bef0 comp rx=0 tx=0).stop 2024-04-17T19:48:26.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.709+0000 7f5523a00640 1 -- 172.21.15.29:0/3821594722 shutdown_connections 2024-04-17T19:48:26.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.709+0000 7f5523a00640 1 --2- 172.21.15.29:0/3821594722 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54fc068380 0x7f54fc06a890 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:26.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.709+0000 7f5523a00640 1 --2- 172.21.15.29:0/3821594722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c15db00 0x7f551c0c5640 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:26.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.709+0000 7f5523a00640 1 --2- 172.21.15.29:0/3821594722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f551c0c7e30 0x7f551c0c4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:26.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.709+0000 7f5523a00640 1 --2- 172.21.15.29:0/3821594722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f551c0c7a20 0x7f551c0c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:26.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.709+0000 7f5523a00640 1 -- 172.21.15.29:0/3821594722 >> 172.21.15.29:0/3821594722 conn(0x7f551c072e50 msgr2=0x7f551c071130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:26.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.709+0000 7f5523a00640 1 -- 172.21.15.29:0/3821594722 shutdown_connections 2024-04-17T19:48:26.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:26.709+0000 7f5523a00640 1 -- 172.21.15.29:0/3821594722 wait complete. 2024-04-17T19:48:29.720 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-17T19:48:29.803 INFO:tasks.ceph.mds.c.smithi029.stderr:2024-04-17T19:48:29.813+0000 7ff3b0200640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-17T19:48:29.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.893+0000 7f6391800640 1 Processor -- start 2024-04-17T19:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.893+0000 7f6391800640 1 -- start start 2024-04-17T19:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.897+0000 7f6391800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f638c15fc80 0x7f638c160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.897+0000 7f6391800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f638c06c900 con 0x7f638c0caf80 2024-04-17T19:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.897+0000 7f6391800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f638c06c690 con 0x7f638c0cc4d0 2024-04-17T19:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.897+0000 7f6391800640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f638c15c050 con 0x7f638c15fc80 2024-04-17T19:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.897+0000 7f6382000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f638c15fc80 0x7f638c160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.897+0000 7f638aa00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f638c0caf80 0x7f638c15c9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48160/0 (socket says 172.21.15.29:48160) 2024-04-17T19:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.897+0000 7f638aa00640 1 -- 172.21.15.29:0/526874257 learned_addr learned my addr 172.21.15.29:0/526874257 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.897+0000 7f6382000640 1 -- 172.21.15.29:0/526874257 >> v1:172.21.15.37:6789/0 conn(0x7f638c0cc4d0 legacy=0x7f638c1668f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.897+0000 7f6382000640 1 -- 172.21.15.29:0/526874257 >> v1:172.21.15.29:6789/0 conn(0x7f638c0caf80 legacy=0x7f638c15c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.897+0000 7f6382000640 1 -- 172.21.15.29:0/526874257 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638c06cb80 con 0x7f638c15fc80 2024-04-17T19:48:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.897+0000 7f638a000640 1 -- 172.21.15.29:0/526874257 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1829542607 0 0) 0x7f637c001120 con 0x7f638c0caf80 2024-04-17T19:48:29.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.901+0000 7f6382000640 1 --2- 172.21.15.29:0/526874257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f638c15fc80 0x7f638c160090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f638c06c860 tx=0x7f637401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b094950c933b3d6e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:29.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.901+0000 7f638a000640 1 -- 172.21.15.29:0/526874257 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63740b5070 con 0x7f638c15fc80 2024-04-17T19:48:29.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.901+0000 7f638a000640 1 -- 172.21.15.29:0/526874257 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63740a8990 con 0x7f638c15fc80 2024-04-17T19:48:29.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.901+0000 7f638a000640 1 -- 172.21.15.29:0/526874257 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63740a8d60 con 0x7f638c15fc80 2024-04-17T19:48:29.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.901+0000 7f6391800640 1 -- 172.21.15.29:0/526874257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f638c15fc80 msgr2=0x7f638c160090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:29.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.901+0000 7f6391800640 1 --2- 172.21.15.29:0/526874257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f638c15fc80 0x7f638c160090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f638c06c860 tx=0x7f637401ec90 comp rx=0 tx=0).stop 2024-04-17T19:48:29.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.901+0000 7f6391800640 1 -- 172.21.15.29:0/526874257 shutdown_connections 2024-04-17T19:48:29.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.901+0000 7f6391800640 1 --2- 172.21.15.29:0/526874257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f638c15fc80 0x7f638c160090 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:29.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.901+0000 7f6391800640 1 -- 172.21.15.29:0/526874257 >> 172.21.15.29:0/526874257 conn(0x7f638c06e3d0 msgr2=0x7f638c06e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:29.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.901+0000 7f6391800640 1 -- 172.21.15.29:0/526874257 shutdown_connections 2024-04-17T19:48:29.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.901+0000 7f6391800640 1 -- 172.21.15.29:0/526874257 wait complete. 2024-04-17T19:48:29.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f6391800640 1 Processor -- start 2024-04-17T19:48:29.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f6391800640 1 -- start start 2024-04-17T19:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f6391800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f638c0caf80 0x7f638c1f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f6391800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f638c0cc4d0 0x7f638c0cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f6391800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f638c15fc80 0x7f638c1533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f6391800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f638c06d420 con 0x7f638c15fc80 2024-04-17T19:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f6391800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f638c06d140 con 0x7f638c0cc4d0 2024-04-17T19:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f6391800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f638c15f760 con 0x7f638c0caf80 2024-04-17T19:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f638b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f638c15fc80 0x7f638c1533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f638aa00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f638c0caf80 0x7f638c1f2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f6382000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f638c0cc4d0 0x7f638c0cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f638b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f638c15fc80 0x7f638c1533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33800/0 (socket says 172.21.15.29:33800) 2024-04-17T19:48:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f638b400640 1 -- 172.21.15.29:0/1596463758 learned_addr learned my addr 172.21.15.29:0/1596463758 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:29.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f638b400640 1 -- 172.21.15.29:0/1596463758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f638c0caf80 msgr2=0x7f638c1f2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f638b400640 1 --2- 172.21.15.29:0/1596463758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f638c0caf80 0x7f638c1f2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f638b400640 1 -- 172.21.15.29:0/1596463758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f638c0cc4d0 msgr2=0x7f638c0cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f638b400640 1 --2- 172.21.15.29:0/1596463758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f638c0cc4d0 0x7f638c0cf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f638b400640 1 -- 172.21.15.29:0/1596463758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f638c1f3370 con 0x7f638c15fc80 2024-04-17T19:48:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f638b400640 1 --2- 172.21.15.29:0/1596463758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f638c15fc80 0x7f638c1533d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f638400c760 tx=0x7f638400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f6383400640 1 -- 172.21.15.29:0/1596463758 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6384003b80 con 0x7f638c15fc80 2024-04-17T19:48:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f6383400640 1 -- 172.21.15.29:0/1596463758 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6384003d20 con 0x7f638c15fc80 2024-04-17T19:48:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f6391800640 1 -- 172.21.15.29:0/1596463758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f638c0b1a90 con 0x7f638c15fc80 2024-04-17T19:48:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f6383400640 1 -- 172.21.15.29:0/1596463758 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638400aa30 con 0x7f638c15fc80 2024-04-17T19:48:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.905+0000 7f6391800640 1 -- 172.21.15.29:0/1596463758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f638c1f3580 con 0x7f638c15fc80 2024-04-17T19:48:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.909+0000 7f6391800640 1 -- 172.21.15.29:0/1596463758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f638c069b90 con 0x7f638c15fc80 2024-04-17T19:48:29.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.909+0000 7f6383400640 1 -- 172.21.15.29:0/1596463758 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f638400abd0 con 0x7f638c15fc80 2024-04-17T19:48:29.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.909+0000 7f6383400640 1 --2- 172.21.15.29:0/1596463758 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6368068380 0x7f636806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:29.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.909+0000 7f6383400640 1 -- 172.21.15.29:0/1596463758 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f6384086bf0 con 0x7f638c15fc80 2024-04-17T19:48:29.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.909+0000 7f638aa00640 1 --2- 172.21.15.29:0/1596463758 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6368068380 0x7f636806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:29.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.909+0000 7f638aa00640 1 --2- 172.21.15.29:0/1596463758 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6368068380 0x7f636806a890 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f637c000f70 tx=0x7f637c007240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:29.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:29.921+0000 7f6383400640 1 -- 172.21.15.29:0/1596463758 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6384056630 con 0x7f638c15fc80 2024-04-17T19:48:30.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.117+0000 7f6391800640 1 -- 172.21.15.29:0/1596463758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f638c0cad60 con 0x7f638c15fc80 2024-04-17T19:48:30.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.117+0000 7f6383400640 1 -- 172.21.15.29:0/1596463758 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f638405a930 con 0x7f638c15fc80 2024-04-17T19:48:30.107 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:30.107 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-17T19:48:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.121+0000 7f6391800640 1 -- 172.21.15.29:0/1596463758 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6368068380 msgr2=0x7f636806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.121+0000 7f6391800640 1 --2- 172.21.15.29:0/1596463758 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6368068380 0x7f636806a890 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f637c000f70 tx=0x7f637c007240 comp rx=0 tx=0).stop 2024-04-17T19:48:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.121+0000 7f6391800640 1 -- 172.21.15.29:0/1596463758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f638c15fc80 msgr2=0x7f638c1533d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.121+0000 7f6391800640 1 --2- 172.21.15.29:0/1596463758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f638c15fc80 0x7f638c1533d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f638400c760 tx=0x7f638400cc20 comp rx=0 tx=0).stop 2024-04-17T19:48:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.121+0000 7f6391800640 1 -- 172.21.15.29:0/1596463758 shutdown_connections 2024-04-17T19:48:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.121+0000 7f6391800640 1 --2- 172.21.15.29:0/1596463758 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6368068380 0x7f636806a890 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.121+0000 7f6391800640 1 --2- 172.21.15.29:0/1596463758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f638c15fc80 0x7f638c1533d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.121+0000 7f6391800640 1 --2- 172.21.15.29:0/1596463758 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f638c0cc4d0 0x7f638c0cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.121+0000 7f6391800640 1 --2- 172.21.15.29:0/1596463758 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f638c0caf80 0x7f638c1f2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.121+0000 7f6391800640 1 -- 172.21.15.29:0/1596463758 >> 172.21.15.29:0/1596463758 conn(0x7f638c06e3d0 msgr2=0x7f638c0cec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:30.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.121+0000 7f6391800640 1 -- 172.21.15.29:0/1596463758 shutdown_connections 2024-04-17T19:48:30.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.121+0000 7f6391800640 1 -- 172.21.15.29:0/1596463758 wait complete. 2024-04-17T19:48:30.129 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-17T19:48:30.129 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:48:30.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.293+0000 7f50ebc00640 1 Processor -- start 2024-04-17T19:48:30.283 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.293+0000 7f50ebc00640 1 -- start start 2024-04-17T19:48:30.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.297+0000 7f50ebc00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f50e4159e60 0x7f50e40cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:30.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.297+0000 7f50ebc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50e4153d20 0x7f50e4071940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:30.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.297+0000 7f50ebc00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50e40c4930 con 0x7f50e4071e80 2024-04-17T19:48:30.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.297+0000 7f50ebc00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f50e40d0320 con 0x7f50e4159e60 2024-04-17T19:48:30.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.297+0000 7f50ebc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f50e40cf620 con 0x7f50e4153d20 2024-04-17T19:48:30.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.297+0000 7f50e9200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f50e4159e60 0x7f50e40cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:30.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.297+0000 7f50e9200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f50e4159e60 0x7f50e40cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33808/0 (socket says 172.21.15.29:33808) 2024-04-17T19:48:30.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.297+0000 7f50e9200640 1 -- 172.21.15.29:0/2227905442 learned_addr learned my addr 172.21.15.29:0/2227905442 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:30.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.297+0000 7f50e3e00640 1 --2- 172.21.15.29:0/2227905442 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50e4153d20 0x7f50e4071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:30.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.297+0000 7f50e9200640 1 -- 172.21.15.29:0/2227905442 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50e4153d20 msgr2=0x7f50e4071940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:30.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.297+0000 7f50e9200640 1 --2- 172.21.15.29:0/2227905442 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50e4153d20 0x7f50e4071940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.297+0000 7f50e9200640 1 -- 172.21.15.29:0/2227905442 >> v1:172.21.15.37:6789/0 conn(0x7f50e4071e80 legacy=0x7f50e40c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:30.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.297+0000 7f50e9200640 1 -- 172.21.15.29:0/2227905442 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50e40c5190 con 0x7f50e4159e60 2024-04-17T19:48:30.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.301+0000 7f50e9200640 1 --2- 172.21.15.29:0/2227905442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50e4159e60 0x7f50e40cfde0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f50d8009870 tx=0x7f50d801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c79b75b5402f028e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:30.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.301+0000 7f50e3400640 1 -- 172.21.15.29:0/2227905442 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d80b0070 con 0x7f50e4159e60 2024-04-17T19:48:30.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.301+0000 7f50e3400640 1 -- 172.21.15.29:0/2227905442 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50d80ab600 con 0x7f50e4159e60 2024-04-17T19:48:30.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.301+0000 7f50e3400640 1 -- 172.21.15.29:0/2227905442 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d80ab9d0 con 0x7f50e4159e60 2024-04-17T19:48:30.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.301+0000 7f50ebc00640 1 -- 172.21.15.29:0/2227905442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50e4159e60 msgr2=0x7f50e40cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:30.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.301+0000 7f50ebc00640 1 --2- 172.21.15.29:0/2227905442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50e4159e60 0x7f50e40cfde0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f50d8009870 tx=0x7f50d801c560 comp rx=0 tx=0).stop 2024-04-17T19:48:30.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.301+0000 7f50ebc00640 1 -- 172.21.15.29:0/2227905442 shutdown_connections 2024-04-17T19:48:30.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.301+0000 7f50ebc00640 1 --2- 172.21.15.29:0/2227905442 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50e4153d20 0x7f50e4071940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.301+0000 7f50ebc00640 1 --2- 172.21.15.29:0/2227905442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50e4159e60 0x7f50e40cfde0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.301+0000 7f50ebc00640 1 -- 172.21.15.29:0/2227905442 >> 172.21.15.29:0/2227905442 conn(0x7f50e4067eb0 msgr2=0x7f50e4070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:30.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.301+0000 7f50ebc00640 1 -- 172.21.15.29:0/2227905442 shutdown_connections 2024-04-17T19:48:30.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.301+0000 7f50ebc00640 1 -- 172.21.15.29:0/2227905442 wait complete. 2024-04-17T19:48:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50ebc00640 1 Processor -- start 2024-04-17T19:48:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50ebc00640 1 -- start start 2024-04-17T19:48:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50ebc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50e4071e80 0x7f50e40cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50ebc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50e4153d20 0x7f50e41f5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50ebc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50e4159e60 0x7f50e406d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50ebc00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f50e40cbd10 con 0x7f50e4071e80 2024-04-17T19:48:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50ebc00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f50e40cbb90 con 0x7f50e4153d20 2024-04-17T19:48:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50ebc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f50e40c4510 con 0x7f50e4159e60 2024-04-17T19:48:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50e3e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50e4153d20 0x7f50e41f5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50e9200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50e4071e80 0x7f50e40cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50e3e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50e4153d20 0x7f50e41f5d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:45192/0 (socket says 172.21.15.29:45192) 2024-04-17T19:48:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50e3e00640 1 -- 172.21.15.29:0/23234430 learned_addr learned my addr 172.21.15.29:0/23234430 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:30.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50e9c00640 1 --2- 172.21.15.29:0/23234430 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50e4159e60 0x7f50e406d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:30.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50e9200640 1 -- 172.21.15.29:0/23234430 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50e4159e60 msgr2=0x7f50e406d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:30.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50e9200640 1 --2- 172.21.15.29:0/23234430 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50e4159e60 0x7f50e406d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50e9200640 1 -- 172.21.15.29:0/23234430 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50e4153d20 msgr2=0x7f50e41f5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:30.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50e9200640 1 --2- 172.21.15.29:0/23234430 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50e4153d20 0x7f50e41f5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50e9200640 1 -- 172.21.15.29:0/23234430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50e41f6250 con 0x7f50e4071e80 2024-04-17T19:48:30.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50e9200640 1 --2- 172.21.15.29:0/23234430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50e4071e80 0x7f50e40cdc20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f50d80ab000 tx=0x7f50d8019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:30.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50e0c00640 1 -- 172.21.15.29:0/23234430 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d80b0040 con 0x7f50e4071e80 2024-04-17T19:48:30.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50e0c00640 1 -- 172.21.15.29:0/23234430 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50d80ab400 con 0x7f50e4071e80 2024-04-17T19:48:30.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50e0c00640 1 -- 172.21.15.29:0/23234430 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d80b46a0 con 0x7f50e4071e80 2024-04-17T19:48:30.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.305+0000 7f50ebc00640 1 -- 172.21.15.29:0/23234430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50e4150d30 con 0x7f50e4071e80 2024-04-17T19:48:30.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.309+0000 7f50ebc00640 1 -- 172.21.15.29:0/23234430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50e40cf1b0 con 0x7f50e4071e80 2024-04-17T19:48:30.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.309+0000 7f50ebc00640 1 -- 172.21.15.29:0/23234430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50e4069b90 con 0x7f50e4071e80 2024-04-17T19:48:30.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.309+0000 7f50e0c00640 1 -- 172.21.15.29:0/23234430 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f50d80bb020 con 0x7f50e4071e80 2024-04-17T19:48:30.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.309+0000 7f50e0c00640 1 --2- 172.21.15.29:0/23234430 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f50c40683b0 0x7f50c406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:30.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.309+0000 7f50e0c00640 1 -- 172.21.15.29:0/23234430 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f50d811fff0 con 0x7f50e4071e80 2024-04-17T19:48:30.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.309+0000 7f50e3e00640 1 --2- 172.21.15.29:0/23234430 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f50c40683b0 0x7f50c406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:30.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.313+0000 7f50e3e00640 1 --2- 172.21.15.29:0/23234430 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f50c40683b0 0x7f50c406a8c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009b80 tx=0x7f50d0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:30.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.313+0000 7f50e0c00640 1 -- 172.21.15.29:0/23234430 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f50d80c2020 con 0x7f50e4071e80 2024-04-17T19:48:30.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.493+0000 7f50ebc00640 1 -- 172.21.15.29:0/23234430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f50e41655d0 con 0x7f50e4071e80 2024-04-17T19:48:30.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.493+0000 7f50e0c00640 1 -- 172.21.15.29:0/23234430 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18780 (secure 0 0 0) 0x7f50d8016150 con 0x7f50e4071e80 2024-04-17T19:48:30.482 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:30.482 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":26,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:48:29.829433+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T19:47:47.658449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T19:47:48.209047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T19:47:51.887909+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:48:30.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.497+0000 7f50ebc00640 1 -- 172.21.15.29:0/23234430 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f50c40683b0 msgr2=0x7f50c406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:30.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.497+0000 7f50ebc00640 1 --2- 172.21.15.29:0/23234430 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f50c40683b0 0x7f50c406a8c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009b80 tx=0x7f50d0009190 comp rx=0 tx=0).stop 2024-04-17T19:48:30.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.497+0000 7f50ebc00640 1 -- 172.21.15.29:0/23234430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50e4071e80 msgr2=0x7f50e40cdc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:30.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.497+0000 7f50ebc00640 1 --2- 172.21.15.29:0/23234430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50e4071e80 0x7f50e40cdc20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f50d80ab000 tx=0x7f50d8019300 comp rx=0 tx=0).stop 2024-04-17T19:48:30.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.497+0000 7f50ebc00640 1 -- 172.21.15.29:0/23234430 shutdown_connections 2024-04-17T19:48:30.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.497+0000 7f50ebc00640 1 --2- 172.21.15.29:0/23234430 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f50c40683b0 0x7f50c406a8c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.497+0000 7f50ebc00640 1 --2- 172.21.15.29:0/23234430 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f50e4159e60 0x7f50e406d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.497+0000 7f50ebc00640 1 --2- 172.21.15.29:0/23234430 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f50e4153d20 0x7f50e41f5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.497+0000 7f50ebc00640 1 --2- 172.21.15.29:0/23234430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50e4071e80 0x7f50e40cdc20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.497+0000 7f50ebc00640 1 -- 172.21.15.29:0/23234430 >> 172.21.15.29:0/23234430 conn(0x7f50e4067eb0 msgr2=0x7f50e4152cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:30.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.497+0000 7f50ebc00640 1 -- 172.21.15.29:0/23234430 shutdown_connections 2024-04-17T19:48:30.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.497+0000 7f50ebc00640 1 -- 172.21.15.29:0/23234430 wait complete. 2024-04-17T19:48:30.504 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T19:47:36.674676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T19:47:47.658449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T19:47:48.209047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T19:47:51.887909+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T19:48:30.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T19:48:30.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.669+0000 7f35b9600640 1 Processor -- start 2024-04-17T19:48:30.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.669+0000 7f35b9600640 1 -- start start 2024-04-17T19:48:30.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.673+0000 7f35b9600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f35b415cbf0 0x7f35b40b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:30.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.673+0000 7f35b9600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35b40c54b0 0x7f35b4166a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:30.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.673+0000 7f35b9600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35b4070b00 con 0x7f35b40c50a0 2024-04-17T19:48:30.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.673+0000 7f35b9600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f35b40cab70 con 0x7f35b415cbf0 2024-04-17T19:48:30.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.673+0000 7f35b9600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f35b40b5520 con 0x7f35b40c54b0 2024-04-17T19:48:30.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.673+0000 7f35b2a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f35b415cbf0 0x7f35b40b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:30.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.673+0000 7f35b2000640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f35b40c50a0 0x7f35b406e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48172/0 (socket says 172.21.15.29:48172) 2024-04-17T19:48:30.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.673+0000 7f35b2000640 1 -- 172.21.15.29:0/1392797106 learned_addr learned my addr 172.21.15.29:0/1392797106 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:30.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.673+0000 7f35b3400640 1 --2- 172.21.15.29:0/1392797106 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35b40c54b0 0x7f35b4166a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:30.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.673+0000 7f35b2a00640 1 -- 172.21.15.29:0/1392797106 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35b40c54b0 msgr2=0x7f35b4166a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:30.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.673+0000 7f35b2a00640 1 --2- 172.21.15.29:0/1392797106 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35b40c54b0 0x7f35b4166a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.673+0000 7f35b2a00640 1 -- 172.21.15.29:0/1392797106 >> v1:172.21.15.29:6789/0 conn(0x7f35b40c50a0 legacy=0x7f35b406e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:30.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.673+0000 7f35b2a00640 1 -- 172.21.15.29:0/1392797106 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b406fd80 con 0x7f35b415cbf0 2024-04-17T19:48:30.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.677+0000 7f35b2a00640 1 --2- 172.21.15.29:0/1392797106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35b415cbf0 0x7f35b40b4fe0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f35a40097c0 tx=0x7f35a401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e42554ae8d238219 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:30.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.677+0000 7f35b1600640 1 -- 172.21.15.29:0/1392797106 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a40b5070 con 0x7f35b415cbf0 2024-04-17T19:48:30.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.677+0000 7f35b1600640 1 -- 172.21.15.29:0/1392797106 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35a40a8990 con 0x7f35b415cbf0 2024-04-17T19:48:30.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.677+0000 7f35b1600640 1 -- 172.21.15.29:0/1392797106 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a40a8d60 con 0x7f35b415cbf0 2024-04-17T19:48:30.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.677+0000 7f35b9600640 1 -- 172.21.15.29:0/1392797106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35b415cbf0 msgr2=0x7f35b40b4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:30.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.677+0000 7f35b9600640 1 --2- 172.21.15.29:0/1392797106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35b415cbf0 0x7f35b40b4fe0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f35a40097c0 tx=0x7f35a401ec90 comp rx=0 tx=0).stop 2024-04-17T19:48:30.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.677+0000 7f35b9600640 1 -- 172.21.15.29:0/1392797106 shutdown_connections 2024-04-17T19:48:30.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.677+0000 7f35b9600640 1 --2- 172.21.15.29:0/1392797106 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35b40c54b0 0x7f35b4166a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.677+0000 7f35b9600640 1 --2- 172.21.15.29:0/1392797106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35b415cbf0 0x7f35b40b4fe0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.677+0000 7f35b9600640 1 -- 172.21.15.29:0/1392797106 >> 172.21.15.29:0/1392797106 conn(0x7f35b40cf770 msgr2=0x7f35b40cfb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:30.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.677+0000 7f35b9600640 1 -- 172.21.15.29:0/1392797106 shutdown_connections 2024-04-17T19:48:30.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.677+0000 7f35b9600640 1 -- 172.21.15.29:0/1392797106 wait complete. 2024-04-17T19:48:30.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b9600640 1 Processor -- start 2024-04-17T19:48:30.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b9600640 1 -- start start 2024-04-17T19:48:30.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b9600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35b40c50a0 0x7f35b4150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:30.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b9600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35b40c54b0 0x7f35b415fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:30.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b9600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35b415cbf0 0x7f35b41f23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:30.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b9600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f35b4070300 con 0x7f35b40c54b0 2024-04-17T19:48:30.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b9600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f35b4070020 con 0x7f35b40c50a0 2024-04-17T19:48:30.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b9600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f35b40705e0 con 0x7f35b415cbf0 2024-04-17T19:48:30.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b2000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35b40c54b0 0x7f35b415fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:30.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b2000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35b40c54b0 0x7f35b415fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33822/0 (socket says 172.21.15.29:33822) 2024-04-17T19:48:30.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b2000640 1 -- 172.21.15.29:0/4253183492 learned_addr learned my addr 172.21.15.29:0/4253183492 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:30.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b2a00640 1 --2- 172.21.15.29:0/4253183492 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35b40c50a0 0x7f35b4150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:30.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b3400640 1 --2- 172.21.15.29:0/4253183492 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35b415cbf0 0x7f35b41f23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:30.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b2000640 1 -- 172.21.15.29:0/4253183492 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35b415cbf0 msgr2=0x7f35b41f23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:30.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b2000640 1 --2- 172.21.15.29:0/4253183492 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35b415cbf0 0x7f35b41f23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b2000640 1 -- 172.21.15.29:0/4253183492 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35b40c50a0 msgr2=0x7f35b4150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:30.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b2000640 1 --2- 172.21.15.29:0/4253183492 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35b40c50a0 0x7f35b4150de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b2000640 1 -- 172.21.15.29:0/4253183492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b4151320 con 0x7f35b40c54b0 2024-04-17T19:48:30.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b2000640 1 --2- 172.21.15.29:0/4253183492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35b40c54b0 0x7f35b415fc40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f35a000c310 tx=0x7f35a000c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:30.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35aaa00640 1 -- 172.21.15.29:0/4253183492 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a0017070 con 0x7f35b40c54b0 2024-04-17T19:48:30.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35aaa00640 1 -- 172.21.15.29:0/4253183492 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35a0005cf0 con 0x7f35b40c54b0 2024-04-17T19:48:30.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35aaa00640 1 -- 172.21.15.29:0/4253183492 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a0007310 con 0x7f35b40c54b0 2024-04-17T19:48:30.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b9600640 1 -- 172.21.15.29:0/4253183492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35b414f8f0 con 0x7f35b40c54b0 2024-04-17T19:48:30.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.681+0000 7f35b9600640 1 -- 172.21.15.29:0/4253183492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35b4150300 con 0x7f35b40c54b0 2024-04-17T19:48:30.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.685+0000 7f35b9600640 1 -- 172.21.15.29:0/4253183492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35b4069b90 con 0x7f35b40c54b0 2024-04-17T19:48:30.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.685+0000 7f35aaa00640 1 -- 172.21.15.29:0/4253183492 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f35a001a070 con 0x7f35b40c54b0 2024-04-17T19:48:30.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.685+0000 7f35aaa00640 1 --2- 172.21.15.29:0/4253183492 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f35940683b0 0x7f359406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:30.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.685+0000 7f35aaa00640 1 -- 172.21.15.29:0/4253183492 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f35a0085fe0 con 0x7f35b40c54b0 2024-04-17T19:48:30.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.685+0000 7f35b2a00640 1 --2- 172.21.15.29:0/4253183492 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f35940683b0 0x7f359406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:30.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.685+0000 7f35b2a00640 1 --2- 172.21.15.29:0/4253183492 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f35940683b0 0x7f359406a8c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f35a40b2000 tx=0x7f35a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:30.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.689+0000 7f35aaa00640 1 -- 172.21.15.29:0/4253183492 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f35a0055af0 con 0x7f35b40c54b0 2024-04-17T19:48:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.857+0000 7f35b9600640 1 -- 172.21.15.29:0/4253183492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f35b40cacf0 con 0x7f35b40c54b0 2024-04-17T19:48:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.857+0000 7f35aaa00640 1 -- 172.21.15.29:0/4253183492 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f35a0059df0 con 0x7f35b40c54b0 2024-04-17T19:48:30.846 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-04-17T19:48:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.861+0000 7f35b9600640 1 -- 172.21.15.29:0/4253183492 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f35940683b0 msgr2=0x7f359406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.861+0000 7f35b9600640 1 --2- 172.21.15.29:0/4253183492 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f35940683b0 0x7f359406a8c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f35a40b2000 tx=0x7f35a4008040 comp rx=0 tx=0).stop 2024-04-17T19:48:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.861+0000 7f35b9600640 1 -- 172.21.15.29:0/4253183492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35b40c54b0 msgr2=0x7f35b415fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.861+0000 7f35b9600640 1 --2- 172.21.15.29:0/4253183492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35b40c54b0 0x7f35b415fc40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f35a000c310 tx=0x7f35a000c7d0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.861+0000 7f35b9600640 1 -- 172.21.15.29:0/4253183492 shutdown_connections 2024-04-17T19:48:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.861+0000 7f35b9600640 1 --2- 172.21.15.29:0/4253183492 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f35940683b0 0x7f359406a8c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.861+0000 7f35b9600640 1 --2- 172.21.15.29:0/4253183492 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35b415cbf0 0x7f35b41f23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.861+0000 7f35b9600640 1 --2- 172.21.15.29:0/4253183492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35b40c54b0 0x7f35b415fc40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.861+0000 7f35b9600640 1 --2- 172.21.15.29:0/4253183492 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35b40c50a0 0x7f35b4150de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.861+0000 7f35b9600640 1 -- 172.21.15.29:0/4253183492 >> 172.21.15.29:0/4253183492 conn(0x7f35b40cf770 msgr2=0x7f35b415aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.861+0000 7f35b9600640 1 -- 172.21.15.29:0/4253183492 shutdown_connections 2024-04-17T19:48:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:30.861+0000 7f35b9600640 1 -- 172.21.15.29:0/4253183492 wait complete. 2024-04-17T19:48:30.869 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T19:48:31.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.037+0000 7f887de00640 1 Processor -- start 2024-04-17T19:48:31.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.037+0000 7f887de00640 1 -- start start 2024-04-17T19:48:31.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.037+0000 7f887de00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8878071ce0 0x7f8878141de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:31.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.037+0000 7f887de00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f88781593d0 0x7f88781597e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:31.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.037+0000 7f887de00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8878142320 con 0x7f88780c9f00 2024-04-17T19:48:31.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.037+0000 7f887de00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f88780c7bd0 con 0x7f8878071ce0 2024-04-17T19:48:31.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.037+0000 7f887de00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f887815c6d0 con 0x7f88781593d0 2024-04-17T19:48:31.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.037+0000 7f8876a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8878071ce0 0x7f8878141de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:31.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.037+0000 7f8876a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8878071ce0 0x7f8878141de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33826/0 (socket says 172.21.15.29:33826) 2024-04-17T19:48:31.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.037+0000 7f8876a00640 1 -- 172.21.15.29:0/2132321191 learned_addr learned my addr 172.21.15.29:0/2132321191 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:31.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.037+0000 7f886e000640 1 --2- 172.21.15.29:0/2132321191 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f88781593d0 0x7f88781597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:31.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.037+0000 7f8876a00640 1 -- 172.21.15.29:0/2132321191 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f88781593d0 msgr2=0x7f88781597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:31.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.037+0000 7f8876a00640 1 --2- 172.21.15.29:0/2132321191 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f88781593d0 0x7f88781597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.037+0000 7f8876a00640 1 -- 172.21.15.29:0/2132321191 >> v1:172.21.15.37:6789/0 conn(0x7f88780c9f00 legacy=0x7f88780c6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:31.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.037+0000 7f8876a00640 1 -- 172.21.15.29:0/2132321191 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f887815c9b0 con 0x7f8878071ce0 2024-04-17T19:48:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.041+0000 7f8876a00640 1 --2- 172.21.15.29:0/2132321191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8878071ce0 0x7f8878141de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8868009870 tx=0x7f886801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da4fade3e112259 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.041+0000 7f8876000640 1 -- 172.21.15.29:0/2132321191 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88680b2070 con 0x7f8878071ce0 2024-04-17T19:48:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.041+0000 7f8876000640 1 -- 172.21.15.29:0/2132321191 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88680a9070 con 0x7f8878071ce0 2024-04-17T19:48:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.041+0000 7f8876000640 1 -- 172.21.15.29:0/2132321191 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88680ad880 con 0x7f8878071ce0 2024-04-17T19:48:31.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.041+0000 7f887de00640 1 -- 172.21.15.29:0/2132321191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8878071ce0 msgr2=0x7f8878141de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:31.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.041+0000 7f887de00640 1 --2- 172.21.15.29:0/2132321191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8878071ce0 0x7f8878141de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8868009870 tx=0x7f886801c560 comp rx=0 tx=0).stop 2024-04-17T19:48:31.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.041+0000 7f887de00640 1 -- 172.21.15.29:0/2132321191 shutdown_connections 2024-04-17T19:48:31.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.041+0000 7f887de00640 1 --2- 172.21.15.29:0/2132321191 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f88781593d0 0x7f88781597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.041+0000 7f887de00640 1 --2- 172.21.15.29:0/2132321191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8878071ce0 0x7f8878141de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.041+0000 7f887de00640 1 -- 172.21.15.29:0/2132321191 >> 172.21.15.29:0/2132321191 conn(0x7f887806bf90 msgr2=0x7f887806c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:31.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.041+0000 7f887de00640 1 -- 172.21.15.29:0/2132321191 shutdown_connections 2024-04-17T19:48:31.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.041+0000 7f887de00640 1 -- 172.21.15.29:0/2132321191 wait complete. 2024-04-17T19:48:31.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.045+0000 7f887de00640 1 Processor -- start 2024-04-17T19:48:31.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.045+0000 7f887de00640 1 -- start start 2024-04-17T19:48:31.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.045+0000 7f887de00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8878071ce0 0x7f8878138980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:31.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.045+0000 7f887de00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88780c9f00 0x7f8878137560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:31.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.045+0000 7f887de00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f88781593d0 0x7f8878151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:31.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.045+0000 7f887de00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f88780720f0 con 0x7f88780c9f00 2024-04-17T19:48:31.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.045+0000 7f887de00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f88780ca720 con 0x7f88781593d0 2024-04-17T19:48:31.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.045+0000 7f887de00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8878072270 con 0x7f8878071ce0 2024-04-17T19:48:31.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.045+0000 7f886e000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88780c9f00 0x7f8878137560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:31.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.045+0000 7f886e000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88780c9f00 0x7f8878137560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33838/0 (socket says 172.21.15.29:33838) 2024-04-17T19:48:31.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.045+0000 7f886e000640 1 -- 172.21.15.29:0/1607185541 learned_addr learned my addr 172.21.15.29:0/1607185541 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:31.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.045+0000 7f8876a00640 1 --2- 172.21.15.29:0/1607185541 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8878071ce0 0x7f8878138980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:31.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.045+0000 7f8877400640 1 --2- 172.21.15.29:0/1607185541 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f88781593d0 0x7f8878151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:31.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.049+0000 7f886e000640 1 -- 172.21.15.29:0/1607185541 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8878071ce0 msgr2=0x7f8878138980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:31.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.049+0000 7f886e000640 1 --2- 172.21.15.29:0/1607185541 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8878071ce0 0x7f8878138980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.049+0000 7f886e000640 1 -- 172.21.15.29:0/1607185541 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f88781593d0 msgr2=0x7f8878151a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:31.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.049+0000 7f886e000640 1 --2- 172.21.15.29:0/1607185541 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f88781593d0 0x7f8878151a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.049+0000 7f886e000640 1 -- 172.21.15.29:0/1607185541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8878138ec0 con 0x7f88780c9f00 2024-04-17T19:48:31.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.049+0000 7f8876a00640 1 --2- 172.21.15.29:0/1607185541 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8878071ce0 0x7f8878138980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:31.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.049+0000 7f8877400640 1 --2- 172.21.15.29:0/1607185541 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f88781593d0 0x7f8878151a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:31.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.049+0000 7f886e000640 1 --2- 172.21.15.29:0/1607185541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88780c9f00 0x7f8878137560 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f886000d620 tx=0x7f886000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:31.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.049+0000 7f886fe00640 1 -- 172.21.15.29:0/1607185541 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886000b620 con 0x7f88780c9f00 2024-04-17T19:48:31.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.049+0000 7f886fe00640 1 -- 172.21.15.29:0/1607185541 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8860019070 con 0x7f88780c9f00 2024-04-17T19:48:31.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.049+0000 7f887de00640 1 -- 172.21.15.29:0/1607185541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f887813f4d0 con 0x7f88780c9f00 2024-04-17T19:48:31.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.049+0000 7f886fe00640 1 -- 172.21.15.29:0/1607185541 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886000b9e0 con 0x7f88780c9f00 2024-04-17T19:48:31.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.049+0000 7f887de00640 1 -- 172.21.15.29:0/1607185541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88781585a0 con 0x7f88780c9f00 2024-04-17T19:48:31.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.049+0000 7f887de00640 1 -- 172.21.15.29:0/1607185541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f887813f7c0 con 0x7f88780c9f00 2024-04-17T19:48:31.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.049+0000 7f886fe00640 1 -- 172.21.15.29:0/1607185541 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f886000bb80 con 0x7f88780c9f00 2024-04-17T19:48:31.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.049+0000 7f886fe00640 1 --2- 172.21.15.29:0/1607185541 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f88540683b0 0x7f885406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:31.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.049+0000 7f886fe00640 1 -- 172.21.15.29:0/1607185541 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f88600872a0 con 0x7f88780c9f00 2024-04-17T19:48:31.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.053+0000 7f8876a00640 1 --2- 172.21.15.29:0/1607185541 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f88540683b0 0x7f885406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:31.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.053+0000 7f8876a00640 1 --2- 172.21.15.29:0/1607185541 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f88540683b0 0x7f885406a8c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f88680af000 tx=0x7f88680a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:31.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.057+0000 7f886fe00640 1 -- 172.21.15.29:0/1607185541 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8860057340 con 0x7f88780c9f00 2024-04-17T19:48:31.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.217+0000 7f887de00640 1 -- 172.21.15.29:0/1607185541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8878069b90 con 0x7f88780c9f00 2024-04-17T19:48:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.217+0000 7f886fe00640 1 -- 172.21.15.29:0/1607185541 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f8860056c40 con 0x7f88780c9f00 2024-04-17T19:48:31.207 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-04-17T19:48:31.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.221+0000 7f887de00640 1 -- 172.21.15.29:0/1607185541 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f88540683b0 msgr2=0x7f885406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:31.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.221+0000 7f887de00640 1 --2- 172.21.15.29:0/1607185541 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f88540683b0 0x7f885406a8c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f88680af000 tx=0x7f88680a7000 comp rx=0 tx=0).stop 2024-04-17T19:48:31.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.221+0000 7f887de00640 1 -- 172.21.15.29:0/1607185541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88780c9f00 msgr2=0x7f8878137560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:31.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.221+0000 7f887de00640 1 --2- 172.21.15.29:0/1607185541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88780c9f00 0x7f8878137560 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f886000d620 tx=0x7f886000dae0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.221+0000 7f887de00640 1 -- 172.21.15.29:0/1607185541 shutdown_connections 2024-04-17T19:48:31.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.221+0000 7f887de00640 1 --2- 172.21.15.29:0/1607185541 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f88540683b0 0x7f885406a8c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.221+0000 7f887de00640 1 --2- 172.21.15.29:0/1607185541 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f88781593d0 0x7f8878151a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.221+0000 7f887de00640 1 --2- 172.21.15.29:0/1607185541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88780c9f00 0x7f8878137560 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.221+0000 7f887de00640 1 --2- 172.21.15.29:0/1607185541 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8878071ce0 0x7f8878138980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.221+0000 7f887de00640 1 -- 172.21.15.29:0/1607185541 >> 172.21.15.29:0/1607185541 conn(0x7f887806bf90 msgr2=0x7f88781410f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:31.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.221+0000 7f887de00640 1 -- 172.21.15.29:0/1607185541 shutdown_connections 2024-04-17T19:48:31.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.221+0000 7f887de00640 1 -- 172.21.15.29:0/1607185541 wait complete. 2024-04-17T19:48:31.229 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T19:48:31.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.397+0000 7fb69fe00640 1 Processor -- start 2024-04-17T19:48:31.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.397+0000 7fb69fe00640 1 -- start start 2024-04-17T19:48:31.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.397+0000 7fb69fe00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb69815cbf0 0x7fb6980b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:31.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.397+0000 7fb69fe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb6980c54b0 0x7fb698166a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:31.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.397+0000 7fb69fe00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb698070b00 con 0x7fb6980c50a0 2024-04-17T19:48:31.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.397+0000 7fb69fe00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fb6980cab70 con 0x7fb6980c54b0 2024-04-17T19:48:31.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.397+0000 7fb69fe00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb6980b5520 con 0x7fb69815cbf0 2024-04-17T19:48:31.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.397+0000 7fb69de00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb6980c54b0 0x7fb698166a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:31.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.397+0000 7fb69ca00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb6980c50a0 0x7fb69806e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48178/0 (socket says 172.21.15.29:48178) 2024-04-17T19:48:31.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.397+0000 7fb69ca00640 1 -- 172.21.15.29:0/1630463173 learned_addr learned my addr 172.21.15.29:0/1630463173 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:31.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.397+0000 7fb69d400640 1 --2- 172.21.15.29:0/1630463173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb69815cbf0 0x7fb6980b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:31.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.401+0000 7fb69de00640 1 -- 172.21.15.29:0/1630463173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb69815cbf0 msgr2=0x7fb6980b4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:31.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.401+0000 7fb69de00640 1 --2- 172.21.15.29:0/1630463173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb69815cbf0 0x7fb6980b4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.401+0000 7fb69de00640 1 -- 172.21.15.29:0/1630463173 >> v1:172.21.15.29:6789/0 conn(0x7fb6980c50a0 legacy=0x7fb69806e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:31.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.401+0000 7fb69de00640 1 -- 172.21.15.29:0/1630463173 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb69806fd80 con 0x7fb6980c54b0 2024-04-17T19:48:31.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.401+0000 7fb697e00640 1 -- 172.21.15.29:0/1630463173 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3129328704 0 0) 0x7fb698070b00 con 0x7fb6980c50a0 2024-04-17T19:48:31.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.401+0000 7fb69de00640 1 --2- 172.21.15.29:0/1630463173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6980c54b0 0x7fb698166a60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb690009d90 tx=0x7fb69001f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e43f0cc63de9e16 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:31.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.401+0000 7fb697e00640 1 -- 172.21.15.29:0/1630463173 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6900b4070 con 0x7fb6980c54b0 2024-04-17T19:48:31.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.401+0000 7fb697e00640 1 -- 172.21.15.29:0/1630463173 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6900af600 con 0x7fb6980c54b0 2024-04-17T19:48:31.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.401+0000 7fb697e00640 1 -- 172.21.15.29:0/1630463173 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6900af9d0 con 0x7fb6980c54b0 2024-04-17T19:48:31.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.405+0000 7fb69fe00640 1 -- 172.21.15.29:0/1630463173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6980c54b0 msgr2=0x7fb698166a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:31.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.405+0000 7fb69fe00640 1 --2- 172.21.15.29:0/1630463173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6980c54b0 0x7fb698166a60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb690009d90 tx=0x7fb69001f150 comp rx=0 tx=0).stop 2024-04-17T19:48:31.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.405+0000 7fb69fe00640 1 -- 172.21.15.29:0/1630463173 shutdown_connections 2024-04-17T19:48:31.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.405+0000 7fb69fe00640 1 --2- 172.21.15.29:0/1630463173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6980c54b0 0x7fb698166a60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.405+0000 7fb69fe00640 1 --2- 172.21.15.29:0/1630463173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb69815cbf0 0x7fb6980b4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.405+0000 7fb69fe00640 1 -- 172.21.15.29:0/1630463173 >> 172.21.15.29:0/1630463173 conn(0x7fb6980cf770 msgr2=0x7fb6980cfb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:31.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.405+0000 7fb69fe00640 1 -- 172.21.15.29:0/1630463173 shutdown_connections 2024-04-17T19:48:31.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.405+0000 7fb69fe00640 1 -- 172.21.15.29:0/1630463173 wait complete. 2024-04-17T19:48:31.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.405+0000 7fb69fe00640 1 Processor -- start 2024-04-17T19:48:31.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.405+0000 7fb69fe00640 1 -- start start 2024-04-17T19:48:31.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.405+0000 7fb69fe00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb6980c50a0 0x7fb69815fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:31.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.405+0000 7fb69fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6980c54b0 0x7fb6981e7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:31.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.405+0000 7fb69fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb69815cbf0 0x7fb6980cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:31.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.405+0000 7fb69fe00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb698070300 con 0x7fb6980c54b0 2024-04-17T19:48:31.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.405+0000 7fb69fe00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb698070020 con 0x7fb69815cbf0 2024-04-17T19:48:31.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.405+0000 7fb69fe00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb6980705e0 con 0x7fb6980c50a0 2024-04-17T19:48:31.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.409+0000 7fb69ca00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6980c54b0 0x7fb6981e7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:31.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.409+0000 7fb69ca00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6980c54b0 0x7fb6981e7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33850/0 (socket says 172.21.15.29:33850) 2024-04-17T19:48:31.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.409+0000 7fb69ca00640 1 -- 172.21.15.29:0/4069449804 learned_addr learned my addr 172.21.15.29:0/4069449804 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:31.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.409+0000 7fb69d400640 1 --2- 172.21.15.29:0/4069449804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb6980c50a0 0x7fb69815fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:31.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.409+0000 7fb69de00640 1 --2- 172.21.15.29:0/4069449804 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb69815cbf0 0x7fb6980cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:31.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.409+0000 7fb69ca00640 1 -- 172.21.15.29:0/4069449804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb6980c50a0 msgr2=0x7fb69815fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:31.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.409+0000 7fb69ca00640 1 --2- 172.21.15.29:0/4069449804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb6980c50a0 0x7fb69815fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.409+0000 7fb69ca00640 1 -- 172.21.15.29:0/4069449804 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb69815cbf0 msgr2=0x7fb6980cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:31.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.409+0000 7fb69ca00640 1 --2- 172.21.15.29:0/4069449804 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb69815cbf0 0x7fb6980cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.409+0000 7fb69ca00640 1 -- 172.21.15.29:0/4069449804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6981e81a0 con 0x7fb6980c54b0 2024-04-17T19:48:31.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.409+0000 7fb69ca00640 1 --2- 172.21.15.29:0/4069449804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6980c54b0 0x7fb6981e7c60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb68400c3e0 tx=0x7fb68400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:31.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.409+0000 7fb695600640 1 -- 172.21.15.29:0/4069449804 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6840074e0 con 0x7fb6980c54b0 2024-04-17T19:48:31.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.409+0000 7fb695600640 1 -- 172.21.15.29:0/4069449804 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6840047f0 con 0x7fb6980c54b0 2024-04-17T19:48:31.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.409+0000 7fb69fe00640 1 -- 172.21.15.29:0/4069449804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6981f2630 con 0x7fb6980c54b0 2024-04-17T19:48:31.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.409+0000 7fb695600640 1 -- 172.21.15.29:0/4069449804 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb684004be0 con 0x7fb6980c54b0 2024-04-17T19:48:31.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.409+0000 7fb69fe00640 1 -- 172.21.15.29:0/4069449804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6981f2bb0 con 0x7fb6980c54b0 2024-04-17T19:48:31.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.409+0000 7fb69fe00640 1 -- 172.21.15.29:0/4069449804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb698069b90 con 0x7fb6980c54b0 2024-04-17T19:48:31.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.409+0000 7fb695600640 1 -- 172.21.15.29:0/4069449804 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb684011070 con 0x7fb6980c54b0 2024-04-17T19:48:31.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.413+0000 7fb695600640 1 --2- 172.21.15.29:0/4069449804 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb678068380 0x7fb67806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:31.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.413+0000 7fb695600640 1 -- 172.21.15.29:0/4069449804 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fb684086940 con 0x7fb6980c54b0 2024-04-17T19:48:31.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.413+0000 7fb69d400640 1 --2- 172.21.15.29:0/4069449804 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb678068380 0x7fb67806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:31.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.413+0000 7fb69d400640 1 --2- 172.21.15.29:0/4069449804 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb678068380 0x7fb67806a890 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009ac0 tx=0x7fb68c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:31.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.417+0000 7fb695600640 1 -- 172.21.15.29:0/4069449804 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb684056400 con 0x7fb6980c54b0 2024-04-17T19:48:31.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.585+0000 7fb69fe00640 1 -- 172.21.15.29:0/4069449804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb6980cacf0 con 0x7fb6980c54b0 2024-04-17T19:48:31.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.585+0000 7fb695600640 1 -- 172.21.15.29:0/4069449804 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fb68405a700 con 0x7fb6980c54b0 2024-04-17T19:48:31.572 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-04-17T19:48:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.585+0000 7fb69fe00640 1 -- 172.21.15.29:0/4069449804 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb678068380 msgr2=0x7fb67806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.585+0000 7fb69fe00640 1 --2- 172.21.15.29:0/4069449804 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb678068380 0x7fb67806a890 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009ac0 tx=0x7fb68c009190 comp rx=0 tx=0).stop 2024-04-17T19:48:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.585+0000 7fb69fe00640 1 -- 172.21.15.29:0/4069449804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6980c54b0 msgr2=0x7fb6981e7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.585+0000 7fb69fe00640 1 --2- 172.21.15.29:0/4069449804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6980c54b0 0x7fb6981e7c60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb68400c3e0 tx=0x7fb68400c8a0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.585+0000 7fb69fe00640 1 -- 172.21.15.29:0/4069449804 shutdown_connections 2024-04-17T19:48:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.585+0000 7fb69fe00640 1 --2- 172.21.15.29:0/4069449804 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb678068380 0x7fb67806a890 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.585+0000 7fb69fe00640 1 --2- 172.21.15.29:0/4069449804 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb69815cbf0 0x7fb6980cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.585+0000 7fb69fe00640 1 --2- 172.21.15.29:0/4069449804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6980c54b0 0x7fb6981e7c60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.585+0000 7fb69fe00640 1 --2- 172.21.15.29:0/4069449804 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb6980c50a0 0x7fb69815fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.585+0000 7fb69fe00640 1 -- 172.21.15.29:0/4069449804 >> 172.21.15.29:0/4069449804 conn(0x7fb6980cf770 msgr2=0x7fb69815c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.585+0000 7fb69fe00640 1 -- 172.21.15.29:0/4069449804 shutdown_connections 2024-04-17T19:48:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.585+0000 7fb69fe00640 1 -- 172.21.15.29:0/4069449804 wait complete. 2024-04-17T19:48:31.595 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T19:48:31.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.761+0000 7f715ca00640 1 Processor -- start 2024-04-17T19:48:31.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.761+0000 7f715ca00640 1 -- start start 2024-04-17T19:48:31.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.761+0000 7f715ca00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f715815cbb0 0x7f715806bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:31.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.761+0000 7f715ca00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7158160040 0x7f7158159610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:31.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.761+0000 7f715ca00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71580ca520 con 0x7f715806c840 2024-04-17T19:48:31.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.761+0000 7f715ca00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f715815c720 con 0x7f715815cbb0 2024-04-17T19:48:31.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.761+0000 7f715ca00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f715806c300 con 0x7f7158160040 2024-04-17T19:48:31.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.761+0000 7f7155e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f715815cbb0 0x7f715806bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:31.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.761+0000 7f714d400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7158160040 0x7f7158159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:31.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.761+0000 7f7155e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f715815cbb0 0x7f715806bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33854/0 (socket says 172.21.15.29:33854) 2024-04-17T19:48:31.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.761+0000 7f7155e00640 1 -- 172.21.15.29:0/3805815261 learned_addr learned my addr 172.21.15.29:0/3805815261 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:31.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.761+0000 7f7156800640 1 --1- 172.21.15.29:0/3805815261 >> v1:172.21.15.37:6789/0 conn(0x7f715806c840 0x7f7158070060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:59504/0 (socket says 172.21.15.29:59504) 2024-04-17T19:48:31.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.765+0000 7f7155e00640 1 -- 172.21.15.29:0/3805815261 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7158160040 msgr2=0x7f7158159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:31.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.765+0000 7f7155e00640 1 --2- 172.21.15.29:0/3805815261 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7158160040 0x7f7158159610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.765+0000 7f7155e00640 1 -- 172.21.15.29:0/3805815261 >> v1:172.21.15.37:6789/0 conn(0x7f715806c840 legacy=0x7f7158070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:31.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.765+0000 7f7155e00640 1 -- 172.21.15.29:0/3805815261 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7158159b50 con 0x7f715815cbb0 2024-04-17T19:48:31.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.765+0000 7f7155e00640 1 --2- 172.21.15.29:0/3805815261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f715815cbb0 0x7f715806bdc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f71480098a0 tx=0x7f714801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffcc02c5d667f7c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:31.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.765+0000 7f7155400640 1 -- 172.21.15.29:0/3805815261 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71480b5070 con 0x7f715815cbb0 2024-04-17T19:48:31.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.765+0000 7f7155400640 1 -- 172.21.15.29:0/3805815261 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71480a8a30 con 0x7f715815cbb0 2024-04-17T19:48:31.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.765+0000 7f7155400640 1 -- 172.21.15.29:0/3805815261 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71480a8e00 con 0x7f715815cbb0 2024-04-17T19:48:31.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.769+0000 7f715ca00640 1 -- 172.21.15.29:0/3805815261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f715815cbb0 msgr2=0x7f715806bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:31.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.769+0000 7f715ca00640 1 --2- 172.21.15.29:0/3805815261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f715815cbb0 0x7f715806bdc0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f71480098a0 tx=0x7f714801ed30 comp rx=0 tx=0).stop 2024-04-17T19:48:31.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.769+0000 7f715ca00640 1 -- 172.21.15.29:0/3805815261 shutdown_connections 2024-04-17T19:48:31.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.769+0000 7f715ca00640 1 --2- 172.21.15.29:0/3805815261 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7158160040 0x7f7158159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.769+0000 7f715ca00640 1 --2- 172.21.15.29:0/3805815261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f715815cbb0 0x7f715806bdc0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.769+0000 7f715ca00640 1 -- 172.21.15.29:0/3805815261 >> 172.21.15.29:0/3805815261 conn(0x7f7158071a60 msgr2=0x7f7158071e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:31.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.769+0000 7f715ca00640 1 -- 172.21.15.29:0/3805815261 shutdown_connections 2024-04-17T19:48:31.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.769+0000 7f715ca00640 1 -- 172.21.15.29:0/3805815261 wait complete. 2024-04-17T19:48:31.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.769+0000 7f715ca00640 1 Processor -- start 2024-04-17T19:48:31.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.769+0000 7f715ca00640 1 -- start start 2024-04-17T19:48:31.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f715ca00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f715806c840 0x7f71581e5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:31.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f715ca00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f715815cbb0 0x7f71581e7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:31.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f715ca00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158160040 0x7f71581e7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:31.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f715ca00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f71580ca790 con 0x7f7158160040 2024-04-17T19:48:31.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f715ca00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f715815c0f0 con 0x7f715806c840 2024-04-17T19:48:31.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f715ca00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f71580caa70 con 0x7f715815cbb0 2024-04-17T19:48:31.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f7156800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158160040 0x7f71581e7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:31.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f7156800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158160040 0x7f71581e7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33868/0 (socket says 172.21.15.29:33868) 2024-04-17T19:48:31.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f7156800640 1 -- 172.21.15.29:0/935047909 learned_addr learned my addr 172.21.15.29:0/935047909 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:31.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f7155e00640 1 --2- 172.21.15.29:0/935047909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f715806c840 0x7f71581e5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:31.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f714d400640 1 --2- 172.21.15.29:0/935047909 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f715815cbb0 0x7f71581e7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:31.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f7156800640 1 -- 172.21.15.29:0/935047909 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f715815cbb0 msgr2=0x7f71581e7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:31.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f7156800640 1 --2- 172.21.15.29:0/935047909 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f715815cbb0 0x7f71581e7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f7156800640 1 -- 172.21.15.29:0/935047909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f715806c840 msgr2=0x7f71581e5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:31.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f7156800640 1 --2- 172.21.15.29:0/935047909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f715806c840 0x7f71581e5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f7156800640 1 -- 172.21.15.29:0/935047909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71581e63d0 con 0x7f7158160040 2024-04-17T19:48:31.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f714d400640 1 --2- 172.21.15.29:0/935047909 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f715815cbb0 0x7f71581e7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:31.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f7156800640 1 --2- 172.21.15.29:0/935047909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158160040 0x7f71581e7560 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f715000c760 tx=0x7f715000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:31.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f714ea00640 1 -- 172.21.15.29:0/935047909 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7150017070 con 0x7f7158160040 2024-04-17T19:48:31.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f714ea00640 1 -- 172.21.15.29:0/935047909 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f715000ce10 con 0x7f7158160040 2024-04-17T19:48:31.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f715ca00640 1 -- 172.21.15.29:0/935047909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71581f3320 con 0x7f7158160040 2024-04-17T19:48:31.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f714ea00640 1 -- 172.21.15.29:0/935047909 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715000a840 con 0x7f7158160040 2024-04-17T19:48:31.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f715ca00640 1 -- 172.21.15.29:0/935047909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71581e5690 con 0x7f7158160040 2024-04-17T19:48:31.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.773+0000 7f715ca00640 1 -- 172.21.15.29:0/935047909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7158069b90 con 0x7f7158160040 2024-04-17T19:48:31.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.777+0000 7f714ea00640 1 -- 172.21.15.29:0/935047909 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f715001a070 con 0x7f7158160040 2024-04-17T19:48:31.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.777+0000 7f714ea00640 1 --2- 172.21.15.29:0/935047909 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f71340683b0 0x7f713406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:31.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.777+0000 7f714ea00640 1 -- 172.21.15.29:0/935047909 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f71500862a0 con 0x7f7158160040 2024-04-17T19:48:31.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.777+0000 7f7155e00640 1 --2- 172.21.15.29:0/935047909 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f71340683b0 0x7f713406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:31.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.781+0000 7f7155e00640 1 --2- 172.21.15.29:0/935047909 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f71340683b0 0x7f713406a8c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f71480b2000 tx=0x7f7148009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:31.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.781+0000 7f714ea00640 1 -- 172.21.15.29:0/935047909 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7150055d70 con 0x7f7158160040 2024-04-17T19:48:31.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.953+0000 7f715ca00640 1 -- 172.21.15.29:0/935047909 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f71580cadc0 con 0x7f7158160040 2024-04-17T19:48:31.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.953+0000 7f714ea00640 1 -- 172.21.15.29:0/935047909 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f715005a070 con 0x7f7158160040 2024-04-17T19:48:31.941 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-04-17T19:48:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.953+0000 7f715ca00640 1 -- 172.21.15.29:0/935047909 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f71340683b0 msgr2=0x7f713406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.953+0000 7f715ca00640 1 --2- 172.21.15.29:0/935047909 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f71340683b0 0x7f713406a8c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f71480b2000 tx=0x7f7148009040 comp rx=0 tx=0).stop 2024-04-17T19:48:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.953+0000 7f715ca00640 1 -- 172.21.15.29:0/935047909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158160040 msgr2=0x7f71581e7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.953+0000 7f715ca00640 1 --2- 172.21.15.29:0/935047909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158160040 0x7f71581e7560 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f715000c760 tx=0x7f715000cc20 comp rx=0 tx=0).stop 2024-04-17T19:48:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.953+0000 7f715ca00640 1 -- 172.21.15.29:0/935047909 shutdown_connections 2024-04-17T19:48:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.953+0000 7f715ca00640 1 --2- 172.21.15.29:0/935047909 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f71340683b0 0x7f713406a8c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.953+0000 7f715ca00640 1 --2- 172.21.15.29:0/935047909 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7158160040 0x7f71581e7560 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.953+0000 7f715ca00640 1 --2- 172.21.15.29:0/935047909 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f715815cbb0 0x7f71581e7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.953+0000 7f715ca00640 1 --2- 172.21.15.29:0/935047909 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f715806c840 0x7f71581e5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:31.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.953+0000 7f715ca00640 1 -- 172.21.15.29:0/935047909 >> 172.21.15.29:0/935047909 conn(0x7f7158071a60 msgr2=0x7f7158155ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:31.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.953+0000 7f715ca00640 1 -- 172.21.15.29:0/935047909 shutdown_connections 2024-04-17T19:48:31.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:31.953+0000 7f715ca00640 1 -- 172.21.15.29:0/935047909 wait complete. 2024-04-17T19:48:31.963 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:48:32.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.125+0000 7f9887c00640 1 Processor -- start 2024-04-17T19:48:32.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.125+0000 7f9887c00640 1 -- start start 2024-04-17T19:48:32.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f9887c00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f98800cfde0 0x7f9880071940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f9887c00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98800c4930 con 0x7f9880071e80 2024-04-17T19:48:32.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f9887c00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98800cf620 con 0x7f9880159e60 2024-04-17T19:48:32.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f9887c00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f98800c5190 con 0x7f98800cfde0 2024-04-17T19:48:32.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f9885c00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9880071e80 0x7f98801647e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48182/0 (socket says 172.21.15.29:48182) 2024-04-17T19:48:32.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f9885c00640 1 -- 172.21.15.29:0/635151441 learned_addr learned my addr 172.21.15.29:0/635151441 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:32.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f987fe00640 1 --2- 172.21.15.29:0/635151441 >> v2:172.21.15.37:3300/0 conn(0x7f98800cfde0 0x7f9880071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f987f400640 1 -- 172.21.15.29:0/635151441 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 95633085 0 0) 0x7f98800c4930 con 0x7f9880071e80 2024-04-17T19:48:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f987f400640 1 -- 172.21.15.29:0/635151441 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9868003530 con 0x7f9880071e80 2024-04-17T19:48:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f987f400640 1 -- 172.21.15.29:0/635151441 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1293260156 0 0) 0x7f9868003530 con 0x7f9880071e80 2024-04-17T19:48:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f987f400640 1 -- 172.21.15.29:0/635151441 >> v2:172.21.15.37:3300/0 conn(0x7f98800cfde0 msgr2=0x7f9880071940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f987f400640 1 --2- 172.21.15.29:0/635151441 >> v2:172.21.15.37:3300/0 conn(0x7f98800cfde0 0x7f9880071940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f987f400640 1 -- 172.21.15.29:0/635151441 >> v1:172.21.15.37:6789/0 conn(0x7f9880159e60 legacy=0x7f9880153d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f987f400640 1 -- 172.21.15.29:0/635151441 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98800c4bd0 con 0x7f9880071e80 2024-04-17T19:48:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f987f400640 1 -- 172.21.15.29:0/635151441 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f9878001a00 con 0x7f9880071e80 2024-04-17T19:48:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f987f400640 1 -- 172.21.15.29:0/635151441 >> v1:172.21.15.29:6789/0 conn(0x7f9880071e80 legacy=0x7f98801647e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f987f400640 1 --2- 172.21.15.29:0/635151441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9868003a30 0x7f9868003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f987f400640 1 --2- 172.21.15.29:0/635151441 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9868004810 0x7f9868004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f987f400640 1 --2- 172.21.15.29:0/635151441 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9868005230 0x7f986800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f987f400640 1 -- 172.21.15.29:0/635151441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f98800c5190 con 0x7f9868003a30 2024-04-17T19:48:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f987f400640 1 -- 172.21.15.29:0/635151441 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f986800c1c0 con 0x7f9868004810 2024-04-17T19:48:32.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f987f400640 1 -- 172.21.15.29:0/635151441 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f986800c3c0 con 0x7f9868005230 2024-04-17T19:48:32.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f9885200640 1 --2- 172.21.15.29:0/635151441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9868003a30 0x7f9868003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f9885c00640 1 --2- 172.21.15.29:0/635151441 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9868005230 0x7f986800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f987fe00640 1 --2- 172.21.15.29:0/635151441 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9868004810 0x7f9868004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f9885200640 1 -- 172.21.15.29:0/635151441 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9868005230 msgr2=0x7f986800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:32.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f9885200640 1 --2- 172.21.15.29:0/635151441 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9868005230 0x7f986800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f9885200640 1 -- 172.21.15.29:0/635151441 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9868004810 msgr2=0x7f9868004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:32.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f9885200640 1 --2- 172.21.15.29:0/635151441 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9868004810 0x7f9868004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f9885200640 1 -- 172.21.15.29:0/635151441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9868003530 con 0x7f9868003a30 2024-04-17T19:48:32.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.129+0000 7f9885c00640 1 -- 172.21.15.29:0/635151441 reap_dead start 2024-04-17T19:48:32.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.133+0000 7f9885200640 1 --2- 172.21.15.29:0/635151441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9868003a30 0x7f9868003ea0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9874001750 tx=0x7f987401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44bde9fa8da21f05 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:32.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.133+0000 7f987f400640 1 -- 172.21.15.29:0/635151441 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98740013a0 con 0x7f9868003a30 2024-04-17T19:48:32.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.133+0000 7f987f400640 1 -- 172.21.15.29:0/635151441 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98740052d0 con 0x7f9868003a30 2024-04-17T19:48:32.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.133+0000 7f987f400640 1 -- 172.21.15.29:0/635151441 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98740054c0 con 0x7f9868003a30 2024-04-17T19:48:32.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.133+0000 7f9887c00640 1 -- 172.21.15.29:0/635151441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9868003a30 msgr2=0x7f9868003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:32.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.133+0000 7f9887c00640 1 --2- 172.21.15.29:0/635151441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9868003a30 0x7f9868003ea0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9874001750 tx=0x7f987401c3e0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.133+0000 7f9887c00640 1 -- 172.21.15.29:0/635151441 shutdown_connections 2024-04-17T19:48:32.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.133+0000 7f9887c00640 1 --2- 172.21.15.29:0/635151441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9868003a30 0x7f9868003ea0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.133+0000 7f9887c00640 1 -- 172.21.15.29:0/635151441 >> 172.21.15.29:0/635151441 conn(0x7f9880067eb0 msgr2=0x7f9880070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:32.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.133+0000 7f9887c00640 1 -- 172.21.15.29:0/635151441 shutdown_connections 2024-04-17T19:48:32.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.133+0000 7f9887c00640 1 -- 172.21.15.29:0/635151441 wait complete. 2024-04-17T19:48:32.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f9887c00640 1 Processor -- start 2024-04-17T19:48:32.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f9887c00640 1 -- start start 2024-04-17T19:48:32.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f9887c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9868003a30 0x7f98800ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f9887c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98801f3900 0x7f9880146ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f9887c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98800cd810 0x7f98800cdc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f9887c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f98800cbd10 con 0x7f9868003a30 2024-04-17T19:48:32.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f9887c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f98800cbb90 con 0x7f98801f3900 2024-04-17T19:48:32.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f9887c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f98800c4510 con 0x7f98800cd810 2024-04-17T19:48:32.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f9885200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9868003a30 0x7f98800ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f9885200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9868003a30 0x7f98800ce5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33884/0 (socket says 172.21.15.29:33884) 2024-04-17T19:48:32.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f9885200640 1 -- 172.21.15.29:0/2812079740 learned_addr learned my addr 172.21.15.29:0/2812079740 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:32.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f9885c00640 1 --2- 172.21.15.29:0/2812079740 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98800cd810 0x7f98800cdc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f987fe00640 1 --2- 172.21.15.29:0/2812079740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98801f3900 0x7f9880146ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f9885200640 1 -- 172.21.15.29:0/2812079740 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98800cd810 msgr2=0x7f98800cdc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:32.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f9885200640 1 --2- 172.21.15.29:0/2812079740 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98800cd810 0x7f98800cdc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f9885200640 1 -- 172.21.15.29:0/2812079740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98801f3900 msgr2=0x7f9880146ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:32.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f9885200640 1 --2- 172.21.15.29:0/2812079740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98801f3900 0x7f9880146ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f9885200640 1 -- 172.21.15.29:0/2812079740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98800cebc0 con 0x7f9868003a30 2024-04-17T19:48:32.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f9885200640 1 --2- 172.21.15.29:0/2812079740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9868003a30 0x7f98800ce5a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9874005040 tx=0x7f9868005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:32.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f987cc00640 1 -- 172.21.15.29:0/2812079740 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98740bc070 con 0x7f9868003a30 2024-04-17T19:48:32.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.137+0000 7f987cc00640 1 -- 172.21.15.29:0/2812079740 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9874005480 con 0x7f9868003a30 2024-04-17T19:48:32.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.141+0000 7f987cc00640 1 -- 172.21.15.29:0/2812079740 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9874007a60 con 0x7f9868003a30 2024-04-17T19:48:32.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.141+0000 7f9887c00640 1 -- 172.21.15.29:0/2812079740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f988015b1a0 con 0x7f9868003a30 2024-04-17T19:48:32.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.141+0000 7f9887c00640 1 -- 172.21.15.29:0/2812079740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f988015b760 con 0x7f9868003a30 2024-04-17T19:48:32.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.141+0000 7f9887c00640 1 -- 172.21.15.29:0/2812079740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9880069b90 con 0x7f9868003a30 2024-04-17T19:48:32.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.141+0000 7f987cc00640 1 -- 172.21.15.29:0/2812079740 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f98740bf070 con 0x7f9868003a30 2024-04-17T19:48:32.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.141+0000 7f987cc00640 1 --2- 172.21.15.29:0/2812079740 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f98600683b0 0x7f986006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.141+0000 7f987cc00640 1 -- 172.21.15.29:0/2812079740 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f98741201b0 con 0x7f9868003a30 2024-04-17T19:48:32.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.141+0000 7f987fe00640 1 --2- 172.21.15.29:0/2812079740 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f98600683b0 0x7f986006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.145+0000 7f987fe00640 1 --2- 172.21.15.29:0/2812079740 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f98600683b0 0x7f986006a8c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f98800b26a0 tx=0x7f986c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:32.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.149+0000 7f987cc00640 1 -- 172.21.15.29:0/2812079740 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f98740160f0 con 0x7f9868003a30 2024-04-17T19:48:32.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.345+0000 7f9887c00640 1 -- 172.21.15.29:0/2812079740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98800cce20 con 0x7f9868003a30 2024-04-17T19:48:32.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.345+0000 7f987cc00640 1 -- 172.21.15.29:0/2812079740 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6898 (secure 0 0 0) 0x7f98740f02e0 con 0x7f9868003a30 2024-04-17T19:48:32.334 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:32.334 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:48:28.833454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4226":{"gid":4226,"name":"b","rank":4,"incarnation":18,"state":"up:creating","state_seq":1,"addr":"172.21.15.37:6838/332907250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":332907250},{"type":"v1","addr":"172.21.15.37:6838","nonce":332907250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229]},"id":1}]} 2024-04-17T19:48:32.335 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 18 2024-04-17T19:48:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.349+0000 7f9887c00640 1 -- 172.21.15.29:0/2812079740 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f98600683b0 msgr2=0x7f986006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.349+0000 7f9887c00640 1 --2- 172.21.15.29:0/2812079740 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f98600683b0 0x7f986006a8c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f98800b26a0 tx=0x7f986c008040 comp rx=0 tx=0).stop 2024-04-17T19:48:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.349+0000 7f9887c00640 1 -- 172.21.15.29:0/2812079740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9868003a30 msgr2=0x7f98800ce5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.349+0000 7f9887c00640 1 --2- 172.21.15.29:0/2812079740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9868003a30 0x7f98800ce5a0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9874005040 tx=0x7f9868005840 comp rx=0 tx=0).stop 2024-04-17T19:48:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.349+0000 7f9887c00640 1 -- 172.21.15.29:0/2812079740 shutdown_connections 2024-04-17T19:48:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.349+0000 7f9887c00640 1 --2- 172.21.15.29:0/2812079740 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f98600683b0 0x7f986006a8c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.349+0000 7f9887c00640 1 --2- 172.21.15.29:0/2812079740 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f98800cd810 0x7f98800cdc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.349+0000 7f9887c00640 1 --2- 172.21.15.29:0/2812079740 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f98801f3900 0x7f9880146ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.349+0000 7f9887c00640 1 --2- 172.21.15.29:0/2812079740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9868003a30 0x7f98800ce5a0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.349+0000 7f9887c00640 1 -- 172.21.15.29:0/2812079740 >> 172.21.15.29:0/2812079740 conn(0x7f9880067eb0 msgr2=0x7f988014a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:32.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.349+0000 7f9887c00640 1 -- 172.21.15.29:0/2812079740 shutdown_connections 2024-04-17T19:48:32.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.349+0000 7f9887c00640 1 -- 172.21.15.29:0/2812079740 wait complete. 2024-04-17T19:48:32.356 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:48:32.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.521+0000 7f672f200640 1 Processor -- start 2024-04-17T19:48:32.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.521+0000 7f672f200640 1 -- start start 2024-04-17T19:48:32.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.521+0000 7f672f200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6730156370 0x7f67300ca110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.521+0000 7f672f200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f673006c730 con 0x7f673015c970 2024-04-17T19:48:32.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.521+0000 7f672f200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f673006c9f0 con 0x7f67300cabe0 2024-04-17T19:48:32.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.521+0000 7f672f200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f673006ccd0 con 0x7f6730156370 2024-04-17T19:48:32.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.521+0000 7f672e800640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f673015c970 0x7f6730166740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48198/0 (socket says 172.21.15.29:48198) 2024-04-17T19:48:32.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.521+0000 7f672e800640 1 -- 172.21.15.29:0/3187243900 learned_addr learned my addr 172.21.15.29:0/3187243900 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:32.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.521+0000 7f672de00640 1 --2- 172.21.15.29:0/3187243900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6730156370 0x7f67300ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.521+0000 7f672ca00640 1 -- 172.21.15.29:0/3187243900 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 439152249 0 0) 0x7f673006c730 con 0x7f673015c970 2024-04-17T19:48:32.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.521+0000 7f672ca00640 1 -- 172.21.15.29:0/3187243900 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6714003530 con 0x7f673015c970 2024-04-17T19:48:32.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.521+0000 7f672ca00640 1 -- 172.21.15.29:0/3187243900 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1799568670 0 0) 0x7f673006c9f0 con 0x7f67300cabe0 2024-04-17T19:48:32.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.521+0000 7f672ca00640 1 -- 172.21.15.29:0/3187243900 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f673006c730 con 0x7f67300cabe0 2024-04-17T19:48:32.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.521+0000 7f672de00640 1 -- 172.21.15.29:0/3187243900 >> v1:172.21.15.37:6789/0 conn(0x7f67300cabe0 legacy=0x7f673015c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:32.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.521+0000 7f672de00640 1 -- 172.21.15.29:0/3187243900 >> v1:172.21.15.29:6789/0 conn(0x7f673015c970 legacy=0x7f6730166740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:32.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.521+0000 7f672de00640 1 -- 172.21.15.29:0/3187243900 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f673006cfb0 con 0x7f6730156370 2024-04-17T19:48:32.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.525+0000 7f672de00640 1 --2- 172.21.15.29:0/3187243900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6730156370 0x7f67300ca110 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f67200097f0 tx=0x7f672001ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6a42e6aaf38a268 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:32.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.525+0000 7f672ca00640 1 -- 172.21.15.29:0/3187243900 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67200b5070 con 0x7f6730156370 2024-04-17T19:48:32.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.525+0000 7f672ca00640 1 -- 172.21.15.29:0/3187243900 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67200a89d0 con 0x7f6730156370 2024-04-17T19:48:32.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.525+0000 7f672ca00640 1 -- 172.21.15.29:0/3187243900 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67200a8d80 con 0x7f6730156370 2024-04-17T19:48:32.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.525+0000 7f672f200640 1 -- 172.21.15.29:0/3187243900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6730156370 msgr2=0x7f67300ca110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:32.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.525+0000 7f672f200640 1 --2- 172.21.15.29:0/3187243900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6730156370 0x7f67300ca110 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f67200097f0 tx=0x7f672001ecd0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.525+0000 7f672f200640 1 -- 172.21.15.29:0/3187243900 shutdown_connections 2024-04-17T19:48:32.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.525+0000 7f672f200640 1 --2- 172.21.15.29:0/3187243900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6730156370 0x7f67300ca110 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.525+0000 7f672f200640 1 -- 172.21.15.29:0/3187243900 >> 172.21.15.29:0/3187243900 conn(0x7f67300680e0 msgr2=0x7f673006dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:32.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.525+0000 7f672f200640 1 -- 172.21.15.29:0/3187243900 shutdown_connections 2024-04-17T19:48:32.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.525+0000 7f672f200640 1 -- 172.21.15.29:0/3187243900 wait complete. 2024-04-17T19:48:32.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.529+0000 7f672f200640 1 Processor -- start 2024-04-17T19:48:32.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.529+0000 7f672f200640 1 -- start start 2024-04-17T19:48:32.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.529+0000 7f672f200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67300cabe0 0x7f67300c8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.529+0000 7f672f200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6730156370 0x7f673014f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.529+0000 7f672f200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f673015c970 0x7f67301f58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.529+0000 7f672f200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f67300ca650 con 0x7f6730156370 2024-04-17T19:48:32.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.529+0000 7f672f200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f673006d570 con 0x7f67300cabe0 2024-04-17T19:48:32.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.529+0000 7f672f200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f673006e7c0 con 0x7f673015c970 2024-04-17T19:48:32.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.529+0000 7f672d400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6730156370 0x7f673014f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.529+0000 7f672d400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6730156370 0x7f673014f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33894/0 (socket says 172.21.15.29:33894) 2024-04-17T19:48:32.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.529+0000 7f672d400640 1 -- 172.21.15.29:0/3140573764 learned_addr learned my addr 172.21.15.29:0/3140573764 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:32.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.529+0000 7f672de00640 1 --2- 172.21.15.29:0/3140573764 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67300cabe0 0x7f67300c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.533+0000 7f672e800640 1 --2- 172.21.15.29:0/3140573764 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f673015c970 0x7f67301f58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.533+0000 7f672d400640 1 -- 172.21.15.29:0/3140573764 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f673015c970 msgr2=0x7f67301f58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:32.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.533+0000 7f672d400640 1 --2- 172.21.15.29:0/3140573764 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f673015c970 0x7f67301f58f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.533+0000 7f672d400640 1 -- 172.21.15.29:0/3140573764 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67300cabe0 msgr2=0x7f67300c8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:32.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.533+0000 7f672d400640 1 --2- 172.21.15.29:0/3140573764 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67300cabe0 0x7f67300c8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.533+0000 7f672d400640 1 -- 172.21.15.29:0/3140573764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67301f66c0 con 0x7f6730156370 2024-04-17T19:48:32.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.533+0000 7f672d400640 1 --2- 172.21.15.29:0/3140573764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6730156370 0x7f673014f8c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f671c00c430 tx=0x7f671c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:32.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.533+0000 7f6726000640 1 -- 172.21.15.29:0/3140573764 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c017070 con 0x7f6730156370 2024-04-17T19:48:32.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.533+0000 7f6726000640 1 -- 172.21.15.29:0/3140573764 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f671c0052f0 con 0x7f6730156370 2024-04-17T19:48:32.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.533+0000 7f6726000640 1 -- 172.21.15.29:0/3140573764 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671c003840 con 0x7f6730156370 2024-04-17T19:48:32.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.533+0000 7f672f200640 1 -- 172.21.15.29:0/3140573764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67301568e0 con 0x7f6730156370 2024-04-17T19:48:32.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.533+0000 7f672f200640 1 -- 172.21.15.29:0/3140573764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67300c95a0 con 0x7f6730156370 2024-04-17T19:48:32.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.533+0000 7f672f200640 1 -- 172.21.15.29:0/3140573764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f673004ff10 con 0x7f6730156370 2024-04-17T19:48:32.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.533+0000 7f6726000640 1 -- 172.21.15.29:0/3140573764 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f671c01a070 con 0x7f6730156370 2024-04-17T19:48:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.537+0000 7f6726000640 1 --2- 172.21.15.29:0/3140573764 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6710068380 0x7f671006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.537+0000 7f6726000640 1 -- 172.21.15.29:0/3140573764 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f671c085ff0 con 0x7f6730156370 2024-04-17T19:48:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.537+0000 7f672de00640 1 --2- 172.21.15.29:0/3140573764 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6710068380 0x7f671006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.537+0000 7f672de00640 1 --2- 172.21.15.29:0/3140573764 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6710068380 0x7f671006a890 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f67200b2000 tx=0x7f67200aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:32.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.541+0000 7f6726000640 1 -- 172.21.15.29:0/3140573764 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f671c055b50 con 0x7f6730156370 2024-04-17T19:48:32.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.705+0000 7f672f200640 1 -- 172.21.15.29:0/3140573764 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f673006dd80 con 0x7f6730156370 2024-04-17T19:48:32.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.705+0000 7f6726000640 1 -- 172.21.15.29:0/3140573764 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18780 (secure 0 0 0) 0x7f671c059e50 con 0x7f6730156370 2024-04-17T19:48:32.696 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:32.696 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":26,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:48:29.829433+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T19:47:47.658449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T19:47:48.209047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T19:47:51.887909+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:48:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.709+0000 7f672f200640 1 -- 172.21.15.29:0/3140573764 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6710068380 msgr2=0x7f671006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.709+0000 7f672f200640 1 --2- 172.21.15.29:0/3140573764 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6710068380 0x7f671006a890 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f67200b2000 tx=0x7f67200aa070 comp rx=0 tx=0).stop 2024-04-17T19:48:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.709+0000 7f672f200640 1 -- 172.21.15.29:0/3140573764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6730156370 msgr2=0x7f673014f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.709+0000 7f672f200640 1 --2- 172.21.15.29:0/3140573764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6730156370 0x7f673014f8c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f671c00c430 tx=0x7f671c00c8f0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.709+0000 7f672f200640 1 -- 172.21.15.29:0/3140573764 shutdown_connections 2024-04-17T19:48:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.709+0000 7f672f200640 1 --2- 172.21.15.29:0/3140573764 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6710068380 0x7f671006a890 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.709+0000 7f672f200640 1 --2- 172.21.15.29:0/3140573764 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f673015c970 0x7f67301f58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.709+0000 7f672f200640 1 --2- 172.21.15.29:0/3140573764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6730156370 0x7f673014f8c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.709+0000 7f672f200640 1 --2- 172.21.15.29:0/3140573764 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f67300cabe0 0x7f67300c8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.709+0000 7f672f200640 1 -- 172.21.15.29:0/3140573764 >> 172.21.15.29:0/3140573764 conn(0x7f67300680e0 msgr2=0x7f67300cf0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:32.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.709+0000 7f672f200640 1 -- 172.21.15.29:0/3140573764 shutdown_connections 2024-04-17T19:48:32.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.709+0000 7f672f200640 1 -- 172.21.15.29:0/3140573764 wait complete. 2024-04-17T19:48:32.718 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T19:47:36.674676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T19:47:47.658449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T19:47:48.209047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T19:47:51.887909+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T19:48:32.718 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T19:48:32.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.885+0000 7f5eb1c00640 1 Processor -- start 2024-04-17T19:48:32.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.885+0000 7f5eb1c00640 1 -- start start 2024-04-17T19:48:32.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.885+0000 7f5eb1c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5eac155df0 0x7f5eac06bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.885+0000 7f5eb1c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eac15fcb0 0x7f5eac0ce880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.885+0000 7f5eb1c00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5eac15bde0 con 0x7f5eac0ce470 2024-04-17T19:48:32.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.885+0000 7f5eb1c00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f5eac0c8ab0 con 0x7f5eac155df0 2024-04-17T19:48:32.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.885+0000 7f5eb1c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5eac156200 con 0x7f5eac15fcb0 2024-04-17T19:48:32.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.885+0000 7f5eabe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5eac155df0 0x7f5eac06bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.885+0000 7f5eab400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5eac0ce470 0x7f5eac15f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48208/0 (socket says 172.21.15.29:48208) 2024-04-17T19:48:32.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.885+0000 7f5eab400640 1 -- 172.21.15.29:0/2016699603 learned_addr learned my addr 172.21.15.29:0/2016699603 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:32.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.889+0000 7f5eb1200640 1 --2- 172.21.15.29:0/2016699603 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eac15fcb0 0x7f5eac0ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.889+0000 7f5eabe00640 1 -- 172.21.15.29:0/2016699603 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eac15fcb0 msgr2=0x7f5eac0ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:32.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.889+0000 7f5eabe00640 1 --2- 172.21.15.29:0/2016699603 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eac15fcb0 0x7f5eac0ce880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.876 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.889+0000 7f5eabe00640 1 -- 172.21.15.29:0/2016699603 >> v1:172.21.15.29:6789/0 conn(0x7f5eac0ce470 legacy=0x7f5eac15f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:32.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.889+0000 7f5eabe00640 1 -- 172.21.15.29:0/2016699603 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eac15c0c0 con 0x7f5eac155df0 2024-04-17T19:48:32.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.889+0000 7f5eb1200640 1 --2- 172.21.15.29:0/2016699603 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eac15fcb0 0x7f5eac0ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.889+0000 7f5eabe00640 1 --2- 172.21.15.29:0/2016699603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac155df0 0x7f5eac06bdc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c0097c0 tx=0x7f5e9c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1428c8cf328c90d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.889+0000 7f5eaaa00640 1 -- 172.21.15.29:0/2016699603 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c0b0070 con 0x7f5eac155df0 2024-04-17T19:48:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.889+0000 7f5eaaa00640 1 -- 172.21.15.29:0/2016699603 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e9c0ab600 con 0x7f5eac155df0 2024-04-17T19:48:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.889+0000 7f5eaaa00640 1 -- 172.21.15.29:0/2016699603 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e9c0ab9d0 con 0x7f5eac155df0 2024-04-17T19:48:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.893+0000 7f5eb1c00640 1 -- 172.21.15.29:0/2016699603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac155df0 msgr2=0x7f5eac06bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.893+0000 7f5eb1c00640 1 --2- 172.21.15.29:0/2016699603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac155df0 0x7f5eac06bdc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c0097c0 tx=0x7f5e9c01c4a0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.893+0000 7f5eb1c00640 1 -- 172.21.15.29:0/2016699603 shutdown_connections 2024-04-17T19:48:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.893+0000 7f5eb1c00640 1 --2- 172.21.15.29:0/2016699603 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eac15fcb0 0x7f5eac0ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.893+0000 7f5eb1c00640 1 --2- 172.21.15.29:0/2016699603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac155df0 0x7f5eac06bdc0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.893+0000 7f5eb1c00640 1 -- 172.21.15.29:0/2016699603 >> 172.21.15.29:0/2016699603 conn(0x7f5eac06ece0 msgr2=0x7f5eac06f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.893+0000 7f5eb1c00640 1 -- 172.21.15.29:0/2016699603 shutdown_connections 2024-04-17T19:48:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.893+0000 7f5eb1c00640 1 -- 172.21.15.29:0/2016699603 wait complete. 2024-04-17T19:48:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.893+0000 7f5eb1c00640 1 Processor -- start 2024-04-17T19:48:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.893+0000 7f5eb1c00640 1 -- start start 2024-04-17T19:48:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eb1c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5eac0ce470 0x7f5eac14bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eb1c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eac155df0 0x7f5eac142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eb1c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac15fcb0 0x7f5eac0c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eb1c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5eac06c300 con 0x7f5eac15fcb0 2024-04-17T19:48:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eb1c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5eac15ccd0 con 0x7f5eac0ce470 2024-04-17T19:48:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eb1c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5eac06c480 con 0x7f5eac155df0 2024-04-17T19:48:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eb1200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac15fcb0 0x7f5eac0c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eb1200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac15fcb0 0x7f5eac0c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33908/0 (socket says 172.21.15.29:33908) 2024-04-17T19:48:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eb1200640 1 -- 172.21.15.29:0/1587757888 learned_addr learned my addr 172.21.15.29:0/1587757888 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:32.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eab400640 1 --2- 172.21.15.29:0/1587757888 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eac155df0 0x7f5eac142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eabe00640 1 --2- 172.21.15.29:0/1587757888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5eac0ce470 0x7f5eac14bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eb1200640 1 -- 172.21.15.29:0/1587757888 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eac155df0 msgr2=0x7f5eac142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eb1200640 1 --2- 172.21.15.29:0/1587757888 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eac155df0 0x7f5eac142ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eb1200640 1 -- 172.21.15.29:0/1587757888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5eac0ce470 msgr2=0x7f5eac14bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eb1200640 1 --2- 172.21.15.29:0/1587757888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5eac0ce470 0x7f5eac14bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eb1200640 1 -- 172.21.15.29:0/1587757888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eac14c460 con 0x7f5eac15fcb0 2024-04-17T19:48:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eabe00640 1 --2- 172.21.15.29:0/1587757888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5eac0ce470 0x7f5eac14bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eb1200640 1 --2- 172.21.15.29:0/1587757888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac15fcb0 0x7f5eac0c7380 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4007e30 tx=0x7f5ea400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5ea3e00640 1 -- 172.21.15.29:0/1587757888 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea401a070 con 0x7f5eac15fcb0 2024-04-17T19:48:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5ea3e00640 1 -- 172.21.15.29:0/1587757888 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ea4015400 con 0x7f5eac15fcb0 2024-04-17T19:48:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eb1c00640 1 -- 172.21.15.29:0/1587757888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eac1494b0 con 0x7f5eac15fcb0 2024-04-17T19:48:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5ea3e00640 1 -- 172.21.15.29:0/1587757888 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea40157d0 con 0x7f5eac15fcb0 2024-04-17T19:48:32.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eb1c00640 1 -- 172.21.15.29:0/1587757888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eac15b8e0 con 0x7f5eac15fcb0 2024-04-17T19:48:32.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5eb1c00640 1 -- 172.21.15.29:0/1587757888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5eac04ff10 con 0x7f5eac15fcb0 2024-04-17T19:48:32.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.897+0000 7f5ea3e00640 1 -- 172.21.15.29:0/1587757888 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5ea401d070 con 0x7f5eac15fcb0 2024-04-17T19:48:32.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.901+0000 7f5ea3e00640 1 --2- 172.21.15.29:0/1587757888 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5e8c0683b0 0x7f5e8c06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:32.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.901+0000 7f5ea3e00640 1 -- 172.21.15.29:0/1587757888 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5ea4086f20 con 0x7f5eac15fcb0 2024-04-17T19:48:32.888 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.901+0000 7f5eabe00640 1 --2- 172.21.15.29:0/1587757888 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5e8c0683b0 0x7f5e8c06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:32.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.901+0000 7f5eabe00640 1 --2- 172.21.15.29:0/1587757888 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5e8c0683b0 0x7f5e8c06a8c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c0ad000 tx=0x7f5e9c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:32.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:32.909+0000 7f5ea3e00640 1 -- 172.21.15.29:0/1587757888 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5ea4010070 con 0x7f5eac15fcb0 2024-04-17T19:48:33.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.081+0000 7f5eb1c00640 1 -- 172.21.15.29:0/1587757888 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5eac159680 con 0x7f5eac15fcb0 2024-04-17T19:48:33.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.085+0000 7f5ea3e00640 1 -- 172.21.15.29:0/1587757888 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f5ea4057170 con 0x7f5eac15fcb0 2024-04-17T19:48:33.072 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-04-17T19:48:33.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.085+0000 7f5eb1c00640 1 -- 172.21.15.29:0/1587757888 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5e8c0683b0 msgr2=0x7f5e8c06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.085+0000 7f5eb1c00640 1 --2- 172.21.15.29:0/1587757888 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5e8c0683b0 0x7f5e8c06a8c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c0ad000 tx=0x7f5e9c008040 comp rx=0 tx=0).stop 2024-04-17T19:48:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.085+0000 7f5eb1c00640 1 -- 172.21.15.29:0/1587757888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac15fcb0 msgr2=0x7f5eac0c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.085+0000 7f5eb1c00640 1 --2- 172.21.15.29:0/1587757888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac15fcb0 0x7f5eac0c7380 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4007e30 tx=0x7f5ea400d4a0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.085+0000 7f5eb1c00640 1 -- 172.21.15.29:0/1587757888 shutdown_connections 2024-04-17T19:48:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.085+0000 7f5eb1c00640 1 --2- 172.21.15.29:0/1587757888 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5e8c0683b0 0x7f5e8c06a8c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.085+0000 7f5eb1c00640 1 --2- 172.21.15.29:0/1587757888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5eac15fcb0 0x7f5eac0c7380 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.085+0000 7f5eb1c00640 1 --2- 172.21.15.29:0/1587757888 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5eac155df0 0x7f5eac142ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.085+0000 7f5eb1c00640 1 --2- 172.21.15.29:0/1587757888 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5eac0ce470 0x7f5eac14bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.085+0000 7f5eb1c00640 1 -- 172.21.15.29:0/1587757888 >> 172.21.15.29:0/1587757888 conn(0x7f5eac06ece0 msgr2=0x7f5eac1550c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.085+0000 7f5eb1c00640 1 -- 172.21.15.29:0/1587757888 shutdown_connections 2024-04-17T19:48:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.085+0000 7f5eb1c00640 1 -- 172.21.15.29:0/1587757888 wait complete. 2024-04-17T19:48:33.094 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T19:48:33.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.261+0000 7f9339200640 1 Processor -- start 2024-04-17T19:48:33.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.261+0000 7f9339200640 1 -- start start 2024-04-17T19:48:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.261+0000 7f9339200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f933406cc40 0x7f933406bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.261+0000 7f9339200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f933415cf10 0x7f933406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.261+0000 7f9339200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f933415fa10 con 0x7f93340c1970 2024-04-17T19:48:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.261+0000 7f9339200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f933415f7e0 con 0x7f933406cc40 2024-04-17T19:48:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.261+0000 7f9339200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f933415c370 con 0x7f933415cf10 2024-04-17T19:48:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.261+0000 7f9331e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f933406cc40 0x7f933406bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.261+0000 7f932be00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f93340c1970 0x7f933415c800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48216/0 (socket says 172.21.15.29:48216) 2024-04-17T19:48:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.261+0000 7f932be00640 1 -- 172.21.15.29:0/1895655611 learned_addr learned my addr 172.21.15.29:0/1895655611 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.261+0000 7f9332800640 1 --2- 172.21.15.29:0/1895655611 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f933415cf10 0x7f933406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.265+0000 7f9331400640 1 -- 172.21.15.29:0/1895655611 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 744556168 0 0) 0x7f933415fa10 con 0x7f93340c1970 2024-04-17T19:48:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.265+0000 7f9331e00640 1 -- 172.21.15.29:0/1895655611 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f933415cf10 msgr2=0x7f933406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.265+0000 7f9331e00640 1 --2- 172.21.15.29:0/1895655611 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f933415cf10 0x7f933406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.265+0000 7f9331e00640 1 -- 172.21.15.29:0/1895655611 >> v1:172.21.15.29:6789/0 conn(0x7f93340c1970 legacy=0x7f933415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.265+0000 7f9331e00640 1 -- 172.21.15.29:0/1895655611 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f933415fcf0 con 0x7f933406cc40 2024-04-17T19:48:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.265+0000 7f9331e00640 1 --2- 172.21.15.29:0/1895655611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f933406cc40 0x7f933406bee0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f93240097c0 tx=0x7f932401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41bc0c163a8c0a62 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.265+0000 7f9331400640 1 -- 172.21.15.29:0/1895655611 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93240b5070 con 0x7f933406cc40 2024-04-17T19:48:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.265+0000 7f9331400640 1 -- 172.21.15.29:0/1895655611 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93240a8990 con 0x7f933406cc40 2024-04-17T19:48:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.265+0000 7f9331400640 1 -- 172.21.15.29:0/1895655611 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93240a8d60 con 0x7f933406cc40 2024-04-17T19:48:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.269+0000 7f9339200640 1 -- 172.21.15.29:0/1895655611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f933406cc40 msgr2=0x7f933406bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.269+0000 7f9339200640 1 --2- 172.21.15.29:0/1895655611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f933406cc40 0x7f933406bee0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f93240097c0 tx=0x7f932401ec90 comp rx=0 tx=0).stop 2024-04-17T19:48:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.269+0000 7f9339200640 1 -- 172.21.15.29:0/1895655611 shutdown_connections 2024-04-17T19:48:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.269+0000 7f9339200640 1 --2- 172.21.15.29:0/1895655611 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f933415cf10 0x7f933406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.269+0000 7f9339200640 1 --2- 172.21.15.29:0/1895655611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f933406cc40 0x7f933406bee0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.269+0000 7f9339200640 1 -- 172.21.15.29:0/1895655611 >> 172.21.15.29:0/1895655611 conn(0x7f93340cace0 msgr2=0x7f93340cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.269+0000 7f9339200640 1 -- 172.21.15.29:0/1895655611 shutdown_connections 2024-04-17T19:48:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.269+0000 7f9339200640 1 -- 172.21.15.29:0/1895655611 wait complete. 2024-04-17T19:48:33.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.269+0000 7f9339200640 1 Processor -- start 2024-04-17T19:48:33.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.269+0000 7f9339200640 1 -- start start 2024-04-17T19:48:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.269+0000 7f9339200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f933406cc40 0x7f93340c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.269+0000 7f9339200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93340c1970 0x7f93340c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.269+0000 7f9339200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f933415cf10 0x7f9334154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.269+0000 7f9339200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9334160590 con 0x7f93340c1970 2024-04-17T19:48:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.269+0000 7f9339200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f93341602b0 con 0x7f933415cf10 2024-04-17T19:48:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.269+0000 7f9339200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f93340c1420 con 0x7f933406cc40 2024-04-17T19:48:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.273+0000 7f932be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93340c1970 0x7f93340c7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.273+0000 7f932be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93340c1970 0x7f93340c7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33912/0 (socket says 172.21.15.29:33912) 2024-04-17T19:48:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.273+0000 7f932be00640 1 -- 172.21.15.29:0/208108542 learned_addr learned my addr 172.21.15.29:0/208108542 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.273+0000 7f9331e00640 1 --2- 172.21.15.29:0/208108542 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f933406cc40 0x7f93340c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.273+0000 7f9332800640 1 --2- 172.21.15.29:0/208108542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f933415cf10 0x7f9334154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.273+0000 7f932be00640 1 -- 172.21.15.29:0/208108542 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f933406cc40 msgr2=0x7f93340c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.273+0000 7f932be00640 1 --2- 172.21.15.29:0/208108542 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f933406cc40 0x7f93340c88d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.273+0000 7f932be00640 1 -- 172.21.15.29:0/208108542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f933415cf10 msgr2=0x7f9334154b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.273+0000 7f932be00640 1 --2- 172.21.15.29:0/208108542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f933415cf10 0x7f9334154b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.273+0000 7f932be00640 1 -- 172.21.15.29:0/208108542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93341f25e0 con 0x7f93340c1970 2024-04-17T19:48:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.273+0000 7f932be00640 1 --2- 172.21.15.29:0/208108542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93340c1970 0x7f93340c7ad0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f931c00c4b0 tx=0x7f931c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:33.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.273+0000 7f932a000640 1 -- 172.21.15.29:0/208108542 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c018070 con 0x7f93340c1970 2024-04-17T19:48:33.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.273+0000 7f932a000640 1 -- 172.21.15.29:0/208108542 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f931c007500 con 0x7f93340c1970 2024-04-17T19:48:33.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.273+0000 7f9339200640 1 -- 172.21.15.29:0/208108542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93341f28a0 con 0x7f93340c1970 2024-04-17T19:48:33.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.273+0000 7f932a000640 1 -- 172.21.15.29:0/208108542 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c004a30 con 0x7f93340c1970 2024-04-17T19:48:33.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.273+0000 7f9339200640 1 -- 172.21.15.29:0/208108542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93341f2e60 con 0x7f93340c1970 2024-04-17T19:48:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.273+0000 7f9339200640 1 -- 172.21.15.29:0/208108542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9334069b90 con 0x7f93340c1970 2024-04-17T19:48:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.273+0000 7f932a000640 1 -- 172.21.15.29:0/208108542 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f931c01b070 con 0x7f93340c1970 2024-04-17T19:48:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.277+0000 7f932a000640 1 --2- 172.21.15.29:0/208108542 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f93100683b0 0x7f931006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:33.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.277+0000 7f932a000640 1 -- 172.21.15.29:0/208108542 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f931c086060 con 0x7f93340c1970 2024-04-17T19:48:33.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.277+0000 7f9331e00640 1 --2- 172.21.15.29:0/208108542 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f93100683b0 0x7f931006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:33.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.277+0000 7f9331e00640 1 --2- 172.21.15.29:0/208108542 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f93100683b0 0x7f931006a8c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f93240b2000 tx=0x7f93240aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.285+0000 7f932a000640 1 -- 172.21.15.29:0/208108542 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f931c055b30 con 0x7f93340c1970 2024-04-17T19:48:33.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.449+0000 7f9339200640 1 -- 172.21.15.29:0/208108542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f933406ca20 con 0x7f93340c1970 2024-04-17T19:48:33.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.449+0000 7f932a000640 1 -- 172.21.15.29:0/208108542 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f931c059e30 con 0x7f93340c1970 2024-04-17T19:48:33.438 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-04-17T19:48:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.453+0000 7f9339200640 1 -- 172.21.15.29:0/208108542 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f93100683b0 msgr2=0x7f931006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.453+0000 7f9339200640 1 --2- 172.21.15.29:0/208108542 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f93100683b0 0x7f931006a8c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f93240b2000 tx=0x7f93240aa070 comp rx=0 tx=0).stop 2024-04-17T19:48:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.453+0000 7f9339200640 1 -- 172.21.15.29:0/208108542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93340c1970 msgr2=0x7f93340c7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.453+0000 7f9339200640 1 --2- 172.21.15.29:0/208108542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93340c1970 0x7f93340c7ad0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f931c00c4b0 tx=0x7f931c00c970 comp rx=0 tx=0).stop 2024-04-17T19:48:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.453+0000 7f9339200640 1 -- 172.21.15.29:0/208108542 shutdown_connections 2024-04-17T19:48:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.453+0000 7f9339200640 1 --2- 172.21.15.29:0/208108542 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f93100683b0 0x7f931006a8c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.453+0000 7f9339200640 1 --2- 172.21.15.29:0/208108542 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f933415cf10 0x7f9334154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.453+0000 7f9339200640 1 --2- 172.21.15.29:0/208108542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93340c1970 0x7f93340c7ad0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.453+0000 7f9339200640 1 --2- 172.21.15.29:0/208108542 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f933406cc40 0x7f93340c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.453+0000 7f9339200640 1 -- 172.21.15.29:0/208108542 >> 172.21.15.29:0/208108542 conn(0x7f93340cace0 msgr2=0x7f9334152e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.453+0000 7f9339200640 1 -- 172.21.15.29:0/208108542 shutdown_connections 2024-04-17T19:48:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.453+0000 7f9339200640 1 -- 172.21.15.29:0/208108542 wait complete. 2024-04-17T19:48:33.460 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T19:48:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.625+0000 7fb2c1200640 1 Processor -- start 2024-04-17T19:48:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.625+0000 7fb2c1200640 1 -- start start 2024-04-17T19:48:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.625+0000 7fb2c1200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb2bc06ccb0 0x7fb2bc152f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.625+0000 7fb2c1200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb2bc06bf50 0x7fb2bc06c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.629+0000 7fb2c1200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2bc15cc30 con 0x7fb2bc0c9080 2024-04-17T19:48:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.629+0000 7fb2c1200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fb2bc15ff10 con 0x7fb2bc06bf50 2024-04-17T19:48:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.629+0000 7fb2c1200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb2bc15cef0 con 0x7fb2bc06ccb0 2024-04-17T19:48:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.629+0000 7fb2ba800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb2bc06bf50 0x7fb2bc06c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.629+0000 7fb2ba800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb2bc06bf50 0x7fb2bc06c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33922/0 (socket says 172.21.15.29:33922) 2024-04-17T19:48:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.629+0000 7fb2ba800640 1 -- 172.21.15.29:0/3625199450 learned_addr learned my addr 172.21.15.29:0/3625199450 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.629+0000 7fb2b9e00640 1 --2- 172.21.15.29:0/3625199450 >> v2:172.21.15.37:3300/0 conn(0x7fb2bc06ccb0 0x7fb2bc152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.629+0000 7fb2ba800640 1 -- 172.21.15.29:0/3625199450 >> v2:172.21.15.37:3300/0 conn(0x7fb2bc06ccb0 msgr2=0x7fb2bc152f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.629+0000 7fb2ba800640 1 --2- 172.21.15.29:0/3625199450 >> v2:172.21.15.37:3300/0 conn(0x7fb2bc06ccb0 0x7fb2bc152f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.629+0000 7fb2ba800640 1 -- 172.21.15.29:0/3625199450 >> v1:172.21.15.37:6789/0 conn(0x7fb2bc0c9080 legacy=0x7fb2bc15c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.629+0000 7fb2ba800640 1 -- 172.21.15.29:0/3625199450 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2bc15f8c0 con 0x7fb2bc06bf50 2024-04-17T19:48:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.629+0000 7fb2ba800640 1 --2- 172.21.15.29:0/3625199450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2bc06bf50 0x7fb2bc06c360 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc0723d0 tx=0x7fb2b401f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc683dd2cdc7a671 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.629+0000 7fb2b9400640 1 -- 172.21.15.29:0/3625199450 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b40b4070 con 0x7fb2bc06bf50 2024-04-17T19:48:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.633+0000 7fb2b9400640 1 -- 172.21.15.29:0/3625199450 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2b40af400 con 0x7fb2bc06bf50 2024-04-17T19:48:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.633+0000 7fb2b9400640 1 -- 172.21.15.29:0/3625199450 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b40af7d0 con 0x7fb2bc06bf50 2024-04-17T19:48:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.633+0000 7fb2c1200640 1 -- 172.21.15.29:0/3625199450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2bc06bf50 msgr2=0x7fb2bc06c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.633+0000 7fb2c1200640 1 --2- 172.21.15.29:0/3625199450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2bc06bf50 0x7fb2bc06c360 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc0723d0 tx=0x7fb2b401f1a0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.633+0000 7fb2c1200640 1 -- 172.21.15.29:0/3625199450 shutdown_connections 2024-04-17T19:48:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.633+0000 7fb2c1200640 1 --2- 172.21.15.29:0/3625199450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2bc06bf50 0x7fb2bc06c360 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.633+0000 7fb2c1200640 1 --2- 172.21.15.29:0/3625199450 >> v2:172.21.15.37:3300/0 conn(0x7fb2bc06ccb0 0x7fb2bc152f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.633+0000 7fb2c1200640 1 -- 172.21.15.29:0/3625199450 >> 172.21.15.29:0/3625199450 conn(0x7fb2bc0ce250 msgr2=0x7fb2bc0ce660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.633+0000 7fb2c1200640 1 -- 172.21.15.29:0/3625199450 shutdown_connections 2024-04-17T19:48:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.633+0000 7fb2c1200640 1 -- 172.21.15.29:0/3625199450 wait complete. 2024-04-17T19:48:33.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.637+0000 7fb2c1200640 1 Processor -- start 2024-04-17T19:48:33.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.637+0000 7fb2c1200640 1 -- start start 2024-04-17T19:48:33.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.637+0000 7fb2c1200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2bc06bf50 0x7fb2bc1f2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:33.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.637+0000 7fb2c1200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb2bc06ccb0 0x7fb2bc15ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:33.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.637+0000 7fb2c1200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2bc0c9080 0x7fb2bc0cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:33.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.637+0000 7fb2c1200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb2bc15c840 con 0x7fb2bc06bf50 2024-04-17T19:48:33.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.637+0000 7fb2c1200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb2bc0c9490 con 0x7fb2bc0c9080 2024-04-17T19:48:33.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.637+0000 7fb2c1200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb2bc15c9c0 con 0x7fb2bc06ccb0 2024-04-17T19:48:33.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.637+0000 7fb2b9e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2bc06bf50 0x7fb2bc1f2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:33.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.637+0000 7fb2b9e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2bc06bf50 0x7fb2bc1f2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33934/0 (socket says 172.21.15.29:33934) 2024-04-17T19:48:33.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.637+0000 7fb2b9e00640 1 -- 172.21.15.29:0/1258154750 learned_addr learned my addr 172.21.15.29:0/1258154750 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:33.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.637+0000 7fb2ba800640 1 --2- 172.21.15.29:0/1258154750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2bc0c9080 0x7fb2bc0cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:33.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.637+0000 7fb2b3e00640 1 --2- 172.21.15.29:0/1258154750 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb2bc06ccb0 0x7fb2bc15ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.637+0000 7fb2b9e00640 1 -- 172.21.15.29:0/1258154750 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb2bc06ccb0 msgr2=0x7fb2bc15ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.637+0000 7fb2b9e00640 1 --2- 172.21.15.29:0/1258154750 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb2bc06ccb0 0x7fb2bc15ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.637+0000 7fb2b9e00640 1 -- 172.21.15.29:0/1258154750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2bc0c9080 msgr2=0x7fb2bc0cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.637+0000 7fb2b9e00640 1 --2- 172.21.15.29:0/1258154750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2bc0c9080 0x7fb2bc0cd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.641+0000 7fb2b9e00640 1 -- 172.21.15.29:0/1258154750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2bc0c6dc0 con 0x7fb2bc06bf50 2024-04-17T19:48:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.641+0000 7fb2ba800640 1 --2- 172.21.15.29:0/1258154750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2bc0c9080 0x7fb2bc0cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.641+0000 7fb2b9e00640 1 --2- 172.21.15.29:0/1258154750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2bc06bf50 0x7fb2bc1f2730 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac00e8a0 tx=0x7fb2ac00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.641+0000 7fb2b2000640 1 -- 172.21.15.29:0/1258154750 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac017070 con 0x7fb2bc06bf50 2024-04-17T19:48:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.641+0000 7fb2b2000640 1 -- 172.21.15.29:0/1258154750 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2ac00c940 con 0x7fb2bc06bf50 2024-04-17T19:48:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.641+0000 7fb2b2000640 1 -- 172.21.15.29:0/1258154750 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac00ccf0 con 0x7fb2bc06bf50 2024-04-17T19:48:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.641+0000 7fb2c1200640 1 -- 172.21.15.29:0/1258154750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2bc14d3d0 con 0x7fb2bc06bf50 2024-04-17T19:48:33.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.641+0000 7fb2c1200640 1 -- 172.21.15.29:0/1258154750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2bc14d990 con 0x7fb2bc06bf50 2024-04-17T19:48:33.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.641+0000 7fb2c1200640 1 -- 172.21.15.29:0/1258154750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2bc069b90 con 0x7fb2bc06bf50 2024-04-17T19:48:33.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.641+0000 7fb2b2000640 1 -- 172.21.15.29:0/1258154750 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb2ac01a070 con 0x7fb2bc06bf50 2024-04-17T19:48:33.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.645+0000 7fb2b2000640 1 --2- 172.21.15.29:0/1258154750 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb298068380 0x7fb29806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:33.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.645+0000 7fb2b2000640 1 -- 172.21.15.29:0/1258154750 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fb2ac087040 con 0x7fb2bc06bf50 2024-04-17T19:48:33.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.645+0000 7fb2b3e00640 1 --2- 172.21.15.29:0/1258154750 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb298068380 0x7fb29806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:33.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.645+0000 7fb2b3e00640 1 --2- 172.21.15.29:0/1258154750 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb298068380 0x7fb29806a890 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc1e60f0 tx=0x7fb2a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:33.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.649+0000 7fb2b2000640 1 -- 172.21.15.29:0/1258154750 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb2ac057200 con 0x7fb2bc06bf50 2024-04-17T19:48:33.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.809+0000 7fb2c1200640 1 -- 172.21.15.29:0/1258154750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb2bc06ca90 con 0x7fb2bc06bf50 2024-04-17T19:48:33.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.809+0000 7fb2b2000640 1 -- 172.21.15.29:0/1258154750 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fb2ac056b00 con 0x7fb2bc06bf50 2024-04-17T19:48:33.799 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-04-17T19:48:33.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.813+0000 7fb2c1200640 1 -- 172.21.15.29:0/1258154750 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb298068380 msgr2=0x7fb29806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:33.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.813+0000 7fb2c1200640 1 --2- 172.21.15.29:0/1258154750 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb298068380 0x7fb29806a890 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc1e60f0 tx=0x7fb2a4008040 comp rx=0 tx=0).stop 2024-04-17T19:48:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.813+0000 7fb2c1200640 1 -- 172.21.15.29:0/1258154750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2bc06bf50 msgr2=0x7fb2bc1f2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.813+0000 7fb2c1200640 1 --2- 172.21.15.29:0/1258154750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2bc06bf50 0x7fb2bc1f2730 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac00e8a0 tx=0x7fb2ac00eca0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.813+0000 7fb2c1200640 1 -- 172.21.15.29:0/1258154750 shutdown_connections 2024-04-17T19:48:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.813+0000 7fb2c1200640 1 --2- 172.21.15.29:0/1258154750 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb298068380 0x7fb29806a890 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.813+0000 7fb2c1200640 1 --2- 172.21.15.29:0/1258154750 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb2bc0c9080 0x7fb2bc0cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.813+0000 7fb2c1200640 1 --2- 172.21.15.29:0/1258154750 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb2bc06ccb0 0x7fb2bc15ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.813+0000 7fb2c1200640 1 --2- 172.21.15.29:0/1258154750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2bc06bf50 0x7fb2bc1f2730 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.813+0000 7fb2c1200640 1 -- 172.21.15.29:0/1258154750 >> 172.21.15.29:0/1258154750 conn(0x7fb2bc0ce250 msgr2=0x7fb2bc150d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.813+0000 7fb2c1200640 1 -- 172.21.15.29:0/1258154750 shutdown_connections 2024-04-17T19:48:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.813+0000 7fb2c1200640 1 -- 172.21.15.29:0/1258154750 wait complete. 2024-04-17T19:48:33.822 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T19:48:33.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.993+0000 7f0873000640 1 Processor -- start 2024-04-17T19:48:33.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.997+0000 7f0873000640 1 -- start start 2024-04-17T19:48:33.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.997+0000 7f0873000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f086c15a7f0 0x7f086c0c4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:33.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.997+0000 7f0873000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f086c153c10 0x7f086c154020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:33.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.997+0000 7f0873000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f086c0c84c0 con 0x7f086c0ca0d0 2024-04-17T19:48:33.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.997+0000 7f0873000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f086c154560 con 0x7f086c153c10 2024-04-17T19:48:33.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.997+0000 7f0873000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f086c0c87a0 con 0x7f086c15a7f0 2024-04-17T19:48:33.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.997+0000 7f086b400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f086c153c10 0x7f086c154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:33.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.997+0000 7f086b400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f086c153c10 0x7f086c154020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33944/0 (socket says 172.21.15.29:33944) 2024-04-17T19:48:33.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.997+0000 7f086b400640 1 -- 172.21.15.29:0/371056205 learned_addr learned my addr 172.21.15.29:0/371056205 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:33.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.997+0000 7f086be00640 1 --2- 172.21.15.29:0/371056205 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f086c15a7f0 0x7f086c0c4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:33.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.997+0000 7f086b400640 1 -- 172.21.15.29:0/371056205 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f086c15a7f0 msgr2=0x7f086c0c4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:33.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.997+0000 7f086b400640 1 --2- 172.21.15.29:0/371056205 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f086c15a7f0 0x7f086c0c4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.997+0000 7f086b400640 1 -- 172.21.15.29:0/371056205 >> v1:172.21.15.37:6789/0 conn(0x7f086c0ca0d0 legacy=0x7f086c0ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:33.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.997+0000 7f086b400640 1 -- 172.21.15.29:0/371056205 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f086c15d390 con 0x7f086c153c10 2024-04-17T19:48:33.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:33.997+0000 7f086be00640 1 --2- 172.21.15.29:0/371056205 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f086c15a7f0 0x7f086c0c4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:33.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.001+0000 7f086b400640 1 --2- 172.21.15.29:0/371056205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f086c153c10 0x7f086c154020 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0858009870 tx=0x7f085801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f726593610c9b84 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:33.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.001+0000 7f086aa00640 1 -- 172.21.15.29:0/371056205 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08580ab650 con 0x7f086c153c10 2024-04-17T19:48:33.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.001+0000 7f086aa00640 1 -- 172.21.15.29:0/371056205 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08580b0070 con 0x7f086c153c10 2024-04-17T19:48:33.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.001+0000 7f086aa00640 1 -- 172.21.15.29:0/371056205 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08580aba10 con 0x7f086c153c10 2024-04-17T19:48:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.001+0000 7f0873000640 1 -- 172.21.15.29:0/371056205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f086c153c10 msgr2=0x7f086c154020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.001+0000 7f0873000640 1 --2- 172.21.15.29:0/371056205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f086c153c10 0x7f086c154020 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0858009870 tx=0x7f085801c560 comp rx=0 tx=0).stop 2024-04-17T19:48:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.001+0000 7f0873000640 1 -- 172.21.15.29:0/371056205 shutdown_connections 2024-04-17T19:48:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.001+0000 7f0873000640 1 --2- 172.21.15.29:0/371056205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f086c153c10 0x7f086c154020 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.001+0000 7f0873000640 1 --2- 172.21.15.29:0/371056205 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f086c15a7f0 0x7f086c0c4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.001+0000 7f0873000640 1 -- 172.21.15.29:0/371056205 >> 172.21.15.29:0/371056205 conn(0x7f086c06bf90 msgr2=0x7f086c06c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.001+0000 7f0873000640 1 -- 172.21.15.29:0/371056205 shutdown_connections 2024-04-17T19:48:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.001+0000 7f0873000640 1 -- 172.21.15.29:0/371056205 wait complete. 2024-04-17T19:48:33.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f0873000640 1 Processor -- start 2024-04-17T19:48:33.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f0873000640 1 -- start start 2024-04-17T19:48:33.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f0873000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f086c0ca0d0 0x7f086c15b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:33.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f0873000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f086c153c10 0x7f086c15c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:33.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f0873000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f086c15a7f0 0x7f086c1637d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:33.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f0873000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f086c15e150 con 0x7f086c153c10 2024-04-17T19:48:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f0873000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f086c15ac00 con 0x7f086c15a7f0 2024-04-17T19:48:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f0873000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f086c0c4f40 con 0x7f086c0ca0d0 2024-04-17T19:48:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f086b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f086c153c10 0x7f086c15c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f086b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f086c153c10 0x7f086c15c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33946/0 (socket says 172.21.15.29:33946) 2024-04-17T19:48:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f086b400640 1 -- 172.21.15.29:0/3239741319 learned_addr learned my addr 172.21.15.29:0/3239741319 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f086be00640 1 --2- 172.21.15.29:0/3239741319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f086c0ca0d0 0x7f086c15b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f0871000640 1 --2- 172.21.15.29:0/3239741319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f086c15a7f0 0x7f086c1637d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f086b400640 1 -- 172.21.15.29:0/3239741319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f086c0ca0d0 msgr2=0x7f086c15b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f086b400640 1 --2- 172.21.15.29:0/3239741319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f086c0ca0d0 0x7f086c15b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f086b400640 1 -- 172.21.15.29:0/3239741319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f086c15a7f0 msgr2=0x7f086c1637d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f086b400640 1 --2- 172.21.15.29:0/3239741319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f086c15a7f0 0x7f086c1637d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f086b400640 1 -- 172.21.15.29:0/3239741319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f086c1f5060 con 0x7f086c153c10 2024-04-17T19:48:33.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f0871000640 1 --2- 172.21.15.29:0/3239741319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f086c15a7f0 0x7f086c1637d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:33.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f086b400640 1 --2- 172.21.15.29:0/3239741319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f086c153c10 0x7f086c15c2c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f08580ad040 tx=0x7f085801ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:33.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f0863e00640 1 -- 172.21.15.29:0/3239741319 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08580be5f0 con 0x7f086c153c10 2024-04-17T19:48:33.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f0863e00640 1 -- 172.21.15.29:0/3239741319 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08580be7b0 con 0x7f086c153c10 2024-04-17T19:48:33.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f0863e00640 1 -- 172.21.15.29:0/3239741319 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08580beba0 con 0x7f086c153c10 2024-04-17T19:48:33.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f0873000640 1 -- 172.21.15.29:0/3239741319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f086c145760 con 0x7f086c153c10 2024-04-17T19:48:33.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.005+0000 7f0873000640 1 -- 172.21.15.29:0/3239741319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f086c146540 con 0x7f086c153c10 2024-04-17T19:48:33.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.009+0000 7f0873000640 1 -- 172.21.15.29:0/3239741319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f086c15af60 con 0x7f086c153c10 2024-04-17T19:48:33.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.009+0000 7f0863e00640 1 -- 172.21.15.29:0/3239741319 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f08580b43f0 con 0x7f086c153c10 2024-04-17T19:48:33.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.009+0000 7f0863e00640 1 --2- 172.21.15.29:0/3239741319 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f084c068380 0x7f084c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:33.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.009+0000 7f0863e00640 1 -- 172.21.15.29:0/3239741319 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f0858121300 con 0x7f086c153c10 2024-04-17T19:48:33.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.009+0000 7f086be00640 1 --2- 172.21.15.29:0/3239741319 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f084c068380 0x7f084c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:33.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.009+0000 7f086be00640 1 --2- 172.21.15.29:0/3239741319 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f084c068380 0x7f084c06a890 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f085c0099e0 tx=0x7f085c003cf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:34.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.013+0000 7f0863e00640 1 -- 172.21.15.29:0/3239741319 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0858016150 con 0x7f086c153c10 2024-04-17T19:48:34.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.177+0000 7f0873000640 1 -- 172.21.15.29:0/3239741319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f086c146780 con 0x7f086c153c10 2024-04-17T19:48:34.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.177+0000 7f0863e00640 1 -- 172.21.15.29:0/3239741319 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f08580f0dd0 con 0x7f086c153c10 2024-04-17T19:48:34.166 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-04-17T19:48:34.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.181+0000 7f0873000640 1 -- 172.21.15.29:0/3239741319 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f084c068380 msgr2=0x7f084c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:34.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.181+0000 7f0873000640 1 --2- 172.21.15.29:0/3239741319 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f084c068380 0x7f084c06a890 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f085c0099e0 tx=0x7f085c003cf0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.181+0000 7f0873000640 1 -- 172.21.15.29:0/3239741319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f086c153c10 msgr2=0x7f086c15c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:34.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.181+0000 7f0873000640 1 --2- 172.21.15.29:0/3239741319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f086c153c10 0x7f086c15c2c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f08580ad040 tx=0x7f085801ada0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.181+0000 7f0873000640 1 -- 172.21.15.29:0/3239741319 shutdown_connections 2024-04-17T19:48:34.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.181+0000 7f0873000640 1 --2- 172.21.15.29:0/3239741319 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f084c068380 0x7f084c06a890 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.181+0000 7f0873000640 1 --2- 172.21.15.29:0/3239741319 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f086c15a7f0 0x7f086c1637d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.181+0000 7f0873000640 1 --2- 172.21.15.29:0/3239741319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f086c153c10 0x7f086c15c2c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.181+0000 7f0873000640 1 --2- 172.21.15.29:0/3239741319 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f086c0ca0d0 0x7f086c15b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.181+0000 7f0873000640 1 -- 172.21.15.29:0/3239741319 >> 172.21.15.29:0/3239741319 conn(0x7f086c06bf90 msgr2=0x7f086c152650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:34.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.181+0000 7f0873000640 1 -- 172.21.15.29:0/3239741319 shutdown_connections 2024-04-17T19:48:34.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.181+0000 7f0873000640 1 -- 172.21.15.29:0/3239741319 wait complete. 2024-04-17T19:48:34.188 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:48:34.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.349+0000 7fdf3f200640 1 Processor -- start 2024-04-17T19:48:34.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.349+0000 7fdf3f200640 1 -- start start 2024-04-17T19:48:34.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.353+0000 7fdf3f200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fdf380cc8c0 0x7fdf380cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:34.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.353+0000 7fdf3f200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf381597a0 0x7fdf381635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:34.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.353+0000 7fdf3f200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf380c88b0 con 0x7fdf38159390 2024-04-17T19:48:34.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.353+0000 7fdf3f200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fdf380c6ce0 con 0x7fdf381597a0 2024-04-17T19:48:34.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.353+0000 7fdf3f200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fdf380c6fc0 con 0x7fdf380cc8c0 2024-04-17T19:48:34.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.353+0000 7fdf3d200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf381597a0 0x7fdf381635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:34.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.353+0000 7fdf2fe00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdf38159390 0x7fdf380ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48224/0 (socket says 172.21.15.29:48224) 2024-04-17T19:48:34.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.353+0000 7fdf2fe00640 1 -- 172.21.15.29:0/1709140354 learned_addr learned my addr 172.21.15.29:0/1709140354 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:34.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.353+0000 7fdf37e00640 1 --2- 172.21.15.29:0/1709140354 >> v2:172.21.15.37:3300/0 conn(0x7fdf380cc8c0 0x7fdf380cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:34.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.353+0000 7fdf3d200640 1 -- 172.21.15.29:0/1709140354 >> v2:172.21.15.37:3300/0 conn(0x7fdf380cc8c0 msgr2=0x7fdf380cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:34.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.353+0000 7fdf3d200640 1 --2- 172.21.15.29:0/1709140354 >> v2:172.21.15.37:3300/0 conn(0x7fdf380cc8c0 0x7fdf380cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.353+0000 7fdf3d200640 1 -- 172.21.15.29:0/1709140354 >> v1:172.21.15.29:6789/0 conn(0x7fdf38159390 legacy=0x7fdf380ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:34.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.353+0000 7fdf3d200640 1 -- 172.21.15.29:0/1709140354 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf380c72a0 con 0x7fdf381597a0 2024-04-17T19:48:34.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.353+0000 7fdf37400640 1 -- 172.21.15.29:0/1709140354 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 6628004 0 0) 0x7fdf380c88b0 con 0x7fdf38159390 2024-04-17T19:48:34.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.357+0000 7fdf3d200640 1 --2- 172.21.15.29:0/1709140354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf381597a0 0x7fdf381635e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdf380721b0 tx=0x7fdf3001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0802f1add2bd52a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:34.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.357+0000 7fdf37400640 1 -- 172.21.15.29:0/1709140354 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf300b2070 con 0x7fdf381597a0 2024-04-17T19:48:34.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.357+0000 7fdf37400640 1 -- 172.21.15.29:0/1709140354 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf300a57f0 con 0x7fdf381597a0 2024-04-17T19:48:34.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.357+0000 7fdf37400640 1 -- 172.21.15.29:0/1709140354 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf300a5bc0 con 0x7fdf381597a0 2024-04-17T19:48:34.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.357+0000 7fdf3f200640 1 -- 172.21.15.29:0/1709140354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf381597a0 msgr2=0x7fdf381635e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:34.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.357+0000 7fdf3f200640 1 --2- 172.21.15.29:0/1709140354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf381597a0 0x7fdf381635e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdf380721b0 tx=0x7fdf3001c930 comp rx=0 tx=0).stop 2024-04-17T19:48:34.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.357+0000 7fdf3f200640 1 -- 172.21.15.29:0/1709140354 shutdown_connections 2024-04-17T19:48:34.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.357+0000 7fdf3f200640 1 --2- 172.21.15.29:0/1709140354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf381597a0 0x7fdf381635e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.357+0000 7fdf3f200640 1 --2- 172.21.15.29:0/1709140354 >> v2:172.21.15.37:3300/0 conn(0x7fdf380cc8c0 0x7fdf380cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.357+0000 7fdf3f200640 1 -- 172.21.15.29:0/1709140354 >> 172.21.15.29:0/1709140354 conn(0x7fdf3806fde0 msgr2=0x7fdf380701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:34.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.357+0000 7fdf3f200640 1 -- 172.21.15.29:0/1709140354 shutdown_connections 2024-04-17T19:48:34.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.357+0000 7fdf3f200640 1 -- 172.21.15.29:0/1709140354 wait complete. 2024-04-17T19:48:34.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3f200640 1 Processor -- start 2024-04-17T19:48:34.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3f200640 1 -- start start 2024-04-17T19:48:34.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3f200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf380cc8c0 0x7fdf38073e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:34.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3f200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf38159390 0x7fdf38073840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:34.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3f200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf381597a0 0x7fdf381f5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:34.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3f200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdf380cabc0 con 0x7fdf381597a0 2024-04-17T19:48:34.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3f200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fdf380caa40 con 0x7fdf38159390 2024-04-17T19:48:34.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3f200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fdf380c6880 con 0x7fdf380cc8c0 2024-04-17T19:48:34.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3d200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf381597a0 0x7fdf381f5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:34.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3d200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf381597a0 0x7fdf381f5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33968/0 (socket says 172.21.15.29:33968) 2024-04-17T19:48:34.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3d200640 1 -- 172.21.15.29:0/750529444 learned_addr learned my addr 172.21.15.29:0/750529444 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:34.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf2fe00640 1 --2- 172.21.15.29:0/750529444 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf38159390 0x7fdf38073840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:34.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf37e00640 1 --2- 172.21.15.29:0/750529444 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf380cc8c0 0x7fdf38073e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:34.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3d200640 1 -- 172.21.15.29:0/750529444 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf380cc8c0 msgr2=0x7fdf38073e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:34.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3d200640 1 --2- 172.21.15.29:0/750529444 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf380cc8c0 0x7fdf38073e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3d200640 1 -- 172.21.15.29:0/750529444 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf38159390 msgr2=0x7fdf38073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:34.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3d200640 1 --2- 172.21.15.29:0/750529444 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf38159390 0x7fdf38073840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3d200640 1 -- 172.21.15.29:0/750529444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf381f5c90 con 0x7fdf381597a0 2024-04-17T19:48:34.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3d200640 1 --2- 172.21.15.29:0/750529444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf381597a0 0x7fdf381f5750 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdf3001ce10 tx=0x7fdf300196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:34.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf34c00640 1 -- 172.21.15.29:0/750529444 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf300b2040 con 0x7fdf381597a0 2024-04-17T19:48:34.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf34c00640 1 -- 172.21.15.29:0/750529444 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf300a55f0 con 0x7fdf381597a0 2024-04-17T19:48:34.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf34c00640 1 -- 172.21.15.29:0/750529444 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf300b8670 con 0x7fdf381597a0 2024-04-17T19:48:34.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3f200640 1 -- 172.21.15.29:0/750529444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf381f8c50 con 0x7fdf381597a0 2024-04-17T19:48:34.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.361+0000 7fdf3f200640 1 -- 172.21.15.29:0/750529444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf381f9210 con 0x7fdf381597a0 2024-04-17T19:48:34.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.365+0000 7fdf3f200640 1 -- 172.21.15.29:0/750529444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf38069b90 con 0x7fdf381597a0 2024-04-17T19:48:34.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.365+0000 7fdf34c00640 1 -- 172.21.15.29:0/750529444 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdf300c1020 con 0x7fdf381597a0 2024-04-17T19:48:34.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.365+0000 7fdf34c00640 1 --2- 172.21.15.29:0/750529444 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fdf140683b0 0x7fdf1406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:34.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.365+0000 7fdf34c00640 1 -- 172.21.15.29:0/750529444 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fdf301217c0 con 0x7fdf381597a0 2024-04-17T19:48:34.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.365+0000 7fdf37e00640 1 --2- 172.21.15.29:0/750529444 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fdf140683b0 0x7fdf1406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:34.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.365+0000 7fdf37e00640 1 --2- 172.21.15.29:0/750529444 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fdf140683b0 0x7fdf1406a8c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009a50 tx=0x7fdf28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:34.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.369+0000 7fdf34c00640 1 -- 172.21.15.29:0/750529444 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdf30016520 con 0x7fdf381597a0 2024-04-17T19:48:34.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.569+0000 7fdf3f200640 1 -- 172.21.15.29:0/750529444 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf3806c860 con 0x7fdf381597a0 2024-04-17T19:48:34.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.569+0000 7fdf34c00640 1 -- 172.21.15.29:0/750529444 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+6898 (secure 0 0 0) 0x7fdf300f1970 con 0x7fdf381597a0 2024-04-17T19:48:34.560 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:34.560 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:48:28.833454+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4226":{"gid":4226,"name":"b","rank":4,"incarnation":18,"state":"up:creating","state_seq":1,"addr":"172.21.15.37:6838/332907250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":332907250},{"type":"v1","addr":"172.21.15.37:6838","nonce":332907250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229]},"id":1}]} 2024-04-17T19:48:34.560 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 18 2024-04-17T19:48:34.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.573+0000 7fdf3f200640 1 -- 172.21.15.29:0/750529444 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fdf140683b0 msgr2=0x7fdf1406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:34.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.573+0000 7fdf3f200640 1 --2- 172.21.15.29:0/750529444 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fdf140683b0 0x7fdf1406a8c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009a50 tx=0x7fdf28008040 comp rx=0 tx=0).stop 2024-04-17T19:48:34.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.573+0000 7fdf3f200640 1 -- 172.21.15.29:0/750529444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf381597a0 msgr2=0x7fdf381f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:34.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.573+0000 7fdf3f200640 1 --2- 172.21.15.29:0/750529444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf381597a0 0x7fdf381f5750 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdf3001ce10 tx=0x7fdf300196d0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.573+0000 7fdf3f200640 1 -- 172.21.15.29:0/750529444 shutdown_connections 2024-04-17T19:48:34.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.573+0000 7fdf3f200640 1 --2- 172.21.15.29:0/750529444 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fdf140683b0 0x7fdf1406a8c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.573+0000 7fdf3f200640 1 --2- 172.21.15.29:0/750529444 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf381597a0 0x7fdf381f5750 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.573+0000 7fdf3f200640 1 --2- 172.21.15.29:0/750529444 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdf38159390 0x7fdf38073840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.573+0000 7fdf3f200640 1 --2- 172.21.15.29:0/750529444 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdf380cc8c0 0x7fdf38073e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.573+0000 7fdf3f200640 1 -- 172.21.15.29:0/750529444 >> 172.21.15.29:0/750529444 conn(0x7fdf3806fde0 msgr2=0x7fdf380cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:34.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.573+0000 7fdf3f200640 1 -- 172.21.15.29:0/750529444 shutdown_connections 2024-04-17T19:48:34.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.573+0000 7fdf3f200640 1 -- 172.21.15.29:0/750529444 wait complete. 2024-04-17T19:48:34.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:48:34.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.745+0000 7f8d30c00640 1 Processor -- start 2024-04-17T19:48:34.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.745+0000 7f8d30c00640 1 -- start start 2024-04-17T19:48:34.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d30c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d2c0c9b20 0x7f8d2c1560d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:34.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d30c00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d2c0c55e0 con 0x7f8d2c06e300 2024-04-17T19:48:34.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d30c00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d2c0c9f30 con 0x7f8d2c15c540 2024-04-17T19:48:34.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d30c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8d2c0c8fa0 con 0x7f8d2c0c9b20 2024-04-17T19:48:34.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d21400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8d2c06e300 0x7f8d2c15be30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48234/0 (socket says 172.21.15.29:48234) 2024-04-17T19:48:34.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d21400640 1 -- 172.21.15.29:0/1804874109 learned_addr learned my addr 172.21.15.29:0/1804874109 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:34.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29e00640 1 --2- 172.21.15.29:0/1804874109 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d2c0c9b20 0x7f8d2c1560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:34.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29400640 1 -- 172.21.15.29:0/1804874109 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 549267617 0 0) 0x7f8d14001120 con 0x7f8d2c06e300 2024-04-17T19:48:34.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29400640 1 -- 172.21.15.29:0/1804874109 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d10003530 con 0x7f8d2c06e300 2024-04-17T19:48:34.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29400640 1 -- 172.21.15.29:0/1804874109 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1713012564 0 0) 0x7f8d2c0c9f30 con 0x7f8d2c15c540 2024-04-17T19:48:34.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29400640 1 -- 172.21.15.29:0/1804874109 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d14001120 con 0x7f8d2c15c540 2024-04-17T19:48:34.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29400640 1 -- 172.21.15.29:0/1804874109 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1903099360 0 0) 0x7f8d10003530 con 0x7f8d2c06e300 2024-04-17T19:48:34.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29400640 1 -- 172.21.15.29:0/1804874109 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d2c0c9b20 msgr2=0x7f8d2c1560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:34.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29400640 1 --2- 172.21.15.29:0/1804874109 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d2c0c9b20 0x7f8d2c1560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29400640 1 -- 172.21.15.29:0/1804874109 >> v1:172.21.15.37:6789/0 conn(0x7f8d2c15c540 legacy=0x7f8d2c1667b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:34.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29400640 1 -- 172.21.15.29:0/1804874109 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d2c0ca1d0 con 0x7f8d2c06e300 2024-04-17T19:48:34.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29400640 1 -- 172.21.15.29:0/1804874109 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f8d14001850 con 0x7f8d2c06e300 2024-04-17T19:48:34.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29e00640 1 --2- 172.21.15.29:0/1804874109 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d2c0c9b20 0x7f8d2c1560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:48:34.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29400640 1 -- 172.21.15.29:0/1804874109 >> v1:172.21.15.29:6789/0 conn(0x7f8d2c06e300 legacy=0x7f8d2c15be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:34.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29400640 1 --2- 172.21.15.29:0/1804874109 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d10003bf0 0x7f8d10004040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:34.740 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29400640 1 --2- 172.21.15.29:0/1804874109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d10004810 0x7f8d10004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:34.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29400640 1 --2- 172.21.15.29:0/1804874109 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d100051b0 0x7f8d1000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:34.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29400640 1 -- 172.21.15.29:0/1804874109 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1000c040 con 0x7f8d10004810 2024-04-17T19:48:34.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29400640 1 -- 172.21.15.29:0/1804874109 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1000c1f0 con 0x7f8d100051b0 2024-04-17T19:48:34.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29400640 1 -- 172.21.15.29:0/1804874109 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8d1000c3f0 con 0x7f8d10003bf0 2024-04-17T19:48:34.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d21400640 1 --2- 172.21.15.29:0/1804874109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d10004810 0x7f8d10004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:34.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d29e00640 1 --2- 172.21.15.29:0/1804874109 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d10003bf0 0x7f8d10004040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:34.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d2a800640 1 --2- 172.21.15.29:0/1804874109 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d100051b0 0x7f8d1000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:34.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d21400640 1 -- 172.21.15.29:0/1804874109 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d10003bf0 msgr2=0x7f8d10004040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:34.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d21400640 1 --2- 172.21.15.29:0/1804874109 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d10003bf0 0x7f8d10004040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d21400640 1 -- 172.21.15.29:0/1804874109 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d100051b0 msgr2=0x7f8d1000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:34.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d21400640 1 --2- 172.21.15.29:0/1804874109 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d100051b0 0x7f8d1000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.749+0000 7f8d21400640 1 -- 172.21.15.29:0/1804874109 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d10003530 con 0x7f8d10004810 2024-04-17T19:48:34.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.753+0000 7f8d21400640 1 --2- 172.21.15.29:0/1804874109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d10004810 0x7f8d10004c20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400c490 tx=0x7f8d1401f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b1db2c809126347 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:34.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.753+0000 7f8d29400640 1 -- 172.21.15.29:0/1804874109 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1400c0e0 con 0x7f8d10004810 2024-04-17T19:48:34.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.753+0000 7f8d29400640 1 -- 172.21.15.29:0/1804874109 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d14005b00 con 0x7f8d10004810 2024-04-17T19:48:34.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.753+0000 7f8d29400640 1 -- 172.21.15.29:0/1804874109 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d14007070 con 0x7f8d10004810 2024-04-17T19:48:34.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.753+0000 7f8d30c00640 1 -- 172.21.15.29:0/1804874109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d10004810 msgr2=0x7f8d10004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:34.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.753+0000 7f8d30c00640 1 --2- 172.21.15.29:0/1804874109 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d10004810 0x7f8d10004c20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400c490 tx=0x7f8d1401f330 comp rx=0 tx=0).stop 2024-04-17T19:48:34.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.753+0000 7f8d2a800640 1 -- 172.21.15.29:0/1804874109 reap_dead start 2024-04-17T19:48:34.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.753+0000 7f8d30c00640 1 -- 172.21.15.29:0/1804874109 shutdown_connections 2024-04-17T19:48:34.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.753+0000 7f8d30c00640 1 -- 172.21.15.29:0/1804874109 >> 172.21.15.29:0/1804874109 conn(0x7f8d2c06c3c0 msgr2=0x7f8d2c0ca3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:34.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.753+0000 7f8d30c00640 1 -- 172.21.15.29:0/1804874109 shutdown_connections 2024-04-17T19:48:34.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.753+0000 7f8d30c00640 1 -- 172.21.15.29:0/1804874109 wait complete. 2024-04-17T19:48:34.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d30c00640 1 Processor -- start 2024-04-17T19:48:34.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d30c00640 1 -- start start 2024-04-17T19:48:34.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d30c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d10004810 0x7f8d2c1f57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d30c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d2c15bc00 0x7f8d2c0c73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d30c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d2c1e73f0 0x7f8d2c151590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d30c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c0c8670 con 0x7f8d2c15bc00 2024-04-17T19:48:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d30c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2c0c8390 con 0x7f8d10004810 2024-04-17T19:48:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d30c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8d2c0c8950 con 0x7f8d2c1e73f0 2024-04-17T19:48:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d21400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d2c15bc00 0x7f8d2c0c73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d21400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d2c15bc00 0x7f8d2c0c73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33982/0 (socket says 172.21.15.29:33982) 2024-04-17T19:48:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d21400640 1 -- 172.21.15.29:0/779925841 learned_addr learned my addr 172.21.15.29:0/779925841 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d2a800640 1 --2- 172.21.15.29:0/779925841 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d2c1e73f0 0x7f8d2c151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d29e00640 1 --2- 172.21.15.29:0/779925841 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d10004810 0x7f8d2c1f57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d21400640 1 -- 172.21.15.29:0/779925841 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d2c1e73f0 msgr2=0x7f8d2c151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d21400640 1 --2- 172.21.15.29:0/779925841 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d2c1e73f0 0x7f8d2c151590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d21400640 1 -- 172.21.15.29:0/779925841 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d10004810 msgr2=0x7f8d2c1f57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d21400640 1 --2- 172.21.15.29:0/779925841 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d10004810 0x7f8d2c1f57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d21400640 1 -- 172.21.15.29:0/779925841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d2c1f21f0 con 0x7f8d2c15bc00 2024-04-17T19:48:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d29e00640 1 --2- 172.21.15.29:0/779925841 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d10004810 0x7f8d2c1f57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T19:48:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d21400640 1 --2- 172.21.15.29:0/779925841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d2c15bc00 0x7f8d2c0c73a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8d1401f160 tx=0x7f8d10005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d22a00640 1 -- 172.21.15.29:0/779925841 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d14005cb0 con 0x7f8d2c15bc00 2024-04-17T19:48:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d22a00640 1 -- 172.21.15.29:0/779925841 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d14005e50 con 0x7f8d2c15bc00 2024-04-17T19:48:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d22a00640 1 -- 172.21.15.29:0/779925841 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d140049f0 con 0x7f8d2c15bc00 2024-04-17T19:48:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d30c00640 1 -- 172.21.15.29:0/779925841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d2c1f2470 con 0x7f8d2c15bc00 2024-04-17T19:48:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.757+0000 7f8d30c00640 1 -- 172.21.15.29:0/779925841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d2c1f2a30 con 0x7f8d2c15bc00 2024-04-17T19:48:34.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.761+0000 7f8d30c00640 1 -- 172.21.15.29:0/779925841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d2c069b90 con 0x7f8d2c15bc00 2024-04-17T19:48:34.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.761+0000 7f8d22a00640 1 -- 172.21.15.29:0/779925841 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8d140ba030 con 0x7f8d2c15bc00 2024-04-17T19:48:34.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.761+0000 7f8d22a00640 1 --2- 172.21.15.29:0/779925841 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8d08068380 0x7f8d0806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:34.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.761+0000 7f8d22a00640 1 -- 172.21.15.29:0/779925841 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8d14123a20 con 0x7f8d2c15bc00 2024-04-17T19:48:34.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.761+0000 7f8d29e00640 1 --2- 172.21.15.29:0/779925841 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8d08068380 0x7f8d0806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:34.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.765+0000 7f8d29e00640 1 --2- 172.21.15.29:0/779925841 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8d08068380 0x7f8d0806a890 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c0090e0 tx=0x7f8d1c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:34.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.765+0000 7f8d22a00640 1 -- 172.21.15.29:0/779925841 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8d2c1f21f0 con 0x7f8d2c15bc00 2024-04-17T19:48:34.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.933+0000 7f8d30c00640 1 -- 172.21.15.29:0/779925841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8d2c1563c0 con 0x7f8d2c15bc00 2024-04-17T19:48:34.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.937+0000 7f8d22a00640 1 -- 172.21.15.29:0/779925841 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18780 (secure 0 0 0) 0x7f8d2c0c55e0 con 0x7f8d2c15bc00 2024-04-17T19:48:34.926 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:34.926 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":26,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:48:29.829433+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T19:47:47.658449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T19:47:48.209047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T19:47:51.887909+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:48:34.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.941+0000 7f8d30c00640 1 -- 172.21.15.29:0/779925841 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8d08068380 msgr2=0x7f8d0806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:34.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.941+0000 7f8d30c00640 1 --2- 172.21.15.29:0/779925841 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8d08068380 0x7f8d0806a890 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c0090e0 tx=0x7f8d1c008040 comp rx=0 tx=0).stop 2024-04-17T19:48:34.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.941+0000 7f8d30c00640 1 -- 172.21.15.29:0/779925841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d2c15bc00 msgr2=0x7f8d2c0c73a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:34.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.941+0000 7f8d30c00640 1 --2- 172.21.15.29:0/779925841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d2c15bc00 0x7f8d2c0c73a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8d1401f160 tx=0x7f8d10005840 comp rx=0 tx=0).stop 2024-04-17T19:48:34.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.941+0000 7f8d30c00640 1 -- 172.21.15.29:0/779925841 shutdown_connections 2024-04-17T19:48:34.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.941+0000 7f8d30c00640 1 --2- 172.21.15.29:0/779925841 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8d08068380 0x7f8d0806a890 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.941+0000 7f8d30c00640 1 --2- 172.21.15.29:0/779925841 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d2c1e73f0 0x7f8d2c151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.941+0000 7f8d30c00640 1 --2- 172.21.15.29:0/779925841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d2c15bc00 0x7f8d2c0c73a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.941+0000 7f8d30c00640 1 --2- 172.21.15.29:0/779925841 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d10004810 0x7f8d2c1f57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:34.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.941+0000 7f8d30c00640 1 -- 172.21.15.29:0/779925841 >> 172.21.15.29:0/779925841 conn(0x7f8d2c06c3c0 msgr2=0x7f8d2c1538c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:34.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.941+0000 7f8d30c00640 1 -- 172.21.15.29:0/779925841 shutdown_connections 2024-04-17T19:48:34.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:34.941+0000 7f8d30c00640 1 -- 172.21.15.29:0/779925841 wait complete. 2024-04-17T19:48:34.953 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:48:35.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.117+0000 7f9c9d000640 1 Processor -- start 2024-04-17T19:48:35.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.117+0000 7f9c9d000640 1 -- start start 2024-04-17T19:48:35.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.117+0000 7f9c9d000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c980cc7f0 0x7f9c980c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:35.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.117+0000 7f9c9d000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9c9815fdb0 0x7f9c981601c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:35.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.117+0000 7f9c9d000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c980cbc20 con 0x7f9c980cae50 2024-04-17T19:48:35.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.117+0000 7f9c9d000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9c9815d140 con 0x7f9c9815fdb0 2024-04-17T19:48:35.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.117+0000 7f9c9d000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9c9815c180 con 0x7f9c980cc7f0 2024-04-17T19:48:35.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.121+0000 7f9c95e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c980cc7f0 0x7f9c980c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:35.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.121+0000 7f9c8fe00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9c9815fdb0 0x7f9c981601c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:35.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.121+0000 7f9c95e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c980cc7f0 0x7f9c980c9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:58920/0 (socket says 172.21.15.29:58920) 2024-04-17T19:48:35.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.121+0000 7f9c95e00640 1 -- 172.21.15.29:0/3794513002 learned_addr learned my addr 172.21.15.29:0/3794513002 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:35.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.121+0000 7f9c96800640 1 --1- 172.21.15.29:0/3794513002 >> v1:172.21.15.37:6789/0 conn(0x7f9c980cae50 0x7f9c980cf480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:59552/0 (socket says 172.21.15.29:59552) 2024-04-17T19:48:35.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.121+0000 7f9c95e00640 1 -- 172.21.15.29:0/3794513002 >> v2:172.21.15.37:3300/0 conn(0x7f9c9815fdb0 msgr2=0x7f9c981601c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:35.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.121+0000 7f9c95e00640 1 --2- 172.21.15.29:0/3794513002 >> v2:172.21.15.37:3300/0 conn(0x7f9c9815fdb0 0x7f9c981601c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:35.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.121+0000 7f9c95e00640 1 -- 172.21.15.29:0/3794513002 >> v1:172.21.15.37:6789/0 conn(0x7f9c980cae50 legacy=0x7f9c980cf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:35.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.121+0000 7f9c95e00640 1 -- 172.21.15.29:0/3794513002 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c9806ca70 con 0x7f9c980cc7f0 2024-04-17T19:48:35.111 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.121+0000 7f9c95e00640 1 --2- 172.21.15.29:0/3794513002 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c980cc7f0 0x7f9c980c9ef0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c88009870 tx=0x7f9c8801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=338681fab965bbb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:35.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.125+0000 7f9c95400640 1 -- 172.21.15.29:0/3794513002 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c880b0070 con 0x7f9c980cc7f0 2024-04-17T19:48:35.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.125+0000 7f9c95400640 1 -- 172.21.15.29:0/3794513002 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c880ab600 con 0x7f9c980cc7f0 2024-04-17T19:48:35.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.125+0000 7f9c95400640 1 -- 172.21.15.29:0/3794513002 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c880ab9d0 con 0x7f9c980cc7f0 2024-04-17T19:48:35.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.125+0000 7f9c9d000640 1 -- 172.21.15.29:0/3794513002 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c980cc7f0 msgr2=0x7f9c980c9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:35.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.125+0000 7f9c9d000640 1 --2- 172.21.15.29:0/3794513002 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c980cc7f0 0x7f9c980c9ef0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9c88009870 tx=0x7f9c8801c440 comp rx=0 tx=0).stop 2024-04-17T19:48:35.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.125+0000 7f9c9d000640 1 -- 172.21.15.29:0/3794513002 shutdown_connections 2024-04-17T19:48:35.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.125+0000 7f9c9d000640 1 --2- 172.21.15.29:0/3794513002 >> v2:172.21.15.37:3300/0 conn(0x7f9c9815fdb0 0x7f9c981601c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:35.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.125+0000 7f9c9d000640 1 --2- 172.21.15.29:0/3794513002 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c980cc7f0 0x7f9c980c9ef0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:35.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.125+0000 7f9c9d000640 1 -- 172.21.15.29:0/3794513002 >> 172.21.15.29:0/3794513002 conn(0x7f9c98067eb0 msgr2=0x7f9c98070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:35.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.125+0000 7f9c9d000640 1 -- 172.21.15.29:0/3794513002 shutdown_connections 2024-04-17T19:48:35.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.125+0000 7f9c9d000640 1 -- 172.21.15.29:0/3794513002 wait complete. 2024-04-17T19:48:35.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.125+0000 7f9c9d000640 1 Processor -- start 2024-04-17T19:48:35.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.125+0000 7f9c9d000640 1 -- start start 2024-04-17T19:48:35.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c9d000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c980cae50 0x7f9c981f6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:35.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c9d000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c980cc7f0 0x7f9c980cd890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:35.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c9d000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c9815fdb0 0x7f9c980cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:35.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c9d000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9806c790 con 0x7f9c980cae50 2024-04-17T19:48:35.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c9d000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9806c4b0 con 0x7f9c980cc7f0 2024-04-17T19:48:35.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c9d000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9c980cb670 con 0x7f9c9815fdb0 2024-04-17T19:48:35.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c95e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c980cae50 0x7f9c981f6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:35.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c95e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c980cae50 0x7f9c981f6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33992/0 (socket says 172.21.15.29:33992) 2024-04-17T19:48:35.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c95e00640 1 -- 172.21.15.29:0/344060191 learned_addr learned my addr 172.21.15.29:0/344060191 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:35.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c8fe00640 1 --2- 172.21.15.29:0/344060191 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c980cc7f0 0x7f9c980cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:35.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c96800640 1 --2- 172.21.15.29:0/344060191 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c9815fdb0 0x7f9c980cddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:35.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c95e00640 1 -- 172.21.15.29:0/344060191 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c9815fdb0 msgr2=0x7f9c980cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:35.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c95e00640 1 --2- 172.21.15.29:0/344060191 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c9815fdb0 0x7f9c980cddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:35.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c95e00640 1 -- 172.21.15.29:0/344060191 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c980cc7f0 msgr2=0x7f9c980cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:35.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c95e00640 1 --2- 172.21.15.29:0/344060191 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c980cc7f0 0x7f9c980cd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:35.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c95e00640 1 -- 172.21.15.29:0/344060191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c981f6760 con 0x7f9c980cae50 2024-04-17T19:48:35.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c95e00640 1 --2- 172.21.15.29:0/344060191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c980cae50 0x7f9c981f6220 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9c88000c10 tx=0x7f9c880191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:35.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c8e000640 1 -- 172.21.15.29:0/344060191 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c880b8020 con 0x7f9c980cae50 2024-04-17T19:48:35.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c8e000640 1 -- 172.21.15.29:0/344060191 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c880b4610 con 0x7f9c980cae50 2024-04-17T19:48:35.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c8e000640 1 -- 172.21.15.29:0/344060191 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c880b0040 con 0x7f9c980cae50 2024-04-17T19:48:35.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c9d000640 1 -- 172.21.15.29:0/344060191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c980b09a0 con 0x7f9c980cae50 2024-04-17T19:48:35.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c9d000640 1 -- 172.21.15.29:0/344060191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c980b0f60 con 0x7f9c980cae50 2024-04-17T19:48:35.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.129+0000 7f9c9d000640 1 -- 172.21.15.29:0/344060191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c98069b90 con 0x7f9c980cae50 2024-04-17T19:48:35.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.133+0000 7f9c8e000640 1 -- 172.21.15.29:0/344060191 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9c880bb020 con 0x7f9c980cae50 2024-04-17T19:48:35.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.133+0000 7f9c8e000640 1 --2- 172.21.15.29:0/344060191 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9c74068380 0x7f9c7406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:35.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.133+0000 7f9c8e000640 1 -- 172.21.15.29:0/344060191 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f9c88120960 con 0x7f9c980cae50 2024-04-17T19:48:35.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.133+0000 7f9c8fe00640 1 --2- 172.21.15.29:0/344060191 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9c74068380 0x7f9c7406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:35.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.133+0000 7f9c8fe00640 1 --2- 172.21.15.29:0/344060191 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9c74068380 0x7f9c7406a890 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9c980b1670 tx=0x7f9c80009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:35.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.137+0000 7f9c8e000640 1 -- 172.21.15.29:0/344060191 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9c880c2030 con 0x7f9c980cae50 2024-04-17T19:48:35.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.337+0000 7f9c9d000640 1 -- 172.21.15.29:0/344060191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c9815b5c0 con 0x7f9c980cae50 2024-04-17T19:48:35.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.337+0000 7f9c8e000640 1 -- 172.21.15.29:0/344060191 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6902 (secure 0 0 0) 0x7f9c88016150 con 0x7f9c980cae50 2024-04-17T19:48:35.325 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:35.325 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:48:34.874038+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4226":{"gid":4226,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.37:6838/332907250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":332907250},{"type":"v1","addr":"172.21.15.37:6838","nonce":332907250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4226,4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:48:35.325 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 19 2024-04-17T19:48:35.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.337+0000 7f9c9d000640 1 -- 172.21.15.29:0/344060191 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9c74068380 msgr2=0x7f9c7406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:35.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.337+0000 7f9c9d000640 1 --2- 172.21.15.29:0/344060191 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9c74068380 0x7f9c7406a890 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9c980b1670 tx=0x7f9c80009190 comp rx=0 tx=0).stop 2024-04-17T19:48:35.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.337+0000 7f9c9d000640 1 -- 172.21.15.29:0/344060191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c980cae50 msgr2=0x7f9c981f6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:35.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.337+0000 7f9c9d000640 1 --2- 172.21.15.29:0/344060191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c980cae50 0x7f9c981f6220 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9c88000c10 tx=0x7f9c880191e0 comp rx=0 tx=0).stop 2024-04-17T19:48:35.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.337+0000 7f9c9d000640 1 -- 172.21.15.29:0/344060191 shutdown_connections 2024-04-17T19:48:35.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.337+0000 7f9c9d000640 1 --2- 172.21.15.29:0/344060191 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9c74068380 0x7f9c7406a890 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:35.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.337+0000 7f9c9d000640 1 --2- 172.21.15.29:0/344060191 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9c9815fdb0 0x7f9c980cddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:35.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.337+0000 7f9c9d000640 1 --2- 172.21.15.29:0/344060191 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9c980cc7f0 0x7f9c980cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:35.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.337+0000 7f9c9d000640 1 --2- 172.21.15.29:0/344060191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c980cae50 0x7f9c981f6220 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:35.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.337+0000 7f9c9d000640 1 -- 172.21.15.29:0/344060191 >> 172.21.15.29:0/344060191 conn(0x7f9c98067eb0 msgr2=0x7f9c9815eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:35.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.337+0000 7f9c9d000640 1 -- 172.21.15.29:0/344060191 shutdown_connections 2024-04-17T19:48:35.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:35.337+0000 7f9c9d000640 1 -- 172.21.15.29:0/344060191 wait complete. 2024-04-17T19:48:35.952 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T19:47:49.315515+0000', 'modified': '2024-04-17T19:48:34.874038+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 4305, 'mds_1': 4302, 'mds_2': 4259, 'mds_3': 4229, 'mds_4': 4226}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4226': {'gid': 4226, 'name': 'b', 'rank': 4, 'incarnation': 18, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.37:6838/332907250', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.37:6835', 'nonce': 332907250}, {'type': 'v1', 'addr': '172.21.15.37:6838', 'nonce': 332907250}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4229': {'gid': 4229, 'name': 'f', 'rank': 3, 'incarnation': 16, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.37:6839/1100598562', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.37:6836', 'nonce': 1100598562}, {'type': 'v1', 'addr': '172.21.15.37:6839', 'nonce': 1100598562}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4259': {'gid': 4259, 'name': 'a', 'rank': 2, 'incarnation': 14, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.29:6836/4184160618', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6832', 'nonce': 4184160618}, {'type': 'v1', 'addr': '172.21.15.29:6836', 'nonce': 4184160618}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4302': {'gid': 4302, 'name': 'e', 'rank': 1, 'incarnation': 10, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.29:6835/270402476', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6834', 'nonce': 270402476}, {'type': 'v1', 'addr': '172.21.15.29:6835', 'nonce': 270402476}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4305': {'gid': 4305, 'name': 'c', 'rank': 0, 'incarnation': 5, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.29:6837/2691590158', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6833', 'nonce': 2691590158}, {'type': 'v1', 'addr': '172.21.15.29:6837', 'nonce': 2691590158}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4305, 'qdb_cluster': [4226, 4229, 4259, 4302, 4305]} 2024-04-17T19:48:35.953 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-17T19:48:35.953 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-17T19:48:36.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.117+0000 7f4409a00640 1 Processor -- start 2024-04-17T19:48:36.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.117+0000 7f4409a00640 1 -- start start 2024-04-17T19:48:36.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.117+0000 7f4409a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44040c8d50 0x7f44040c9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:36.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.117+0000 7f4409a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4404067ce0 0x7f44040680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:36.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.117+0000 7f4409a00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44040703d0 con 0x7f440415c990 2024-04-17T19:48:36.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.117+0000 7f4409a00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f440406c800 con 0x7f4404067ce0 2024-04-17T19:48:36.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.117+0000 7f4409a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f44040706b0 con 0x7f44040c8d50 2024-04-17T19:48:36.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.121+0000 7f4403400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4404067ce0 0x7f44040680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:36.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.121+0000 7f4402a00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f440415c990 0x7f440406bf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48238/0 (socket says 172.21.15.29:48238) 2024-04-17T19:48:36.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.121+0000 7f4402a00640 1 -- 172.21.15.29:0/2876423188 learned_addr learned my addr 172.21.15.29:0/2876423188 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:36.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.121+0000 7f43fbe00640 1 --2- 172.21.15.29:0/2876423188 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44040c8d50 0x7f44040c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:36.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.121+0000 7f4403400640 1 -- 172.21.15.29:0/2876423188 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44040c8d50 msgr2=0x7f44040c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:36.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.121+0000 7f4403400640 1 --2- 172.21.15.29:0/2876423188 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44040c8d50 0x7f44040c9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:36.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.121+0000 7f4403400640 1 -- 172.21.15.29:0/2876423188 >> v1:172.21.15.29:6789/0 conn(0x7f440415c990 legacy=0x7f440406bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:36.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.121+0000 7f4403400640 1 -- 172.21.15.29:0/2876423188 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4404070990 con 0x7f4404067ce0 2024-04-17T19:48:36.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.121+0000 7f4402000640 1 -- 172.21.15.29:0/2876423188 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1866835303 0 0) 0x7f44040703d0 con 0x7f440415c990 2024-04-17T19:48:36.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.125+0000 7f4403400640 1 --2- 172.21.15.29:0/2876423188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4404067ce0 0x7f44040680f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009d90 tx=0x7f43fc01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bbab221cd4f58 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:36.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.125+0000 7f4402000640 1 -- 172.21.15.29:0/2876423188 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43fc0b2070 con 0x7f4404067ce0 2024-04-17T19:48:36.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.125+0000 7f4402000640 1 -- 172.21.15.29:0/2876423188 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43fc0a5830 con 0x7f4404067ce0 2024-04-17T19:48:36.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.125+0000 7f4402000640 1 -- 172.21.15.29:0/2876423188 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43fc0a5c00 con 0x7f4404067ce0 2024-04-17T19:48:36.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.125+0000 7f4409a00640 1 -- 172.21.15.29:0/2876423188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4404067ce0 msgr2=0x7f44040680f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:36.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.125+0000 7f4409a00640 1 --2- 172.21.15.29:0/2876423188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4404067ce0 0x7f44040680f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009d90 tx=0x7f43fc01c970 comp rx=0 tx=0).stop 2024-04-17T19:48:36.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.125+0000 7f4409a00640 1 -- 172.21.15.29:0/2876423188 shutdown_connections 2024-04-17T19:48:36.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.125+0000 7f4409a00640 1 --2- 172.21.15.29:0/2876423188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4404067ce0 0x7f44040680f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:36.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.125+0000 7f4409a00640 1 --2- 172.21.15.29:0/2876423188 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44040c8d50 0x7f44040c9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:36.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.125+0000 7f4409a00640 1 -- 172.21.15.29:0/2876423188 >> 172.21.15.29:0/2876423188 conn(0x7f44040ce350 msgr2=0x7f44040ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:36.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.125+0000 7f4409a00640 1 -- 172.21.15.29:0/2876423188 shutdown_connections 2024-04-17T19:48:36.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.125+0000 7f4409a00640 1 -- 172.21.15.29:0/2876423188 wait complete. 2024-04-17T19:48:36.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4409a00640 1 Processor -- start 2024-04-17T19:48:36.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4409a00640 1 -- start start 2024-04-17T19:48:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4409a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4404067ce0 0x7f4404159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4409a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44040c8d50 0x7f440414af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4409a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f440415c990 0x7f440414b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4409a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4404169230 con 0x7f4404067ce0 2024-04-17T19:48:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4409a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f44041690b0 con 0x7f440415c990 2024-04-17T19:48:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4409a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f44041693b0 con 0x7f44040c8d50 2024-04-17T19:48:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4402a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4404067ce0 0x7f4404159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4402a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4404067ce0 0x7f4404159040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34008/0 (socket says 172.21.15.29:34008) 2024-04-17T19:48:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4402a00640 1 -- 172.21.15.29:0/2538221338 learned_addr learned my addr 172.21.15.29:0/2538221338 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4403400640 1 --2- 172.21.15.29:0/2538221338 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f440415c990 0x7f440414b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:36.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f43fbe00640 1 --2- 172.21.15.29:0/2538221338 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44040c8d50 0x7f440414af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:36.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4402a00640 1 -- 172.21.15.29:0/2538221338 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44040c8d50 msgr2=0x7f440414af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:36.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4402a00640 1 --2- 172.21.15.29:0/2538221338 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44040c8d50 0x7f440414af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:36.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4402a00640 1 -- 172.21.15.29:0/2538221338 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f440415c990 msgr2=0x7f440414b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:36.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4402a00640 1 --2- 172.21.15.29:0/2538221338 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f440415c990 0x7f440414b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:36.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4402a00640 1 -- 172.21.15.29:0/2538221338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440414c530 con 0x7f4404067ce0 2024-04-17T19:48:36.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4403400640 1 --2- 172.21.15.29:0/2538221338 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f440415c990 0x7f440414b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:36.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4402a00640 1 --2- 172.21.15.29:0/2538221338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4404067ce0 0x7f4404159040 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f43f400c480 tx=0x7f43f400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:36.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f43faa00640 1 -- 172.21.15.29:0/2538221338 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f4018070 con 0x7f4404067ce0 2024-04-17T19:48:36.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f43faa00640 1 -- 172.21.15.29:0/2538221338 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f4007500 con 0x7f4404067ce0 2024-04-17T19:48:36.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f43faa00640 1 -- 172.21.15.29:0/2538221338 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f4004a30 con 0x7f4404067ce0 2024-04-17T19:48:36.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4409a00640 1 -- 172.21.15.29:0/2538221338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4404156bb0 con 0x7f4404067ce0 2024-04-17T19:48:36.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.129+0000 7f4409a00640 1 -- 172.21.15.29:0/2538221338 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44041589c0 con 0x7f4404067ce0 2024-04-17T19:48:36.121 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.133+0000 7f43faa00640 1 -- 172.21.15.29:0/2538221338 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f43f4004bd0 con 0x7f4404067ce0 2024-04-17T19:48:36.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.133+0000 7f43faa00640 1 --2- 172.21.15.29:0/2538221338 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f43e0068380 0x7f43e006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:36.122 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.133+0000 7f43faa00640 1 -- 172.21.15.29:0/2538221338 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f43f4086090 con 0x7f4404067ce0 2024-04-17T19:48:36.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.133+0000 7f43fbe00640 1 --2- 172.21.15.29:0/2538221338 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f43e0068380 0x7f43e006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:36.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.137+0000 7f43fbe00640 1 --2- 172.21.15.29:0/2538221338 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f43e0068380 0x7f43e006a890 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4404155870 tx=0x7f43ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:36.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.137+0000 7f4411194000 1 Processor -- start 2024-04-17T19:48:36.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.137+0000 7f4411194000 1 -- start start 2024-04-17T19:48:36.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.137+0000 7f4411194000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e39135c10 0x564e3929f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:36.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.137+0000 7f4411194000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e3912e930 0x564e3929f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:36.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.137+0000 7f4411194000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e38d2e280 0x564e3929fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:36.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.137+0000 7f4411194000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564e390ac4f0 con 0x564e39135c10 2024-04-17T19:48:36.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.137+0000 7f4411194000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x564e39046c10 con 0x564e3912e930 2024-04-17T19:48:36.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.137+0000 7f4411194000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x564e390a84b0 con 0x564e38d2e280 2024-04-17T19:48:36.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.137+0000 7f4402a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e39135c10 0x564e3929f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:36.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.137+0000 7f4402a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e39135c10 0x564e3929f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34022/0 (socket says 172.21.15.29:34022) 2024-04-17T19:48:36.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.137+0000 7f4402a00640 1 -- 172.21.15.29:0/2473124723 learned_addr learned my addr 172.21.15.29:0/2473124723 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:36.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.137+0000 7f4403400640 1 --2- 172.21.15.29:0/2473124723 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e38d2e280 0x564e3929fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:36.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.137+0000 7f43fbe00640 1 --2- 172.21.15.29:0/2473124723 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e3912e930 0x564e3929f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:36.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f4402a00640 1 -- 172.21.15.29:0/2473124723 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e38d2e280 msgr2=0x564e3929fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:36.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f4402a00640 1 --2- 172.21.15.29:0/2473124723 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e38d2e280 0x564e3929fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:36.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f4402a00640 1 -- 172.21.15.29:0/2473124723 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e3912e930 msgr2=0x564e3929f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:36.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f4402a00640 1 --2- 172.21.15.29:0/2473124723 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e3912e930 0x564e3929f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:36.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f4402a00640 1 -- 172.21.15.29:0/2473124723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e39046e60 con 0x564e39135c10 2024-04-17T19:48:36.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f4403400640 1 --2- 172.21.15.29:0/2473124723 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e38d2e280 0x564e3929fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:36.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f4402a00640 1 --2- 172.21.15.29:0/2473124723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e39135c10 0x564e3929f1b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f43f40840f0 tx=0x7f43f4055790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d5db1f76f58b46 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:36.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f43f3400640 1 -- 172.21.15.29:0/2473124723 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f4053e40 con 0x564e39135c10 2024-04-17T19:48:36.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f43f3400640 1 -- 172.21.15.29:0/2473124723 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f4001050 con 0x564e39135c10 2024-04-17T19:48:36.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f43f3400640 1 -- 172.21.15.29:0/2473124723 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f4054f90 con 0x564e39135c10 2024-04-17T19:48:36.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f4411194000 1 -- 172.21.15.29:0/2473124723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e39135c10 msgr2=0x564e3929f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:36.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f4411194000 1 --2- 172.21.15.29:0/2473124723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e39135c10 0x564e3929f1b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f43f40840f0 tx=0x7f43f4055790 comp rx=0 tx=0).stop 2024-04-17T19:48:36.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f4411194000 1 -- 172.21.15.29:0/2473124723 shutdown_connections 2024-04-17T19:48:36.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f4411194000 1 --2- 172.21.15.29:0/2473124723 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e38d2e280 0x564e3929fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:36.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f4411194000 1 --2- 172.21.15.29:0/2473124723 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e3912e930 0x564e3929f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:36.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f4411194000 1 --2- 172.21.15.29:0/2473124723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e39135c10 0x564e3929f1b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:36.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f4411194000 1 -- 172.21.15.29:0/2473124723 >> 172.21.15.29:0/2473124723 conn(0x564e39177670 msgr2=0x564e3929cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:36.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f4411194000 1 -- 172.21.15.29:0/2473124723 shutdown_connections 2024-04-17T19:48:36.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f4411194000 1 -- 172.21.15.29:0/2473124723 wait complete. 2024-04-17T19:48:36.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f4411194000 1 -- start start 2024-04-17T19:48:36.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.141+0000 7f4411194000 1 Processor -- start 2024-04-17T19:48:36.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4411194000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e38d2e280 0x564e392aa440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:36.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4411194000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e3912e930 0x564e392aa980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:36.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4411194000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e39135c10 0x564e39260b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:36.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4411194000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564e390ed9e0 con 0x564e3912e930 2024-04-17T19:48:36.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4411194000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x564e39069a80 con 0x564e39135c10 2024-04-17T19:48:36.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4411194000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x564e39067d10 con 0x564e38d2e280 2024-04-17T19:48:36.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4411194000 10 client.0 _ll_register_callbacks cb 0x564e39249b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T19:48:36.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f43fbe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e3912e930 0x564e392aa980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:36.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f43fbe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e3912e930 0x564e392aa980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34038/0 (socket says 172.21.15.29:34038) 2024-04-17T19:48:36.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f43fbe00640 1 -- 172.21.15.29:0/3128749847 learned_addr learned my addr 172.21.15.29:0/3128749847 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:36.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4403400640 1 --2- 172.21.15.29:0/3128749847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e39135c10 0x564e39260b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:36.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4402a00640 1 --2- 172.21.15.29:0/3128749847 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e38d2e280 0x564e392aa440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:36.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f43fbe00640 1 -- 172.21.15.29:0/3128749847 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e38d2e280 msgr2=0x564e392aa440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:36.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f43fbe00640 1 --2- 172.21.15.29:0/3128749847 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e38d2e280 0x564e392aa440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:36.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f43fbe00640 1 -- 172.21.15.29:0/3128749847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e39135c10 msgr2=0x564e39260b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:36.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f43fbe00640 1 --2- 172.21.15.29:0/3128749847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e39135c10 0x564e39260b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:36.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f43fbe00640 1 -- 172.21.15.29:0/3128749847 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e39077e20 con 0x564e3912e930 2024-04-17T19:48:36.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4402a00640 1 --2- 172.21.15.29:0/3128749847 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e38d2e280 0x564e392aa440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:36.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4403400640 1 --2- 172.21.15.29:0/3128749847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e39135c10 0x564e39260b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:36.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f43fbe00640 1 --2- 172.21.15.29:0/3128749847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e3912e930 0x564e392aa980 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f43ec0156f0 tx=0x7f43ec0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f8c7e09e346f37 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:36.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f43e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-17T19:48:36.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f43e7e00640 1 -- 172.21.15.29:0/3128749847 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec026070 con 0x564e3912e930 2024-04-17T19:48:36.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4411194000 1 -- 172.21.15.29:0/3128749847 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43f4001050 con 0x564e3912e930 2024-04-17T19:48:36.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f43e7e00640 1 -- 172.21.15.29:0/3128749847 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f43ec014730 con 0x564e3912e930 2024-04-17T19:48:36.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f43e7e00640 1 -- 172.21.15.29:0/3128749847 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f43ec031050 con 0x564e3912e930 2024-04-17T19:48:36.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4411194000 10 client.4875 fetch_fsmap learned FSMap version 19 2024-04-17T19:48:36.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4411194000 1 -- 172.21.15.29:0/3128749847 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x564e39177a80 con 0x564e3912e930 2024-04-17T19:48:36.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f43e7e00640 1 -- 172.21.15.29:0/3128749847 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f4001050 con 0x564e3912e930 2024-04-17T19:48:36.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f43e7e00640 1 -- 172.21.15.29:0/3128749847 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7f43ec034020 con 0x564e3912e930 2024-04-17T19:48:36.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4411194000 10 client.4875 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T19:48:36.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4411194000 10 client.4875 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-17T19:48:36.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4411194000 20 client.4875 populate_metadata read hostname 'smithi029' 2024-04-17T19:48:36.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4411194000 1 --2- 172.21.15.29:0/3128749847 >> [v2:172.21.15.37:6835/332907250,v1:172.21.15.37:6838/332907250] conn(0x564e38d527d0 0x564e39261860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:36.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4411194000 4 client.4875 mds_command: new command op to 4226 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T19:48:36.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4411194000 1 -- 172.21.15.29:0/3128749847 --> [v2:172.21.15.37:6835/332907250,v1:172.21.15.37:6838/332907250] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e39045be0 con 0x564e38d527d0 2024-04-17T19:48:36.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4402a00640 1 --2- 172.21.15.29:0/3128749847 >> [v2:172.21.15.37:6835/332907250,v1:172.21.15.37:6838/332907250] conn(0x564e38d527d0 0x564e39261860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:36.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f4402a00640 1 --2- 172.21.15.29:0/3128749847 >> [v2:172.21.15.37:6835/332907250,v1:172.21.15.37:6838/332907250] conn(0x564e38d527d0 0x564e39261860 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3768ea74b44e60f5 server_cookie=a6826bea5eeea7a0 in_seq=0 out_seq=0 2024-04-17T19:48:36.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:36.145+0000 7f43e7e00640 10 client.4875 ms_handle_connect on v2:172.21.15.37:6835/332907250 2024-04-17T19:48:39.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.884+0000 7f43e7e00640 1 -- 172.21.15.29:0/3128749847 <== mds.4 v2:172.21.15.37:6835/332907250 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564e39045be0 con 0x564e38d527d0 2024-04-17T19:48:39.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.884+0000 7f43e7e00640 10 client.4875 handle_command_reply: tid=0 2024-04-17T19:48:39.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.884+0000 7f4411194000 1 client.4875 shutdown 2024-04-17T19:48:39.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.884+0000 7f4411194000 1 -- 172.21.15.29:0/3128749847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e3912e930 msgr2=0x564e392aa980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:39.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.884+0000 7f4411194000 1 --2- 172.21.15.29:0/3128749847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e3912e930 0x564e392aa980 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f43ec0156f0 tx=0x7f43ec0046d0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.884+0000 7f4411194000 1 -- 172.21.15.29:0/3128749847 shutdown_connections 2024-04-17T19:48:39.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.884+0000 7f4411194000 1 --2- 172.21.15.29:0/3128749847 >> [v2:172.21.15.37:6835/332907250,v1:172.21.15.37:6838/332907250] conn(0x564e38d527d0 0x564e39261860 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.884+0000 7f4411194000 1 --2- 172.21.15.29:0/3128749847 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e39135c10 0x564e39260b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.884+0000 7f4411194000 1 --2- 172.21.15.29:0/3128749847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e3912e930 0x564e392aa980 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.884+0000 7f4411194000 1 --2- 172.21.15.29:0/3128749847 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e38d2e280 0x564e392aa440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.884+0000 7f43e7e00640 0 client.4875 ms_handle_reset on v2:172.21.15.37:6835/332907250 2024-04-17T19:48:39.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.884+0000 7f4411194000 1 -- 172.21.15.29:0/3128749847 >> 172.21.15.29:0/3128749847 conn(0x564e39177670 msgr2=0x564e3929a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:39.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.884+0000 7f4411194000 1 -- 172.21.15.29:0/3128749847 shutdown_connections 2024-04-17T19:48:39.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.884+0000 7f4411194000 1 -- 172.21.15.29:0/3128749847 wait complete. 2024-04-17T19:48:39.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.884+0000 7f4411194000 20 client.4875 trim_cache size 0 max 16384 2024-04-17T19:48:39.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.900+0000 7f4411194000 1 Processor -- start 2024-04-17T19:48:39.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.900+0000 7f4411194000 1 -- start start 2024-04-17T19:48:39.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4411194000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e38d2e280 0x564e3925d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:39.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4411194000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e3912e930 0x564e3929a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:39.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4411194000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e39135c10 0x564e3929a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:39.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4411194000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4404203340 con 0x564e38d2e280 2024-04-17T19:48:39.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4411194000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f44041458c0 con 0x564e3912e930 2024-04-17T19:48:39.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4411194000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x564e390ab800 con 0x564e39135c10 2024-04-17T19:48:39.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4402a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e38d2e280 0x564e3925d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:39.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4402a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e38d2e280 0x564e3925d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33082/0 (socket says 172.21.15.29:33082) 2024-04-17T19:48:39.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4402a00640 1 -- 172.21.15.29:0/2884787157 learned_addr learned my addr 172.21.15.29:0/2884787157 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:39.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4403400640 1 --2- 172.21.15.29:0/2884787157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e39135c10 0x564e3929a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:39.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f43fbe00640 1 --2- 172.21.15.29:0/2884787157 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e3912e930 0x564e3929a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:39.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4402a00640 1 -- 172.21.15.29:0/2884787157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e39135c10 msgr2=0x564e3929a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:39.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4402a00640 1 --2- 172.21.15.29:0/2884787157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e39135c10 0x564e3929a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4402a00640 1 -- 172.21.15.29:0/2884787157 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e3912e930 msgr2=0x564e3929a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:39.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4402a00640 1 --2- 172.21.15.29:0/2884787157 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e3912e930 0x564e3929a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4402a00640 1 -- 172.21.15.29:0/2884787157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e39105e30 con 0x564e38d2e280 2024-04-17T19:48:39.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4402a00640 1 --2- 172.21.15.29:0/2884787157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e38d2e280 0x564e3925d190 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f43f4051fa0 tx=0x7f43f4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f826a9ef843e996 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:39.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f43f2a00640 1 -- 172.21.15.29:0/2884787157 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f4079070 con 0x564e38d2e280 2024-04-17T19:48:39.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f43f2a00640 1 -- 172.21.15.29:0/2884787157 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f4001050 con 0x564e38d2e280 2024-04-17T19:48:39.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f43f2a00640 1 -- 172.21.15.29:0/2884787157 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f4074030 con 0x564e38d2e280 2024-04-17T19:48:39.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4411194000 1 -- 172.21.15.29:0/2884787157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e38d2e280 msgr2=0x564e3925d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:39.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4411194000 1 --2- 172.21.15.29:0/2884787157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e38d2e280 0x564e3925d190 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f43f4051fa0 tx=0x7f43f4013000 comp rx=0 tx=0).stop 2024-04-17T19:48:39.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4411194000 1 -- 172.21.15.29:0/2884787157 shutdown_connections 2024-04-17T19:48:39.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4411194000 1 --2- 172.21.15.29:0/2884787157 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e39135c10 0x564e3929a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4411194000 1 --2- 172.21.15.29:0/2884787157 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e3912e930 0x564e3929a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4411194000 1 --2- 172.21.15.29:0/2884787157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e38d2e280 0x564e3925d190 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4411194000 1 -- 172.21.15.29:0/2884787157 >> 172.21.15.29:0/2884787157 conn(0x564e39177670 msgr2=0x564e392af4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:39.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4411194000 1 -- 172.21.15.29:0/2884787157 shutdown_connections 2024-04-17T19:48:39.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.904+0000 7f4411194000 1 -- 172.21.15.29:0/2884787157 wait complete. 2024-04-17T19:48:39.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4411194000 1 -- start start 2024-04-17T19:48:39.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4411194000 1 Processor -- start 2024-04-17T19:48:39.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4411194000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e38d2e280 0x564e39310af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:39.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4411194000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e3912e930 0x564e39311030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:39.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4411194000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e39135c10 0x564e392ad080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:39.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4411194000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x564e390b9700 con 0x564e39135c10 2024-04-17T19:48:39.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4411194000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x564e390b0b20 con 0x564e38d2e280 2024-04-17T19:48:39.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4411194000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x564e390427d0 con 0x564e3912e930 2024-04-17T19:48:39.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4411194000 10 client.0 _ll_register_callbacks cb 0x7f43ec011e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T19:48:39.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4403400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e39135c10 0x564e392ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:39.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4403400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e39135c10 0x564e392ad080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33092/0 (socket says 172.21.15.29:33092) 2024-04-17T19:48:39.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4403400640 1 -- 172.21.15.29:0/4213927579 learned_addr learned my addr 172.21.15.29:0/4213927579 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:39.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4402a00640 1 --2- 172.21.15.29:0/4213927579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e38d2e280 0x564e39310af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:39.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f43fbe00640 1 --2- 172.21.15.29:0/4213927579 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e3912e930 0x564e39311030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:39.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4403400640 1 -- 172.21.15.29:0/4213927579 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e3912e930 msgr2=0x564e39311030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:39.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4403400640 1 --2- 172.21.15.29:0/4213927579 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e3912e930 0x564e39311030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4403400640 1 -- 172.21.15.29:0/4213927579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e38d2e280 msgr2=0x564e39310af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:39.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4403400640 1 --2- 172.21.15.29:0/4213927579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e38d2e280 0x564e39310af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4403400640 1 -- 172.21.15.29:0/4213927579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e3903e7e0 con 0x564e39135c10 2024-04-17T19:48:39.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f43fbe00640 1 --2- 172.21.15.29:0/4213927579 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e3912e930 0x564e39311030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:39.900 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4403400640 1 --2- 172.21.15.29:0/4213927579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e39135c10 0x564e392ad080 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009d90 tx=0x7f43fc01b1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda88fe5345f2aa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:39.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f43e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-17T19:48:39.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f43e7e00640 1 -- 172.21.15.29:0/4213927579 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43fc0b2040 con 0x564e39135c10 2024-04-17T19:48:39.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f43e7e00640 1 -- 172.21.15.29:0/4213927579 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f43fc004240 con 0x564e39135c10 2024-04-17T19:48:39.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4411194000 1 -- 172.21.15.29:0/4213927579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f43f4001050 con 0x564e39135c10 2024-04-17T19:48:39.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f43e7e00640 1 -- 172.21.15.29:0/4213927579 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f43fc004440 con 0x564e39135c10 2024-04-17T19:48:39.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4411194000 10 client.4881 fetch_fsmap learned FSMap version 19 2024-04-17T19:48:39.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4411194000 1 -- 172.21.15.29:0/4213927579 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x564e39051670 con 0x564e39135c10 2024-04-17T19:48:39.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f43e7e00640 1 -- 172.21.15.29:0/4213927579 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f4001050 con 0x564e39135c10 2024-04-17T19:48:39.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f43e7e00640 1 -- 172.21.15.29:0/4213927579 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7f43fc0cc050 con 0x564e39135c10 2024-04-17T19:48:39.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4411194000 10 client.4881 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T19:48:39.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4411194000 10 client.4881 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-17T19:48:39.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4411194000 20 client.4881 populate_metadata read hostname 'smithi029' 2024-04-17T19:48:39.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.908+0000 7f4411194000 1 --2- 172.21.15.29:0/4213927579 >> [v2:172.21.15.37:6835/332907250,v1:172.21.15.37:6838/332907250] conn(0x564e38d40830 0x564e392adbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:39.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4411194000 4 client.4881 mds_command: new command op to 4226 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T19:48:39.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4411194000 1 -- 172.21.15.29:0/4213927579 --> [v2:172.21.15.37:6835/332907250,v1:172.21.15.37:6838/332907250] -- command(tid 0: {"prefix": "status"}) -- 0x564e390b9f10 con 0x564e38d40830 2024-04-17T19:48:39.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4402a00640 1 --2- 172.21.15.29:0/4213927579 >> [v2:172.21.15.37:6835/332907250,v1:172.21.15.37:6838/332907250] conn(0x564e38d40830 0x564e392adbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:39.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4402a00640 1 --2- 172.21.15.29:0/4213927579 >> [v2:172.21.15.37:6835/332907250,v1:172.21.15.37:6838/332907250] conn(0x564e38d40830 0x564e392adbc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=aee4552f659aaed1 server_cookie=8d646cc4c2b25ae0 in_seq=0 out_seq=0 2024-04-17T19:48:39.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f43e7e00640 10 client.4881 ms_handle_connect on v2:172.21.15.37:6835/332907250 2024-04-17T19:48:39.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f43e7e00640 1 -- 172.21.15.29:0/4213927579 <== mds.4 v2:172.21.15.37:6835/332907250 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x564e390b9f10 con 0x564e38d40830 2024-04-17T19:48:39.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f43e7e00640 10 client.4881 handle_command_reply: tid=0 2024-04-17T19:48:39.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4411194000 1 client.4881 shutdown 2024-04-17T19:48:39.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4411194000 1 -- 172.21.15.29:0/4213927579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e39135c10 msgr2=0x564e392ad080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:39.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4411194000 1 --2- 172.21.15.29:0/4213927579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e39135c10 0x564e392ad080 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009d90 tx=0x7f43fc01b1b0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4411194000 1 -- 172.21.15.29:0/4213927579 shutdown_connections 2024-04-17T19:48:39.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4411194000 1 --2- 172.21.15.29:0/4213927579 >> [v2:172.21.15.37:6835/332907250,v1:172.21.15.37:6838/332907250] conn(0x564e38d40830 0x564e392adbc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4411194000 1 --2- 172.21.15.29:0/4213927579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564e39135c10 0x564e392ad080 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4411194000 1 --2- 172.21.15.29:0/4213927579 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x564e3912e930 0x564e39311030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4411194000 1 --2- 172.21.15.29:0/4213927579 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x564e38d2e280 0x564e39310af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f43e7e00640 0 client.4881 ms_handle_reset on v2:172.21.15.37:6835/332907250 2024-04-17T19:48:39.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4411194000 1 -- 172.21.15.29:0/4213927579 >> 172.21.15.29:0/4213927579 conn(0x564e39177670 msgr2=0x564e392608d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:39.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4411194000 1 -- 172.21.15.29:0/4213927579 shutdown_connections 2024-04-17T19:48:39.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4411194000 1 -- 172.21.15.29:0/4213927579 wait complete. 2024-04-17T19:48:39.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4411194000 20 client.4881 trim_cache size 0 max 16384 2024-04-17T19:48:39.904 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-17T19:48:39.904 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "a680ee90-875c-4283-a901-86d7638393c3", 2024-04-17T19:48:39.904 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 4, 2024-04-17T19:48:39.904 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 4226, 2024-04-17T19:48:39.905 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-04-17T19:48:39.905 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-04-17T19:48:39.905 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-04-17T19:48:39.905 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 11.058340905, 2024-04-17T19:48:39.905 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 19, 2024-04-17T19:48:39.905 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 26, 2024-04-17T19:48:39.905 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 26, 2024-04-17T19:48:39.905 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 57.499911025999999 2024-04-17T19:48:39.905 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-17T19:48:39.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4409a00640 1 -- 172.21.15.29:0/2538221338 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f43e0068380 msgr2=0x7f43e006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:39.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4409a00640 1 --2- 172.21.15.29:0/2538221338 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f43e0068380 0x7f43e006a890 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4404155870 tx=0x7f43ec008040 comp rx=0 tx=0).stop 2024-04-17T19:48:39.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4409a00640 1 -- 172.21.15.29:0/2538221338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4404067ce0 msgr2=0x7f4404159040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:39.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4409a00640 1 --2- 172.21.15.29:0/2538221338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4404067ce0 0x7f4404159040 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f43f400c480 tx=0x7f43f400c940 comp rx=0 tx=0).stop 2024-04-17T19:48:39.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4409a00640 1 -- 172.21.15.29:0/2538221338 shutdown_connections 2024-04-17T19:48:39.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4409a00640 1 --2- 172.21.15.29:0/2538221338 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f43e0068380 0x7f43e006a890 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4409a00640 1 --2- 172.21.15.29:0/2538221338 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f440415c990 0x7f440414b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4409a00640 1 --2- 172.21.15.29:0/2538221338 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44040c8d50 0x7f440414af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4409a00640 1 --2- 172.21.15.29:0/2538221338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4404067ce0 0x7f4404159040 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:39.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4409a00640 1 -- 172.21.15.29:0/2538221338 >> 172.21.15.29:0/2538221338 conn(0x7f44040ce350 msgr2=0x7f4404159d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:39.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4409a00640 1 -- 172.21.15.29:0/2538221338 shutdown_connections 2024-04-17T19:48:39.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:39.912+0000 7f4409a00640 1 -- 172.21.15.29:0/2538221338 wait complete. 2024-04-17T19:48:39.923 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-04-17T19:48:40.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.088+0000 7f5fafa00640 1 Processor -- start 2024-04-17T19:48:40.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.088+0000 7f5fafa00640 1 -- start start 2024-04-17T19:48:40.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.088+0000 7f5fafa00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5fa8071ce0 0x7f5fa8141de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.088+0000 7f5fafa00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fa8142320 con 0x7f5fa81597e0 2024-04-17T19:48:40.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.088+0000 7f5fafa00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fa815c6d0 con 0x7f5fa81593d0 2024-04-17T19:48:40.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.088+0000 7f5fafa00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5fa80c7bd0 con 0x7f5fa8071ce0 2024-04-17T19:48:40.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.088+0000 7f5fada00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5fa81597e0 0x7f5fa8163620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52212/0 (socket says 172.21.15.29:52212) 2024-04-17T19:48:40.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.088+0000 7f5fada00640 1 -- 172.21.15.29:0/3102541184 learned_addr learned my addr 172.21.15.29:0/3102541184 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:40.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.088+0000 7f5fad000640 1 --2- 172.21.15.29:0/3102541184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5fa8071ce0 0x7f5fa8141de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.088+0000 7f5fa7e00640 1 -- 172.21.15.29:0/3102541184 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1768704195 0 0) 0x7f5fa8142320 con 0x7f5fa81597e0 2024-04-17T19:48:40.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.088+0000 7f5fa7e00640 1 -- 172.21.15.29:0/3102541184 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f8c003530 con 0x7f5fa81597e0 2024-04-17T19:48:40.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.088+0000 7f5fa7e00640 1 -- 172.21.15.29:0/3102541184 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4033848675 0 0) 0x7f5f8c003530 con 0x7f5fa81597e0 2024-04-17T19:48:40.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.088+0000 7f5fa7e00640 1 -- 172.21.15.29:0/3102541184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5fa8071ce0 msgr2=0x7f5fa8141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.088+0000 7f5fa7e00640 1 --2- 172.21.15.29:0/3102541184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5fa8071ce0 0x7f5fa8141de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.088+0000 7f5fa7e00640 1 -- 172.21.15.29:0/3102541184 >> v1:172.21.15.37:6789/0 conn(0x7f5fa81593d0 legacy=0x7f5fa80c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.088+0000 7f5fa7e00640 1 -- 172.21.15.29:0/3102541184 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa815c9b0 con 0x7f5fa81597e0 2024-04-17T19:48:40.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.088+0000 7f5fa7e00640 1 -- 172.21.15.29:0/3102541184 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f5fa0001a00 con 0x7f5fa81597e0 2024-04-17T19:48:40.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5fa7e00640 1 -- 172.21.15.29:0/3102541184 >> v1:172.21.15.29:6789/0 conn(0x7f5fa81597e0 legacy=0x7f5fa8163620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5fa7e00640 1 --2- 172.21.15.29:0/3102541184 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f8c0039f0 0x7f5f8c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5fa7e00640 1 --2- 172.21.15.29:0/3102541184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5fa8071ce0 0x7f5f8c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5fa7e00640 1 --2- 172.21.15.29:0/3102541184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f8c004d50 0x7f5f8c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5fa7e00640 1 -- 172.21.15.29:0/3102541184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5f8c00dce0 con 0x7f5f8c004d50 2024-04-17T19:48:40.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5fa7e00640 1 -- 172.21.15.29:0/3102541184 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5f8c00de90 con 0x7f5f8c0039f0 2024-04-17T19:48:40.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5fa7e00640 1 -- 172.21.15.29:0/3102541184 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5f8c00e090 con 0x7f5fa8071ce0 2024-04-17T19:48:40.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5fada00640 1 --2- 172.21.15.29:0/3102541184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f8c004d50 0x7f5f8c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5fad000640 1 --2- 172.21.15.29:0/3102541184 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f8c0039f0 0x7f5f8c003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5f9fe00640 1 --2- 172.21.15.29:0/3102541184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5fa8071ce0 0x7f5f8c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5fada00640 1 -- 172.21.15.29:0/3102541184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5fa8071ce0 msgr2=0x7f5f8c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5fada00640 1 --2- 172.21.15.29:0/3102541184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5fa8071ce0 0x7f5f8c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5fada00640 1 -- 172.21.15.29:0/3102541184 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f8c0039f0 msgr2=0x7f5f8c003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5fada00640 1 --2- 172.21.15.29:0/3102541184 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f8c0039f0 0x7f5f8c003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5fada00640 1 -- 172.21.15.29:0/3102541184 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f8c003530 con 0x7f5f8c004d50 2024-04-17T19:48:40.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5f9fe00640 1 --2- 172.21.15.29:0/3102541184 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5fa8071ce0 0x7f5f8c004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:40.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5fada00640 1 --2- 172.21.15.29:0/3102541184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f8c004d50 0x7f5f8c005160 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8071ae0 tx=0x7f5fa001d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e64d379615db50b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5fa7e00640 1 -- 172.21.15.29:0/3102541184 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa000a480 con 0x7f5f8c004d50 2024-04-17T19:48:40.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5fa7e00640 1 -- 172.21.15.29:0/3102541184 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa00ad630 con 0x7f5f8c004d50 2024-04-17T19:48:40.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.092+0000 7f5fa7e00640 1 -- 172.21.15.29:0/3102541184 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa00ada60 con 0x7f5f8c004d50 2024-04-17T19:48:40.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.096+0000 7f5fafa00640 1 -- 172.21.15.29:0/3102541184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f8c004d50 msgr2=0x7f5f8c005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.096+0000 7f5fafa00640 1 --2- 172.21.15.29:0/3102541184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f8c004d50 0x7f5f8c005160 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8071ae0 tx=0x7f5fa001d320 comp rx=0 tx=0).stop 2024-04-17T19:48:40.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.096+0000 7f5fada00640 1 -- 172.21.15.29:0/3102541184 reap_dead start 2024-04-17T19:48:40.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.096+0000 7f5fafa00640 1 -- 172.21.15.29:0/3102541184 shutdown_connections 2024-04-17T19:48:40.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.096+0000 7f5fafa00640 1 -- 172.21.15.29:0/3102541184 >> 172.21.15.29:0/3102541184 conn(0x7f5fa806bf90 msgr2=0x7f5fa806c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.096+0000 7f5fafa00640 1 -- 172.21.15.29:0/3102541184 shutdown_connections 2024-04-17T19:48:40.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.096+0000 7f5fafa00640 1 -- 172.21.15.29:0/3102541184 wait complete. 2024-04-17T19:48:40.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.096+0000 7f5fafa00640 1 Processor -- start 2024-04-17T19:48:40.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.096+0000 7f5fafa00640 1 -- start start 2024-04-17T19:48:40.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5fafa00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f8c004d50 0x7f5fa81f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5fafa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fa8153f80 0x7f5fa8155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5fafa00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5fa813ebf0 0x7f5fa8158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5fafa00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa8158eb0 con 0x7f5fa8153f80 2024-04-17T19:48:40.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5fafa00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa815cf70 con 0x7f5fa813ebf0 2024-04-17T19:48:40.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5fafa00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5fa8159bf0 con 0x7f5f8c004d50 2024-04-17T19:48:40.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5f9fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fa8153f80 0x7f5fa8155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5f9fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fa8153f80 0x7f5fa8155840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33108/0 (socket says 172.21.15.29:33108) 2024-04-17T19:48:40.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5f9fe00640 1 -- 172.21.15.29:0/2059133774 learned_addr learned my addr 172.21.15.29:0/2059133774 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5fada00640 1 --2- 172.21.15.29:0/2059133774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5fa813ebf0 0x7f5fa8158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5fad000640 1 --2- 172.21.15.29:0/2059133774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f8c004d50 0x7f5fa81f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5f9fe00640 1 -- 172.21.15.29:0/2059133774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f8c004d50 msgr2=0x7f5fa81f1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5f9fe00640 1 --2- 172.21.15.29:0/2059133774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f8c004d50 0x7f5fa81f1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5f9fe00640 1 -- 172.21.15.29:0/2059133774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5fa813ebf0 msgr2=0x7f5fa8158240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5f9fe00640 1 --2- 172.21.15.29:0/2059133774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5fa813ebf0 0x7f5fa8158240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5f9fe00640 1 -- 172.21.15.29:0/2059133774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa8157520 con 0x7f5fa8153f80 2024-04-17T19:48:40.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5f9fe00640 1 --2- 172.21.15.29:0/2059133774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fa8153f80 0x7f5fa8155840 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000e3c0 tx=0x7f5f9000e880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5fa5600640 1 -- 172.21.15.29:0/2059133774 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f90019070 con 0x7f5fa8153f80 2024-04-17T19:48:40.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5fa5600640 1 -- 172.21.15.29:0/2059133774 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f900052e0 con 0x7f5fa8153f80 2024-04-17T19:48:40.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5fa5600640 1 -- 172.21.15.29:0/2059133774 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f90003800 con 0x7f5fa8153f80 2024-04-17T19:48:40.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5fafa00640 1 -- 172.21.15.29:0/2059133774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fa8156760 con 0x7f5fa8153f80 2024-04-17T19:48:40.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.100+0000 7f5fafa00640 1 -- 172.21.15.29:0/2059133774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fa8156d20 con 0x7f5fa8153f80 2024-04-17T19:48:40.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.104+0000 7f5fa5600640 1 -- 172.21.15.29:0/2059133774 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5f9001c070 con 0x7f5fa8153f80 2024-04-17T19:48:40.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.104+0000 7f5fa5600640 1 --2- 172.21.15.29:0/2059133774 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5f840683b0 0x7f5f8406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.104+0000 7f5fa5600640 1 -- 172.21.15.29:0/2059133774 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5f900862b0 con 0x7f5fa8153f80 2024-04-17T19:48:40.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.104+0000 7f5fad000640 1 --2- 172.21.15.29:0/2059133774 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5f840683b0 0x7f5f8406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.104+0000 7f5fb6fa9000 1 Processor -- start 2024-04-17T19:48:40.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.104+0000 7f5fb6fa9000 1 -- start start 2024-04-17T19:48:40.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.104+0000 7f5fad000640 1 --2- 172.21.15.29:0/2059133774 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5f840683b0 0x7f5f8406a8c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5f98009aa0 tx=0x7f5f98008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.104+0000 7f5fb6fa9000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b8c10 0x55bcfb2221b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.104+0000 7f5fb6fa9000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfb0b1930 0x55bcfb2226f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.104+0000 7f5fb6fa9000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfacb1280 0x55bcfb222c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.104+0000 7f5fb6fa9000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bcfb02f4f0 con 0x55bcfacb1280 2024-04-17T19:48:40.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.104+0000 7f5fb6fa9000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x55bcfafc9c10 con 0x55bcfb0b1930 2024-04-17T19:48:40.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.104+0000 7f5fb6fa9000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x55bcfb02b4b0 con 0x55bcfb0b8c10 2024-04-17T19:48:40.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.104+0000 7f5fada00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfacb1280 0x55bcfb222c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.104+0000 7f5fada00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfacb1280 0x55bcfb222c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33120/0 (socket says 172.21.15.29:33120) 2024-04-17T19:48:40.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.104+0000 7f5fada00640 1 -- 172.21.15.29:0/4160495983 learned_addr learned my addr 172.21.15.29:0/4160495983 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.104+0000 7f5f9fe00640 1 --2- 172.21.15.29:0/4160495983 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfb0b1930 0x55bcfb2226f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.104+0000 7f5fad000640 1 --2- 172.21.15.29:0/4160495983 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b8c10 0x55bcfb2221b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5fada00640 1 -- 172.21.15.29:0/4160495983 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b8c10 msgr2=0x55bcfb2221b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5fada00640 1 --2- 172.21.15.29:0/4160495983 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b8c10 0x55bcfb2221b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5fada00640 1 -- 172.21.15.29:0/4160495983 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfb0b1930 msgr2=0x55bcfb2226f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5fada00640 1 --2- 172.21.15.29:0/4160495983 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfb0b1930 0x55bcfb2226f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5fada00640 1 -- 172.21.15.29:0/4160495983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcfafc9e60 con 0x55bcfacb1280 2024-04-17T19:48:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5fada00640 1 --2- 172.21.15.29:0/4160495983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfacb1280 0x55bcfb222c30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5fa001d800 tx=0x7f5fa00a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e0751dc6d03ece1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5f9d600640 1 -- 172.21.15.29:0/4160495983 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa00bd070 con 0x55bcfacb1280 2024-04-17T19:48:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5f9d600640 1 -- 172.21.15.29:0/4160495983 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa00011d0 con 0x55bcfacb1280 2024-04-17T19:48:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5f9d600640 1 -- 172.21.15.29:0/4160495983 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa000a480 con 0x55bcfacb1280 2024-04-17T19:48:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/4160495983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfacb1280 msgr2=0x55bcfb222c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/4160495983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfacb1280 0x55bcfb222c30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5fa001d800 tx=0x7f5fa00a7000 comp rx=0 tx=0).stop 2024-04-17T19:48:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/4160495983 shutdown_connections 2024-04-17T19:48:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/4160495983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfacb1280 0x55bcfb222c30 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/4160495983 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfb0b1930 0x55bcfb2226f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/4160495983 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b8c10 0x55bcfb2221b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/4160495983 >> 172.21.15.29:0/4160495983 conn(0x55bcfb0fa670 msgr2=0x55bcfb21ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/4160495983 shutdown_connections 2024-04-17T19:48:40.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/4160495983 wait complete. 2024-04-17T19:48:40.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5fb6fa9000 1 -- start start 2024-04-17T19:48:40.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.108+0000 7f5fb6fa9000 1 Processor -- start 2024-04-17T19:48:40.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fb6fa9000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfacb1280 0x55bcfb21bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fb6fa9000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfb0b1930 0x55bcfb21c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fb6fa9000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b8c10 0x55bcfb1e3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fb6fa9000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bcfb0709e0 con 0x55bcfacb1280 2024-04-17T19:48:40.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fb6fa9000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x55bcfafeca80 con 0x55bcfb0b1930 2024-04-17T19:48:40.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fb6fa9000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x55bcfafead10 con 0x55bcfb0b8c10 2024-04-17T19:48:40.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fb6fa9000 10 client.0 _ll_register_callbacks cb 0x55bcfb1ccb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T19:48:40.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fad000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfacb1280 0x55bcfb21bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fad000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfacb1280 0x55bcfb21bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33128/0 (socket says 172.21.15.29:33128) 2024-04-17T19:48:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fad000640 1 -- 172.21.15.29:0/889483309 learned_addr learned my addr 172.21.15.29:0/889483309 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5f9fe00640 1 --2- 172.21.15.29:0/889483309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfb0b1930 0x55bcfb21c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fada00640 1 --2- 172.21.15.29:0/889483309 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b8c10 0x55bcfb1e3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fad000640 1 -- 172.21.15.29:0/889483309 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b8c10 msgr2=0x55bcfb1e3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fad000640 1 --2- 172.21.15.29:0/889483309 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b8c10 0x55bcfb1e3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fad000640 1 -- 172.21.15.29:0/889483309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfb0b1930 msgr2=0x55bcfb21c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fad000640 1 --2- 172.21.15.29:0/889483309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfb0b1930 0x55bcfb21c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fad000640 1 -- 172.21.15.29:0/889483309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcfaffae20 con 0x55bcfacb1280 2024-04-17T19:48:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5f9fe00640 1 --2- 172.21.15.29:0/889483309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfb0b1930 0x55bcfb21c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fad000640 1 --2- 172.21.15.29:0/889483309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfacb1280 0x55bcfb21bf10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5f98016a90 tx=0x7f5f98004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f19a84ba6899975f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5f96000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-17T19:48:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5f96000640 1 -- 172.21.15.29:0/889483309 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f98023c10 con 0x55bcfacb1280 2024-04-17T19:48:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5f96000640 1 -- 172.21.15.29:0/889483309 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5f98023db0 con 0x55bcfacb1280 2024-04-17T19:48:40.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/889483309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fa00011d0 con 0x55bcfacb1280 2024-04-17T19:48:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5f96000640 1 -- 172.21.15.29:0/889483309 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f5f98036050 con 0x55bcfacb1280 2024-04-17T19:48:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fb6fa9000 10 client.4893 fetch_fsmap learned FSMap version 19 2024-04-17T19:48:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/889483309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x55bcfb0faa80 con 0x55bcfacb1280 2024-04-17T19:48:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5f96000640 1 -- 172.21.15.29:0/889483309 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa00011d0 con 0x55bcfacb1280 2024-04-17T19:48:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5f96000640 1 -- 172.21.15.29:0/889483309 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7f5f98039020 con 0x55bcfacb1280 2024-04-17T19:48:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fb6fa9000 10 client.4893 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T19:48:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fb6fa9000 10 client.4893 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-17T19:48:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fb6fa9000 20 client.4893 populate_metadata read hostname 'smithi029' 2024-04-17T19:48:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/889483309 >> [v2:172.21.15.37:6836/1100598562,v1:172.21.15.37:6839/1100598562] conn(0x55bcfacd57d0 0x55bcfb1e4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fb6fa9000 4 client.4893 mds_command: new command op to 4229 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T19:48:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/889483309 --> [v2:172.21.15.37:6836/1100598562,v1:172.21.15.37:6839/1100598562] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bcfafc8be0 con 0x55bcfacd57d0 2024-04-17T19:48:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.112+0000 7f5f9fe00640 1 --2- 172.21.15.29:0/889483309 >> [v2:172.21.15.37:6836/1100598562,v1:172.21.15.37:6839/1100598562] conn(0x55bcfacd57d0 0x55bcfb1e4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.116+0000 7f5f96000640 10 client.4893 ms_handle_connect on v2:172.21.15.37:6836/1100598562 2024-04-17T19:48:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.116+0000 7f5f9fe00640 1 --2- 172.21.15.29:0/889483309 >> [v2:172.21.15.37:6836/1100598562,v1:172.21.15.37:6839/1100598562] conn(0x55bcfacd57d0 0x55bcfb1e4900 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=23ea42c7c1e40c4e server_cookie=94dce46390bb5380 in_seq=0 out_seq=0 2024-04-17T19:48:40.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.116+0000 7f5f96000640 1 -- 172.21.15.29:0/889483309 <== mds.3 v2:172.21.15.37:6836/1100598562 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55bcfafc8be0 con 0x55bcfacd57d0 2024-04-17T19:48:40.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.116+0000 7f5f96000640 10 client.4893 handle_command_reply: tid=0 2024-04-17T19:48:40.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.116+0000 7f5fb6fa9000 1 client.4893 shutdown 2024-04-17T19:48:40.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.116+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/889483309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfacb1280 msgr2=0x55bcfb21bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.116+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/889483309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfacb1280 0x55bcfb21bf10 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5f98016a90 tx=0x7f5f98004740 comp rx=0 tx=0).stop 2024-04-17T19:48:40.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.116+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/889483309 shutdown_connections 2024-04-17T19:48:40.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.116+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/889483309 >> [v2:172.21.15.37:6836/1100598562,v1:172.21.15.37:6839/1100598562] conn(0x55bcfacd57d0 0x55bcfb1e4900 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.116+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/889483309 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b8c10 0x55bcfb1e3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.116+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/889483309 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfb0b1930 0x55bcfb21c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.116+0000 7f5f96000640 0 client.4893 ms_handle_reset on v2:172.21.15.37:6836/1100598562 2024-04-17T19:48:40.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.116+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/889483309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfacb1280 0x55bcfb21bf10 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.116+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/889483309 >> 172.21.15.29:0/889483309 conn(0x55bcfb0fa670 msgr2=0x55bcfb21d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.116+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/889483309 shutdown_connections 2024-04-17T19:48:40.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.116+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/889483309 wait complete. 2024-04-17T19:48:40.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.116+0000 7f5fb6fa9000 20 client.4893 trim_cache size 0 max 16384 2024-04-17T19:48:40.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.136+0000 7f5fb6fa9000 1 Processor -- start 2024-04-17T19:48:40.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.136+0000 7f5fb6fa9000 1 -- start start 2024-04-17T19:48:40.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.136+0000 7f5fb6fa9000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfacb1280 0x55bcfb1df680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.136+0000 7f5fb6fa9000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b1930 0x55bcfb1dfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.136+0000 7f5fb6fa9000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfb0b8c10 0x55bcfb23d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.136+0000 7f5fb6fa9000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa81fed10 con 0x55bcfb0b8c10 2024-04-17T19:48:40.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.136+0000 7f5fb6fa9000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x55bcfb02e800 con 0x55bcfacb1280 2024-04-17T19:48:40.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.136+0000 7f5fb6fa9000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x55bcfb033b20 con 0x55bcfb0b1930 2024-04-17T19:48:40.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.136+0000 7f5fada00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfb0b8c10 0x55bcfb23d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.136+0000 7f5fada00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfb0b8c10 0x55bcfb23d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33134/0 (socket says 172.21.15.29:33134) 2024-04-17T19:48:40.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.136+0000 7f5fada00640 1 -- 172.21.15.29:0/1611509027 learned_addr learned my addr 172.21.15.29:0/1611509027 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.136+0000 7f5fad000640 1 --2- 172.21.15.29:0/1611509027 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfacb1280 0x55bcfb1df680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.136+0000 7f5f9fe00640 1 --2- 172.21.15.29:0/1611509027 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b1930 0x55bcfb1dfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.136+0000 7f5fada00640 1 -- 172.21.15.29:0/1611509027 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b1930 msgr2=0x55bcfb1dfbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.136+0000 7f5fada00640 1 --2- 172.21.15.29:0/1611509027 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b1930 0x55bcfb1dfbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.136+0000 7f5fada00640 1 -- 172.21.15.29:0/1611509027 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfacb1280 msgr2=0x55bcfb1df680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.136+0000 7f5fada00640 1 --2- 172.21.15.29:0/1611509027 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfacb1280 0x55bcfb1df680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.136+0000 7f5fada00640 1 -- 172.21.15.29:0/1611509027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcfb088e30 con 0x55bcfb0b8c10 2024-04-17T19:48:40.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fada00640 1 --2- 172.21.15.29:0/1611509027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfb0b8c10 0x55bcfb23d810 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5fa001d800 tx=0x7f5fa815cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35cbccb056a83700 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5f9cc00640 1 -- 172.21.15.29:0/1611509027 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa00c7070 con 0x55bcfb0b8c10 2024-04-17T19:48:40.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5f9cc00640 1 -- 172.21.15.29:0/1611509027 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa0004b70 con 0x55bcfb0b8c10 2024-04-17T19:48:40.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5f9cc00640 1 -- 172.21.15.29:0/1611509027 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fa00bd040 con 0x55bcfb0b8c10 2024-04-17T19:48:40.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/1611509027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfb0b8c10 msgr2=0x55bcfb23d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/1611509027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfb0b8c10 0x55bcfb23d810 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5fa001d800 tx=0x7f5fa815cb80 comp rx=0 tx=0).stop 2024-04-17T19:48:40.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/1611509027 shutdown_connections 2024-04-17T19:48:40.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/1611509027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfb0b8c10 0x55bcfb23d810 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/1611509027 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b1930 0x55bcfb1dfbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/1611509027 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfacb1280 0x55bcfb1df680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/1611509027 >> 172.21.15.29:0/1611509027 conn(0x55bcfb0fa670 msgr2=0x55bcfb1def70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/1611509027 shutdown_connections 2024-04-17T19:48:40.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/1611509027 wait complete. 2024-04-17T19:48:40.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fb6fa9000 1 -- start start 2024-04-17T19:48:40.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fb6fa9000 1 Processor -- start 2024-04-17T19:48:40.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fb6fa9000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfacb1280 0x55bcfb2938e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fb6fa9000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfb0b1930 0x55bcfb293e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fb6fa9000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b8c10 0x55bcfb23b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fb6fa9000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55bcfafc57d0 con 0x55bcfb0b1930 2024-04-17T19:48:40.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fb6fa9000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x55bcfb03c700 con 0x55bcfacb1280 2024-04-17T19:48:40.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fb6fa9000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x55bcfaff5b10 con 0x55bcfb0b8c10 2024-04-17T19:48:40.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.140+0000 7f5fb6fa9000 10 client.0 _ll_register_callbacks cb 0x7f5f98014a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T19:48:40.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f9fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfb0b1930 0x55bcfb293e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f9fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfb0b1930 0x55bcfb293e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33142/0 (socket says 172.21.15.29:33142) 2024-04-17T19:48:40.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f9fe00640 1 -- 172.21.15.29:0/2742985698 learned_addr learned my addr 172.21.15.29:0/2742985698 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fad000640 1 --2- 172.21.15.29:0/2742985698 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfacb1280 0x55bcfb2938e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fada00640 1 --2- 172.21.15.29:0/2742985698 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b8c10 0x55bcfb23b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f9fe00640 1 -- 172.21.15.29:0/2742985698 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b8c10 msgr2=0x55bcfb23b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f9fe00640 1 --2- 172.21.15.29:0/2742985698 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b8c10 0x55bcfb23b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f9fe00640 1 -- 172.21.15.29:0/2742985698 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfacb1280 msgr2=0x55bcfb2938e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f9fe00640 1 --2- 172.21.15.29:0/2742985698 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfacb1280 0x55bcfb2938e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f9fe00640 1 -- 172.21.15.29:0/2742985698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcfafc17e0 con 0x55bcfb0b1930 2024-04-17T19:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f9fe00640 1 --2- 172.21.15.29:0/2742985698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfb0b1930 0x55bcfb293e20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000ed80 tx=0x7f5f90055970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5273149b38d0e29f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f96000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-17T19:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f96000640 1 -- 172.21.15.29:0/2742985698 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f90081070 con 0x55bcfb0b1930 2024-04-17T19:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f96000640 1 -- 172.21.15.29:0/2742985698 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5f9006e1c0 con 0x55bcfb0b1930 2024-04-17T19:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/2742985698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5fa0004b70 con 0x55bcfb0b1930 2024-04-17T19:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f96000640 1 -- 172.21.15.29:0/2742985698 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f5f900523f0 con 0x55bcfb0b1930 2024-04-17T19:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 10 client.4899 fetch_fsmap learned FSMap version 19 2024-04-17T19:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/2742985698 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x55bcfafd4670 con 0x55bcfb0b1930 2024-04-17T19:48:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f96000640 1 -- 172.21.15.29:0/2742985698 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fa0004b70 con 0x55bcfb0b1930 2024-04-17T19:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f96000640 1 -- 172.21.15.29:0/2742985698 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7f5f90074b60 con 0x55bcfb0b1930 2024-04-17T19:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 10 client.4899 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T19:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 10 client.4899 resolve_mds: resolved name 'f' to daemon mds.f 2024-04-17T19:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 20 client.4899 populate_metadata read hostname 'smithi029' 2024-04-17T19:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/2742985698 >> [v2:172.21.15.37:6836/1100598562,v1:172.21.15.37:6839/1100598562] conn(0x55bcfacc3830 0x55bcfb23c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 4 client.4899 mds_command: new command op to 4229 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T19:48:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/2742985698 --> [v2:172.21.15.37:6836/1100598562,v1:172.21.15.37:6839/1100598562] -- command(tid 0: {"prefix": "status"}) -- 0x55bcfb03cf10 con 0x55bcfacc3830 2024-04-17T19:48:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fad000640 1 --2- 172.21.15.29:0/2742985698 >> [v2:172.21.15.37:6836/1100598562,v1:172.21.15.37:6839/1100598562] conn(0x55bcfacc3830 0x55bcfb23c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fad000640 1 --2- 172.21.15.29:0/2742985698 >> [v2:172.21.15.37:6836/1100598562,v1:172.21.15.37:6839/1100598562] conn(0x55bcfacc3830 0x55bcfb23c220 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6729f33a4998aed4 server_cookie=aecf176b80db8813 in_seq=0 out_seq=0 2024-04-17T19:48:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f96000640 10 client.4899 ms_handle_connect on v2:172.21.15.37:6836/1100598562 2024-04-17T19:48:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f96000640 1 -- 172.21.15.29:0/2742985698 <== mds.3 v2:172.21.15.37:6836/1100598562 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55bcfb03cf10 con 0x55bcfacc3830 2024-04-17T19:48:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f96000640 10 client.4899 handle_command_reply: tid=0 2024-04-17T19:48:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 1 client.4899 shutdown 2024-04-17T19:48:40.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/2742985698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfb0b1930 msgr2=0x55bcfb293e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/2742985698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfb0b1930 0x55bcfb293e20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000ed80 tx=0x7f5f90055970 comp rx=0 tx=0).stop 2024-04-17T19:48:40.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/2742985698 shutdown_connections 2024-04-17T19:48:40.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/2742985698 >> [v2:172.21.15.37:6836/1100598562,v1:172.21.15.37:6839/1100598562] conn(0x55bcfacc3830 0x55bcfb23c220 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/2742985698 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55bcfb0b8c10 0x55bcfb23b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/2742985698 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55bcfb0b1930 0x55bcfb293e20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 1 --2- 172.21.15.29:0/2742985698 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55bcfacb1280 0x55bcfb2938e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5f96000640 0 client.4899 ms_handle_reset on v2:172.21.15.37:6836/1100598562 2024-04-17T19:48:40.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/2742985698 >> 172.21.15.29:0/2742985698 conn(0x55bcfb0fa670 msgr2=0x55bcfb1df9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/2742985698 shutdown_connections 2024-04-17T19:48:40.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 1 -- 172.21.15.29:0/2742985698 wait complete. 2024-04-17T19:48:40.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.144+0000 7f5fb6fa9000 20 client.4899 trim_cache size 0 max 16384 2024-04-17T19:48:40.138 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-17T19:48:40.138 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "a680ee90-875c-4283-a901-86d7638393c3", 2024-04-17T19:48:40.138 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 3, 2024-04-17T19:48:40.138 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 4229, 2024-04-17T19:48:40.138 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-04-17T19:48:40.138 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-04-17T19:48:40.138 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-04-17T19:48:40.138 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 17.338653605000001, 2024-04-17T19:48:40.138 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 19, 2024-04-17T19:48:40.138 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 26, 2024-04-17T19:48:40.138 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 26, 2024-04-17T19:48:40.138 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 57.732142652999997 2024-04-17T19:48:40.138 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-17T19:48:40.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.148+0000 7f5fafa00640 1 -- 172.21.15.29:0/2059133774 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5f840683b0 msgr2=0x7f5f8406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.148+0000 7f5fafa00640 1 --2- 172.21.15.29:0/2059133774 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5f840683b0 0x7f5f8406a8c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5f98009aa0 tx=0x7f5f98008040 comp rx=0 tx=0).stop 2024-04-17T19:48:40.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.148+0000 7f5fafa00640 1 -- 172.21.15.29:0/2059133774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fa8153f80 msgr2=0x7f5fa8155840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.148+0000 7f5fafa00640 1 --2- 172.21.15.29:0/2059133774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fa8153f80 0x7f5fa8155840 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000e3c0 tx=0x7f5f9000e880 comp rx=0 tx=0).stop 2024-04-17T19:48:40.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.148+0000 7f5fafa00640 1 -- 172.21.15.29:0/2059133774 shutdown_connections 2024-04-17T19:48:40.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.148+0000 7f5fafa00640 1 --2- 172.21.15.29:0/2059133774 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5f840683b0 0x7f5f8406a8c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.148+0000 7f5fafa00640 1 --2- 172.21.15.29:0/2059133774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5fa813ebf0 0x7f5fa8158240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.148+0000 7f5fafa00640 1 --2- 172.21.15.29:0/2059133774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fa8153f80 0x7f5fa8155840 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.148+0000 7f5fafa00640 1 --2- 172.21.15.29:0/2059133774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f8c004d50 0x7f5fa81f1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.148+0000 7f5fafa00640 1 -- 172.21.15.29:0/2059133774 >> 172.21.15.29:0/2059133774 conn(0x7f5fa806bf90 msgr2=0x7f5fa8073a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.148+0000 7f5fafa00640 1 -- 172.21.15.29:0/2059133774 shutdown_connections 2024-04-17T19:48:40.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.148+0000 7f5fafa00640 1 -- 172.21.15.29:0/2059133774 wait complete. 2024-04-17T19:48:40.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-17T19:48:40.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.320+0000 7fa597a00640 1 Processor -- start 2024-04-17T19:48:40.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.320+0000 7fa597a00640 1 -- start start 2024-04-17T19:48:40.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.320+0000 7fa597a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5900c91e0 0x7fa59015cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.320+0000 7fa597a00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa5900700d0 0x7fa590155b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.320+0000 7fa597a00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa59015d060 con 0x7fa5900704e0 2024-04-17T19:48:40.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.320+0000 7fa597a00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fa5900744a0 con 0x7fa5900c91e0 2024-04-17T19:48:40.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.320+0000 7fa597a00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa5900c95f0 con 0x7fa5900700d0 2024-04-17T19:48:40.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.320+0000 7fa595000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5900c91e0 0x7fa59015cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.320+0000 7fa595a00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa5900704e0 0x7fa59015f690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52220/0 (socket says 172.21.15.29:52220) 2024-04-17T19:48:40.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.320+0000 7fa595a00640 1 -- 172.21.15.29:0/558439741 learned_addr learned my addr 172.21.15.29:0/558439741 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:40.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.320+0000 7fa58fe00640 1 --2- 172.21.15.29:0/558439741 >> v2:172.21.15.37:3300/0 conn(0x7fa5900700d0 0x7fa590155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.320+0000 7fa595000640 1 -- 172.21.15.29:0/558439741 >> v2:172.21.15.37:3300/0 conn(0x7fa5900700d0 msgr2=0x7fa590155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.320+0000 7fa595000640 1 --2- 172.21.15.29:0/558439741 >> v2:172.21.15.37:3300/0 conn(0x7fa5900700d0 0x7fa590155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.320+0000 7fa595000640 1 -- 172.21.15.29:0/558439741 >> v1:172.21.15.29:6789/0 conn(0x7fa5900704e0 legacy=0x7fa59015f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.320+0000 7fa595000640 1 -- 172.21.15.29:0/558439741 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5900cafd0 con 0x7fa5900c91e0 2024-04-17T19:48:40.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.324+0000 7fa595000640 1 --2- 172.21.15.29:0/558439741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5900c91e0 0x7fa59015cb20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa5900c4a30 tx=0x7fa58401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3b474d6aa2b53e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.324+0000 7fa58f400640 1 -- 172.21.15.29:0/558439741 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5840b0070 con 0x7fa5900c91e0 2024-04-17T19:48:40.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.324+0000 7fa58f400640 1 -- 172.21.15.29:0/558439741 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5840ab600 con 0x7fa5900c91e0 2024-04-17T19:48:40.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.324+0000 7fa58f400640 1 -- 172.21.15.29:0/558439741 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5840ab9d0 con 0x7fa5900c91e0 2024-04-17T19:48:40.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.324+0000 7fa597a00640 1 -- 172.21.15.29:0/558439741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5900c91e0 msgr2=0x7fa59015cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.324+0000 7fa597a00640 1 --2- 172.21.15.29:0/558439741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5900c91e0 0x7fa59015cb20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa5900c4a30 tx=0x7fa58401c480 comp rx=0 tx=0).stop 2024-04-17T19:48:40.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.324+0000 7fa597a00640 1 -- 172.21.15.29:0/558439741 shutdown_connections 2024-04-17T19:48:40.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.324+0000 7fa597a00640 1 --2- 172.21.15.29:0/558439741 >> v2:172.21.15.37:3300/0 conn(0x7fa5900700d0 0x7fa590155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.324+0000 7fa597a00640 1 --2- 172.21.15.29:0/558439741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5900c91e0 0x7fa59015cb20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.324+0000 7fa597a00640 1 -- 172.21.15.29:0/558439741 >> 172.21.15.29:0/558439741 conn(0x7fa59006c420 msgr2=0x7fa5900ca380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.324+0000 7fa597a00640 1 -- 172.21.15.29:0/558439741 shutdown_connections 2024-04-17T19:48:40.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.324+0000 7fa597a00640 1 -- 172.21.15.29:0/558439741 wait complete. 2024-04-17T19:48:40.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa597a00640 1 Processor -- start 2024-04-17T19:48:40.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa597a00640 1 -- start start 2024-04-17T19:48:40.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa597a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5900700d0 0x7fa5900c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa597a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa5900704e0 0x7fa590147440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa597a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa5900c91e0 0x7fa59015e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa597a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa5901564c0 con 0x7fa5900700d0 2024-04-17T19:48:40.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa597a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa5900c8cc0 con 0x7fa5900704e0 2024-04-17T19:48:40.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa597a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa5900c9f60 con 0x7fa5900c91e0 2024-04-17T19:48:40.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa595000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5900700d0 0x7fa5900c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa595000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5900700d0 0x7fa5900c6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33158/0 (socket says 172.21.15.29:33158) 2024-04-17T19:48:40.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa595000640 1 -- 172.21.15.29:0/4013137318 learned_addr learned my addr 172.21.15.29:0/4013137318 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa595a00640 1 --2- 172.21.15.29:0/4013137318 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa5900c91e0 0x7fa59015e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa58fe00640 1 --2- 172.21.15.29:0/4013137318 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa5900704e0 0x7fa590147440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa595000640 1 -- 172.21.15.29:0/4013137318 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa5900c91e0 msgr2=0x7fa59015e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa595000640 1 --2- 172.21.15.29:0/4013137318 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa5900c91e0 0x7fa59015e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa595000640 1 -- 172.21.15.29:0/4013137318 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa5900704e0 msgr2=0x7fa590147440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa595000640 1 --2- 172.21.15.29:0/4013137318 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa5900704e0 0x7fa590147440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa595000640 1 -- 172.21.15.29:0/4013137318 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5900c7500 con 0x7fa5900700d0 2024-04-17T19:48:40.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa595000640 1 --2- 172.21.15.29:0/4013137318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5900700d0 0x7fa5900c6fc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa5840ae040 tx=0x7fa584019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa58cc00640 1 -- 172.21.15.29:0/4013137318 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5840b0040 con 0x7fa5900700d0 2024-04-17T19:48:40.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa58cc00640 1 -- 172.21.15.29:0/4013137318 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5840ba070 con 0x7fa5900700d0 2024-04-17T19:48:40.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa58cc00640 1 -- 172.21.15.29:0/4013137318 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5840b4650 con 0x7fa5900700d0 2024-04-17T19:48:40.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.328+0000 7fa597a00640 1 -- 172.21.15.29:0/4013137318 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5900c7710 con 0x7fa5900700d0 2024-04-17T19:48:40.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.332+0000 7fa597a00640 1 -- 172.21.15.29:0/4013137318 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5900c64e0 con 0x7fa5900700d0 2024-04-17T19:48:40.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.332+0000 7fa58cc00640 1 -- 172.21.15.29:0/4013137318 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa5840b47f0 con 0x7fa5900700d0 2024-04-17T19:48:40.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.332+0000 7fa58cc00640 1 --2- 172.21.15.29:0/4013137318 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa570068380 0x7fa57006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.332+0000 7fa58cc00640 1 -- 172.21.15.29:0/4013137318 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa584120ad0 con 0x7fa5900700d0 2024-04-17T19:48:40.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.332+0000 7fa58fe00640 1 --2- 172.21.15.29:0/4013137318 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa570068380 0x7fa57006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa59f08e000 1 Processor -- start 2024-04-17T19:48:40.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa59f08e000 1 -- start start 2024-04-17T19:48:40.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa58fe00640 1 --2- 172.21.15.29:0/4013137318 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa570068380 0x7fa57006a890 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa59015f0f0 tx=0x7fa57c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa59f08e000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b81c6c10 0x5572b83301b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa59f08e000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b81bf930 0x5572b83306f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa59f08e000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b7dbf280 0x5572b8330c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa59f08e000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5572b813d4f0 con 0x5572b81c6c10 2024-04-17T19:48:40.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa59f08e000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x5572b80d7c10 con 0x5572b7dbf280 2024-04-17T19:48:40.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa59f08e000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x5572b81394b0 con 0x5572b81bf930 2024-04-17T19:48:40.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa595000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b81c6c10 0x5572b83301b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa595000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b81c6c10 0x5572b83301b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33164/0 (socket says 172.21.15.29:33164) 2024-04-17T19:48:40.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa595000640 1 -- 172.21.15.29:0/613318753 learned_addr learned my addr 172.21.15.29:0/613318753 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa595a00640 1 --2- 172.21.15.29:0/613318753 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b7dbf280 0x5572b8330c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa58fe00640 1 --2- 172.21.15.29:0/613318753 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b81bf930 0x5572b83306f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa595000640 1 -- 172.21.15.29:0/613318753 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b81bf930 msgr2=0x5572b83306f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa595000640 1 --2- 172.21.15.29:0/613318753 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b81bf930 0x5572b83306f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa595000640 1 -- 172.21.15.29:0/613318753 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b7dbf280 msgr2=0x5572b8330c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa595000640 1 --2- 172.21.15.29:0/613318753 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b7dbf280 0x5572b8330c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa595000640 1 -- 172.21.15.29:0/613318753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572b80d7e60 con 0x5572b81c6c10 2024-04-17T19:48:40.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa595000640 1 --2- 172.21.15.29:0/613318753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b81c6c10 0x5572b83301b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa5840ec950 tx=0x7fa5840ec8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4d38c9da61e9869 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa581600640 1 -- 172.21.15.29:0/613318753 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5840f48a0 con 0x5572b81c6c10 2024-04-17T19:48:40.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa581600640 1 -- 172.21.15.29:0/613318753 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5840f90d0 con 0x5572b81c6c10 2024-04-17T19:48:40.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa581600640 1 -- 172.21.15.29:0/613318753 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5840ecb60 con 0x5572b81c6c10 2024-04-17T19:48:40.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa59f08e000 1 -- 172.21.15.29:0/613318753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b81c6c10 msgr2=0x5572b83301b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa59f08e000 1 --2- 172.21.15.29:0/613318753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b81c6c10 0x5572b83301b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa5840ec950 tx=0x7fa5840ec8c0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa59f08e000 1 -- 172.21.15.29:0/613318753 shutdown_connections 2024-04-17T19:48:40.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa59f08e000 1 --2- 172.21.15.29:0/613318753 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b7dbf280 0x5572b8330c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa59f08e000 1 --2- 172.21.15.29:0/613318753 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b81bf930 0x5572b83306f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa59f08e000 1 --2- 172.21.15.29:0/613318753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b81c6c10 0x5572b83301b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.336+0000 7fa59f08e000 1 -- 172.21.15.29:0/613318753 >> 172.21.15.29:0/613318753 conn(0x5572b8208670 msgr2=0x5572b832df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa59f08e000 1 -- 172.21.15.29:0/613318753 shutdown_connections 2024-04-17T19:48:40.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa59f08e000 1 -- 172.21.15.29:0/613318753 wait complete. 2024-04-17T19:48:40.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa59f08e000 1 -- start start 2024-04-17T19:48:40.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa59f08e000 1 Processor -- start 2024-04-17T19:48:40.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa59f08e000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b7dbf280 0x5572b8329f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa59f08e000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b81bf930 0x5572b832a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa59f08e000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b81c6c10 0x5572b82f1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa59f08e000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5572b817e9e0 con 0x5572b7dbf280 2024-04-17T19:48:40.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa59f08e000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x5572b80faa80 con 0x5572b81c6c10 2024-04-17T19:48:40.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa59f08e000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x5572b80f8d10 con 0x5572b81bf930 2024-04-17T19:48:40.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa59f08e000 10 client.0 _ll_register_callbacks cb 0x5572b82dab20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T19:48:40.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa595000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b7dbf280 0x5572b8329f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa595000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b7dbf280 0x5572b8329f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33176/0 (socket says 172.21.15.29:33176) 2024-04-17T19:48:40.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa595000640 1 -- 172.21.15.29:0/2509062137 learned_addr learned my addr 172.21.15.29:0/2509062137 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa595a00640 1 --2- 172.21.15.29:0/2509062137 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b81c6c10 0x5572b82f1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa58fe00640 1 --2- 172.21.15.29:0/2509062137 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b81bf930 0x5572b832a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa595000640 1 -- 172.21.15.29:0/2509062137 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b81bf930 msgr2=0x5572b832a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa595000640 1 --2- 172.21.15.29:0/2509062137 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b81bf930 0x5572b832a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa595000640 1 -- 172.21.15.29:0/2509062137 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b81c6c10 msgr2=0x5572b82f1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa595000640 1 --2- 172.21.15.29:0/2509062137 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b81c6c10 0x5572b82f1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa595000640 1 -- 172.21.15.29:0/2509062137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572b8108e20 con 0x5572b7dbf280 2024-04-17T19:48:40.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa595000640 1 --2- 172.21.15.29:0/2509062137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b7dbf280 0x5572b8329f10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa58401c9f0 tx=0x7fa5840b4000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f01106e41f4112 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa576000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-17T19:48:40.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa576000640 1 -- 172.21.15.29:0/2509062137 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58411e060 con 0x5572b7dbf280 2024-04-17T19:48:40.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa576000640 1 -- 172.21.15.29:0/2509062137 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa5840f8e40 con 0x5572b7dbf280 2024-04-17T19:48:40.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.340+0000 7fa59f08e000 1 -- 172.21.15.29:0/2509062137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa5840f90d0 con 0x5572b7dbf280 2024-04-17T19:48:40.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa576000640 1 -- 172.21.15.29:0/2509062137 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa58410f050 con 0x5572b7dbf280 2024-04-17T19:48:40.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 10 client.4914 fetch_fsmap learned FSMap version 19 2024-04-17T19:48:40.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 1 -- 172.21.15.29:0/2509062137 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x5572b8208a80 con 0x5572b7dbf280 2024-04-17T19:48:40.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa576000640 1 -- 172.21.15.29:0/2509062137 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5840f90d0 con 0x5572b7dbf280 2024-04-17T19:48:40.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa576000640 1 -- 172.21.15.29:0/2509062137 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7fa5840b3050 con 0x5572b7dbf280 2024-04-17T19:48:40.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 10 client.4914 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T19:48:40.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 10 client.4914 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T19:48:40.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 20 client.4914 populate_metadata read hostname 'smithi029' 2024-04-17T19:48:40.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 1 --2- 172.21.15.29:0/2509062137 >> [v2:172.21.15.29:6832/4184160618,v1:172.21.15.29:6836/4184160618] conn(0x5572b7de37d0 0x5572b82f2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 4 client.4914 mds_command: new command op to 4259 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T19:48:40.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 1 -- 172.21.15.29:0/2509062137 --> [v2:172.21.15.29:6832/4184160618,v1:172.21.15.29:6836/4184160618] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572b80d6be0 con 0x5572b7de37d0 2024-04-17T19:48:40.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa58fe00640 1 --2- 172.21.15.29:0/2509062137 >> [v2:172.21.15.29:6832/4184160618,v1:172.21.15.29:6836/4184160618] conn(0x5572b7de37d0 0x5572b82f2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa58fe00640 1 --2- 172.21.15.29:0/2509062137 >> [v2:172.21.15.29:6832/4184160618,v1:172.21.15.29:6836/4184160618] conn(0x5572b7de37d0 0x5572b82f2900 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d4c6bff4b5b14691 server_cookie=be8cd044a34a5398 in_seq=0 out_seq=0 2024-04-17T19:48:40.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa576000640 10 client.4914 ms_handle_connect on v2:172.21.15.29:6832/4184160618 2024-04-17T19:48:40.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa576000640 1 -- 172.21.15.29:0/2509062137 <== mds.2 v2:172.21.15.29:6832/4184160618 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5572b80d6be0 con 0x5572b7de37d0 2024-04-17T19:48:40.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa576000640 10 client.4914 handle_command_reply: tid=0 2024-04-17T19:48:40.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 1 client.4914 shutdown 2024-04-17T19:48:40.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 1 -- 172.21.15.29:0/2509062137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b7dbf280 msgr2=0x5572b8329f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 1 --2- 172.21.15.29:0/2509062137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b7dbf280 0x5572b8329f10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa58401c9f0 tx=0x7fa5840b4000 comp rx=0 tx=0).stop 2024-04-17T19:48:40.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 1 -- 172.21.15.29:0/2509062137 shutdown_connections 2024-04-17T19:48:40.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 1 --2- 172.21.15.29:0/2509062137 >> [v2:172.21.15.29:6832/4184160618,v1:172.21.15.29:6836/4184160618] conn(0x5572b7de37d0 0x5572b82f2900 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 1 --2- 172.21.15.29:0/2509062137 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b81c6c10 0x5572b82f1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 1 --2- 172.21.15.29:0/2509062137 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b81bf930 0x5572b832a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa576000640 0 client.4914 ms_handle_reset on v2:172.21.15.29:6832/4184160618 2024-04-17T19:48:40.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 1 --2- 172.21.15.29:0/2509062137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b7dbf280 0x5572b8329f10 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 1 -- 172.21.15.29:0/2509062137 >> 172.21.15.29:0/2509062137 conn(0x5572b8208670 msgr2=0x5572b832b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 1 -- 172.21.15.29:0/2509062137 shutdown_connections 2024-04-17T19:48:40.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 1 -- 172.21.15.29:0/2509062137 wait complete. 2024-04-17T19:48:40.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.344+0000 7fa59f08e000 20 client.4914 trim_cache size 0 max 16384 2024-04-17T19:48:40.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa59f08e000 1 Processor -- start 2024-04-17T19:48:40.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa59f08e000 1 -- start start 2024-04-17T19:48:40.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa59f08e000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b8208670 0x5572b8328160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa59f08e000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b7dbf280 0x5572b83286a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa59f08e000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b81bf930 0x5572b8329010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa59f08e000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5572b814a700 con 0x5572b81bf930 2024-04-17T19:48:40.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa59f08e000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x5572b80d37d0 con 0x5572b7dbf280 2024-04-17T19:48:40.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa59f08e000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x5572b82f87b0 con 0x5572b8208670 2024-04-17T19:48:40.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa595a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b81bf930 0x5572b8329010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa595a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b81bf930 0x5572b8329010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33184/0 (socket says 172.21.15.29:33184) 2024-04-17T19:48:40.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa595a00640 1 -- 172.21.15.29:0/313546239 learned_addr learned my addr 172.21.15.29:0/313546239 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa595000640 1 --2- 172.21.15.29:0/313546239 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b8208670 0x5572b8328160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa58fe00640 1 --2- 172.21.15.29:0/313546239 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b7dbf280 0x5572b83286a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa595a00640 1 -- 172.21.15.29:0/313546239 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b8208670 msgr2=0x5572b8328160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa595a00640 1 --2- 172.21.15.29:0/313546239 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b8208670 0x5572b8328160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa595a00640 1 -- 172.21.15.29:0/313546239 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b7dbf280 msgr2=0x5572b83286a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa595a00640 1 --2- 172.21.15.29:0/313546239 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b7dbf280 0x5572b83286a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa595a00640 1 -- 172.21.15.29:0/313546239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572b8196e30 con 0x5572b81bf930 2024-04-17T19:48:40.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.364+0000 7fa595a00640 1 --2- 172.21.15.29:0/313546239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b81bf930 0x5572b8329010 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa588001f40 tx=0x7fa588007420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f9d9eb7ebfed93 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa580c00640 1 -- 172.21.15.29:0/313546239 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa588002030 con 0x5572b81bf930 2024-04-17T19:48:40.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa580c00640 1 -- 172.21.15.29:0/313546239 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa58801b070 con 0x5572b81bf930 2024-04-17T19:48:40.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa580c00640 1 -- 172.21.15.29:0/313546239 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa588016680 con 0x5572b81bf930 2024-04-17T19:48:40.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa59f08e000 1 -- 172.21.15.29:0/313546239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b81bf930 msgr2=0x5572b8329010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa59f08e000 1 --2- 172.21.15.29:0/313546239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b81bf930 0x5572b8329010 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa588001f40 tx=0x7fa588007420 comp rx=0 tx=0).stop 2024-04-17T19:48:40.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa59f08e000 1 -- 172.21.15.29:0/313546239 shutdown_connections 2024-04-17T19:48:40.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa59f08e000 1 --2- 172.21.15.29:0/313546239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b81bf930 0x5572b8329010 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa59f08e000 1 --2- 172.21.15.29:0/313546239 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b7dbf280 0x5572b83286a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa59f08e000 1 --2- 172.21.15.29:0/313546239 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b8208670 0x5572b8328160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa59f08e000 1 -- 172.21.15.29:0/313546239 >> 172.21.15.29:0/313546239 conn(0x5572b82f5640 msgr2=0x5572b834ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa59f08e000 1 -- 172.21.15.29:0/313546239 shutdown_connections 2024-04-17T19:48:40.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa59f08e000 1 -- 172.21.15.29:0/313546239 wait complete. 2024-04-17T19:48:40.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa59f08e000 1 -- start start 2024-04-17T19:48:40.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa59f08e000 1 Processor -- start 2024-04-17T19:48:40.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa59f08e000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b7dbf280 0x5572b83a5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa59f08e000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b81bf930 0x5572b83a5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa59f08e000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b8208670 0x5572b8349dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa59f08e000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5572b80faa80 con 0x5572b8208670 2024-04-17T19:48:40.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa59f08e000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x5572b817e9e0 con 0x5572b81bf930 2024-04-17T19:48:40.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa59f08e000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x5572b80f8d10 con 0x5572b7dbf280 2024-04-17T19:48:40.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa59f08e000 10 client.0 _ll_register_callbacks cb 0x7fa5840f0350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T19:48:40.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa595a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b8208670 0x5572b8349dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa595a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b8208670 0x5572b8349dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33198/0 (socket says 172.21.15.29:33198) 2024-04-17T19:48:40.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa595a00640 1 -- 172.21.15.29:0/3194320247 learned_addr learned my addr 172.21.15.29:0/3194320247 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa58fe00640 1 --2- 172.21.15.29:0/3194320247 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b81bf930 0x5572b83a5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.368+0000 7fa595000640 1 --2- 172.21.15.29:0/3194320247 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b7dbf280 0x5572b83a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa595a00640 1 -- 172.21.15.29:0/3194320247 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b7dbf280 msgr2=0x5572b83a5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa595a00640 1 --2- 172.21.15.29:0/3194320247 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b7dbf280 0x5572b83a5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa595a00640 1 -- 172.21.15.29:0/3194320247 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b81bf930 msgr2=0x5572b83a5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa595a00640 1 --2- 172.21.15.29:0/3194320247 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b81bf930 0x5572b83a5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa595a00640 1 -- 172.21.15.29:0/3194320247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572b80cf7e0 con 0x5572b8208670 2024-04-17T19:48:40.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa595a00640 1 --2- 172.21.15.29:0/3194320247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b8208670 0x5572b8349dd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa588005c10 tx=0x7fa588001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7e3bb6130cbcbf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa576000640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-17T19:48:40.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa576000640 1 -- 172.21.15.29:0/3194320247 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa588001be0 con 0x5572b8208670 2024-04-17T19:48:40.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa576000640 1 -- 172.21.15.29:0/3194320247 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa588016ca0 con 0x5572b8208670 2024-04-17T19:48:40.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 1 -- 172.21.15.29:0/3194320247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa58801b070 con 0x5572b8208670 2024-04-17T19:48:40.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa576000640 1 -- 172.21.15.29:0/3194320247 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fa588032050 con 0x5572b8208670 2024-04-17T19:48:40.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 10 client.4920 fetch_fsmap learned FSMap version 19 2024-04-17T19:48:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 1 -- 172.21.15.29:0/3194320247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x5572b80e2670 con 0x5572b8208670 2024-04-17T19:48:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa576000640 1 -- 172.21.15.29:0/3194320247 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa58801b070 con 0x5572b8208670 2024-04-17T19:48:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa576000640 1 -- 172.21.15.29:0/3194320247 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7fa588035020 con 0x5572b8208670 2024-04-17T19:48:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 10 client.4920 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T19:48:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 10 client.4920 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T19:48:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 20 client.4920 populate_metadata read hostname 'smithi029' 2024-04-17T19:48:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 1 --2- 172.21.15.29:0/3194320247 >> [v2:172.21.15.29:6832/4184160618,v1:172.21.15.29:6836/4184160618] conn(0x5572b7dfce80 0x5572b834a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 4 client.4920 mds_command: new command op to 4259 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T19:48:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 1 -- 172.21.15.29:0/3194320247 --> [v2:172.21.15.29:6832/4184160618,v1:172.21.15.29:6836/4184160618] -- command(tid 0: {"prefix": "status"}) -- 0x5572b814af10 con 0x5572b7dfce80 2024-04-17T19:48:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa595000640 1 --2- 172.21.15.29:0/3194320247 >> [v2:172.21.15.29:6832/4184160618,v1:172.21.15.29:6836/4184160618] conn(0x5572b7dfce80 0x5572b834a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa595000640 1 --2- 172.21.15.29:0/3194320247 >> [v2:172.21.15.29:6832/4184160618,v1:172.21.15.29:6836/4184160618] conn(0x5572b7dfce80 0x5572b834a730 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b52338c9e2408f62 server_cookie=7577a4b72de14448 in_seq=0 out_seq=0 2024-04-17T19:48:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa576000640 10 client.4920 ms_handle_connect on v2:172.21.15.29:6832/4184160618 2024-04-17T19:48:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa576000640 1 -- 172.21.15.29:0/3194320247 <== mds.2 v2:172.21.15.29:6832/4184160618 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x5572b814af10 con 0x5572b7dfce80 2024-04-17T19:48:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa576000640 10 client.4920 handle_command_reply: tid=0 2024-04-17T19:48:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 1 client.4920 shutdown 2024-04-17T19:48:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 1 -- 172.21.15.29:0/3194320247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b8208670 msgr2=0x5572b8349dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 1 --2- 172.21.15.29:0/3194320247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b8208670 0x5572b8349dd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa588005c10 tx=0x7fa588001260 comp rx=0 tx=0).stop 2024-04-17T19:48:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 1 -- 172.21.15.29:0/3194320247 shutdown_connections 2024-04-17T19:48:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 1 --2- 172.21.15.29:0/3194320247 >> [v2:172.21.15.29:6832/4184160618,v1:172.21.15.29:6836/4184160618] conn(0x5572b7dfce80 0x5572b834a730 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 1 --2- 172.21.15.29:0/3194320247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5572b8208670 0x5572b8349dd0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 1 --2- 172.21.15.29:0/3194320247 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5572b81bf930 0x5572b83a5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 1 --2- 172.21.15.29:0/3194320247 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5572b7dbf280 0x5572b83a5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa576000640 0 client.4920 ms_handle_reset on v2:172.21.15.29:6832/4184160618 2024-04-17T19:48:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 1 -- 172.21.15.29:0/3194320247 >> 172.21.15.29:0/3194320247 conn(0x5572b82f5640 msgr2=0x5572b82ed9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 1 -- 172.21.15.29:0/3194320247 shutdown_connections 2024-04-17T19:48:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 1 -- 172.21.15.29:0/3194320247 wait complete. 2024-04-17T19:48:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa59f08e000 20 client.4920 trim_cache size 0 max 16384 2024-04-17T19:48:40.365 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-17T19:48:40.366 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "a680ee90-875c-4283-a901-86d7638393c3", 2024-04-17T19:48:40.366 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 2, 2024-04-17T19:48:40.366 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 4259, 2024-04-17T19:48:40.366 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-04-17T19:48:40.366 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-04-17T19:48:40.366 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-04-17T19:48:40.366 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 23.599226327, 2024-04-17T19:48:40.366 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 19, 2024-04-17T19:48:40.366 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 26, 2024-04-17T19:48:40.366 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 26, 2024-04-17T19:48:40.366 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 55.932917379999999 2024-04-17T19:48:40.366 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-17T19:48:40.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa597a00640 1 -- 172.21.15.29:0/4013137318 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa570068380 msgr2=0x7fa57006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa597a00640 1 --2- 172.21.15.29:0/4013137318 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa570068380 0x7fa57006a890 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa59015f0f0 tx=0x7fa57c008040 comp rx=0 tx=0).stop 2024-04-17T19:48:40.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa597a00640 1 -- 172.21.15.29:0/4013137318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5900700d0 msgr2=0x7fa5900c6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.372+0000 7fa597a00640 1 --2- 172.21.15.29:0/4013137318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5900700d0 0x7fa5900c6fc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa5840ae040 tx=0x7fa584019220 comp rx=0 tx=0).stop 2024-04-17T19:48:40.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.376+0000 7fa597a00640 1 -- 172.21.15.29:0/4013137318 shutdown_connections 2024-04-17T19:48:40.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.376+0000 7fa597a00640 1 --2- 172.21.15.29:0/4013137318 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa570068380 0x7fa57006a890 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.376+0000 7fa597a00640 1 --2- 172.21.15.29:0/4013137318 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa5900c91e0 0x7fa59015e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.376+0000 7fa597a00640 1 --2- 172.21.15.29:0/4013137318 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa5900704e0 0x7fa590147440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.376+0000 7fa597a00640 1 --2- 172.21.15.29:0/4013137318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5900700d0 0x7fa5900c6fc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.376+0000 7fa597a00640 1 -- 172.21.15.29:0/4013137318 >> 172.21.15.29:0/4013137318 conn(0x7fa59006c420 msgr2=0x7fa590155820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.376+0000 7fa597a00640 1 -- 172.21.15.29:0/4013137318 shutdown_connections 2024-04-17T19:48:40.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.376+0000 7fa597a00640 1 -- 172.21.15.29:0/4013137318 wait complete. 2024-04-17T19:48:40.382 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-04-17T19:48:40.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.544+0000 7fb01dc00640 1 Processor -- start 2024-04-17T19:48:40.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.544+0000 7fb01dc00640 1 -- start start 2024-04-17T19:48:40.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.548+0000 7fb01dc00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb01815a7f0 0x7fb0180c4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.548+0000 7fb01dc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb018153c10 0x7fb018154020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.548+0000 7fb01dc00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0180ca0d0 0x7fb0180ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.548+0000 7fb01dc00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fb018154560 con 0x7fb0180ca0d0 2024-04-17T19:48:40.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.548+0000 7fb01dc00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb0180c84c0 con 0x7fb01815a7f0 2024-04-17T19:48:40.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.548+0000 7fb01dc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb0180c87a0 con 0x7fb018153c10 2024-04-17T19:48:40.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.548+0000 7fb017400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0180ca0d0 0x7fb0180ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.548+0000 7fb016a00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb01815a7f0 0x7fb0180c4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.548+0000 7fb017400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0180ca0d0 0x7fb0180ca4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33202/0 (socket says 172.21.15.29:33202) 2024-04-17T19:48:40.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.548+0000 7fb017400640 1 -- 172.21.15.29:0/1115463900 learned_addr learned my addr 172.21.15.29:0/1115463900 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.548+0000 7fb00fe00640 1 --2- 172.21.15.29:0/1115463900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb018153c10 0x7fb018154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.548+0000 7fb017400640 1 -- 172.21.15.29:0/1115463900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb018153c10 msgr2=0x7fb018154020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.548+0000 7fb017400640 1 --2- 172.21.15.29:0/1115463900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb018153c10 0x7fb018154020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.548+0000 7fb017400640 1 -- 172.21.15.29:0/1115463900 >> v2:172.21.15.37:3300/0 conn(0x7fb01815a7f0 msgr2=0x7fb0180c4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.548+0000 7fb017400640 1 --2- 172.21.15.29:0/1115463900 >> v2:172.21.15.37:3300/0 conn(0x7fb01815a7f0 0x7fb0180c4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.548+0000 7fb017400640 1 -- 172.21.15.29:0/1115463900 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb01815d390 con 0x7fb0180ca0d0 2024-04-17T19:48:40.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.552+0000 7fb017400640 1 --2- 172.21.15.29:0/1115463900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0180ca0d0 0x7fb0180ca4e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb018070820 tx=0x7fb01001c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbd7e923e8ba31fd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.552+0000 7fb016000640 1 -- 172.21.15.29:0/1115463900 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0100a5840 con 0x7fb0180ca0d0 2024-04-17T19:48:40.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.552+0000 7fb016000640 1 -- 172.21.15.29:0/1115463900 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0100a59e0 con 0x7fb0180ca0d0 2024-04-17T19:48:40.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.552+0000 7fb016000640 1 -- 172.21.15.29:0/1115463900 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0100a5db0 con 0x7fb0180ca0d0 2024-04-17T19:48:40.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.552+0000 7fb01dc00640 1 -- 172.21.15.29:0/1115463900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0180ca0d0 msgr2=0x7fb0180ca4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.552+0000 7fb01dc00640 1 --2- 172.21.15.29:0/1115463900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0180ca0d0 0x7fb0180ca4e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb018070820 tx=0x7fb01001c9c0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.552+0000 7fb01dc00640 1 -- 172.21.15.29:0/1115463900 shutdown_connections 2024-04-17T19:48:40.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.552+0000 7fb01dc00640 1 --2- 172.21.15.29:0/1115463900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0180ca0d0 0x7fb0180ca4e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.552+0000 7fb01dc00640 1 --2- 172.21.15.29:0/1115463900 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb018153c10 0x7fb018154020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.552+0000 7fb01dc00640 1 --2- 172.21.15.29:0/1115463900 >> v2:172.21.15.37:3300/0 conn(0x7fb01815a7f0 0x7fb0180c4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.552+0000 7fb01dc00640 1 -- 172.21.15.29:0/1115463900 >> 172.21.15.29:0/1115463900 conn(0x7fb01806bf90 msgr2=0x7fb01806c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.552+0000 7fb01dc00640 1 -- 172.21.15.29:0/1115463900 shutdown_connections 2024-04-17T19:48:40.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.552+0000 7fb01dc00640 1 -- 172.21.15.29:0/1115463900 wait complete. 2024-04-17T19:48:40.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb01dc00640 1 Processor -- start 2024-04-17T19:48:40.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb01dc00640 1 -- start start 2024-04-17T19:48:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb01dc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0180ca0d0 0x7fb0181f8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb01dc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb018153c10 0x7fb0181f6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb01dc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb01815a7f0 0x7fb01815c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb01dc00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb01815e150 con 0x7fb0180ca0d0 2024-04-17T19:48:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb01dc00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb01815ac00 con 0x7fb018153c10 2024-04-17T19:48:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb01dc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb0180caa20 con 0x7fb01815a7f0 2024-04-17T19:48:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb016a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0180ca0d0 0x7fb0181f8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb016a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0180ca0d0 0x7fb0181f8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33204/0 (socket says 172.21.15.29:33204) 2024-04-17T19:48:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb016a00640 1 -- 172.21.15.29:0/805527 learned_addr learned my addr 172.21.15.29:0/805527 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb00fe00640 1 --2- 172.21.15.29:0/805527 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb018153c10 0x7fb0181f6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb017400640 1 --2- 172.21.15.29:0/805527 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb01815a7f0 0x7fb01815c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb016a00640 1 -- 172.21.15.29:0/805527 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb01815a7f0 msgr2=0x7fb01815c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb016a00640 1 --2- 172.21.15.29:0/805527 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb01815a7f0 0x7fb01815c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb016a00640 1 -- 172.21.15.29:0/805527 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb018153c10 msgr2=0x7fb0181f6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb016a00640 1 --2- 172.21.15.29:0/805527 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb018153c10 0x7fb0181f6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb016a00640 1 -- 172.21.15.29:0/805527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0181f65d0 con 0x7fb0180ca0d0 2024-04-17T19:48:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb016a00640 1 --2- 172.21.15.29:0/805527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0180ca0d0 0x7fb0181f8c60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb00800e870 tx=0x7fb00800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb00ea00640 1 -- 172.21.15.29:0/805527 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb008017070 con 0x7fb0180ca0d0 2024-04-17T19:48:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb00ea00640 1 -- 172.21.15.29:0/805527 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb00800c940 con 0x7fb0180ca0d0 2024-04-17T19:48:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb00ea00640 1 -- 172.21.15.29:0/805527 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb00800ccf0 con 0x7fb0180ca0d0 2024-04-17T19:48:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb01dc00640 1 -- 172.21.15.29:0/805527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0181f92f0 con 0x7fb0180ca0d0 2024-04-17T19:48:40.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.556+0000 7fb01dc00640 1 -- 172.21.15.29:0/805527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0181f9890 con 0x7fb0180ca0d0 2024-04-17T19:48:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.560+0000 7fb00ea00640 1 -- 172.21.15.29:0/805527 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb00801a070 con 0x7fb0180ca0d0 2024-04-17T19:48:40.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.560+0000 7fb00ea00640 1 --2- 172.21.15.29:0/805527 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faff40683b0 0x7faff406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.560+0000 7fb00ea00640 1 -- 172.21.15.29:0/805527 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fb008086f90 con 0x7fb0180ca0d0 2024-04-17T19:48:40.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.560+0000 7fb02537d000 1 Processor -- start 2024-04-17T19:48:40.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.560+0000 7fb00fe00640 1 --2- 172.21.15.29:0/805527 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faff40683b0 0x7faff406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.560+0000 7fb02537d000 1 -- start start 2024-04-17T19:48:40.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb00fe00640 1 --2- 172.21.15.29:0/805527 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faff40683b0 0x7faff406a8c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb018072e10 tx=0x7fb000009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb02537d000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d401b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb02537d000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 0x55c3b0d406f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb02537d000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b07cf280 0x55c3b0d40c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb02537d000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c3b0b4d4f0 con 0x55c3b0bd6c10 2024-04-17T19:48:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb02537d000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x55c3b0ae7c10 con 0x55c3b07cf280 2024-04-17T19:48:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb02537d000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x55c3b0b494b0 con 0x55c3b0bcf930 2024-04-17T19:48:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb016a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb016a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d401b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33214/0 (socket says 172.21.15.29:33214) 2024-04-17T19:48:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb016a00640 1 -- 172.21.15.29:0/13089796 learned_addr learned my addr 172.21.15.29:0/13089796 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb017400640 1 --2- 172.21.15.29:0/13089796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b07cf280 0x55c3b0d40c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb00fe00640 1 --2- 172.21.15.29:0/13089796 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 0x55c3b0d406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb016a00640 1 -- 172.21.15.29:0/13089796 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 msgr2=0x55c3b0d406f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb016a00640 1 --2- 172.21.15.29:0/13089796 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 0x55c3b0d406f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb016a00640 1 -- 172.21.15.29:0/13089796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b07cf280 msgr2=0x55c3b0d40c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb016a00640 1 --2- 172.21.15.29:0/13089796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b07cf280 0x55c3b0d40c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb016a00640 1 -- 172.21.15.29:0/13089796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3b0ae7e60 con 0x55c3b0bd6c10 2024-04-17T19:48:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb017400640 1 --2- 172.21.15.29:0/13089796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b07cf280 0x55c3b0d40c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T19:48:40.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb016a00640 1 --2- 172.21.15.29:0/13089796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d401b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb008084f90 tx=0x7fb008056a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e88a3782f25fd588 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb007400640 1 -- 172.21.15.29:0/13089796 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb008021040 con 0x55c3b0bd6c10 2024-04-17T19:48:40.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb007400640 1 -- 172.21.15.29:0/13089796 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb00805f590 con 0x55c3b0bd6c10 2024-04-17T19:48:40.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb007400640 1 -- 172.21.15.29:0/13089796 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0080530d0 con 0x55c3b0bd6c10 2024-04-17T19:48:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb02537d000 1 -- 172.21.15.29:0/13089796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 msgr2=0x55c3b0d401b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb02537d000 1 --2- 172.21.15.29:0/13089796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d401b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb008084f90 tx=0x7fb008056a60 comp rx=0 tx=0).stop 2024-04-17T19:48:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb02537d000 1 -- 172.21.15.29:0/13089796 shutdown_connections 2024-04-17T19:48:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb02537d000 1 --2- 172.21.15.29:0/13089796 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b07cf280 0x55c3b0d40c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb02537d000 1 --2- 172.21.15.29:0/13089796 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 0x55c3b0d406f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb02537d000 1 --2- 172.21.15.29:0/13089796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d401b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb02537d000 1 -- 172.21.15.29:0/13089796 >> 172.21.15.29:0/13089796 conn(0x55c3b0c18670 msgr2=0x55c3b0d3df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb02537d000 1 -- 172.21.15.29:0/13089796 shutdown_connections 2024-04-17T19:48:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.564+0000 7fb02537d000 1 -- 172.21.15.29:0/13089796 wait complete. 2024-04-17T19:48:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb02537d000 1 -- start start 2024-04-17T19:48:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb02537d000 1 Processor -- start 2024-04-17T19:48:40.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb02537d000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b07cf280 0x55c3b0d39f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb02537d000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 0x55c3b0d3a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb02537d000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d01b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb02537d000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c3b0b8e9e0 con 0x55c3b07cf280 2024-04-17T19:48:40.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb02537d000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x55c3b0b0aa80 con 0x55c3b0bd6c10 2024-04-17T19:48:40.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb02537d000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x55c3b0b08d10 con 0x55c3b0bcf930 2024-04-17T19:48:40.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb02537d000 10 client.0 _ll_register_callbacks cb 0x55c3b0ceab20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T19:48:40.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb016a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b07cf280 0x55c3b0d39f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb016a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b07cf280 0x55c3b0d39f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33224/0 (socket says 172.21.15.29:33224) 2024-04-17T19:48:40.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb016a00640 1 -- 172.21.15.29:0/2126900718 learned_addr learned my addr 172.21.15.29:0/2126900718 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb017400640 1 --2- 172.21.15.29:0/2126900718 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d01b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb00fe00640 1 --2- 172.21.15.29:0/2126900718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 0x55c3b0d3a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb016a00640 1 -- 172.21.15.29:0/2126900718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 msgr2=0x55c3b0d3a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb016a00640 1 --2- 172.21.15.29:0/2126900718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 0x55c3b0d3a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb016a00640 1 -- 172.21.15.29:0/2126900718 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b0bd6c10 msgr2=0x55c3b0d01b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb016a00640 1 --2- 172.21.15.29:0/2126900718 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d01b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb016a00640 1 -- 172.21.15.29:0/2126900718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3b0b18e20 con 0x55c3b07cf280 2024-04-17T19:48:40.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb017400640 1 --2- 172.21.15.29:0/2126900718 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d01b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:40.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb016a00640 1 --2- 172.21.15.29:0/2126900718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b07cf280 0x55c3b0d39f10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb00800c520 tx=0x7fb008000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d634fbb30820c70 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7faffbe00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-17T19:48:40.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7faffbe00640 1 -- 172.21.15.29:0/2126900718 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb008021040 con 0x55c3b07cf280 2024-04-17T19:48:40.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb02537d000 1 -- 172.21.15.29:0/2126900718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb00805f590 con 0x55c3b07cf280 2024-04-17T19:48:40.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7faffbe00640 1 -- 172.21.15.29:0/2126900718 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fb00805f310 con 0x55c3b07cf280 2024-04-17T19:48:40.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7faffbe00640 1 -- 172.21.15.29:0/2126900718 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb008055eb0 con 0x55c3b07cf280 2024-04-17T19:48:40.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb02537d000 10 client.4932 fetch_fsmap learned FSMap version 19 2024-04-17T19:48:40.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb02537d000 1 -- 172.21.15.29:0/2126900718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x55c3b0c18a80 con 0x55c3b07cf280 2024-04-17T19:48:40.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7faffbe00640 1 -- 172.21.15.29:0/2126900718 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb00805f590 con 0x55c3b07cf280 2024-04-17T19:48:40.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7faffbe00640 1 -- 172.21.15.29:0/2126900718 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7fb008057ce0 con 0x55c3b07cf280 2024-04-17T19:48:40.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb02537d000 10 client.4932 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T19:48:40.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb02537d000 10 client.4932 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-17T19:48:40.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb02537d000 20 client.4932 populate_metadata read hostname 'smithi029' 2024-04-17T19:48:40.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb02537d000 1 --2- 172.21.15.29:0/2126900718 >> [v2:172.21.15.29:6834/270402476,v1:172.21.15.29:6835/270402476] conn(0x55c3b07f37d0 0x55c3b0d02840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb02537d000 4 client.4932 mds_command: new command op to 4302 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T19:48:40.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb02537d000 1 -- 172.21.15.29:0/2126900718 --> [v2:172.21.15.29:6834/270402476,v1:172.21.15.29:6835/270402476] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c3b0ae6be0 con 0x55c3b07f37d0 2024-04-17T19:48:40.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.568+0000 7fb00fe00640 1 --2- 172.21.15.29:0/2126900718 >> [v2:172.21.15.29:6834/270402476,v1:172.21.15.29:6835/270402476] conn(0x55c3b07f37d0 0x55c3b0d02840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.572+0000 7fb00fe00640 1 --2- 172.21.15.29:0/2126900718 >> [v2:172.21.15.29:6834/270402476,v1:172.21.15.29:6835/270402476] conn(0x55c3b07f37d0 0x55c3b0d02840 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=710a09794fc1a5d1 server_cookie=2ab935371fc53605 in_seq=0 out_seq=0 2024-04-17T19:48:40.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.572+0000 7faffbe00640 10 client.4932 ms_handle_connect on v2:172.21.15.29:6834/270402476 2024-04-17T19:48:40.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.572+0000 7faffbe00640 1 -- 172.21.15.29:0/2126900718 <== mds.1 v2:172.21.15.29:6834/270402476 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c3b0ae6be0 con 0x55c3b07f37d0 2024-04-17T19:48:40.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.572+0000 7faffbe00640 10 client.4932 handle_command_reply: tid=0 2024-04-17T19:48:40.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.572+0000 7fb02537d000 1 client.4932 shutdown 2024-04-17T19:48:40.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.572+0000 7fb02537d000 1 -- 172.21.15.29:0/2126900718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b07cf280 msgr2=0x55c3b0d39f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.572+0000 7fb02537d000 1 --2- 172.21.15.29:0/2126900718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b07cf280 0x55c3b0d39f10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb00800c520 tx=0x7fb008000bd0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.572+0000 7fb02537d000 1 -- 172.21.15.29:0/2126900718 shutdown_connections 2024-04-17T19:48:40.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.572+0000 7fb02537d000 1 --2- 172.21.15.29:0/2126900718 >> [v2:172.21.15.29:6834/270402476,v1:172.21.15.29:6835/270402476] conn(0x55c3b07f37d0 0x55c3b0d02840 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.572+0000 7fb02537d000 1 --2- 172.21.15.29:0/2126900718 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d01b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.572+0000 7fb02537d000 1 --2- 172.21.15.29:0/2126900718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 0x55c3b0d3a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.572+0000 7faffbe00640 0 client.4932 ms_handle_reset on v2:172.21.15.29:6834/270402476 2024-04-17T19:48:40.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.572+0000 7fb02537d000 1 --2- 172.21.15.29:0/2126900718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b07cf280 0x55c3b0d39f10 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.572+0000 7fb02537d000 1 -- 172.21.15.29:0/2126900718 >> 172.21.15.29:0/2126900718 conn(0x55c3b0c18670 msgr2=0x55c3b0d3b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.572+0000 7fb02537d000 1 -- 172.21.15.29:0/2126900718 shutdown_connections 2024-04-17T19:48:40.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.572+0000 7fb02537d000 1 -- 172.21.15.29:0/2126900718 wait complete. 2024-04-17T19:48:40.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.572+0000 7fb02537d000 20 client.4932 trim_cache size 0 max 16384 2024-04-17T19:48:40.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb02537d000 1 Processor -- start 2024-04-17T19:48:40.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb02537d000 1 -- start start 2024-04-17T19:48:40.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb02537d000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b07cf280 0x55c3b0cfe3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb02537d000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 0x55c3b0d38870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb02537d000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d38db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb02537d000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c3b0b5a700 con 0x55c3b0bd6c10 2024-04-17T19:48:40.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb02537d000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x55c3b0ae37d0 con 0x55c3b07cf280 2024-04-17T19:48:40.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb02537d000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x55c3b0b13b10 con 0x55c3b0bcf930 2024-04-17T19:48:40.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb017400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d38db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb017400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d38db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33230/0 (socket says 172.21.15.29:33230) 2024-04-17T19:48:40.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb017400640 1 -- 172.21.15.29:0/3795904080 learned_addr learned my addr 172.21.15.29:0/3795904080 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb00fe00640 1 --2- 172.21.15.29:0/3795904080 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 0x55c3b0d38870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb016a00640 1 --2- 172.21.15.29:0/3795904080 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b07cf280 0x55c3b0cfe3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb017400640 1 -- 172.21.15.29:0/3795904080 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 msgr2=0x55c3b0d38870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb017400640 1 --2- 172.21.15.29:0/3795904080 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 0x55c3b0d38870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb017400640 1 -- 172.21.15.29:0/3795904080 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b07cf280 msgr2=0x55c3b0cfe3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb017400640 1 --2- 172.21.15.29:0/3795904080 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b07cf280 0x55c3b0cfe3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb017400640 1 -- 172.21.15.29:0/3795904080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3b0cffc10 con 0x55c3b0bd6c10 2024-04-17T19:48:40.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb016a00640 1 --2- 172.21.15.29:0/3795904080 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b07cf280 0x55c3b0cfe3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:40.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb017400640 1 --2- 172.21.15.29:0/3795904080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d38db0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb0100af040 tx=0x7fb010004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd33790aa7e049b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb006a00640 1 -- 172.21.15.29:0/3795904080 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0100a5840 con 0x55c3b0bd6c10 2024-04-17T19:48:40.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb006a00640 1 -- 172.21.15.29:0/3795904080 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0100a59e0 con 0x55c3b0bd6c10 2024-04-17T19:48:40.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.592+0000 7fb006a00640 1 -- 172.21.15.29:0/3795904080 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0100a5db0 con 0x55c3b0bd6c10 2024-04-17T19:48:40.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb02537d000 1 -- 172.21.15.29:0/3795904080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 msgr2=0x55c3b0d38db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb02537d000 1 --2- 172.21.15.29:0/3795904080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d38db0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fb0100af040 tx=0x7fb010004ac0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb02537d000 1 -- 172.21.15.29:0/3795904080 shutdown_connections 2024-04-17T19:48:40.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb02537d000 1 --2- 172.21.15.29:0/3795904080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d38db0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb02537d000 1 --2- 172.21.15.29:0/3795904080 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 0x55c3b0d38870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb02537d000 1 --2- 172.21.15.29:0/3795904080 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b07cf280 0x55c3b0cfe3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb02537d000 1 -- 172.21.15.29:0/3795904080 >> 172.21.15.29:0/3795904080 conn(0x55c3b0c18670 msgr2=0x55c3b0d38160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb02537d000 1 -- 172.21.15.29:0/3795904080 shutdown_connections 2024-04-17T19:48:40.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb02537d000 1 -- 172.21.15.29:0/3795904080 wait complete. 2024-04-17T19:48:40.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb02537d000 1 -- start start 2024-04-17T19:48:40.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb02537d000 1 Processor -- start 2024-04-17T19:48:40.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb02537d000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b07cf280 0x55c3b0cfe000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb02537d000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 0x55c3b0cfe540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb02537d000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d57810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb02537d000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x55c3b0b0aa80 con 0x55c3b0bd6c10 2024-04-17T19:48:40.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb02537d000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x55c3b0b8e9e0 con 0x55c3b07cf280 2024-04-17T19:48:40.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb02537d000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x55c3b0b08d10 con 0x55c3b0bcf930 2024-04-17T19:48:40.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb02537d000 10 client.0 _ll_register_callbacks cb 0x7fb008056630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T19:48:40.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb017400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d57810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb017400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d57810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33246/0 (socket says 172.21.15.29:33246) 2024-04-17T19:48:40.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb017400640 1 -- 172.21.15.29:0/644580163 learned_addr learned my addr 172.21.15.29:0/644580163 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb00fe00640 1 --2- 172.21.15.29:0/644580163 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 0x55c3b0cfe540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.596+0000 7fb016a00640 1 --2- 172.21.15.29:0/644580163 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b07cf280 0x55c3b0cfe000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb017400640 1 -- 172.21.15.29:0/644580163 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 msgr2=0x55c3b0cfe540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb017400640 1 --2- 172.21.15.29:0/644580163 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 0x55c3b0cfe540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb017400640 1 -- 172.21.15.29:0/644580163 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b07cf280 msgr2=0x55c3b0cfe000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb017400640 1 --2- 172.21.15.29:0/644580163 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b07cf280 0x55c3b0cfe000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb017400640 1 -- 172.21.15.29:0/644580163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3b0ba6e30 con 0x55c3b0bd6c10 2024-04-17T19:48:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb00fe00640 1 --2- 172.21.15.29:0/644580163 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 0x55c3b0cfe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb016a00640 1 --2- 172.21.15.29:0/644580163 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b07cf280 0x55c3b0cfe000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb017400640 1 --2- 172.21.15.29:0/644580163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d57810 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb01000c090 tx=0x7fb01001b200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8c84d38096d58a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7faffbe00640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-17T19:48:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7faffbe00640 1 -- 172.21.15.29:0/644580163 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0100a5b90 con 0x55c3b0bd6c10 2024-04-17T19:48:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7faffbe00640 1 -- 172.21.15.29:0/644580163 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fb0100a5d30 con 0x55c3b0bd6c10 2024-04-17T19:48:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 1 -- 172.21.15.29:0/644580163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0100a59e0 con 0x55c3b0bd6c10 2024-04-17T19:48:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7faffbe00640 1 -- 172.21.15.29:0/644580163 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fb0100039a0 con 0x55c3b0bd6c10 2024-04-17T19:48:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 10 client.4938 fetch_fsmap learned FSMap version 19 2024-04-17T19:48:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 1 -- 172.21.15.29:0/644580163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x55c3b0adf7e0 con 0x55c3b0bd6c10 2024-04-17T19:48:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7faffbe00640 1 -- 172.21.15.29:0/644580163 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0100a59e0 con 0x55c3b0bd6c10 2024-04-17T19:48:40.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7faffbe00640 1 -- 172.21.15.29:0/644580163 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7fb0100bd070 con 0x55c3b0bd6c10 2024-04-17T19:48:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 10 client.4938 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T19:48:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 10 client.4938 resolve_mds: resolved name 'e' to daemon mds.e 2024-04-17T19:48:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 20 client.4938 populate_metadata read hostname 'smithi029' 2024-04-17T19:48:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 1 --2- 172.21.15.29:0/644580163 >> [v2:172.21.15.29:6834/270402476,v1:172.21.15.29:6835/270402476] conn(0x55c3b07e1830 0x55c3b0d581d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 4 client.4938 mds_command: new command op to 4302 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T19:48:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 1 -- 172.21.15.29:0/644580163 --> [v2:172.21.15.29:6834/270402476,v1:172.21.15.29:6835/270402476] -- command(tid 0: {"prefix": "status"}) -- 0x55c3b0b5af10 con 0x55c3b07e1830 2024-04-17T19:48:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb016a00640 1 --2- 172.21.15.29:0/644580163 >> [v2:172.21.15.29:6834/270402476,v1:172.21.15.29:6835/270402476] conn(0x55c3b07e1830 0x55c3b0d581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb016a00640 1 --2- 172.21.15.29:0/644580163 >> [v2:172.21.15.29:6834/270402476,v1:172.21.15.29:6835/270402476] conn(0x55c3b07e1830 0x55c3b0d581d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d65003bbe50db4aa server_cookie=d20ff2d6e1e39f1c in_seq=0 out_seq=0 2024-04-17T19:48:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7faffbe00640 10 client.4938 ms_handle_connect on v2:172.21.15.29:6834/270402476 2024-04-17T19:48:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7faffbe00640 1 -- 172.21.15.29:0/644580163 <== mds.1 v2:172.21.15.29:6834/270402476 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55c3b0b5af10 con 0x55c3b07e1830 2024-04-17T19:48:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7faffbe00640 10 client.4938 handle_command_reply: tid=0 2024-04-17T19:48:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 1 client.4938 shutdown 2024-04-17T19:48:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 1 -- 172.21.15.29:0/644580163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 msgr2=0x55c3b0d57810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 1 --2- 172.21.15.29:0/644580163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d57810 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb01000c090 tx=0x7fb01001b200 comp rx=0 tx=0).stop 2024-04-17T19:48:40.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 1 -- 172.21.15.29:0/644580163 shutdown_connections 2024-04-17T19:48:40.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 1 --2- 172.21.15.29:0/644580163 >> [v2:172.21.15.29:6834/270402476,v1:172.21.15.29:6835/270402476] conn(0x55c3b07e1830 0x55c3b0d581d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 1 --2- 172.21.15.29:0/644580163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55c3b0bd6c10 0x55c3b0d57810 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 1 --2- 172.21.15.29:0/644580163 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x55c3b0bcf930 0x55c3b0cfe540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7faffbe00640 0 client.4938 ms_handle_reset on v2:172.21.15.29:6834/270402476 2024-04-17T19:48:40.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 1 --2- 172.21.15.29:0/644580163 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x55c3b07cf280 0x55c3b0cfe000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 1 -- 172.21.15.29:0/644580163 >> 172.21.15.29:0/644580163 conn(0x55c3b0c18670 msgr2=0x55c3b0d3e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 1 -- 172.21.15.29:0/644580163 shutdown_connections 2024-04-17T19:48:40.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 1 -- 172.21.15.29:0/644580163 wait complete. 2024-04-17T19:48:40.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb02537d000 20 client.4938 trim_cache size 0 max 16384 2024-04-17T19:48:40.593 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-17T19:48:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "a680ee90-875c-4283-a901-86d7638393c3", 2024-04-17T19:48:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 1, 2024-04-17T19:48:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 4302, 2024-04-17T19:48:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-04-17T19:48:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-04-17T19:48:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-04-17T19:48:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 29.866136572999999, 2024-04-17T19:48:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 19, 2024-04-17T19:48:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 26, 2024-04-17T19:48:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 26, 2024-04-17T19:48:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 56.161197991000002 2024-04-17T19:48:40.594 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-17T19:48:40.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb01dc00640 1 -- 172.21.15.29:0/805527 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faff40683b0 msgr2=0x7faff406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb01dc00640 1 --2- 172.21.15.29:0/805527 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faff40683b0 0x7faff406a8c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb018072e10 tx=0x7fb000009190 comp rx=0 tx=0).stop 2024-04-17T19:48:40.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb01dc00640 1 -- 172.21.15.29:0/805527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0180ca0d0 msgr2=0x7fb0181f8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb01dc00640 1 --2- 172.21.15.29:0/805527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0180ca0d0 0x7fb0181f8c60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb00800e870 tx=0x7fb00800ec70 comp rx=0 tx=0).stop 2024-04-17T19:48:40.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb01dc00640 1 -- 172.21.15.29:0/805527 shutdown_connections 2024-04-17T19:48:40.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb01dc00640 1 --2- 172.21.15.29:0/805527 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faff40683b0 0x7faff406a8c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb01dc00640 1 --2- 172.21.15.29:0/805527 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb01815a7f0 0x7fb01815c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb01dc00640 1 --2- 172.21.15.29:0/805527 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb018153c10 0x7fb0181f6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb01dc00640 1 --2- 172.21.15.29:0/805527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0180ca0d0 0x7fb0181f8c60 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.600+0000 7fb01dc00640 1 -- 172.21.15.29:0/805527 >> 172.21.15.29:0/805527 conn(0x7fb01806bf90 msgr2=0x7fb01813d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.604+0000 7fb01dc00640 1 -- 172.21.15.29:0/805527 shutdown_connections 2024-04-17T19:48:40.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.604+0000 7fb01dc00640 1 -- 172.21.15.29:0/805527 wait complete. 2024-04-17T19:48:40.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-17T19:48:40.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.772+0000 7fcc13600640 1 Processor -- start 2024-04-17T19:48:40.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.772+0000 7fcc13600640 1 -- start start 2024-04-17T19:48:40.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.776+0000 7fcc13600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcc0c071ce0 0x7fcc0c141de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.776+0000 7fcc13600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c1597e0 0x7fcc0c163620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.776+0000 7fcc13600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc0c142320 con 0x7fcc0c1593d0 2024-04-17T19:48:40.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.776+0000 7fcc13600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fcc0c0c7bd0 con 0x7fcc0c071ce0 2024-04-17T19:48:40.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.776+0000 7fcc13600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcc0c15c6d0 con 0x7fcc0c1597e0 2024-04-17T19:48:40.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.776+0000 7fcc10c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcc0c071ce0 0x7fcc0c141de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.776+0000 7fcc03e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcc0c1593d0 0x7fcc0c0c9f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52228/0 (socket says 172.21.15.29:52228) 2024-04-17T19:48:40.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.776+0000 7fcc03e00640 1 -- 172.21.15.29:0/759225051 learned_addr learned my addr 172.21.15.29:0/759225051 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:40.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.776+0000 7fcc11600640 1 --2- 172.21.15.29:0/759225051 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c1597e0 0x7fcc0c163620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.776+0000 7fcc10c00640 1 -- 172.21.15.29:0/759225051 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c1597e0 msgr2=0x7fcc0c163620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.776+0000 7fcc10c00640 1 --2- 172.21.15.29:0/759225051 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c1597e0 0x7fcc0c163620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.776+0000 7fcc10c00640 1 -- 172.21.15.29:0/759225051 >> v1:172.21.15.29:6789/0 conn(0x7fcc0c1593d0 legacy=0x7fcc0c0c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.776+0000 7fcc0be00640 1 -- 172.21.15.29:0/759225051 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 212471860 0 0) 0x7fcc0c142320 con 0x7fcc0c1593d0 2024-04-17T19:48:40.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.776+0000 7fcc10c00640 1 -- 172.21.15.29:0/759225051 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0c15c9b0 con 0x7fcc0c071ce0 2024-04-17T19:48:40.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.780+0000 7fcc10c00640 1 --2- 172.21.15.29:0/759225051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c071ce0 0x7fcc0c141de0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc0097c0 tx=0x7fcbfc01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa5e7f8b1a97bb20 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.780+0000 7fcc0be00640 1 -- 172.21.15.29:0/759225051 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc0b0070 con 0x7fcc0c071ce0 2024-04-17T19:48:40.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.780+0000 7fcc0be00640 1 -- 172.21.15.29:0/759225051 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbfc0ab600 con 0x7fcc0c071ce0 2024-04-17T19:48:40.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.780+0000 7fcc0be00640 1 -- 172.21.15.29:0/759225051 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc0ab9d0 con 0x7fcc0c071ce0 2024-04-17T19:48:40.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.780+0000 7fcc13600640 1 -- 172.21.15.29:0/759225051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c071ce0 msgr2=0x7fcc0c141de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.780+0000 7fcc13600640 1 --2- 172.21.15.29:0/759225051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c071ce0 0x7fcc0c141de0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc0097c0 tx=0x7fcbfc01c4a0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.780+0000 7fcc13600640 1 -- 172.21.15.29:0/759225051 shutdown_connections 2024-04-17T19:48:40.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.780+0000 7fcc13600640 1 --2- 172.21.15.29:0/759225051 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c1597e0 0x7fcc0c163620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.780+0000 7fcc13600640 1 --2- 172.21.15.29:0/759225051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c071ce0 0x7fcc0c141de0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.780+0000 7fcc13600640 1 -- 172.21.15.29:0/759225051 >> 172.21.15.29:0/759225051 conn(0x7fcc0c06bf90 msgr2=0x7fcc0c06c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.780+0000 7fcc13600640 1 -- 172.21.15.29:0/759225051 shutdown_connections 2024-04-17T19:48:40.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.780+0000 7fcc13600640 1 -- 172.21.15.29:0/759225051 wait complete. 2024-04-17T19:48:40.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc13600640 1 Processor -- start 2024-04-17T19:48:40.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc13600640 1 -- start start 2024-04-17T19:48:40.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc13600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c071ce0 0x7fcc0c157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc13600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc0c1593d0 0x7fcc0c151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc13600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c1597e0 0x7fcc0c1524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc13600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c0720f0 con 0x7fcc0c1597e0 2024-04-17T19:48:40.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc13600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c159bf0 con 0x7fcc0c1593d0 2024-04-17T19:48:40.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc13600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcc0c072270 con 0x7fcc0c071ce0 2024-04-17T19:48:40.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc11600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c1597e0 0x7fcc0c1524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc11600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c1597e0 0x7fcc0c1524e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33262/0 (socket says 172.21.15.29:33262) 2024-04-17T19:48:40.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc11600640 1 -- 172.21.15.29:0/1505216449 learned_addr learned my addr 172.21.15.29:0/1505216449 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc03e00640 1 --2- 172.21.15.29:0/1505216449 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc0c1593d0 0x7fcc0c151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc10c00640 1 --2- 172.21.15.29:0/1505216449 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c071ce0 0x7fcc0c157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc11600640 1 -- 172.21.15.29:0/1505216449 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c071ce0 msgr2=0x7fcc0c157110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc11600640 1 --2- 172.21.15.29:0/1505216449 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c071ce0 0x7fcc0c157110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc11600640 1 -- 172.21.15.29:0/1505216449 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc0c1593d0 msgr2=0x7fcc0c151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc11600640 1 --2- 172.21.15.29:0/1505216449 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc0c1593d0 0x7fcc0c151fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc11600640 1 -- 172.21.15.29:0/1505216449 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0c135010 con 0x7fcc0c1597e0 2024-04-17T19:48:40.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc11600640 1 --2- 172.21.15.29:0/1505216449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c1597e0 0x7fcc0c1524e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fcc04007e60 tx=0x7fcc0400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc09600640 1 -- 172.21.15.29:0/1505216449 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc04014420 con 0x7fcc0c1597e0 2024-04-17T19:48:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc09600640 1 -- 172.21.15.29:0/1505216449 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc04019070 con 0x7fcc0c1597e0 2024-04-17T19:48:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc13600640 1 -- 172.21.15.29:0/1505216449 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc0c13a0b0 con 0x7fcc0c1597e0 2024-04-17T19:48:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc09600640 1 -- 172.21.15.29:0/1505216449 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc04014820 con 0x7fcc0c1597e0 2024-04-17T19:48:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.784+0000 7fcc13600640 1 -- 172.21.15.29:0/1505216449 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc0c13a650 con 0x7fcc0c1597e0 2024-04-17T19:48:40.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.788+0000 7fcc09600640 1 -- 172.21.15.29:0/1505216449 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcc040149c0 con 0x7fcc0c1597e0 2024-04-17T19:48:40.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.788+0000 7fcc09600640 1 --2- 172.21.15.29:0/1505216449 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbe80683b0 0x7fcbe806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.788+0000 7fcc09600640 1 -- 172.21.15.29:0/1505216449 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fcc04087740 con 0x7fcc0c1597e0 2024-04-17T19:48:40.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.788+0000 7fcc10c00640 1 --2- 172.21.15.29:0/1505216449 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbe80683b0 0x7fcbe806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.788+0000 7fcc1ac52000 1 Processor -- start 2024-04-17T19:48:40.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.788+0000 7fcc1ac52000 1 -- start start 2024-04-17T19:48:40.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.788+0000 7fcc10c00640 1 --2- 172.21.15.29:0/1505216449 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbe80683b0 0x7fcbe806a8c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc0ad000 tx=0x7fcbfc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc1ac52000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fcb85c10 0x5569fccef1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc1ac52000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fcb7e930 0x5569fccef6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc1ac52000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fc77e280 0x5569fccefc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc1ac52000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5569fcafc4f0 con 0x5569fc77e280 2024-04-17T19:48:40.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc1ac52000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x5569fca96c10 con 0x5569fcb85c10 2024-04-17T19:48:40.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc1ac52000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x5569fcaf84b0 con 0x5569fcb7e930 2024-04-17T19:48:40.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc11600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fc77e280 0x5569fccefc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc11600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fc77e280 0x5569fccefc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33274/0 (socket says 172.21.15.29:33274) 2024-04-17T19:48:40.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc11600640 1 -- 172.21.15.29:0/2685743942 learned_addr learned my addr 172.21.15.29:0/2685743942 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc10c00640 1 --2- 172.21.15.29:0/2685743942 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fcb85c10 0x5569fccef1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc03e00640 1 --2- 172.21.15.29:0/2685743942 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fcb7e930 0x5569fccef6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc11600640 1 -- 172.21.15.29:0/2685743942 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fcb7e930 msgr2=0x5569fccef6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc11600640 1 --2- 172.21.15.29:0/2685743942 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fcb7e930 0x5569fccef6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc11600640 1 -- 172.21.15.29:0/2685743942 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fcb85c10 msgr2=0x5569fccef1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc11600640 1 --2- 172.21.15.29:0/2685743942 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fcb85c10 0x5569fccef1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc11600640 1 -- 172.21.15.29:0/2685743942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569fca96e60 con 0x5569fc77e280 2024-04-17T19:48:40.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc11600640 1 --2- 172.21.15.29:0/2685743942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fc77e280 0x5569fccefc30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400c040 tx=0x7fcc040577d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2a6a588eaf6b31 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc01200640 1 -- 172.21.15.29:0/2685743942 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0405b480 con 0x5569fc77e280 2024-04-17T19:48:40.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc01200640 1 -- 172.21.15.29:0/2685743942 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc04081070 con 0x5569fc77e280 2024-04-17T19:48:40.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc01200640 1 -- 172.21.15.29:0/2685743942 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0405fd40 con 0x5569fc77e280 2024-04-17T19:48:40.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc1ac52000 1 -- 172.21.15.29:0/2685743942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fc77e280 msgr2=0x5569fccefc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/2685743942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fc77e280 0x5569fccefc30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400c040 tx=0x7fcc040577d0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc1ac52000 1 -- 172.21.15.29:0/2685743942 shutdown_connections 2024-04-17T19:48:40.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/2685743942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fc77e280 0x5569fccefc30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/2685743942 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fcb7e930 0x5569fccef6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/2685743942 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fcb85c10 0x5569fccef1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc1ac52000 1 -- 172.21.15.29:0/2685743942 >> 172.21.15.29:0/2685743942 conn(0x5569fcbc7670 msgr2=0x5569fccecf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc1ac52000 1 -- 172.21.15.29:0/2685743942 shutdown_connections 2024-04-17T19:48:40.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc1ac52000 1 -- 172.21.15.29:0/2685743942 wait complete. 2024-04-17T19:48:40.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc1ac52000 1 -- start start 2024-04-17T19:48:40.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.792+0000 7fcc1ac52000 1 Processor -- start 2024-04-17T19:48:40.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc1ac52000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fc77e280 0x5569fcce8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc1ac52000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 0x5569fcce9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc1ac52000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fcb85c10 0x5569fccb0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc1ac52000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5569fcb3d9e0 con 0x5569fcb7e930 2024-04-17T19:48:40.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc1ac52000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x5569fcab9a80 con 0x5569fc77e280 2024-04-17T19:48:40.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc1ac52000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x5569fcab7d10 con 0x5569fcb85c10 2024-04-17T19:48:40.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc1ac52000 10 client.0 _ll_register_callbacks cb 0x5569fcc99b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T19:48:40.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc03e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 0x5569fcce9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc03e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 0x5569fcce9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33286/0 (socket says 172.21.15.29:33286) 2024-04-17T19:48:40.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc03e00640 1 -- 172.21.15.29:0/3400381563 learned_addr learned my addr 172.21.15.29:0/3400381563 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc10c00640 1 --2- 172.21.15.29:0/3400381563 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fc77e280 0x5569fcce8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc11600640 1 --2- 172.21.15.29:0/3400381563 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fcb85c10 0x5569fccb0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc03e00640 1 -- 172.21.15.29:0/3400381563 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fcb85c10 msgr2=0x5569fccb0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc03e00640 1 --2- 172.21.15.29:0/3400381563 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fcb85c10 0x5569fccb0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc03e00640 1 -- 172.21.15.29:0/3400381563 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fc77e280 msgr2=0x5569fcce8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc03e00640 1 --2- 172.21.15.29:0/3400381563 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fc77e280 0x5569fcce8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc03e00640 1 -- 172.21.15.29:0/3400381563 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569fcac7e20 con 0x5569fcb7e930 2024-04-17T19:48:40.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc03e00640 1 --2- 172.21.15.29:0/3400381563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 0x5569fcce9450 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4005fd0 tx=0x7fcbf40014c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d269d0aaf876c54 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcbf9600640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-17T19:48:40.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcbf9600640 1 -- 172.21.15.29:0/3400381563 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4007670 con 0x5569fcb7e930 2024-04-17T19:48:40.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcbf9600640 1 -- 172.21.15.29:0/3400381563 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fcbf4014400 con 0x5569fcb7e930 2024-04-17T19:48:40.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc1ac52000 1 -- 172.21.15.29:0/3400381563 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc04081070 con 0x5569fcb7e930 2024-04-17T19:48:40.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcbf9600640 1 -- 172.21.15.29:0/3400381563 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fcbf4021050 con 0x5569fcb7e930 2024-04-17T19:48:40.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc1ac52000 10 client.4953 fetch_fsmap learned FSMap version 19 2024-04-17T19:48:40.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc1ac52000 1 -- 172.21.15.29:0/3400381563 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x5569fcbc7a80 con 0x5569fcb7e930 2024-04-17T19:48:40.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcbf9600640 1 -- 172.21.15.29:0/3400381563 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc04081070 con 0x5569fcb7e930 2024-04-17T19:48:40.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcbf9600640 1 -- 172.21.15.29:0/3400381563 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7fcbf4024020 con 0x5569fcb7e930 2024-04-17T19:48:40.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc1ac52000 10 client.4953 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T19:48:40.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc1ac52000 10 client.4953 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T19:48:40.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc1ac52000 20 client.4953 populate_metadata read hostname 'smithi029' 2024-04-17T19:48:40.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/3400381563 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x5569fc7a27d0 0x5569fccb1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc1ac52000 4 client.4953 mds_command: new command op to 4305 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T19:48:40.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc1ac52000 1 -- 172.21.15.29:0/3400381563 --> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569fca95be0 con 0x5569fc7a27d0 2024-04-17T19:48:40.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.796+0000 7fcc10c00640 1 --2- 172.21.15.29:0/3400381563 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x5569fc7a27d0 0x5569fccb1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.800+0000 7fcbf9600640 10 client.4953 ms_handle_connect on v2:172.21.15.29:6833/2691590158 2024-04-17T19:48:40.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.800+0000 7fcc10c00640 1 --2- 172.21.15.29:0/3400381563 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x5569fc7a27d0 0x5569fccb1900 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34d1eef231f5e314 server_cookie=fd3038455cbfaa6f in_seq=0 out_seq=0 2024-04-17T19:48:40.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.800+0000 7fcbf9600640 1 -- 172.21.15.29:0/3400381563 <== mds.0 v2:172.21.15.29:6833/2691590158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5569fca95be0 con 0x5569fc7a27d0 2024-04-17T19:48:40.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.800+0000 7fcbf9600640 10 client.4953 handle_command_reply: tid=0 2024-04-17T19:48:40.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.800+0000 7fcc1ac52000 1 client.4953 shutdown 2024-04-17T19:48:40.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.800+0000 7fcc1ac52000 1 -- 172.21.15.29:0/3400381563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 msgr2=0x5569fcce9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.800+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/3400381563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 0x5569fcce9450 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4005fd0 tx=0x7fcbf40014c0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.800+0000 7fcc1ac52000 1 -- 172.21.15.29:0/3400381563 shutdown_connections 2024-04-17T19:48:40.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.800+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/3400381563 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x5569fc7a27d0 0x5569fccb1900 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.800+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/3400381563 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fcb85c10 0x5569fccb0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.800+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/3400381563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 0x5569fcce9450 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.800+0000 7fcbf9600640 0 client.4953 ms_handle_reset on v2:172.21.15.29:6833/2691590158 2024-04-17T19:48:40.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.800+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/3400381563 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fc77e280 0x5569fcce8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.800+0000 7fcc1ac52000 1 -- 172.21.15.29:0/3400381563 >> 172.21.15.29:0/3400381563 conn(0x5569fcbc7670 msgr2=0x5569fccea720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.800+0000 7fcc1ac52000 1 -- 172.21.15.29:0/3400381563 shutdown_connections 2024-04-17T19:48:40.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.800+0000 7fcc1ac52000 1 -- 172.21.15.29:0/3400381563 wait complete. 2024-04-17T19:48:40.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.800+0000 7fcc1ac52000 20 client.4953 trim_cache size 0 max 16384 2024-04-17T19:48:40.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc1ac52000 1 Processor -- start 2024-04-17T19:48:40.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc1ac52000 1 -- start start 2024-04-17T19:48:40.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc1ac52000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fc77e280 0x5569fcd09bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc1ac52000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 0x5569fcd0a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc1ac52000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fcb85c10 0x5569fcce7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc1ac52000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c200b10 con 0x5569fcb7e930 2024-04-17T19:48:40.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc1ac52000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c2011e0 con 0x5569fcb85c10 2024-04-17T19:48:40.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc1ac52000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcc0c1ff260 con 0x5569fc77e280 2024-04-17T19:48:40.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc03e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 0x5569fcd0a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc03e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 0x5569fcd0a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33294/0 (socket says 172.21.15.29:33294) 2024-04-17T19:48:40.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc03e00640 1 -- 172.21.15.29:0/3165915715 learned_addr learned my addr 172.21.15.29:0/3165915715 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc11600640 1 --2- 172.21.15.29:0/3165915715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fcb85c10 0x5569fcce7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc10c00640 1 --2- 172.21.15.29:0/3165915715 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fc77e280 0x5569fcd09bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc03e00640 1 -- 172.21.15.29:0/3165915715 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fc77e280 msgr2=0x5569fcd09bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc03e00640 1 --2- 172.21.15.29:0/3165915715 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fc77e280 0x5569fcd09bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc03e00640 1 -- 172.21.15.29:0/3165915715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fcb85c10 msgr2=0x5569fcce7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc03e00640 1 --2- 172.21.15.29:0/3165915715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fcb85c10 0x5569fcce7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc03e00640 1 -- 172.21.15.29:0/3165915715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569fcb55e30 con 0x5569fcb7e930 2024-04-17T19:48:40.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc11600640 1 --2- 172.21.15.29:0/3165915715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fcb85c10 0x5569fcce7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:40.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcc03e00640 1 --2- 172.21.15.29:0/3165915715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 0x5569fcd0a110 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4007e20 tx=0x7fcbf400e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=969b3384f43c5979 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcbfbe00640 1 -- 172.21.15.29:0/3165915715 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf40019c0 con 0x5569fcb7e930 2024-04-17T19:48:40.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcbfbe00640 1 -- 172.21.15.29:0/3165915715 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbf4029020 con 0x5569fcb7e930 2024-04-17T19:48:40.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.820+0000 7fcbfbe00640 1 -- 172.21.15.29:0/3165915715 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4014400 con 0x5569fcb7e930 2024-04-17T19:48:40.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc1ac52000 1 -- 172.21.15.29:0/3165915715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 msgr2=0x5569fcd0a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/3165915715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 0x5569fcd0a110 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4007e20 tx=0x7fcbf400e000 comp rx=0 tx=0).stop 2024-04-17T19:48:40.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc1ac52000 1 -- 172.21.15.29:0/3165915715 shutdown_connections 2024-04-17T19:48:40.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/3165915715 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fcb85c10 0x5569fcce7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/3165915715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 0x5569fcd0a110 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/3165915715 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fc77e280 0x5569fcd09bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc1ac52000 1 -- 172.21.15.29:0/3165915715 >> 172.21.15.29:0/3165915715 conn(0x5569fcbc7670 msgr2=0x5569fcd0ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc1ac52000 1 -- 172.21.15.29:0/3165915715 shutdown_connections 2024-04-17T19:48:40.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc1ac52000 1 -- 172.21.15.29:0/3165915715 wait complete. 2024-04-17T19:48:40.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc1ac52000 1 -- start start 2024-04-17T19:48:40.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc1ac52000 1 Processor -- start 2024-04-17T19:48:40.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc1ac52000 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fc77e280 0x5569fcd2b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc1ac52000 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 0x5569fcd2bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc1ac52000 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fcb85c10 0x5569fcd2c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc1ac52000 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x5569fcb00b20 con 0x5569fcb7e930 2024-04-17T19:48:40.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc1ac52000 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x5569fcafb800 con 0x5569fc77e280 2024-04-17T19:48:40.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc1ac52000 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x5569fcb09700 con 0x5569fcb85c10 2024-04-17T19:48:40.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc1ac52000 10 client.0 _ll_register_callbacks cb 0x7fcbf4005800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T19:48:40.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc03e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 0x5569fcd2bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc03e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 0x5569fcd2bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33310/0 (socket says 172.21.15.29:33310) 2024-04-17T19:48:40.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc03e00640 1 -- 172.21.15.29:0/781004049 learned_addr learned my addr 172.21.15.29:0/781004049 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:40.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc10c00640 1 --2- 172.21.15.29:0/781004049 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fc77e280 0x5569fcd2b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.824+0000 7fcc11600640 1 --2- 172.21.15.29:0/781004049 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fcb85c10 0x5569fcd2c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc03e00640 1 -- 172.21.15.29:0/781004049 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fcb85c10 msgr2=0x5569fcd2c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc03e00640 1 --2- 172.21.15.29:0/781004049 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fcb85c10 0x5569fcd2c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc03e00640 1 -- 172.21.15.29:0/781004049 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fc77e280 msgr2=0x5569fcd2b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:40.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc03e00640 1 --2- 172.21.15.29:0/781004049 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fc77e280 0x5569fcd2b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc03e00640 1 -- 172.21.15.29:0/781004049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569fca8e7e0 con 0x5569fcb7e930 2024-04-17T19:48:40.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc03e00640 1 --2- 172.21.15.29:0/781004049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 0x5569fcd2bcd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x5569fcac29c0 tx=0x7fcbf4000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=974f83976d54f70b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:40.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcbf9600640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2024-04-17T19:48:40.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcbf9600640 1 -- 172.21.15.29:0/781004049 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4032060 con 0x5569fcb7e930 2024-04-17T19:48:40.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcbf9600640 1 -- 172.21.15.29:0/781004049 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fcbf4014b70 con 0x5569fcb7e930 2024-04-17T19:48:40.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 1 -- 172.21.15.29:0/781004049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbf4029020 con 0x5569fcb7e930 2024-04-17T19:48:40.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcbf9600640 1 -- 172.21.15.29:0/781004049 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fcbf4038050 con 0x5569fcb7e930 2024-04-17T19:48:40.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 10 client.4959 fetch_fsmap learned FSMap version 19 2024-04-17T19:48:40.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 1 -- 172.21.15.29:0/781004049 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) -- 0x5569fcaa1670 con 0x5569fcb7e930 2024-04-17T19:48:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcbf9600640 1 -- 172.21.15.29:0/781004049 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbf4029020 con 0x5569fcb7e930 2024-04-17T19:48:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcbf9600640 1 -- 172.21.15.29:0/781004049 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 19) ==== 4792+0+0 (secure 0 0 0) 0x7fcbf403b020 con 0x5569fcb7e930 2024-04-17T19:48:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 10 client.4959 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T19:48:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 10 client.4959 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T19:48:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 20 client.4959 populate_metadata read hostname 'smithi029' 2024-04-17T19:48:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/781004049 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x5569fc790830 0x5569fcd2cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 4 client.4959 mds_command: new command op to 4305 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T19:48:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 1 -- 172.21.15.29:0/781004049 --> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] -- command(tid 0: {"prefix": "status"}) -- 0x5569fcb09f10 con 0x5569fc790830 2024-04-17T19:48:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc10c00640 1 --2- 172.21.15.29:0/781004049 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x5569fc790830 0x5569fcd2cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc10c00640 1 --2- 172.21.15.29:0/781004049 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x5569fc790830 0x5569fcd2cb40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb352514cf59444a server_cookie=e09e969c14ed15a2 in_seq=0 out_seq=0 2024-04-17T19:48:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcbf9600640 10 client.4959 ms_handle_connect on v2:172.21.15.29:6833/2691590158 2024-04-17T19:48:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcbf9600640 1 -- 172.21.15.29:0/781004049 <== mds.0 v2:172.21.15.29:6833/2691590158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5569fcb09f10 con 0x5569fc790830 2024-04-17T19:48:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcbf9600640 10 client.4959 handle_command_reply: tid=0 2024-04-17T19:48:40.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 1 client.4959 shutdown 2024-04-17T19:48:40.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 1 -- 172.21.15.29:0/781004049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 msgr2=0x5569fcd2bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/781004049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 0x5569fcd2bcd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x5569fcac29c0 tx=0x7fcbf4000ee0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 1 -- 172.21.15.29:0/781004049 shutdown_connections 2024-04-17T19:48:40.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/781004049 >> [v2:172.21.15.29:6833/2691590158,v1:172.21.15.29:6837/2691590158] conn(0x5569fc790830 0x5569fcd2cb40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/781004049 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x5569fcb85c10 0x5569fcd2c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/781004049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5569fcb7e930 0x5569fcd2bcd0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 1 --2- 172.21.15.29:0/781004049 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x5569fc77e280 0x5569fcd2b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcbf9600640 0 client.4959 ms_handle_reset on v2:172.21.15.29:6833/2691590158 2024-04-17T19:48:40.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 1 -- 172.21.15.29:0/781004049 >> 172.21.15.29:0/781004049 conn(0x5569fcbc7670 msgr2=0x5569fcce8380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 1 -- 172.21.15.29:0/781004049 shutdown_connections 2024-04-17T19:48:40.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 1 -- 172.21.15.29:0/781004049 wait complete. 2024-04-17T19:48:40.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc1ac52000 20 client.4959 trim_cache size 0 max 16384 2024-04-17T19:48:40.822 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-04-17T19:48:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "a680ee90-875c-4283-a901-86d7638393c3", 2024-04-17T19:48:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2024-04-17T19:48:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 4305, 2024-04-17T19:48:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2024-04-17T19:48:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2024-04-17T19:48:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2024-04-17T19:48:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 50.995571413999997, 2024-04-17T19:48:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 19, 2024-04-17T19:48:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 26, 2024-04-17T19:48:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 26, 2024-04-17T19:48:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 56.388508577000003 2024-04-17T19:48:40.822 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-04-17T19:48:40.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc13600640 1 -- 172.21.15.29:0/1505216449 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbe80683b0 msgr2=0x7fcbe806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc13600640 1 --2- 172.21.15.29:0/1505216449 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbe80683b0 0x7fcbe806a8c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc0ad000 tx=0x7fcbfc008040 comp rx=0 tx=0).stop 2024-04-17T19:48:40.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc13600640 1 -- 172.21.15.29:0/1505216449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c1597e0 msgr2=0x7fcc0c1524e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:40.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc13600640 1 --2- 172.21.15.29:0/1505216449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c1597e0 0x7fcc0c1524e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fcc04007e60 tx=0x7fcc0400e4b0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc13600640 1 -- 172.21.15.29:0/1505216449 shutdown_connections 2024-04-17T19:48:40.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc13600640 1 --2- 172.21.15.29:0/1505216449 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbe80683b0 0x7fcbe806a8c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc13600640 1 --2- 172.21.15.29:0/1505216449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c1597e0 0x7fcc0c1524e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc13600640 1 --2- 172.21.15.29:0/1505216449 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc0c1593d0 0x7fcc0c151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc13600640 1 --2- 172.21.15.29:0/1505216449 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c071ce0 0x7fcc0c157110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:40.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc13600640 1 -- 172.21.15.29:0/1505216449 >> 172.21.15.29:0/1505216449 conn(0x7fcc0c06bf90 msgr2=0x7fcc0c0739b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:40.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.828+0000 7fcc13600640 1 -- 172.21.15.29:0/1505216449 shutdown_connections 2024-04-17T19:48:40.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:40.832+0000 7fcc13600640 1 -- 172.21.15.29:0/1505216449 wait complete. 2024-04-17T19:48:40.839 INFO:teuthology.run_tasks:Running task sequential... 2024-04-17T19:48:40.849 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-17T19:48:40.849 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-17T19:48:40.850 INFO:teuthology.task.print:Enabling mgr modules 2024-04-17T19:48:40.850 INFO:teuthology.run_tasks:Running task kclient... 2024-04-17T19:48:40.888 INFO:tasks.kclient:Mounting kernel clients... 2024-04-17T19:48:40.888 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-17T19:48:40.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:40.888 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-04-17T19:48:40.897 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-17T19:48:40.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:40.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-17T19:48:40.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:40.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2024-04-17T19:48:41.134 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-04-17T19:48:41.134 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-17T19:48:41.134 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T19:48:41.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T19:48:41.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.296+0000 7f7471000640 1 Processor -- start 2024-04-17T19:48:41.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.296+0000 7f7471000640 1 -- start start 2024-04-17T19:48:41.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.300+0000 7f7471000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f746c0cc8c0 0x7f746c0cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:41.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.300+0000 7f7471000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f746c1597a0 0x7f746c1635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:41.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.300+0000 7f7471000640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f746c0c88b0 con 0x7f746c159390 2024-04-17T19:48:41.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.300+0000 7f7471000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f746c0c6ce0 con 0x7f746c0cc8c0 2024-04-17T19:48:41.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.300+0000 7f7471000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f746c0c6fc0 con 0x7f746c1597a0 2024-04-17T19:48:41.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.300+0000 7f7461400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f746c159390 0x7f746c0ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52242/0 (socket says 172.21.15.29:52242) 2024-04-17T19:48:41.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.300+0000 7f7461400640 1 -- 172.21.15.29:0/4234094121 learned_addr learned my addr 172.21.15.29:0/4234094121 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:41.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.300+0000 7f7469e00640 1 --2- 172.21.15.29:0/4234094121 >> v2:172.21.15.37:3300/0 conn(0x7f746c0cc8c0 0x7f746c0cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:41.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.300+0000 7f746a800640 1 --2- 172.21.15.29:0/4234094121 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f746c1597a0 0x7f746c1635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:41.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.300+0000 7f7469400640 1 -- 172.21.15.29:0/4234094121 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2605821329 0 0) 0x7f746c0c88b0 con 0x7f746c159390 2024-04-17T19:48:41.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.300+0000 7f7469400640 1 -- 172.21.15.29:0/4234094121 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7450003530 con 0x7f746c159390 2024-04-17T19:48:41.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.300+0000 7f7469400640 1 -- 172.21.15.29:0/4234094121 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2944707227 0 0) 0x7f7450003530 con 0x7f746c159390 2024-04-17T19:48:41.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.300+0000 7f746a800640 1 -- 172.21.15.29:0/4234094121 >> v2:172.21.15.37:3300/0 conn(0x7f746c0cc8c0 msgr2=0x7f746c0cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:41.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.300+0000 7f746a800640 1 --2- 172.21.15.29:0/4234094121 >> v2:172.21.15.37:3300/0 conn(0x7f746c0cc8c0 0x7f746c0cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:41.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.300+0000 7f746a800640 1 -- 172.21.15.29:0/4234094121 >> v1:172.21.15.29:6789/0 conn(0x7f746c159390 legacy=0x7f746c0ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:41.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.300+0000 7f746a800640 1 -- 172.21.15.29:0/4234094121 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746c0c72a0 con 0x7f746c1597a0 2024-04-17T19:48:41.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.304+0000 7f746a800640 1 --2- 172.21.15.29:0/4234094121 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f746c1597a0 0x7f746c1635e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f74640097c0 tx=0x7f746401c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=deeee0adae097176 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:41.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.304+0000 7f7469400640 1 -- 172.21.15.29:0/4234094121 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74640b0070 con 0x7f746c1597a0 2024-04-17T19:48:41.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.304+0000 7f7469400640 1 -- 172.21.15.29:0/4234094121 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74640ab400 con 0x7f746c1597a0 2024-04-17T19:48:41.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.304+0000 7f7469400640 1 -- 172.21.15.29:0/4234094121 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7454001680 con 0x7f746c1597a0 2024-04-17T19:48:41.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.304+0000 7f7471000640 1 -- 172.21.15.29:0/4234094121 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f746c1597a0 msgr2=0x7f746c1635e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:41.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.304+0000 7f7471000640 1 --2- 172.21.15.29:0/4234094121 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f746c1597a0 0x7f746c1635e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f74640097c0 tx=0x7f746401c3c0 comp rx=0 tx=0).stop 2024-04-17T19:48:41.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.304+0000 7f7471000640 1 -- 172.21.15.29:0/4234094121 shutdown_connections 2024-04-17T19:48:41.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.304+0000 7f7471000640 1 --2- 172.21.15.29:0/4234094121 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f746c1597a0 0x7f746c1635e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:41.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.304+0000 7f7471000640 1 --2- 172.21.15.29:0/4234094121 >> v2:172.21.15.37:3300/0 conn(0x7f746c0cc8c0 0x7f746c0cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:41.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.304+0000 7f7471000640 1 -- 172.21.15.29:0/4234094121 >> 172.21.15.29:0/4234094121 conn(0x7f746c06fde0 msgr2=0x7f746c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:41.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.304+0000 7f7471000640 1 -- 172.21.15.29:0/4234094121 shutdown_connections 2024-04-17T19:48:41.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.304+0000 7f7471000640 1 -- 172.21.15.29:0/4234094121 wait complete. 2024-04-17T19:48:41.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f7471000640 1 Processor -- start 2024-04-17T19:48:41.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f7471000640 1 -- start start 2024-04-17T19:48:41.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f7471000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f746c0cc8c0 0x7f746c1f5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:41.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f7471000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f746c159390 0x7f746c13be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:41.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f7471000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f746c1597a0 0x7f746c073910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:41.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f7471000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f746c0cabc0 con 0x7f746c1597a0 2024-04-17T19:48:41.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f7471000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f746c0caa40 con 0x7f746c0cc8c0 2024-04-17T19:48:41.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f7471000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f746c0c6880 con 0x7f746c159390 2024-04-17T19:48:41.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f746a800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f746c1597a0 0x7f746c073910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:41.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f746a800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f746c1597a0 0x7f746c073910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33324/0 (socket says 172.21.15.29:33324) 2024-04-17T19:48:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f746a800640 1 -- 172.21.15.29:0/409401575 learned_addr learned my addr 172.21.15.29:0/409401575 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f7469e00640 1 --2- 172.21.15.29:0/409401575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f746c0cc8c0 0x7f746c1f5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f7461400640 1 --2- 172.21.15.29:0/409401575 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f746c159390 0x7f746c13be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f746a800640 1 -- 172.21.15.29:0/409401575 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f746c159390 msgr2=0x7f746c13be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f746a800640 1 --2- 172.21.15.29:0/409401575 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f746c159390 0x7f746c13be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f746a800640 1 -- 172.21.15.29:0/409401575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f746c0cc8c0 msgr2=0x7f746c1f5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f746a800640 1 --2- 172.21.15.29:0/409401575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f746c0cc8c0 0x7f746c1f5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f746a800640 1 -- 172.21.15.29:0/409401575 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746c1f6120 con 0x7f746c1597a0 2024-04-17T19:48:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f7461400640 1 --2- 172.21.15.29:0/409401575 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f746c159390 0x7f746c13be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f7469e00640 1 --2- 172.21.15.29:0/409401575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f746c0cc8c0 0x7f746c1f5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f746a800640 1 --2- 172.21.15.29:0/409401575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f746c1597a0 0x7f746c073910 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f74640ae040 tx=0x7f7464019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f7462a00640 1 -- 172.21.15.29:0/409401575 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74640ba070 con 0x7f746c1597a0 2024-04-17T19:48:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f7462a00640 1 -- 172.21.15.29:0/409401575 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7464003820 con 0x7f746c1597a0 2024-04-17T19:48:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f7471000640 1 -- 172.21.15.29:0/409401575 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f746c162260 con 0x7f746c1597a0 2024-04-17T19:48:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f7462a00640 1 -- 172.21.15.29:0/409401575 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74640b0040 con 0x7f746c1597a0 2024-04-17T19:48:41.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.308+0000 7f7471000640 1 -- 172.21.15.29:0/409401575 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f746c162500 con 0x7f746c1597a0 2024-04-17T19:48:41.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.312+0000 7f7471000640 1 -- 172.21.15.29:0/409401575 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f746c069b90 con 0x7f746c1597a0 2024-04-17T19:48:41.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.312+0000 7f7462a00640 1 -- 172.21.15.29:0/409401575 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f74640bd070 con 0x7f746c1597a0 2024-04-17T19:48:41.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.312+0000 7f7462a00640 1 --2- 172.21.15.29:0/409401575 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7448068380 0x7f744806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:41.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.312+0000 7f7462a00640 1 -- 172.21.15.29:0/409401575 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f746411fce0 con 0x7f746c1597a0 2024-04-17T19:48:41.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.312+0000 7f7469e00640 1 --2- 172.21.15.29:0/409401575 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7448068380 0x7f744806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:41.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.316+0000 7f7469e00640 1 --2- 172.21.15.29:0/409401575 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7448068380 0x7f744806a890 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f745c009fd0 tx=0x7f745c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:41.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.320+0000 7f7462a00640 1 -- 172.21.15.29:0/409401575 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7464015fb0 con 0x7f746c1597a0 2024-04-17T19:48:41.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.520+0000 7f7471000640 1 -- 172.21.15.29:0/409401575 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f746c1f9780 con 0x7f746c1597a0 2024-04-17T19:48:41.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.520+0000 7f7462a00640 1 -- 172.21.15.29:0/409401575 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v19) ==== 53+0+88 (secure 0 0 0) 0x7f74640efe10 con 0x7f746c1597a0 2024-04-17T19:48:41.509 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-17T19:48:41.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.520+0000 7f7471000640 1 -- 172.21.15.29:0/409401575 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7448068380 msgr2=0x7f744806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:41.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.520+0000 7f7471000640 1 --2- 172.21.15.29:0/409401575 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7448068380 0x7f744806a890 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f745c009fd0 tx=0x7f745c00b040 comp rx=0 tx=0).stop 2024-04-17T19:48:41.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.520+0000 7f7471000640 1 -- 172.21.15.29:0/409401575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f746c1597a0 msgr2=0x7f746c073910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:41.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.520+0000 7f7471000640 1 --2- 172.21.15.29:0/409401575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f746c1597a0 0x7f746c073910 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f74640ae040 tx=0x7f7464019160 comp rx=0 tx=0).stop 2024-04-17T19:48:41.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.520+0000 7f7471000640 1 -- 172.21.15.29:0/409401575 shutdown_connections 2024-04-17T19:48:41.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.520+0000 7f7471000640 1 --2- 172.21.15.29:0/409401575 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7448068380 0x7f744806a890 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:41.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.520+0000 7f7471000640 1 --2- 172.21.15.29:0/409401575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f746c1597a0 0x7f746c073910 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:41.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.520+0000 7f7471000640 1 --2- 172.21.15.29:0/409401575 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f746c159390 0x7f746c13be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:41.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.520+0000 7f7471000640 1 --2- 172.21.15.29:0/409401575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f746c0cc8c0 0x7f746c1f5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:41.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.520+0000 7f7471000640 1 -- 172.21.15.29:0/409401575 >> 172.21.15.29:0/409401575 conn(0x7f746c06fde0 msgr2=0x7f746c0cba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:41.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.520+0000 7f7471000640 1 -- 172.21.15.29:0/409401575 shutdown_connections 2024-04-17T19:48:41.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41.520+0000 7f7471000640 1 -- 172.21.15.29:0/409401575 wait complete. 2024-04-17T19:48:41.531 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-17T19:48:41.531 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T19:48:41.531 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2024-04-17T19:48:41.531 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T19:48:41.531 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T19:48:41.531 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-17T19:48:41.532 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-17T19:48:41.532 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T19:48:41.532 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T19:48:41.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:41.532 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2024-04-17T19:48:41.541 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T19:48:41.541 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T19:48:41.541 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T19:48:41.541 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-04-17T19:48:41.541 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2024-04-17T19:48:41.541 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-04-17T19:48:41.541 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T19:48:41.541 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2024-04-17T19:48:41.541 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2024-04-17T19:48:41.541 INFO:teuthology.orchestra.run.smithi029.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T19:48:41.541 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2024-04-17T19:48:41.541 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2024-04-17T19:48:41.541 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-17T19:48:41.541 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 42319sec preferred_lft 42319sec 2024-04-17T19:48:41.542 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link 2024-04-17T19:48:41.542 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-04-17T19:48:41.542 INFO:teuthology.orchestra.run.smithi029.stdout:4: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T19:48:41.542 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2024-04-17T19:48:41.542 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2024-04-17T19:48:41.542 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T19:48:41.542 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2024-04-17T19:48:41.542 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2024-04-17T19:48:41.542 INFO:teuthology.orchestra.run.smithi029.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-17T19:48:41.542 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 02:42:9c:5a:03:85 brd ff:ff:ff:ff:ff:ff 2024-04-17T19:48:41.542 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-17T19:48:41.542 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2024-04-17T19:48:41.543 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-17T19:48:41.543 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T19:48:41.543 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-04-17T19:48:41.544 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add name ceph-brx type bridge 2024-04-17T19:48:41.544 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr flush dev ceph-brx 2024-04-17T19:48:41.544 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set ceph-brx up 2024-04-17T19:48:41.544 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-17T19:48:41.544 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-04-17T19:48:41.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T19:48:41.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T19:48:41.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:41.705 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T19:48:41.770 INFO:teuthology.orchestra.run.smithi029.stdout:1 2024-04-17T19:48:41.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:41.772 DEBUG:teuthology.orchestra.run.smithi029:> route 2024-04-17T19:48:41.829 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2024-04-17T19:48:41.829 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T19:48:41.829 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 0 0 0 ens1f0 2024-04-17T19:48:41.829 INFO:teuthology.orchestra.run.smithi029.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-17T19:48:41.829 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 ens1f0 2024-04-17T19:48:41.829 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T19:48:41.829 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T19:48:41.829 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-04-17T19:48:41.829 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-17T19:48:41.829 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-17T19:48:41.830 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-17T19:48:41.830 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-04-17T19:48:41.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T19:48:41.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T19:48:41.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:41.971 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2024-04-17T19:48:41.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:41.981 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list-id 2024-04-17T19:48:42.035 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T19:48:42.035 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-04-17T19:48:42.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T19:48:42.036 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-17T19:48:42.036 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-04-17T19:48:42.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T19:48:42.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T19:48:42.163 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-17T19:48:42.163 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T19:48:42.163 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-04-17T19:48:42.163 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-17T19:48:42.163 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-17T19:48:42.163 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-17T19:48:42.164 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-17T19:48:42.164 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-17T19:48:42.164 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-04-17T19:48:42.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T19:48:42.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T19:48:42.352 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-17T19:48:42.353 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2024-04-17T19:48:42.353 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2024-04-17T19:48:42.353 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set dev brx.0 master ceph-brx 2024-04-17T19:48:42.353 DEBUG:teuthology.orchestra.run.smithi029:> ') 2024-04-17T19:48:42.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T19:48:42.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T19:48:42.437 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T19:48:42.437 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T19:48:42.446 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T19:48:42.446 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T19:48:42.446 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T19:48:42.494 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-04-17T19:48:42.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:42.495 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-17T19:48:42.600 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-17T19:48:42.600 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-17T19:48:42.600 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2024-04-17T19:48:43.242 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-17T19:48:43.242 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-17T19:48:43.242 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2024-04-17T19:48:43.242 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.37:3300,172.21.15.37:3301" 2024-04-17T19:48:43.242 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-17T19:48:43.511 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-04-17T19:48:43.512 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-04-17T19:48:43.512 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2024-04-17T19:48:43.512 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.37:3300,172.21.15.37:3301" 2024-04-17T19:48:43.512 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2024-04-17T19:48:43.512 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.37:6789,172.21.15.37:6790" 2024-04-17T19:48:43.512 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.37:6789,172.21.15.37:6790:/ 2024-04-17T19:48:43.512 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=a680ee90-875c-4283-a901-86d7638393c3" will pass to kernel 2024-04-17T19:48:43.516 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T19:48:43.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T19:48:43.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T19:48:43.536 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T19:48:43.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T19:48:43.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T19:48:43.642 INFO:teuthology.orchestra.run.smithi029.stdout:client4972 2024-04-17T19:48:43.642 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:48:43.642 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T19:48:43.651 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T19:48:43.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T19:48:43.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T19:48:43.753 INFO:teuthology.orchestra.run.smithi029.stdout:client4972 2024-04-17T19:48:43.753 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T19:48:43.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-17T19:48:43.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T19:48:43.811 INFO:teuthology.orchestra.run.smithi029.stdout:client4972 2024-04-17T19:48:43.811 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/a680ee90-875c-4283-a901-86d7638393c3.client4972/status') 2024-04-17T19:48:43.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-17T19:48:43.872 INFO:teuthology.orchestra.run.smithi029.stdout:instance: client.4972 (3)172.21.15.29:0/2994692149 2024-04-17T19:48:43.872 INFO:teuthology.orchestra.run.smithi029.stdout:blocklisted: false 2024-04-17T19:48:43.872 INFO:teuthology.orchestra.run.smithi029.stderr:0+1 records in 2024-04-17T19:48:43.872 INFO:teuthology.orchestra.run.smithi029.stderr:0+1 records out 2024-04-17T19:48:43.872 INFO:teuthology.orchestra.run.smithi029.stderr:70 bytes copied, 7.0908e-05 s, 987 kB/s 2024-04-17T19:48:43.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T19:48:43.880 INFO:teuthology.run_tasks:Running task mds_thrash... 2024-04-17T19:48:43.896 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:48:44.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.060+0000 7ff491800640 1 Processor -- start 2024-04-17T19:48:44.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.060+0000 7ff491800640 1 -- start start 2024-04-17T19:48:44.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.060+0000 7ff491800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff48c0c6f70 0x7ff48c0c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:44.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.060+0000 7ff491800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff48c15a0a0 con 0x7ff48c06c660 2024-04-17T19:48:44.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.060+0000 7ff491800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff48c15abe0 con 0x7ff48c159c90 2024-04-17T19:48:44.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.060+0000 7ff491800640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7ff48c15a340 con 0x7ff48c0c6f70 2024-04-17T19:48:44.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.060+0000 7ff483e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff48c0c6f70 0x7ff48c0c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:44.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.060+0000 7ff48b400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff48c06c660 0x7ff48c06ca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52246/0 (socket says 172.21.15.29:52246) 2024-04-17T19:48:44.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.060+0000 7ff48b400640 1 -- 172.21.15.29:0/3016721592 learned_addr learned my addr 172.21.15.29:0/3016721592 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:44.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.060+0000 7ff483e00640 1 -- 172.21.15.29:0/3016721592 >> v1:172.21.15.37:6789/0 conn(0x7ff48c159c90 legacy=0x7ff48c0ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:44.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.060+0000 7ff483e00640 1 -- 172.21.15.29:0/3016721592 >> v1:172.21.15.29:6789/0 conn(0x7ff48c06c660 legacy=0x7ff48c06ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:44.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.060+0000 7ff483e00640 1 -- 172.21.15.29:0/3016721592 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48c15a620 con 0x7ff48c0c6f70 2024-04-17T19:48:44.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.064+0000 7ff483e00640 1 --2- 172.21.15.29:0/3016721592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff48c0c6f70 0x7ff48c0c7380 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff4740097c0 tx=0x7ff47401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7cdc955ef99bd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:44.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.064+0000 7ff48a000640 1 -- 172.21.15.29:0/3016721592 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4740b0070 con 0x7ff48c0c6f70 2024-04-17T19:48:44.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.064+0000 7ff48a000640 1 -- 172.21.15.29:0/3016721592 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4740ab600 con 0x7ff48c0c6f70 2024-04-17T19:48:44.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.064+0000 7ff48a000640 1 -- 172.21.15.29:0/3016721592 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4740ab9d0 con 0x7ff48c0c6f70 2024-04-17T19:48:44.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.064+0000 7ff491800640 1 -- 172.21.15.29:0/3016721592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff48c0c6f70 msgr2=0x7ff48c0c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:44.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.064+0000 7ff491800640 1 --2- 172.21.15.29:0/3016721592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff48c0c6f70 0x7ff48c0c7380 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff4740097c0 tx=0x7ff47401c4a0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.064+0000 7ff491800640 1 -- 172.21.15.29:0/3016721592 shutdown_connections 2024-04-17T19:48:44.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.064+0000 7ff491800640 1 --2- 172.21.15.29:0/3016721592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff48c0c6f70 0x7ff48c0c7380 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.064+0000 7ff491800640 1 -- 172.21.15.29:0/3016721592 >> 172.21.15.29:0/3016721592 conn(0x7ff48c06fde0 msgr2=0x7ff48c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:44.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.064+0000 7ff491800640 1 -- 172.21.15.29:0/3016721592 shutdown_connections 2024-04-17T19:48:44.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.064+0000 7ff491800640 1 -- 172.21.15.29:0/3016721592 wait complete. 2024-04-17T19:48:44.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff491800640 1 Processor -- start 2024-04-17T19:48:44.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff491800640 1 -- start start 2024-04-17T19:48:44.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff491800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff48c06c660 0x7ff48c154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:44.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff491800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff48c0c6f70 0x7ff48c1f65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:44.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff491800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff48c159c90 0x7ff48c073520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:44.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff491800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c0cab40 con 0x7ff48c0c6f70 2024-04-17T19:48:44.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff491800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c0ca9c0 con 0x7ff48c06c660 2024-04-17T19:48:44.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff491800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff48c167e40 con 0x7ff48c159c90 2024-04-17T19:48:44.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff483e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff48c0c6f70 0x7ff48c1f65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:44.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff483e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff48c0c6f70 0x7ff48c1f65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33346/0 (socket says 172.21.15.29:33346) 2024-04-17T19:48:44.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff483e00640 1 -- 172.21.15.29:0/2543082413 learned_addr learned my addr 172.21.15.29:0/2543082413 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:44.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff48aa00640 1 --2- 172.21.15.29:0/2543082413 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff48c06c660 0x7ff48c154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:44.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff48b400640 1 --2- 172.21.15.29:0/2543082413 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff48c159c90 0x7ff48c073520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:44.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff483e00640 1 -- 172.21.15.29:0/2543082413 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff48c159c90 msgr2=0x7ff48c073520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:44.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff483e00640 1 --2- 172.21.15.29:0/2543082413 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff48c159c90 0x7ff48c073520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff483e00640 1 -- 172.21.15.29:0/2543082413 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff48c06c660 msgr2=0x7ff48c154f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:44.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff483e00640 1 --2- 172.21.15.29:0/2543082413 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff48c06c660 0x7ff48c154f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff483e00640 1 -- 172.21.15.29:0/2543082413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48c1f6af0 con 0x7ff48c0c6f70 2024-04-17T19:48:44.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.068+0000 7ff483e00640 1 --2- 172.21.15.29:0/2543082413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff48c0c6f70 0x7ff48c1f65b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff4740190b0 tx=0x7ff47401ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:44.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.072+0000 7ff482a00640 1 -- 172.21.15.29:0/2543082413 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4740b0050 con 0x7ff48c0c6f70 2024-04-17T19:48:44.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.072+0000 7ff482a00640 1 -- 172.21.15.29:0/2543082413 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4740bc070 con 0x7ff48c0c6f70 2024-04-17T19:48:44.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.072+0000 7ff482a00640 1 -- 172.21.15.29:0/2543082413 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4740b5a30 con 0x7ff48c0c6f70 2024-04-17T19:48:44.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.072+0000 7ff491800640 1 -- 172.21.15.29:0/2543082413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff48c1f9f10 con 0x7ff48c0c6f70 2024-04-17T19:48:44.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.072+0000 7ff491800640 1 -- 172.21.15.29:0/2543082413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff48c1f5dd0 con 0x7ff48c0c6f70 2024-04-17T19:48:44.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.072+0000 7ff491800640 1 -- 172.21.15.29:0/2543082413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff48c069b90 con 0x7ff48c0c6f70 2024-04-17T19:48:44.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.072+0000 7ff482a00640 1 -- 172.21.15.29:0/2543082413 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff4740ab4c0 con 0x7ff48c0c6f70 2024-04-17T19:48:44.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.072+0000 7ff482a00640 1 --2- 172.21.15.29:0/2543082413 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff468068380 0x7ff46806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:44.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.072+0000 7ff482a00640 1 -- 172.21.15.29:0/2543082413 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7ff4741215f0 con 0x7ff48c0c6f70 2024-04-17T19:48:44.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.072+0000 7ff48aa00640 1 --2- 172.21.15.29:0/2543082413 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff468068380 0x7ff46806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:44.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.076+0000 7ff48aa00640 1 --2- 172.21.15.29:0/2543082413 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff468068380 0x7ff46806a890 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff48c0c8e90 tx=0x7ff47c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:44.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.080+0000 7ff482a00640 1 -- 172.21.15.29:0/2543082413 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff4740c6020 con 0x7ff48c0c6f70 2024-04-17T19:48:44.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.244+0000 7ff491800640 1 -- 172.21.15.29:0/2543082413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff48c164ca0 con 0x7ff48c0c6f70 2024-04-17T19:48:44.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.244+0000 7ff482a00640 1 -- 172.21.15.29:0/2543082413 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18780 (secure 0 0 0) 0x7ff474016090 con 0x7ff48c0c6f70 2024-04-17T19:48:44.234 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:44.235 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":26,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:48:29.829433+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T19:47:47.658449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T19:47:48.209047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T19:47:51.887909+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:48:44.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.248+0000 7ff491800640 1 -- 172.21.15.29:0/2543082413 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff468068380 msgr2=0x7ff46806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:44.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.248+0000 7ff491800640 1 --2- 172.21.15.29:0/2543082413 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff468068380 0x7ff46806a890 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff48c0c8e90 tx=0x7ff47c009040 comp rx=0 tx=0).stop 2024-04-17T19:48:44.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.248+0000 7ff491800640 1 -- 172.21.15.29:0/2543082413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff48c0c6f70 msgr2=0x7ff48c1f65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:44.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.248+0000 7ff491800640 1 --2- 172.21.15.29:0/2543082413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff48c0c6f70 0x7ff48c1f65b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff4740190b0 tx=0x7ff47401ace0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.248+0000 7ff491800640 1 -- 172.21.15.29:0/2543082413 shutdown_connections 2024-04-17T19:48:44.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.248+0000 7ff491800640 1 --2- 172.21.15.29:0/2543082413 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff468068380 0x7ff46806a890 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.248+0000 7ff491800640 1 --2- 172.21.15.29:0/2543082413 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff48c159c90 0x7ff48c073520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.248+0000 7ff491800640 1 --2- 172.21.15.29:0/2543082413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff48c0c6f70 0x7ff48c1f65b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.248+0000 7ff491800640 1 --2- 172.21.15.29:0/2543082413 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff48c06c660 0x7ff48c154f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.248+0000 7ff491800640 1 -- 172.21.15.29:0/2543082413 >> 172.21.15.29:0/2543082413 conn(0x7ff48c06fde0 msgr2=0x7ff48c153390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:44.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.248+0000 7ff491800640 1 -- 172.21.15.29:0/2543082413 shutdown_connections 2024-04-17T19:48:44.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.248+0000 7ff491800640 1 -- 172.21.15.29:0/2543082413 wait complete. 2024-04-17T19:48:44.257 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T19:47:36.674676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T19:47:47.658449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T19:47:48.209047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T19:47:51.887909+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T19:48:44.258 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T19:48:44.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.424+0000 7f6a33600640 1 Processor -- start 2024-04-17T19:48:44.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.424+0000 7f6a33600640 1 -- start start 2024-04-17T19:48:44.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.424+0000 7f6a33600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6a2c1593d0 0x7f6a2c1597e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:44.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.424+0000 7f6a33600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a2c0c9f00 0x7f6a2c0ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:44.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.424+0000 7f6a33600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a2c15c6d0 con 0x7f6a2c071ce0 2024-04-17T19:48:44.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.424+0000 7f6a33600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f6a2c0c7bd0 con 0x7f6a2c1593d0 2024-04-17T19:48:44.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.424+0000 7f6a33600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6a2c15c9b0 con 0x7f6a2c0c9f00 2024-04-17T19:48:44.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.424+0000 7f6a30c00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6a2c071ce0 0x7f6a2c141de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52260/0 (socket says 172.21.15.29:52260) 2024-04-17T19:48:44.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.424+0000 7f6a30c00640 1 -- 172.21.15.29:0/3989344585 learned_addr learned my addr 172.21.15.29:0/3989344585 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:44.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.424+0000 7f6a2be00640 1 --2- 172.21.15.29:0/3989344585 >> v2:172.21.15.29:3300/0 conn(0x7f6a2c1593d0 0x7f6a2c1597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:44.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.424+0000 7f6a31600640 1 --2- 172.21.15.29:0/3989344585 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a2c0c9f00 0x7f6a2c0ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:44.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.424+0000 7f6a2b400640 1 -- 172.21.15.29:0/3989344585 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 994598756 0 0) 0x7f6a2c15c6d0 con 0x7f6a2c071ce0 2024-04-17T19:48:44.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.424+0000 7f6a2b400640 1 -- 172.21.15.29:0/3989344585 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a14003530 con 0x7f6a2c071ce0 2024-04-17T19:48:44.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.424+0000 7f6a2be00640 1 -- 172.21.15.29:0/3989344585 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a2c0c9f00 msgr2=0x7f6a2c0ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:44.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.424+0000 7f6a2be00640 1 --2- 172.21.15.29:0/3989344585 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a2c0c9f00 0x7f6a2c0ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.424+0000 7f6a2be00640 1 -- 172.21.15.29:0/3989344585 >> v1:172.21.15.29:6789/0 conn(0x7f6a2c071ce0 legacy=0x7f6a2c141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:44.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.424+0000 7f6a2be00640 1 -- 172.21.15.29:0/3989344585 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a2c15cc90 con 0x7f6a2c1593d0 2024-04-17T19:48:44.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.428+0000 7f6a2be00640 1 --2- 172.21.15.29:0/3989344585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a2c1593d0 0x7f6a2c1597e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6a180097c0 tx=0x7f6a1801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65ababce3cc1dff server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:44.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.428+0000 7f6a2b400640 1 -- 172.21.15.29:0/3989344585 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a180b1070 con 0x7f6a2c1593d0 2024-04-17T19:48:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.428+0000 7f6a2b400640 1 -- 172.21.15.29:0/3989344585 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a180ac410 con 0x7f6a2c1593d0 2024-04-17T19:48:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.428+0000 7f6a2b400640 1 -- 172.21.15.29:0/3989344585 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a180ac7c0 con 0x7f6a2c1593d0 2024-04-17T19:48:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.428+0000 7f6a33600640 1 -- 172.21.15.29:0/3989344585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a2c1593d0 msgr2=0x7f6a2c1597e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.428+0000 7f6a33600640 1 --2- 172.21.15.29:0/3989344585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a2c1593d0 0x7f6a2c1597e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6a180097c0 tx=0x7f6a1801c4a0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.428+0000 7f6a33600640 1 -- 172.21.15.29:0/3989344585 shutdown_connections 2024-04-17T19:48:44.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.428+0000 7f6a33600640 1 --2- 172.21.15.29:0/3989344585 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a2c0c9f00 0x7f6a2c0ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.428+0000 7f6a33600640 1 --2- 172.21.15.29:0/3989344585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a2c1593d0 0x7f6a2c1597e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.428+0000 7f6a33600640 1 -- 172.21.15.29:0/3989344585 >> 172.21.15.29:0/3989344585 conn(0x7f6a2c06bf90 msgr2=0x7f6a2c06c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:44.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.428+0000 7f6a33600640 1 -- 172.21.15.29:0/3989344585 shutdown_connections 2024-04-17T19:48:44.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.428+0000 7f6a33600640 1 -- 172.21.15.29:0/3989344585 wait complete. 2024-04-17T19:48:44.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a33600640 1 Processor -- start 2024-04-17T19:48:44.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a33600640 1 -- start start 2024-04-17T19:48:44.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a33600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a2c071ce0 0x7f6a2c1383f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:44.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a33600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a2c0c9f00 0x7f6a2c151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:44.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a33600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a2c1593d0 0x7f6a2c13fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:44.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a33600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2c15c3c0 con 0x7f6a2c071ce0 2024-04-17T19:48:44.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a33600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6a2c15c240 con 0x7f6a2c1593d0 2024-04-17T19:48:44.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a33600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6a2c165e40 con 0x7f6a2c0c9f00 2024-04-17T19:48:44.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a30c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a2c071ce0 0x7f6a2c1383f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:44.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a30c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a2c071ce0 0x7f6a2c1383f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33360/0 (socket says 172.21.15.29:33360) 2024-04-17T19:48:44.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a30c00640 1 -- 172.21.15.29:0/492031782 learned_addr learned my addr 172.21.15.29:0/492031782 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:44.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a2be00640 1 --2- 172.21.15.29:0/492031782 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a2c0c9f00 0x7f6a2c151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:44.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a31600640 1 --2- 172.21.15.29:0/492031782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a2c1593d0 0x7f6a2c13fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:44.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a30c00640 1 -- 172.21.15.29:0/492031782 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a2c0c9f00 msgr2=0x7f6a2c151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:44.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a30c00640 1 --2- 172.21.15.29:0/492031782 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a2c0c9f00 0x7f6a2c151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a30c00640 1 -- 172.21.15.29:0/492031782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a2c1593d0 msgr2=0x7f6a2c13fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:44.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a30c00640 1 --2- 172.21.15.29:0/492031782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a2c1593d0 0x7f6a2c13fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a30c00640 1 -- 172.21.15.29:0/492031782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a2c138930 con 0x7f6a2c071ce0 2024-04-17T19:48:44.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a2be00640 1 --2- 172.21.15.29:0/492031782 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a2c0c9f00 0x7f6a2c151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:44.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a31600640 1 --2- 172.21.15.29:0/492031782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a2c1593d0 0x7f6a2c13fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:44.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.432+0000 7f6a30c00640 1 --2- 172.21.15.29:0/492031782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a2c071ce0 0x7f6a2c1383f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6a2000c3f0 tx=0x7f6a2000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:44.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.436+0000 7f6a28c00640 1 -- 172.21.15.29:0/492031782 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a20017070 con 0x7f6a2c071ce0 2024-04-17T19:48:44.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.436+0000 7f6a28c00640 1 -- 172.21.15.29:0/492031782 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a20005cf0 con 0x7f6a2c071ce0 2024-04-17T19:48:44.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.436+0000 7f6a28c00640 1 -- 172.21.15.29:0/492031782 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a200072f0 con 0x7f6a2c071ce0 2024-04-17T19:48:44.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.436+0000 7f6a33600640 1 -- 172.21.15.29:0/492031782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a2c161930 con 0x7f6a2c071ce0 2024-04-17T19:48:44.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.436+0000 7f6a33600640 1 -- 172.21.15.29:0/492031782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a2c161bd0 con 0x7f6a2c071ce0 2024-04-17T19:48:44.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.436+0000 7f6a33600640 1 -- 172.21.15.29:0/492031782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a2c069b90 con 0x7f6a2c071ce0 2024-04-17T19:48:44.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.436+0000 7f6a28c00640 1 -- 172.21.15.29:0/492031782 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6a2001a070 con 0x7f6a2c071ce0 2024-04-17T19:48:44.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.436+0000 7f6a28c00640 1 --2- 172.21.15.29:0/492031782 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6a0c068380 0x7f6a0c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:44.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.436+0000 7f6a28c00640 1 -- 172.21.15.29:0/492031782 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f6a200862c0 con 0x7f6a2c071ce0 2024-04-17T19:48:44.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.436+0000 7f6a2be00640 1 --2- 172.21.15.29:0/492031782 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6a0c068380 0x7f6a0c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:44.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.440+0000 7f6a2be00640 1 --2- 172.21.15.29:0/492031782 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6a0c068380 0x7f6a0c06a890 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6a180ae000 tx=0x7f6a18008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:44.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.440+0000 7f6a28c00640 1 -- 172.21.15.29:0/492031782 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6a200563e0 con 0x7f6a2c071ce0 2024-04-17T19:48:44.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.608+0000 7f6a33600640 1 -- 172.21.15.29:0/492031782 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6a2c1528b0 con 0x7f6a2c071ce0 2024-04-17T19:48:44.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.608+0000 7f6a28c00640 1 -- 172.21.15.29:0/492031782 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f6a20055ce0 con 0x7f6a2c071ce0 2024-04-17T19:48:44.597 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-04-17T19:48:44.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.608+0000 7f6a33600640 1 -- 172.21.15.29:0/492031782 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6a0c068380 msgr2=0x7f6a0c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:44.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.608+0000 7f6a33600640 1 --2- 172.21.15.29:0/492031782 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6a0c068380 0x7f6a0c06a890 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f6a180ae000 tx=0x7f6a18008040 comp rx=0 tx=0).stop 2024-04-17T19:48:44.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.608+0000 7f6a33600640 1 -- 172.21.15.29:0/492031782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a2c071ce0 msgr2=0x7f6a2c1383f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:44.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.608+0000 7f6a33600640 1 --2- 172.21.15.29:0/492031782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a2c071ce0 0x7f6a2c1383f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6a2000c3f0 tx=0x7f6a2000c8b0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.608+0000 7f6a33600640 1 -- 172.21.15.29:0/492031782 shutdown_connections 2024-04-17T19:48:44.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.608+0000 7f6a33600640 1 --2- 172.21.15.29:0/492031782 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6a0c068380 0x7f6a0c06a890 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.608+0000 7f6a33600640 1 --2- 172.21.15.29:0/492031782 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6a2c1593d0 0x7f6a2c13fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.608+0000 7f6a33600640 1 --2- 172.21.15.29:0/492031782 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6a2c0c9f00 0x7f6a2c151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.608+0000 7f6a33600640 1 --2- 172.21.15.29:0/492031782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a2c071ce0 0x7f6a2c1383f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.608+0000 7f6a33600640 1 -- 172.21.15.29:0/492031782 >> 172.21.15.29:0/492031782 conn(0x7f6a2c06bf90 msgr2=0x7f6a2c13cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:44.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.608+0000 7f6a33600640 1 -- 172.21.15.29:0/492031782 shutdown_connections 2024-04-17T19:48:44.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.608+0000 7f6a33600640 1 -- 172.21.15.29:0/492031782 wait complete. 2024-04-17T19:48:44.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T19:48:44.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.784+0000 7f70ab200640 1 Processor -- start 2024-04-17T19:48:44.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.784+0000 7f70ab200640 1 -- start start 2024-04-17T19:48:44.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.788+0000 7f70ab200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f70a406ccb0 0x7f70a4152f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:44.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.788+0000 7f70ab200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f70a40c9080 0x7f70a415c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:44.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.788+0000 7f70ab200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a415cc30 con 0x7f70a415c670 2024-04-17T19:48:44.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.788+0000 7f70ab200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f70a415ff10 con 0x7f70a40c9080 2024-04-17T19:48:44.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.788+0000 7f70ab200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f70a415cef0 con 0x7f70a406ccb0 2024-04-17T19:48:44.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.788+0000 7f709b400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f70a40c9080 0x7f70a415c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:44.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.788+0000 7f709b400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f70a40c9080 0x7f70a415c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33376/0 (socket says 172.21.15.29:33376) 2024-04-17T19:48:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.788+0000 7f709b400640 1 -- 172.21.15.29:0/472984154 learned_addr learned my addr 172.21.15.29:0/472984154 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.788+0000 7f70a3e00640 1 --2- 172.21.15.29:0/472984154 >> v2:172.21.15.37:3300/0 conn(0x7f70a406ccb0 0x7f70a4152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.788+0000 7f709b400640 1 -- 172.21.15.29:0/472984154 >> v2:172.21.15.37:3300/0 conn(0x7f70a406ccb0 msgr2=0x7f70a4152f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.788+0000 7f709b400640 1 --2- 172.21.15.29:0/472984154 >> v2:172.21.15.37:3300/0 conn(0x7f70a406ccb0 0x7f70a4152f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.788+0000 7f709b400640 1 -- 172.21.15.29:0/472984154 >> v1:172.21.15.37:6789/0 conn(0x7f70a415c670 legacy=0x7f70a406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:44.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.788+0000 7f709b400640 1 -- 172.21.15.29:0/472984154 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a415f8c0 con 0x7f70a40c9080 2024-04-17T19:48:44.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.792+0000 7f709b400640 1 --2- 172.21.15.29:0/472984154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40c9080 0x7f70a415c130 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f70a40723d0 tx=0x7f708c01ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8a77dcbb5ccdda2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:44.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.792+0000 7f70a3400640 1 -- 172.21.15.29:0/472984154 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c0b5070 con 0x7f70a40c9080 2024-04-17T19:48:44.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.792+0000 7f70a3400640 1 -- 172.21.15.29:0/472984154 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708c0a89d0 con 0x7f70a40c9080 2024-04-17T19:48:44.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.792+0000 7f70a3400640 1 -- 172.21.15.29:0/472984154 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c0a8da0 con 0x7f70a40c9080 2024-04-17T19:48:44.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.792+0000 7f70ab200640 1 -- 172.21.15.29:0/472984154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40c9080 msgr2=0x7f70a415c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:44.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.792+0000 7f70ab200640 1 --2- 172.21.15.29:0/472984154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40c9080 0x7f70a415c130 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f70a40723d0 tx=0x7f708c01ecd0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.792+0000 7f70ab200640 1 -- 172.21.15.29:0/472984154 shutdown_connections 2024-04-17T19:48:44.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.792+0000 7f70ab200640 1 --2- 172.21.15.29:0/472984154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40c9080 0x7f70a415c130 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.792+0000 7f70ab200640 1 --2- 172.21.15.29:0/472984154 >> v2:172.21.15.37:3300/0 conn(0x7f70a406ccb0 0x7f70a4152f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.792+0000 7f70ab200640 1 -- 172.21.15.29:0/472984154 >> 172.21.15.29:0/472984154 conn(0x7f70a40ce250 msgr2=0x7f70a40ce660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:44.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.792+0000 7f70ab200640 1 -- 172.21.15.29:0/472984154 shutdown_connections 2024-04-17T19:48:44.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.792+0000 7f70ab200640 1 -- 172.21.15.29:0/472984154 wait complete. 2024-04-17T19:48:44.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.792+0000 7f70ab200640 1 Processor -- start 2024-04-17T19:48:44.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.792+0000 7f70ab200640 1 -- start start 2024-04-17T19:48:44.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f70ab200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f70a406ccb0 0x7f70a40c82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:44.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f70ab200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40c9080 0x7f70a41f2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:44.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f70ab200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f70a415c670 0x7f70a41e68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:44.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f70ab200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f70a406c660 con 0x7f70a40c9080 2024-04-17T19:48:44.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f70ab200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f70a40c9490 con 0x7f70a406ccb0 2024-04-17T19:48:44.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f70ab200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f70a406c7e0 con 0x7f70a415c670 2024-04-17T19:48:44.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f709b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40c9080 0x7f70a41f2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:44.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f709b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40c9080 0x7f70a41f2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33386/0 (socket says 172.21.15.29:33386) 2024-04-17T19:48:44.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f709b400640 1 -- 172.21.15.29:0/3741421994 learned_addr learned my addr 172.21.15.29:0/3741421994 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:44.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f70a3e00640 1 --2- 172.21.15.29:0/3741421994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f70a406ccb0 0x7f70a40c82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:44.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f70a9200640 1 --2- 172.21.15.29:0/3741421994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f70a415c670 0x7f70a41e68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:44.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f709b400640 1 -- 172.21.15.29:0/3741421994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f70a415c670 msgr2=0x7f70a41e68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:44.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f709b400640 1 --2- 172.21.15.29:0/3741421994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f70a415c670 0x7f70a41e68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f709b400640 1 -- 172.21.15.29:0/3741421994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f70a406ccb0 msgr2=0x7f70a40c82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:44.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f709b400640 1 --2- 172.21.15.29:0/3741421994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f70a406ccb0 0x7f70a40c82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f709b400640 1 -- 172.21.15.29:0/3741421994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a41f3110 con 0x7f70a40c9080 2024-04-17T19:48:44.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f709b400640 1 --2- 172.21.15.29:0/3741421994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40c9080 0x7f70a41f2bd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f708c0b3040 tx=0x7f708c01ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:44.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f70a0c00640 1 -- 172.21.15.29:0/3741421994 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c0c0070 con 0x7f70a40c9080 2024-04-17T19:48:44.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f70a0c00640 1 -- 172.21.15.29:0/3741421994 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f708c0b0a50 con 0x7f70a40c9080 2024-04-17T19:48:44.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f70ab200640 1 -- 172.21.15.29:0/3741421994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a41f60c0 con 0x7f70a40c9080 2024-04-17T19:48:44.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f70a0c00640 1 -- 172.21.15.29:0/3741421994 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f708c0b5070 con 0x7f70a40c9080 2024-04-17T19:48:44.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f70ab200640 1 -- 172.21.15.29:0/3741421994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a41f6680 con 0x7f70a40c9080 2024-04-17T19:48:44.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.796+0000 7f70ab200640 1 -- 172.21.15.29:0/3741421994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a4069b90 con 0x7f70a40c9080 2024-04-17T19:48:44.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.800+0000 7f70a0c00640 1 -- 172.21.15.29:0/3741421994 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f708c0c3070 con 0x7f70a40c9080 2024-04-17T19:48:44.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.800+0000 7f70a0c00640 1 --2- 172.21.15.29:0/3741421994 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7080068380 0x7f708006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:44.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.800+0000 7f70a0c00640 1 -- 172.21.15.29:0/3741421994 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f708c123eb0 con 0x7f70a40c9080 2024-04-17T19:48:44.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.800+0000 7f70a3e00640 1 --2- 172.21.15.29:0/3741421994 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7080068380 0x7f708006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:44.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.800+0000 7f70a3e00640 1 --2- 172.21.15.29:0/3741421994 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7080068380 0x7f708006a890 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f70940079c0 tx=0x7f7094008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:44.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.804+0000 7f70a0c00640 1 -- 172.21.15.29:0/3741421994 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f708c018830 con 0x7f70a40c9080 2024-04-17T19:48:44.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.972+0000 7f70ab200640 1 -- 172.21.15.29:0/3741421994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f70a414fa80 con 0x7f70a40c9080 2024-04-17T19:48:44.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.972+0000 7f70a0c00640 1 -- 172.21.15.29:0/3741421994 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f708c0f38f0 con 0x7f70a40c9080 2024-04-17T19:48:44.962 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-04-17T19:48:44.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.976+0000 7f70ab200640 1 -- 172.21.15.29:0/3741421994 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7080068380 msgr2=0x7f708006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:44.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.976+0000 7f70ab200640 1 --2- 172.21.15.29:0/3741421994 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7080068380 0x7f708006a890 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f70940079c0 tx=0x7f7094008040 comp rx=0 tx=0).stop 2024-04-17T19:48:44.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.976+0000 7f70ab200640 1 -- 172.21.15.29:0/3741421994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40c9080 msgr2=0x7f70a41f2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:44.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.976+0000 7f70ab200640 1 --2- 172.21.15.29:0/3741421994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40c9080 0x7f70a41f2bd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f708c0b3040 tx=0x7f708c01ba70 comp rx=0 tx=0).stop 2024-04-17T19:48:44.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.976+0000 7f70ab200640 1 -- 172.21.15.29:0/3741421994 shutdown_connections 2024-04-17T19:48:44.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.976+0000 7f70ab200640 1 --2- 172.21.15.29:0/3741421994 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7080068380 0x7f708006a890 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.976+0000 7f70ab200640 1 --2- 172.21.15.29:0/3741421994 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f70a415c670 0x7f70a41e68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.976+0000 7f70ab200640 1 --2- 172.21.15.29:0/3741421994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70a40c9080 0x7f70a41f2bd0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.976+0000 7f70ab200640 1 --2- 172.21.15.29:0/3741421994 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f70a406ccb0 0x7f70a40c82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:44.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.976+0000 7f70ab200640 1 -- 172.21.15.29:0/3741421994 >> 172.21.15.29:0/3741421994 conn(0x7f70a40ce250 msgr2=0x7f70a4149090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:44.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.976+0000 7f70ab200640 1 -- 172.21.15.29:0/3741421994 shutdown_connections 2024-04-17T19:48:44.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:44.976+0000 7f70ab200640 1 -- 172.21.15.29:0/3741421994 wait complete. 2024-04-17T19:48:44.984 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T19:48:45.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.148+0000 7fd9eba00640 1 Processor -- start 2024-04-17T19:48:45.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.148+0000 7fd9eba00640 1 -- start start 2024-04-17T19:48:45.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.148+0000 7fd9eba00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd9e4071ce0 0x7fd9e4141de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:45.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.152+0000 7fd9eba00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9e41593d0 0x7fd9e41597e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:45.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.152+0000 7fd9eba00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9e40c9f00 0x7fd9e40ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:45.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.152+0000 7fd9eba00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fd9e40c7bd0 con 0x7fd9e41593d0 2024-04-17T19:48:45.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.152+0000 7fd9eba00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd9e4142320 con 0x7fd9e4071ce0 2024-04-17T19:48:45.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.152+0000 7fd9eba00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd9e415c6d0 con 0x7fd9e40c9f00 2024-04-17T19:48:45.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.152+0000 7fd9e3e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9e41593d0 0x7fd9e41597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:45.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.152+0000 7fd9e3e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9e41593d0 0x7fd9e41597e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33396/0 (socket says 172.21.15.29:33396) 2024-04-17T19:48:45.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.152+0000 7fd9e3e00640 1 -- 172.21.15.29:0/1405039595 learned_addr learned my addr 172.21.15.29:0/1405039595 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:45.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.152+0000 7fd9e9000640 1 --2- 172.21.15.29:0/1405039595 >> v2:172.21.15.37:3300/0 conn(0x7fd9e4071ce0 0x7fd9e4141de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:45.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.152+0000 7fd9e9a00640 1 --2- 172.21.15.29:0/1405039595 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9e40c9f00 0x7fd9e40ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:45.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.152+0000 7fd9e3e00640 1 -- 172.21.15.29:0/1405039595 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9e40c9f00 msgr2=0x7fd9e40ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:45.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.152+0000 7fd9e3e00640 1 --2- 172.21.15.29:0/1405039595 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9e40c9f00 0x7fd9e40ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.152+0000 7fd9e3e00640 1 -- 172.21.15.29:0/1405039595 >> v2:172.21.15.37:3300/0 conn(0x7fd9e4071ce0 msgr2=0x7fd9e4141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:45.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.152+0000 7fd9e3e00640 1 --2- 172.21.15.29:0/1405039595 >> v2:172.21.15.37:3300/0 conn(0x7fd9e4071ce0 0x7fd9e4141de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.152+0000 7fd9e3e00640 1 -- 172.21.15.29:0/1405039595 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e415c9b0 con 0x7fd9e41593d0 2024-04-17T19:48:45.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.152+0000 7fd9e3e00640 1 --2- 172.21.15.29:0/1405039595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e41593d0 0x7fd9e41597e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd9e40cad90 tx=0x7fd9d001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6354f9d6b26aa9a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:45.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.152+0000 7fd9e3400640 1 -- 172.21.15.29:0/1405039595 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d00b1070 con 0x7fd9e41593d0 2024-04-17T19:48:45.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.156+0000 7fd9e3400640 1 -- 172.21.15.29:0/1405039595 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9d00ac630 con 0x7fd9e41593d0 2024-04-17T19:48:45.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.156+0000 7fd9e3400640 1 -- 172.21.15.29:0/1405039595 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d00a8050 con 0x7fd9e41593d0 2024-04-17T19:48:45.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.156+0000 7fd9eba00640 1 -- 172.21.15.29:0/1405039595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e41593d0 msgr2=0x7fd9e41597e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:45.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.156+0000 7fd9eba00640 1 --2- 172.21.15.29:0/1405039595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e41593d0 0x7fd9e41597e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd9e40cad90 tx=0x7fd9d001c500 comp rx=0 tx=0).stop 2024-04-17T19:48:45.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.156+0000 7fd9eba00640 1 -- 172.21.15.29:0/1405039595 shutdown_connections 2024-04-17T19:48:45.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.156+0000 7fd9eba00640 1 --2- 172.21.15.29:0/1405039595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e41593d0 0x7fd9e41597e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.156+0000 7fd9eba00640 1 --2- 172.21.15.29:0/1405039595 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9e40c9f00 0x7fd9e40ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.156+0000 7fd9eba00640 1 --2- 172.21.15.29:0/1405039595 >> v2:172.21.15.37:3300/0 conn(0x7fd9e4071ce0 0x7fd9e4141de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.156+0000 7fd9eba00640 1 -- 172.21.15.29:0/1405039595 >> 172.21.15.29:0/1405039595 conn(0x7fd9e406bf90 msgr2=0x7fd9e406c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:45.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.156+0000 7fd9eba00640 1 -- 172.21.15.29:0/1405039595 shutdown_connections 2024-04-17T19:48:45.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.156+0000 7fd9eba00640 1 -- 172.21.15.29:0/1405039595 wait complete. 2024-04-17T19:48:45.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.156+0000 7fd9eba00640 1 Processor -- start 2024-04-17T19:48:45.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.156+0000 7fd9eba00640 1 -- start start 2024-04-17T19:48:45.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9eba00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e4071ce0 0x7fd9e415bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:45.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9eba00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9e40c9f00 0x7fd9e414d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:45.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9eba00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9e41593d0 0x7fd9e4134350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:45.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9eba00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e4072270 con 0x7fd9e4071ce0 2024-04-17T19:48:45.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9eba00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e40720f0 con 0x7fd9e41593d0 2024-04-17T19:48:45.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9eba00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd9e415c240 con 0x7fd9e40c9f00 2024-04-17T19:48:45.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9e9000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e4071ce0 0x7fd9e415bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:45.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9e9000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e4071ce0 0x7fd9e415bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33408/0 (socket says 172.21.15.29:33408) 2024-04-17T19:48:45.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9e9000640 1 -- 172.21.15.29:0/1261682607 learned_addr learned my addr 172.21.15.29:0/1261682607 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:45.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9e3e00640 1 --2- 172.21.15.29:0/1261682607 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9e40c9f00 0x7fd9e414d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:45.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9e9a00640 1 --2- 172.21.15.29:0/1261682607 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9e41593d0 0x7fd9e4134350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:45.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9e9000640 1 -- 172.21.15.29:0/1261682607 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9e40c9f00 msgr2=0x7fd9e414d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:45.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9e9000640 1 --2- 172.21.15.29:0/1261682607 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9e40c9f00 0x7fd9e414d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9e9000640 1 -- 172.21.15.29:0/1261682607 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9e41593d0 msgr2=0x7fd9e4134350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:45.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9e9000640 1 --2- 172.21.15.29:0/1261682607 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9e41593d0 0x7fd9e4134350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9e9000640 1 -- 172.21.15.29:0/1261682607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e414dec0 con 0x7fd9e4071ce0 2024-04-17T19:48:45.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9e9000640 1 --2- 172.21.15.29:0/1261682607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e4071ce0 0x7fd9e415bcb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800e8a0 tx=0x7fd9d800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:45.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9e0c00640 1 -- 172.21.15.29:0/1261682607 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8017070 con 0x7fd9e4071ce0 2024-04-17T19:48:45.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9e0c00640 1 -- 172.21.15.29:0/1261682607 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9d800c940 con 0x7fd9e4071ce0 2024-04-17T19:48:45.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9eba00640 1 -- 172.21.15.29:0/1261682607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9e415ac10 con 0x7fd9e4071ce0 2024-04-17T19:48:45.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9e0c00640 1 -- 172.21.15.29:0/1261682607 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d800ccf0 con 0x7fd9e4071ce0 2024-04-17T19:48:45.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9eba00640 1 -- 172.21.15.29:0/1261682607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9e4152780 con 0x7fd9e4071ce0 2024-04-17T19:48:45.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9eba00640 1 -- 172.21.15.29:0/1261682607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9e4069b90 con 0x7fd9e4071ce0 2024-04-17T19:48:45.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.160+0000 7fd9e0c00640 1 -- 172.21.15.29:0/1261682607 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd9d801a070 con 0x7fd9e4071ce0 2024-04-17T19:48:45.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.164+0000 7fd9e0c00640 1 --2- 172.21.15.29:0/1261682607 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd9c4068380 0x7fd9c406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:45.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.164+0000 7fd9e0c00640 1 -- 172.21.15.29:0/1261682607 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd9d8086fc0 con 0x7fd9e4071ce0 2024-04-17T19:48:45.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.164+0000 7fd9e3e00640 1 --2- 172.21.15.29:0/1261682607 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd9c4068380 0x7fd9c406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:45.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.164+0000 7fd9e3e00640 1 --2- 172.21.15.29:0/1261682607 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd9c4068380 0x7fd9c406a890 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00ae000 tx=0x7fd9d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:45.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.168+0000 7fd9e0c00640 1 -- 172.21.15.29:0/1261682607 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd9d8057210 con 0x7fd9e4071ce0 2024-04-17T19:48:45.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.332+0000 7fd9eba00640 1 -- 172.21.15.29:0/1261682607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd9e4141f70 con 0x7fd9e4071ce0 2024-04-17T19:48:45.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.332+0000 7fd9e0c00640 1 -- 172.21.15.29:0/1261682607 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fd9d8056b10 con 0x7fd9e4071ce0 2024-04-17T19:48:45.321 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-04-17T19:48:45.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.332+0000 7fd9eba00640 1 -- 172.21.15.29:0/1261682607 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd9c4068380 msgr2=0x7fd9c406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:45.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.332+0000 7fd9eba00640 1 --2- 172.21.15.29:0/1261682607 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd9c4068380 0x7fd9c406a890 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00ae000 tx=0x7fd9d0008040 comp rx=0 tx=0).stop 2024-04-17T19:48:45.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.332+0000 7fd9eba00640 1 -- 172.21.15.29:0/1261682607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e4071ce0 msgr2=0x7fd9e415bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:45.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.332+0000 7fd9eba00640 1 --2- 172.21.15.29:0/1261682607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e4071ce0 0x7fd9e415bcb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800e8a0 tx=0x7fd9d800eca0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.332+0000 7fd9eba00640 1 -- 172.21.15.29:0/1261682607 shutdown_connections 2024-04-17T19:48:45.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.332+0000 7fd9eba00640 1 --2- 172.21.15.29:0/1261682607 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd9c4068380 0x7fd9c406a890 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.332+0000 7fd9eba00640 1 --2- 172.21.15.29:0/1261682607 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9e41593d0 0x7fd9e4134350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.332+0000 7fd9eba00640 1 --2- 172.21.15.29:0/1261682607 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9e40c9f00 0x7fd9e414d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.332+0000 7fd9eba00640 1 --2- 172.21.15.29:0/1261682607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e4071ce0 0x7fd9e415bcb0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.332+0000 7fd9eba00640 1 -- 172.21.15.29:0/1261682607 >> 172.21.15.29:0/1261682607 conn(0x7fd9e406bf90 msgr2=0x7fd9e4073b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:45.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.332+0000 7fd9eba00640 1 -- 172.21.15.29:0/1261682607 shutdown_connections 2024-04-17T19:48:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.332+0000 7fd9eba00640 1 -- 172.21.15.29:0/1261682607 wait complete. 2024-04-17T19:48:45.348 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T19:48:45.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.512+0000 7f3d53000640 1 Processor -- start 2024-04-17T19:48:45.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.512+0000 7f3d53000640 1 -- start start 2024-04-17T19:48:45.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.516+0000 7f3d53000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3d4c153c10 0x7f3d4c154020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:45.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.516+0000 7f3d53000640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d4c0c5110 con 0x7f3d4c0ca0d0 2024-04-17T19:48:45.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.516+0000 7f3d53000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d4c0c84c0 con 0x7f3d4c15a7f0 2024-04-17T19:48:45.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.516+0000 7f3d53000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f3d4c154560 con 0x7f3d4c153c10 2024-04-17T19:48:45.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.516+0000 7f3d51000640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3d4c0ca0d0 0x7f3d4c0ca4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52262/0 (socket says 172.21.15.29:52262) 2024-04-17T19:48:45.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.516+0000 7f3d51000640 1 -- 172.21.15.29:0/4085562472 learned_addr learned my addr 172.21.15.29:0/4085562472 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:45.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.516+0000 7f3d43e00640 1 --2- 172.21.15.29:0/4085562472 >> v2:172.21.15.29:3300/0 conn(0x7f3d4c153c10 0x7f3d4c154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:45.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.516+0000 7f3d4b400640 1 -- 172.21.15.29:0/4085562472 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3043327611 0 0) 0x7f3d4c0c5110 con 0x7f3d4c0ca0d0 2024-04-17T19:48:45.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.516+0000 7f3d4b400640 1 -- 172.21.15.29:0/4085562472 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d30003530 con 0x7f3d4c0ca0d0 2024-04-17T19:48:45.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.516+0000 7f3d43e00640 1 -- 172.21.15.29:0/4085562472 >> v1:172.21.15.37:6789/0 conn(0x7f3d4c15a7f0 legacy=0x7f3d4c0c4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:45.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.516+0000 7f3d43e00640 1 -- 172.21.15.29:0/4085562472 >> v1:172.21.15.29:6789/0 conn(0x7f3d4c0ca0d0 legacy=0x7f3d4c0ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:45.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.516+0000 7f3d43e00640 1 -- 172.21.15.29:0/4085562472 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d4c0c87a0 con 0x7f3d4c153c10 2024-04-17T19:48:45.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.520+0000 7f3d43e00640 1 --2- 172.21.15.29:0/4085562472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d4c153c10 0x7f3d4c154020 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3d340097c0 tx=0x7f3d3401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5764e968e69d9398 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:45.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.520+0000 7f3d4b400640 1 -- 172.21.15.29:0/4085562472 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d340b1070 con 0x7f3d4c153c10 2024-04-17T19:48:45.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.520+0000 7f3d4b400640 1 -- 172.21.15.29:0/4085562472 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d340ac5f0 con 0x7f3d4c153c10 2024-04-17T19:48:45.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.520+0000 7f3d4b400640 1 -- 172.21.15.29:0/4085562472 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d340ac9c0 con 0x7f3d4c153c10 2024-04-17T19:48:45.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.520+0000 7f3d53000640 1 -- 172.21.15.29:0/4085562472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d4c153c10 msgr2=0x7f3d4c154020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:45.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.520+0000 7f3d53000640 1 --2- 172.21.15.29:0/4085562472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d4c153c10 0x7f3d4c154020 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3d340097c0 tx=0x7f3d3401c4a0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.520+0000 7f3d53000640 1 -- 172.21.15.29:0/4085562472 shutdown_connections 2024-04-17T19:48:45.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.520+0000 7f3d53000640 1 --2- 172.21.15.29:0/4085562472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d4c153c10 0x7f3d4c154020 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.520+0000 7f3d53000640 1 -- 172.21.15.29:0/4085562472 >> 172.21.15.29:0/4085562472 conn(0x7f3d4c06bf90 msgr2=0x7f3d4c06c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.520+0000 7f3d53000640 1 -- 172.21.15.29:0/4085562472 shutdown_connections 2024-04-17T19:48:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.520+0000 7f3d53000640 1 -- 172.21.15.29:0/4085562472 wait complete. 2024-04-17T19:48:45.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d53000640 1 Processor -- start 2024-04-17T19:48:45.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d53000640 1 -- start start 2024-04-17T19:48:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d53000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d4c0ca0d0 0x7f3d4c147e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d53000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d4c153c10 0x7f3d4c1f6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d53000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d4c15a7f0 0x7f3d4c156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d53000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4c15ac00 con 0x7f3d4c0ca0d0 2024-04-17T19:48:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d53000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4c15dc30 con 0x7f3d4c15a7f0 2024-04-17T19:48:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d53000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3d4c15e150 con 0x7f3d4c153c10 2024-04-17T19:48:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d4be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d4c0ca0d0 0x7f3d4c147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d4be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d4c0ca0d0 0x7f3d4c147e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33418/0 (socket says 172.21.15.29:33418) 2024-04-17T19:48:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d4be00640 1 -- 172.21.15.29:0/1962394757 learned_addr learned my addr 172.21.15.29:0/1962394757 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d43e00640 1 --2- 172.21.15.29:0/1962394757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d4c153c10 0x7f3d4c1f6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d51000640 1 --2- 172.21.15.29:0/1962394757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d4c15a7f0 0x7f3d4c156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d4be00640 1 -- 172.21.15.29:0/1962394757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d4c153c10 msgr2=0x7f3d4c1f6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:45.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d4be00640 1 --2- 172.21.15.29:0/1962394757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d4c153c10 0x7f3d4c1f6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d4be00640 1 -- 172.21.15.29:0/1962394757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d4c15a7f0 msgr2=0x7f3d4c156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:45.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d4be00640 1 --2- 172.21.15.29:0/1962394757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d4c15a7f0 0x7f3d4c156280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d4be00640 1 -- 172.21.15.29:0/1962394757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d4c1567c0 con 0x7f3d4c0ca0d0 2024-04-17T19:48:45.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d43e00640 1 --2- 172.21.15.29:0/1962394757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d4c153c10 0x7f3d4c1f6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:45.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d51000640 1 --2- 172.21.15.29:0/1962394757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d4c15a7f0 0x7f3d4c156280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:45.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d4be00640 1 --2- 172.21.15.29:0/1962394757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d4c0ca0d0 0x7f3d4c147e30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00c3f0 tx=0x7f3d3c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:45.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d48c00640 1 -- 172.21.15.29:0/1962394757 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c017070 con 0x7f3d4c0ca0d0 2024-04-17T19:48:45.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d48c00640 1 -- 172.21.15.29:0/1962394757 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d3c0052d0 con 0x7f3d4c0ca0d0 2024-04-17T19:48:45.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d48c00640 1 -- 172.21.15.29:0/1962394757 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c003840 con 0x7f3d4c0ca0d0 2024-04-17T19:48:45.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d53000640 1 -- 172.21.15.29:0/1962394757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d4c1477c0 con 0x7f3d4c0ca0d0 2024-04-17T19:48:45.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.524+0000 7f3d53000640 1 -- 172.21.15.29:0/1962394757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d4c1f6790 con 0x7f3d4c0ca0d0 2024-04-17T19:48:45.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.528+0000 7f3d53000640 1 -- 172.21.15.29:0/1962394757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d4c069b90 con 0x7f3d4c0ca0d0 2024-04-17T19:48:45.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.528+0000 7f3d48c00640 1 -- 172.21.15.29:0/1962394757 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3d3c01a070 con 0x7f3d4c0ca0d0 2024-04-17T19:48:45.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.528+0000 7f3d48c00640 1 --2- 172.21.15.29:0/1962394757 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3d28068380 0x7f3d2806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:45.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.528+0000 7f3d48c00640 1 -- 172.21.15.29:0/1962394757 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f3d3c086160 con 0x7f3d4c0ca0d0 2024-04-17T19:48:45.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.528+0000 7f3d43e00640 1 --2- 172.21.15.29:0/1962394757 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3d28068380 0x7f3d2806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:45.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.528+0000 7f3d43e00640 1 --2- 172.21.15.29:0/1962394757 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3d28068380 0x7f3d2806a890 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3d340ae000 tx=0x7f3d34008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:45.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.532+0000 7f3d48c00640 1 -- 172.21.15.29:0/1962394757 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3d3c005480 con 0x7f3d4c0ca0d0 2024-04-17T19:48:45.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.696+0000 7f3d53000640 1 -- 172.21.15.29:0/1962394757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3d4c0c43b0 con 0x7f3d4c0ca0d0 2024-04-17T19:48:45.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.700+0000 7f3d48c00640 1 -- 172.21.15.29:0/1962394757 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f3d3c056310 con 0x7f3d4c0ca0d0 2024-04-17T19:48:45.688 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-04-17T19:48:45.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.700+0000 7f3d53000640 1 -- 172.21.15.29:0/1962394757 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3d28068380 msgr2=0x7f3d2806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:45.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.700+0000 7f3d53000640 1 --2- 172.21.15.29:0/1962394757 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3d28068380 0x7f3d2806a890 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3d340ae000 tx=0x7f3d34008040 comp rx=0 tx=0).stop 2024-04-17T19:48:45.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.700+0000 7f3d53000640 1 -- 172.21.15.29:0/1962394757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d4c0ca0d0 msgr2=0x7f3d4c147e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:45.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.700+0000 7f3d53000640 1 --2- 172.21.15.29:0/1962394757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d4c0ca0d0 0x7f3d4c147e30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c00c3f0 tx=0x7f3d3c00c8b0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.700+0000 7f3d53000640 1 -- 172.21.15.29:0/1962394757 shutdown_connections 2024-04-17T19:48:45.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.700+0000 7f3d53000640 1 --2- 172.21.15.29:0/1962394757 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3d28068380 0x7f3d2806a890 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.700+0000 7f3d53000640 1 --2- 172.21.15.29:0/1962394757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3d4c15a7f0 0x7f3d4c156280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.700+0000 7f3d53000640 1 --2- 172.21.15.29:0/1962394757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3d4c153c10 0x7f3d4c1f6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.700+0000 7f3d53000640 1 --2- 172.21.15.29:0/1962394757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d4c0ca0d0 0x7f3d4c147e30 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.700+0000 7f3d53000640 1 -- 172.21.15.29:0/1962394757 >> 172.21.15.29:0/1962394757 conn(0x7f3d4c06bf90 msgr2=0x7f3d4c150ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:45.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.700+0000 7f3d53000640 1 -- 172.21.15.29:0/1962394757 shutdown_connections 2024-04-17T19:48:45.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.700+0000 7f3d53000640 1 -- 172.21.15.29:0/1962394757 wait complete. 2024-04-17T19:48:45.710 INFO:tasks.mds_thrash:mds thrasher using random seed: 1713383325 2024-04-17T19:48:45.710 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:48:45.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.876+0000 7f5d0ae00640 1 Processor -- start 2024-04-17T19:48:45.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.876+0000 7f5d0ae00640 1 -- start start 2024-04-17T19:48:45.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.880+0000 7f5d0ae00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d04155da0 0x7f5d041561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:45.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.880+0000 7f5d0ae00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5d0415c3d0 0x7f5d0415c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:45.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.880+0000 7f5d0ae00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d0415bf10 con 0x7f5d0406e960 2024-04-17T19:48:45.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.880+0000 7f5d0ae00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f5d041566f0 con 0x7f5d0415c3d0 2024-04-17T19:48:45.868 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.880+0000 7f5d0ae00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5d04071480 con 0x7f5d04155da0 2024-04-17T19:48:45.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.880+0000 7f5d03e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5d0406e960 0x7f5d04070d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52270/0 (socket says 172.21.15.29:52270) 2024-04-17T19:48:45.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.880+0000 7f5d03e00640 1 -- 172.21.15.29:0/3161826996 learned_addr learned my addr 172.21.15.29:0/3161826996 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:45.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.880+0000 7f5d08e00640 1 --2- 172.21.15.29:0/3161826996 >> v2:172.21.15.29:3300/0 conn(0x7f5d0415c3d0 0x7f5d0415c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:45.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.880+0000 7f5d03400640 1 --2- 172.21.15.29:0/3161826996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d04155da0 0x7f5d041561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:45.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.880+0000 7f5d02a00640 1 -- 172.21.15.29:0/3161826996 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 773256045 0 0) 0x7f5cf4001120 con 0x7f5d0406e960 2024-04-17T19:48:45.869 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.880+0000 7f5d02a00640 1 -- 172.21.15.29:0/3161826996 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ce8003530 con 0x7f5d0406e960 2024-04-17T19:48:45.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.880+0000 7f5d08e00640 1 -- 172.21.15.29:0/3161826996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d04155da0 msgr2=0x7f5d041561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:45.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.880+0000 7f5d08e00640 1 --2- 172.21.15.29:0/3161826996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d04155da0 0x7f5d041561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.880+0000 7f5d08e00640 1 -- 172.21.15.29:0/3161826996 >> v1:172.21.15.29:6789/0 conn(0x7f5d0406e960 legacy=0x7f5d04070d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:45.870 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.880+0000 7f5d08e00640 1 -- 172.21.15.29:0/3161826996 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d040c98e0 con 0x7f5d0415c3d0 2024-04-17T19:48:45.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.884+0000 7f5d08e00640 1 --2- 172.21.15.29:0/3161826996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0415c3d0 0x7f5d0415c7e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc009d90 tx=0x7f5cfc01f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c11d5ac3d8174ff server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:45.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.884+0000 7f5d02a00640 1 -- 172.21.15.29:0/3161826996 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc0b4070 con 0x7f5d0415c3d0 2024-04-17T19:48:45.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.884+0000 7f5d02a00640 1 -- 172.21.15.29:0/3161826996 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cfc0af600 con 0x7f5d0415c3d0 2024-04-17T19:48:45.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.884+0000 7f5d02a00640 1 -- 172.21.15.29:0/3161826996 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cfc0af9b0 con 0x7f5d0415c3d0 2024-04-17T19:48:45.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.884+0000 7f5d0ae00640 1 -- 172.21.15.29:0/3161826996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0415c3d0 msgr2=0x7f5d0415c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:45.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.884+0000 7f5d0ae00640 1 --2- 172.21.15.29:0/3161826996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0415c3d0 0x7f5d0415c7e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc009d90 tx=0x7f5cfc01f150 comp rx=0 tx=0).stop 2024-04-17T19:48:45.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.884+0000 7f5d0ae00640 1 -- 172.21.15.29:0/3161826996 shutdown_connections 2024-04-17T19:48:45.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.884+0000 7f5d0ae00640 1 --2- 172.21.15.29:0/3161826996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0415c3d0 0x7f5d0415c7e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.884+0000 7f5d0ae00640 1 --2- 172.21.15.29:0/3161826996 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d04155da0 0x7f5d041561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.884+0000 7f5d0ae00640 1 -- 172.21.15.29:0/3161826996 >> 172.21.15.29:0/3161826996 conn(0x7f5d040ce5e0 msgr2=0x7f5d040ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:45.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.884+0000 7f5d0ae00640 1 -- 172.21.15.29:0/3161826996 shutdown_connections 2024-04-17T19:48:45.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.884+0000 7f5d0ae00640 1 -- 172.21.15.29:0/3161826996 wait complete. 2024-04-17T19:48:45.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d0ae00640 1 Processor -- start 2024-04-17T19:48:45.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d0ae00640 1 -- start start 2024-04-17T19:48:45.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d0ae00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0406e960 0x7f5d04142da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:45.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d0ae00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d04155da0 0x7f5d04153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:45.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d0ae00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d0415c3d0 0x7f5d041542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:45.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d0ae00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5d040ca180 con 0x7f5d0406e960 2024-04-17T19:48:45.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d0ae00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5d040c9ea0 con 0x7f5d04155da0 2024-04-17T19:48:45.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d0ae00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5d040ca460 con 0x7f5d0415c3d0 2024-04-17T19:48:45.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d03e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0406e960 0x7f5d04142da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:45.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d03e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0406e960 0x7f5d04142da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33440/0 (socket says 172.21.15.29:33440) 2024-04-17T19:48:45.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d03e00640 1 -- 172.21.15.29:0/2179938033 learned_addr learned my addr 172.21.15.29:0/2179938033 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:45.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d03400640 1 --2- 172.21.15.29:0/2179938033 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d04155da0 0x7f5d04153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:45.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d08e00640 1 --2- 172.21.15.29:0/2179938033 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d0415c3d0 0x7f5d041542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:45.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d03e00640 1 -- 172.21.15.29:0/2179938033 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d0415c3d0 msgr2=0x7f5d041542c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:45.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d03e00640 1 --2- 172.21.15.29:0/2179938033 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d0415c3d0 0x7f5d041542c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d03e00640 1 -- 172.21.15.29:0/2179938033 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d04155da0 msgr2=0x7f5d04153d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:45.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d03e00640 1 --2- 172.21.15.29:0/2179938033 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d04155da0 0x7f5d04153d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:45.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d03e00640 1 -- 172.21.15.29:0/2179938033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d0415bc40 con 0x7f5d0406e960 2024-04-17T19:48:45.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d03e00640 1 --2- 172.21.15.29:0/2179938033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0406e960 0x7f5d04142da0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5cf400c3f0 tx=0x7f5cf400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:45.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5cfbe00640 1 -- 172.21.15.29:0/2179938033 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4017070 con 0x7f5d0406e960 2024-04-17T19:48:45.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5cfbe00640 1 -- 172.21.15.29:0/2179938033 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf4005cf0 con 0x7f5d0406e960 2024-04-17T19:48:45.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d0ae00640 1 -- 172.21.15.29:0/2179938033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d04159290 con 0x7f5d0406e960 2024-04-17T19:48:45.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5cfbe00640 1 -- 172.21.15.29:0/2179938033 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4007310 con 0x7f5d0406e960 2024-04-17T19:48:45.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d0ae00640 1 -- 172.21.15.29:0/2179938033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d04156a20 con 0x7f5d0406e960 2024-04-17T19:48:45.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.888+0000 7f5d0ae00640 1 -- 172.21.15.29:0/2179938033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d041597b0 con 0x7f5d0406e960 2024-04-17T19:48:45.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.892+0000 7f5cfbe00640 1 -- 172.21.15.29:0/2179938033 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5cf401a070 con 0x7f5d0406e960 2024-04-17T19:48:45.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.892+0000 7f5cfbe00640 1 --2- 172.21.15.29:0/2179938033 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5ce40683b0 0x7f5ce406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:45.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.892+0000 7f5cfbe00640 1 -- 172.21.15.29:0/2179938033 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5cf4086200 con 0x7f5d0406e960 2024-04-17T19:48:45.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.892+0000 7f5d03400640 1 --2- 172.21.15.29:0/2179938033 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5ce40683b0 0x7f5ce406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:45.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.892+0000 7f5d03400640 1 --2- 172.21.15.29:0/2179938033 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5ce40683b0 0x7f5ce406a8c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5d040cee40 tx=0x7f5cf0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:45.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:45.896+0000 7f5cfbe00640 1 -- 172.21.15.29:0/2179938033 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5d0415bf10 con 0x7f5d0406e960 2024-04-17T19:48:46.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.064+0000 7f5d0ae00640 1 -- 172.21.15.29:0/2179938033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d04069b90 con 0x7f5d0406e960 2024-04-17T19:48:46.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.064+0000 7f5cfbe00640 1 -- 172.21.15.29:0/2179938033 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18780 (secure 0 0 0) 0x7f5cf40563b0 con 0x7f5d0406e960 2024-04-17T19:48:46.055 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:46.055 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":26,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:48:29.829433+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T19:47:47.658449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T19:47:48.209047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T19:47:51.887909+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:48:46.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.068+0000 7f5d0ae00640 1 -- 172.21.15.29:0/2179938033 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5ce40683b0 msgr2=0x7f5ce406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:46.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.068+0000 7f5d0ae00640 1 --2- 172.21.15.29:0/2179938033 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5ce40683b0 0x7f5ce406a8c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5d040cee40 tx=0x7f5cf0008040 comp rx=0 tx=0).stop 2024-04-17T19:48:46.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.068+0000 7f5d0ae00640 1 -- 172.21.15.29:0/2179938033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0406e960 msgr2=0x7f5d04142da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:46.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.068+0000 7f5d0ae00640 1 --2- 172.21.15.29:0/2179938033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0406e960 0x7f5d04142da0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5cf400c3f0 tx=0x7f5cf400c8b0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.068+0000 7f5d0ae00640 1 -- 172.21.15.29:0/2179938033 shutdown_connections 2024-04-17T19:48:46.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.068+0000 7f5d0ae00640 1 --2- 172.21.15.29:0/2179938033 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5ce40683b0 0x7f5ce406a8c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.068+0000 7f5d0ae00640 1 --2- 172.21.15.29:0/2179938033 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5d0415c3d0 0x7f5d041542c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.068+0000 7f5d0ae00640 1 --2- 172.21.15.29:0/2179938033 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5d04155da0 0x7f5d04153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.068+0000 7f5d0ae00640 1 --2- 172.21.15.29:0/2179938033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d0406e960 0x7f5d04142da0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.068+0000 7f5d0ae00640 1 -- 172.21.15.29:0/2179938033 >> 172.21.15.29:0/2179938033 conn(0x7f5d040ce5e0 msgr2=0x7f5d040c6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:46.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.068+0000 7f5d0ae00640 1 -- 172.21.15.29:0/2179938033 shutdown_connections 2024-04-17T19:48:46.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.068+0000 7f5d0ae00640 1 -- 172.21.15.29:0/2179938033 wait complete. 2024-04-17T19:48:46.077 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T19:47:36.674676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T19:47:47.658449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T19:47:48.209047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T19:47:51.887909+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T19:48:46.078 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T19:48:46.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.244+0000 7fcbbae00640 1 Processor -- start 2024-04-17T19:48:46.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.244+0000 7fcbbae00640 1 -- start start 2024-04-17T19:48:46.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.244+0000 7fcbbae00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbb41593d0 0x7fcbb41597e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.244+0000 7fcbbae00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcbb40c9f00 0x7fcbb40ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.244+0000 7fcbbae00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbb415c6d0 con 0x7fcbb4071ce0 2024-04-17T19:48:46.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.244+0000 7fcbbae00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fcbb40c7bd0 con 0x7fcbb40c9f00 2024-04-17T19:48:46.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.244+0000 7fcbbae00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcbb415c9b0 con 0x7fcbb41593d0 2024-04-17T19:48:46.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.244+0000 7fcbb3e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcbb4071ce0 0x7fcbb4141de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52286/0 (socket says 172.21.15.29:52286) 2024-04-17T19:48:46.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.244+0000 7fcbb3e00640 1 -- 172.21.15.29:0/1913113085 learned_addr learned my addr 172.21.15.29:0/1913113085 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:46.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.244+0000 7fcbb8e00640 1 --2- 172.21.15.29:0/1913113085 >> v2:172.21.15.29:3300/0 conn(0x7fcbb40c9f00 0x7fcbb40ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.244+0000 7fcbabe00640 1 --2- 172.21.15.29:0/1913113085 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbb41593d0 0x7fcbb41597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.244+0000 7fcbb3400640 1 -- 172.21.15.29:0/1913113085 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 507075743 0 0) 0x7fcbb415c6d0 con 0x7fcbb4071ce0 2024-04-17T19:48:46.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.244+0000 7fcbb3400640 1 -- 172.21.15.29:0/1913113085 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb98003530 con 0x7fcbb4071ce0 2024-04-17T19:48:46.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.244+0000 7fcbb8e00640 1 -- 172.21.15.29:0/1913113085 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbb41593d0 msgr2=0x7fcbb41597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:46.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.244+0000 7fcbb8e00640 1 --2- 172.21.15.29:0/1913113085 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbb41593d0 0x7fcbb41597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.244+0000 7fcbb8e00640 1 -- 172.21.15.29:0/1913113085 >> v1:172.21.15.29:6789/0 conn(0x7fcbb4071ce0 legacy=0x7fcbb4141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:46.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.244+0000 7fcbb8e00640 1 -- 172.21.15.29:0/1913113085 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb415cc90 con 0x7fcbb40c9f00 2024-04-17T19:48:46.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.248+0000 7fcbb8e00640 1 --2- 172.21.15.29:0/1913113085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb40c9f00 0x7fcbb40ca310 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcbac009d90 tx=0x7fcbac01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=773d879cf4304cf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.248+0000 7fcbb3400640 1 -- 172.21.15.29:0/1913113085 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbac0b2070 con 0x7fcbb40c9f00 2024-04-17T19:48:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.248+0000 7fcbb3400640 1 -- 172.21.15.29:0/1913113085 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbac0a5940 con 0x7fcbb40c9f00 2024-04-17T19:48:46.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.248+0000 7fcbb3400640 1 -- 172.21.15.29:0/1913113085 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbac0a5cf0 con 0x7fcbb40c9f00 2024-04-17T19:48:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.248+0000 7fcbbae00640 1 -- 172.21.15.29:0/1913113085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb40c9f00 msgr2=0x7fcbb40ca310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.248+0000 7fcbbae00640 1 --2- 172.21.15.29:0/1913113085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb40c9f00 0x7fcbb40ca310 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcbac009d90 tx=0x7fcbac01c970 comp rx=0 tx=0).stop 2024-04-17T19:48:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.248+0000 7fcbbae00640 1 -- 172.21.15.29:0/1913113085 shutdown_connections 2024-04-17T19:48:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.248+0000 7fcbbae00640 1 --2- 172.21.15.29:0/1913113085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb40c9f00 0x7fcbb40ca310 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.248+0000 7fcbbae00640 1 --2- 172.21.15.29:0/1913113085 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbb41593d0 0x7fcbb41597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.248+0000 7fcbbae00640 1 -- 172.21.15.29:0/1913113085 >> 172.21.15.29:0/1913113085 conn(0x7fcbb406bf90 msgr2=0x7fcbb406c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.248+0000 7fcbbae00640 1 -- 172.21.15.29:0/1913113085 shutdown_connections 2024-04-17T19:48:46.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.248+0000 7fcbbae00640 1 -- 172.21.15.29:0/1913113085 wait complete. 2024-04-17T19:48:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.252+0000 7fcbbae00640 1 Processor -- start 2024-04-17T19:48:46.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.252+0000 7fcbbae00640 1 -- start start 2024-04-17T19:48:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.252+0000 7fcbbae00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbb4071ce0 0x7fcbb41f8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.252+0000 7fcbbae00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb40c9f00 0x7fcbb413f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.252+0000 7fcbbae00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbb41593d0 0x7fcbb4138bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.252+0000 7fcbbae00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb415c3c0 con 0x7fcbb40c9f00 2024-04-17T19:48:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.252+0000 7fcbbae00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcbb415c240 con 0x7fcbb41593d0 2024-04-17T19:48:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.252+0000 7fcbbae00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcbb4165e40 con 0x7fcbb4071ce0 2024-04-17T19:48:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.252+0000 7fcbabe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb40c9f00 0x7fcbb413f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.252+0000 7fcbabe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb40c9f00 0x7fcbb413f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33458/0 (socket says 172.21.15.29:33458) 2024-04-17T19:48:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.252+0000 7fcbabe00640 1 -- 172.21.15.29:0/352380176 learned_addr learned my addr 172.21.15.29:0/352380176 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.252+0000 7fcbb3e00640 1 --2- 172.21.15.29:0/352380176 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbb4071ce0 0x7fcbb41f8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.252+0000 7fcbb8e00640 1 --2- 172.21.15.29:0/352380176 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbb41593d0 0x7fcbb4138bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.252+0000 7fcbabe00640 1 -- 172.21.15.29:0/352380176 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbb4071ce0 msgr2=0x7fcbb41f8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.252+0000 7fcbabe00640 1 --2- 172.21.15.29:0/352380176 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbb4071ce0 0x7fcbb41f8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.252+0000 7fcbabe00640 1 -- 172.21.15.29:0/352380176 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbb41593d0 msgr2=0x7fcbb4138bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.252+0000 7fcbabe00640 1 --2- 172.21.15.29:0/352380176 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbb41593d0 0x7fcbb4138bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.252+0000 7fcbabe00640 1 -- 172.21.15.29:0/352380176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbb4156c30 con 0x7fcbb40c9f00 2024-04-17T19:48:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.256+0000 7fcbabe00640 1 --2- 172.21.15.29:0/352380176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb40c9f00 0x7fcbb413f7b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c00d5f0 tx=0x7fcb9c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.256+0000 7fcbb0c00640 1 -- 172.21.15.29:0/352380176 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c0125f0 con 0x7fcbb40c9f00 2024-04-17T19:48:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.256+0000 7fcbb0c00640 1 -- 172.21.15.29:0/352380176 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb9c019070 con 0x7fcbb40c9f00 2024-04-17T19:48:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.256+0000 7fcbb0c00640 1 -- 172.21.15.29:0/352380176 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c0129b0 con 0x7fcbb40c9f00 2024-04-17T19:48:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.256+0000 7fcbbae00640 1 -- 172.21.15.29:0/352380176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbb41f9520 con 0x7fcbb40c9f00 2024-04-17T19:48:46.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.256+0000 7fcbbae00640 1 -- 172.21.15.29:0/352380176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbb41f87e0 con 0x7fcbb40c9f00 2024-04-17T19:48:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.256+0000 7fcbbae00640 1 -- 172.21.15.29:0/352380176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbb4069b90 con 0x7fcbb40c9f00 2024-04-17T19:48:46.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.256+0000 7fcbb0c00640 1 -- 172.21.15.29:0/352380176 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcb9c012b50 con 0x7fcbb40c9f00 2024-04-17T19:48:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.260+0000 7fcbb0c00640 1 --2- 172.21.15.29:0/352380176 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcb900683d0 0x7fcb9006a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.260+0000 7fcbb0c00640 1 -- 172.21.15.29:0/352380176 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fcb9c087200 con 0x7fcbb40c9f00 2024-04-17T19:48:46.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.260+0000 7fcbb3e00640 1 --2- 172.21.15.29:0/352380176 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcb900683d0 0x7fcb9006a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.260+0000 7fcbb3e00640 1 --2- 172.21.15.29:0/352380176 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcb900683d0 0x7fcb9006a8e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcba4005890 tx=0x7fcba4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:46.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.264+0000 7fcbb0c00640 1 -- 172.21.15.29:0/352380176 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fcb9c0573a0 con 0x7fcbb40c9f00 2024-04-17T19:48:46.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.428+0000 7fcbbae00640 1 -- 172.21.15.29:0/352380176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcbb41570e0 con 0x7fcbb40c9f00 2024-04-17T19:48:46.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.428+0000 7fcbb0c00640 1 -- 172.21.15.29:0/352380176 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fcb9c056ca0 con 0x7fcbb40c9f00 2024-04-17T19:48:46.418 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-04-17T19:48:46.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.432+0000 7fcbbae00640 1 -- 172.21.15.29:0/352380176 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcb900683d0 msgr2=0x7fcb9006a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:46.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.432+0000 7fcbbae00640 1 --2- 172.21.15.29:0/352380176 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcb900683d0 0x7fcb9006a8e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fcba4005890 tx=0x7fcba4009040 comp rx=0 tx=0).stop 2024-04-17T19:48:46.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.432+0000 7fcbbae00640 1 -- 172.21.15.29:0/352380176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb40c9f00 msgr2=0x7fcbb413f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:46.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.432+0000 7fcbbae00640 1 --2- 172.21.15.29:0/352380176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb40c9f00 0x7fcbb413f7b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c00d5f0 tx=0x7fcb9c00dab0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.432+0000 7fcbbae00640 1 -- 172.21.15.29:0/352380176 shutdown_connections 2024-04-17T19:48:46.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.432+0000 7fcbbae00640 1 --2- 172.21.15.29:0/352380176 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcb900683d0 0x7fcb9006a8e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.432+0000 7fcbbae00640 1 --2- 172.21.15.29:0/352380176 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbb41593d0 0x7fcbb4138bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.432+0000 7fcbbae00640 1 --2- 172.21.15.29:0/352380176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb40c9f00 0x7fcbb413f7b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.432+0000 7fcbbae00640 1 --2- 172.21.15.29:0/352380176 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbb4071ce0 0x7fcbb41f8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.432+0000 7fcbbae00640 1 -- 172.21.15.29:0/352380176 >> 172.21.15.29:0/352380176 conn(0x7fcbb406bf90 msgr2=0x7fcbb4161660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:46.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.432+0000 7fcbbae00640 1 -- 172.21.15.29:0/352380176 shutdown_connections 2024-04-17T19:48:46.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.432+0000 7fcbbae00640 1 -- 172.21.15.29:0/352380176 wait complete. 2024-04-17T19:48:46.440 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T19:48:46.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.604+0000 7f4a2f200640 1 Processor -- start 2024-04-17T19:48:46.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.604+0000 7f4a2f200640 1 -- start start 2024-04-17T19:48:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.608+0000 7f4a2f200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f4a280c91e0 0x7f4a2815cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.608+0000 7f4a2f200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4a280704e0 0x7f4a28166a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.608+0000 7f4a2f200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a2815d060 con 0x7f4a280700d0 2024-04-17T19:48:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.608+0000 7f4a2f200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f4a280744a0 con 0x7f4a280704e0 2024-04-17T19:48:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.608+0000 7f4a2f200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f4a280c95f0 con 0x7f4a280c91e0 2024-04-17T19:48:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.608+0000 7f4a1fe00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4a280700d0 0x7f4a28155b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52292/0 (socket says 172.21.15.29:52292) 2024-04-17T19:48:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.608+0000 7f4a1fe00640 1 -- 172.21.15.29:0/2213979336 learned_addr learned my addr 172.21.15.29:0/2213979336 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.608+0000 7f4a2d200640 1 --2- 172.21.15.29:0/2213979336 >> v2:172.21.15.29:3300/0 conn(0x7f4a280704e0 0x7f4a28166a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.608+0000 7f4a27e00640 1 --2- 172.21.15.29:0/2213979336 >> v2:172.21.15.37:3300/0 conn(0x7f4a280c91e0 0x7f4a2815cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.608+0000 7f4a2d200640 1 -- 172.21.15.29:0/2213979336 >> v2:172.21.15.37:3300/0 conn(0x7f4a280c91e0 msgr2=0x7f4a2815cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:46.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.608+0000 7f4a2d200640 1 --2- 172.21.15.29:0/2213979336 >> v2:172.21.15.37:3300/0 conn(0x7f4a280c91e0 0x7f4a2815cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.608+0000 7f4a2d200640 1 -- 172.21.15.29:0/2213979336 >> v1:172.21.15.29:6789/0 conn(0x7f4a280700d0 legacy=0x7f4a28155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:46.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.608+0000 7f4a2d200640 1 -- 172.21.15.29:0/2213979336 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a280cafd0 con 0x7f4a280704e0 2024-04-17T19:48:46.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.608+0000 7f4a27400640 1 -- 172.21.15.29:0/2213979336 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2980980363 0 0) 0x7f4a10001120 con 0x7f4a280700d0 2024-04-17T19:48:46.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.608+0000 7f4a27e00640 1 --2- 172.21.15.29:0/2213979336 >> v2:172.21.15.37:3300/0 conn(0x7f4a280c91e0 0x7f4a2815cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:46.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.612+0000 7f4a2d200640 1 --2- 172.21.15.29:0/2213979336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a280704e0 0x7f4a28166a20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4a2806cdd0 tx=0x7f4a2001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce14be182f489ed0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:46.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.612+0000 7f4a27400640 1 -- 172.21.15.29:0/2213979336 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a200b2070 con 0x7f4a280704e0 2024-04-17T19:48:46.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.612+0000 7f4a27400640 1 -- 172.21.15.29:0/2213979336 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a200a57f0 con 0x7f4a280704e0 2024-04-17T19:48:46.600 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.612+0000 7f4a27400640 1 -- 172.21.15.29:0/2213979336 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a200a5bc0 con 0x7f4a280704e0 2024-04-17T19:48:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.612+0000 7f4a2f200640 1 -- 172.21.15.29:0/2213979336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a280704e0 msgr2=0x7f4a28166a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.612+0000 7f4a2f200640 1 --2- 172.21.15.29:0/2213979336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a280704e0 0x7f4a28166a20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4a2806cdd0 tx=0x7f4a2001c930 comp rx=0 tx=0).stop 2024-04-17T19:48:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.612+0000 7f4a2f200640 1 -- 172.21.15.29:0/2213979336 shutdown_connections 2024-04-17T19:48:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.612+0000 7f4a2f200640 1 --2- 172.21.15.29:0/2213979336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a280704e0 0x7f4a28166a20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.612+0000 7f4a2f200640 1 --2- 172.21.15.29:0/2213979336 >> v2:172.21.15.37:3300/0 conn(0x7f4a280c91e0 0x7f4a2815cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.612+0000 7f4a2f200640 1 -- 172.21.15.29:0/2213979336 >> 172.21.15.29:0/2213979336 conn(0x7f4a2806c420 msgr2=0x7f4a280ca380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.612+0000 7f4a2f200640 1 -- 172.21.15.29:0/2213979336 shutdown_connections 2024-04-17T19:48:46.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.612+0000 7f4a2f200640 1 -- 172.21.15.29:0/2213979336 wait complete. 2024-04-17T19:48:46.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.612+0000 7f4a2f200640 1 Processor -- start 2024-04-17T19:48:46.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.612+0000 7f4a2f200640 1 -- start start 2024-04-17T19:48:46.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a2f200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a280700d0 0x7f4a281f9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a2f200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a280704e0 0x7f4a2815ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a2f200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a280c91e0 0x7f4a2814a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a2f200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4a280c9f60 con 0x7f4a280c91e0 2024-04-17T19:48:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a2f200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4a280c8cc0 con 0x7f4a280704e0 2024-04-17T19:48:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a2f200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4a280ca0e0 con 0x7f4a280700d0 2024-04-17T19:48:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a2d200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a280c91e0 0x7f4a2814a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a2d200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a280c91e0 0x7f4a2814a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33476/0 (socket says 172.21.15.29:33476) 2024-04-17T19:48:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a2d200640 1 -- 172.21.15.29:0/461895 learned_addr learned my addr 172.21.15.29:0/461895 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a27e00640 1 --2- 172.21.15.29:0/461895 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a280700d0 0x7f4a281f9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a1fe00640 1 --2- 172.21.15.29:0/461895 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a280704e0 0x7f4a2815ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a2d200640 1 -- 172.21.15.29:0/461895 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a280700d0 msgr2=0x7f4a281f9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a2d200640 1 --2- 172.21.15.29:0/461895 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a280700d0 0x7f4a281f9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a2d200640 1 -- 172.21.15.29:0/461895 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a280704e0 msgr2=0x7f4a2815ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a2d200640 1 --2- 172.21.15.29:0/461895 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a280704e0 0x7f4a2815ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a2d200640 1 -- 172.21.15.29:0/461895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a281f9a50 con 0x7f4a280c91e0 2024-04-17T19:48:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a27e00640 1 --2- 172.21.15.29:0/461895 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a280700d0 0x7f4a281f9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a1fe00640 1 --2- 172.21.15.29:0/461895 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a280704e0 0x7f4a2815ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a2d200640 1 --2- 172.21.15.29:0/461895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a280c91e0 0x7f4a2814a450 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4a200b0040 tx=0x7f4a200196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:46.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a24c00640 1 -- 172.21.15.29:0/461895 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a200b2050 con 0x7f4a280c91e0 2024-04-17T19:48:46.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a24c00640 1 -- 172.21.15.29:0/461895 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a200b3040 con 0x7f4a280c91e0 2024-04-17T19:48:46.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a2f200640 1 -- 172.21.15.29:0/461895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a280c4700 con 0x7f4a280c91e0 2024-04-17T19:48:46.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a24c00640 1 -- 172.21.15.29:0/461895 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a200ad850 con 0x7f4a280c91e0 2024-04-17T19:48:46.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a2f200640 1 -- 172.21.15.29:0/461895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a281f8d10 con 0x7f4a280c91e0 2024-04-17T19:48:46.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.616+0000 7f4a2f200640 1 -- 172.21.15.29:0/461895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a2814ebd0 con 0x7f4a280c91e0 2024-04-17T19:48:46.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.620+0000 7f4a24c00640 1 -- 172.21.15.29:0/461895 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4a200c0020 con 0x7f4a280c91e0 2024-04-17T19:48:46.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.620+0000 7f4a24c00640 1 --2- 172.21.15.29:0/461895 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4a04068380 0x7f4a0406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.620+0000 7f4a24c00640 1 -- 172.21.15.29:0/461895 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f4a20121800 con 0x7f4a280c91e0 2024-04-17T19:48:46.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.620+0000 7f4a27e00640 1 --2- 172.21.15.29:0/461895 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4a04068380 0x7f4a0406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.620+0000 7f4a27e00640 1 --2- 172.21.15.29:0/461895 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4a04068380 0x7f4a0406a890 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4a1800b990 tx=0x7f4a18008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:46.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.624+0000 7f4a24c00640 1 -- 172.21.15.29:0/461895 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4a2814ebd0 con 0x7f4a280c91e0 2024-04-17T19:48:46.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.788+0000 7f4a2f200640 1 -- 172.21.15.29:0/461895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4a2815b4a0 con 0x7f4a280c91e0 2024-04-17T19:48:46.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.788+0000 7f4a24c00640 1 -- 172.21.15.29:0/461895 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f4a20016520 con 0x7f4a280c91e0 2024-04-17T19:48:46.779 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-04-17T19:48:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.792+0000 7f4a2f200640 1 -- 172.21.15.29:0/461895 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4a04068380 msgr2=0x7f4a0406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.792+0000 7f4a2f200640 1 --2- 172.21.15.29:0/461895 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4a04068380 0x7f4a0406a890 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4a1800b990 tx=0x7f4a18008040 comp rx=0 tx=0).stop 2024-04-17T19:48:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.792+0000 7f4a2f200640 1 -- 172.21.15.29:0/461895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a280c91e0 msgr2=0x7f4a2814a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.792+0000 7f4a2f200640 1 --2- 172.21.15.29:0/461895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a280c91e0 0x7f4a2814a450 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4a200b0040 tx=0x7f4a200196d0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.792+0000 7f4a2f200640 1 -- 172.21.15.29:0/461895 shutdown_connections 2024-04-17T19:48:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.792+0000 7f4a2f200640 1 --2- 172.21.15.29:0/461895 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4a04068380 0x7f4a0406a890 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.792+0000 7f4a2f200640 1 --2- 172.21.15.29:0/461895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a280c91e0 0x7f4a2814a450 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.792+0000 7f4a2f200640 1 --2- 172.21.15.29:0/461895 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a280704e0 0x7f4a2815ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.792+0000 7f4a2f200640 1 --2- 172.21.15.29:0/461895 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a280700d0 0x7f4a281f9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.792+0000 7f4a2f200640 1 -- 172.21.15.29:0/461895 >> 172.21.15.29:0/461895 conn(0x7f4a2806c420 msgr2=0x7f4a28150080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:46.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.792+0000 7f4a2f200640 1 -- 172.21.15.29:0/461895 shutdown_connections 2024-04-17T19:48:46.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.792+0000 7f4a2f200640 1 -- 172.21.15.29:0/461895 wait complete. 2024-04-17T19:48:46.804 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T19:48:46.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.972+0000 7f53c0400640 1 Processor -- start 2024-04-17T19:48:46.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.972+0000 7f53c0400640 1 -- start start 2024-04-17T19:48:46.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.972+0000 7f53c0400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53b8071ce0 0x7f53b8141de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.972+0000 7f53c0400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f53b81593d0 0x7f53b81597e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.972+0000 7f53c0400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f53b80c9f00 0x7f53b80ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.972+0000 7f53c0400640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f53b80c7bd0 con 0x7f53b80c9f00 2024-04-17T19:48:46.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.972+0000 7f53c0400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f53b8142320 con 0x7f53b81593d0 2024-04-17T19:48:46.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.972+0000 7f53c0400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f53b815c6d0 con 0x7f53b8071ce0 2024-04-17T19:48:46.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.972+0000 7f53be400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f53b80c9f00 0x7f53b80ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.972+0000 7f53be400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f53b80c9f00 0x7f53b80ca310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33486/0 (socket says 172.21.15.29:33486) 2024-04-17T19:48:46.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.972+0000 7f53be400640 1 -- 172.21.15.29:0/971808981 learned_addr learned my addr 172.21.15.29:0/971808981 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:46.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.972+0000 7f53bd000640 1 --2- 172.21.15.29:0/971808981 >> v2:172.21.15.37:3300/0 conn(0x7f53b81593d0 0x7f53b81597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.972+0000 7f53bda00640 1 --2- 172.21.15.29:0/971808981 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53b8071ce0 0x7f53b8141de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.972+0000 7f53be400640 1 -- 172.21.15.29:0/971808981 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53b8071ce0 msgr2=0x7f53b8141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:46.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.972+0000 7f53be400640 1 --2- 172.21.15.29:0/971808981 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53b8071ce0 0x7f53b8141de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.972+0000 7f53be400640 1 -- 172.21.15.29:0/971808981 >> v2:172.21.15.37:3300/0 conn(0x7f53b81593d0 msgr2=0x7f53b81597e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:46.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.972+0000 7f53be400640 1 --2- 172.21.15.29:0/971808981 >> v2:172.21.15.37:3300/0 conn(0x7f53b81593d0 0x7f53b81597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.972+0000 7f53be400640 1 -- 172.21.15.29:0/971808981 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b815c9b0 con 0x7f53b80c9f00 2024-04-17T19:48:46.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.976+0000 7f53be400640 1 --2- 172.21.15.29:0/971808981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53b80c9f00 0x7f53b80ca310 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f53b806cd00 tx=0x7f53b001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fff2644b0587c8c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:46.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.976+0000 7f53b7e00640 1 -- 172.21.15.29:0/971808981 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b00a5760 con 0x7f53b80c9f00 2024-04-17T19:48:46.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.976+0000 7f53b7e00640 1 -- 172.21.15.29:0/971808981 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53b00a5900 con 0x7f53b80c9f00 2024-04-17T19:48:46.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.976+0000 7f53b7e00640 1 -- 172.21.15.29:0/971808981 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b00a5cd0 con 0x7f53b80c9f00 2024-04-17T19:48:46.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.976+0000 7f53c0400640 1 -- 172.21.15.29:0/971808981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53b80c9f00 msgr2=0x7f53b80ca310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:46.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.976+0000 7f53c0400640 1 --2- 172.21.15.29:0/971808981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53b80c9f00 0x7f53b80ca310 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f53b806cd00 tx=0x7f53b001c9d0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.976+0000 7f53c0400640 1 -- 172.21.15.29:0/971808981 shutdown_connections 2024-04-17T19:48:46.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.976+0000 7f53c0400640 1 --2- 172.21.15.29:0/971808981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53b80c9f00 0x7f53b80ca310 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.976+0000 7f53c0400640 1 --2- 172.21.15.29:0/971808981 >> v2:172.21.15.37:3300/0 conn(0x7f53b81593d0 0x7f53b81597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.976+0000 7f53c0400640 1 --2- 172.21.15.29:0/971808981 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53b8071ce0 0x7f53b8141de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.976+0000 7f53c0400640 1 -- 172.21.15.29:0/971808981 >> 172.21.15.29:0/971808981 conn(0x7f53b806bf90 msgr2=0x7f53b806c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:46.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.976+0000 7f53c0400640 1 -- 172.21.15.29:0/971808981 shutdown_connections 2024-04-17T19:48:46.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.976+0000 7f53c0400640 1 -- 172.21.15.29:0/971808981 wait complete. 2024-04-17T19:48:46.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53c0400640 1 Processor -- start 2024-04-17T19:48:46.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53c0400640 1 -- start start 2024-04-17T19:48:46.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53c0400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53b8071ce0 0x7f53b8135890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53c0400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53b80c9f00 0x7f53b81f8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53c0400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53b81593d0 0x7f53b8072da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53c0400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f53b8072270 con 0x7f53b80c9f00 2024-04-17T19:48:46.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53c0400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f53b80720f0 con 0x7f53b8071ce0 2024-04-17T19:48:46.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53c0400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f53b815c240 con 0x7f53b81593d0 2024-04-17T19:48:46.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53bd000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53b80c9f00 0x7f53b81f8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53bd000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53b80c9f00 0x7f53b81f8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33490/0 (socket says 172.21.15.29:33490) 2024-04-17T19:48:46.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53bd000640 1 -- 172.21.15.29:0/1384007483 learned_addr learned my addr 172.21.15.29:0/1384007483 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:46.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53be400640 1 --2- 172.21.15.29:0/1384007483 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53b81593d0 0x7f53b8072da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53bda00640 1 --2- 172.21.15.29:0/1384007483 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53b8071ce0 0x7f53b8135890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53bd000640 1 -- 172.21.15.29:0/1384007483 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53b81593d0 msgr2=0x7f53b8072da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:46.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53bd000640 1 --2- 172.21.15.29:0/1384007483 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53b81593d0 0x7f53b8072da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53bd000640 1 -- 172.21.15.29:0/1384007483 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53b8071ce0 msgr2=0x7f53b8135890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:46.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53bd000640 1 --2- 172.21.15.29:0/1384007483 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53b8071ce0 0x7f53b8135890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:46.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53bd000640 1 -- 172.21.15.29:0/1384007483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b81f5000 con 0x7f53b80c9f00 2024-04-17T19:48:46.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53bda00640 1 --2- 172.21.15.29:0/1384007483 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53b8071ce0 0x7f53b8135890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:46.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53bd000640 1 --2- 172.21.15.29:0/1384007483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53b80c9f00 0x7f53b81f8d70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f53a400d600 tx=0x7f53a400dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:46.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53b5600640 1 -- 172.21.15.29:0/1384007483 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a40125f0 con 0x7f53b80c9f00 2024-04-17T19:48:46.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53b5600640 1 -- 172.21.15.29:0/1384007483 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a4019070 con 0x7f53b80c9f00 2024-04-17T19:48:46.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53b5600640 1 -- 172.21.15.29:0/1384007483 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a40129b0 con 0x7f53b80c9f00 2024-04-17T19:48:46.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.980+0000 7f53c0400640 1 -- 172.21.15.29:0/1384007483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53b81f52e0 con 0x7f53b80c9f00 2024-04-17T19:48:46.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.984+0000 7f53c0400640 1 -- 172.21.15.29:0/1384007483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53b81f58a0 con 0x7f53b80c9f00 2024-04-17T19:48:46.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.984+0000 7f53c0400640 1 -- 172.21.15.29:0/1384007483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53b8069b90 con 0x7f53b80c9f00 2024-04-17T19:48:46.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.984+0000 7f53b5600640 1 -- 172.21.15.29:0/1384007483 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f53a4012b50 con 0x7f53b80c9f00 2024-04-17T19:48:46.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.984+0000 7f53b5600640 1 --2- 172.21.15.29:0/1384007483 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5398068380 0x7f539806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:46.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.984+0000 7f53b5600640 1 -- 172.21.15.29:0/1384007483 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f53a40870d0 con 0x7f53b80c9f00 2024-04-17T19:48:46.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.984+0000 7f53bda00640 1 --2- 172.21.15.29:0/1384007483 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5398068380 0x7f539806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:46.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.988+0000 7f53bda00640 1 --2- 172.21.15.29:0/1384007483 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5398068380 0x7f539806a890 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00b9f0 tx=0x7f53ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:46.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:46.988+0000 7f53b5600640 1 -- 172.21.15.29:0/1384007483 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f53a4057170 con 0x7f53b80c9f00 2024-04-17T19:48:47.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.152+0000 7f53c0400640 1 -- 172.21.15.29:0/1384007483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f53b80c5f90 con 0x7f53b80c9f00 2024-04-17T19:48:47.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.152+0000 7f53b5600640 1 -- 172.21.15.29:0/1384007483 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f53a4056a70 con 0x7f53b80c9f00 2024-04-17T19:48:47.141 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-04-17T19:48:47.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.152+0000 7f53c0400640 1 -- 172.21.15.29:0/1384007483 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5398068380 msgr2=0x7f539806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:47.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.152+0000 7f53c0400640 1 --2- 172.21.15.29:0/1384007483 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5398068380 0x7f539806a890 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f53ac00b9f0 tx=0x7f53ac008040 comp rx=0 tx=0).stop 2024-04-17T19:48:47.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.152+0000 7f53c0400640 1 -- 172.21.15.29:0/1384007483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53b80c9f00 msgr2=0x7f53b81f8d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:47.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.152+0000 7f53c0400640 1 --2- 172.21.15.29:0/1384007483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53b80c9f00 0x7f53b81f8d70 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f53a400d600 tx=0x7f53a400dac0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.152+0000 7f53c0400640 1 -- 172.21.15.29:0/1384007483 shutdown_connections 2024-04-17T19:48:47.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.152+0000 7f53c0400640 1 --2- 172.21.15.29:0/1384007483 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5398068380 0x7f539806a890 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.152+0000 7f53c0400640 1 --2- 172.21.15.29:0/1384007483 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53b81593d0 0x7f53b8072da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.152+0000 7f53c0400640 1 --2- 172.21.15.29:0/1384007483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53b80c9f00 0x7f53b81f8d70 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.152+0000 7f53c0400640 1 --2- 172.21.15.29:0/1384007483 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53b8071ce0 0x7f53b8135890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.152+0000 7f53c0400640 1 -- 172.21.15.29:0/1384007483 >> 172.21.15.29:0/1384007483 conn(0x7f53b806bf90 msgr2=0x7f53b806c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:47.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.152+0000 7f53c0400640 1 -- 172.21.15.29:0/1384007483 shutdown_connections 2024-04-17T19:48:47.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.156+0000 7f53c0400640 1 -- 172.21.15.29:0/1384007483 wait complete. 2024-04-17T19:48:47.163 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T19:48:47.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.328+0000 7fd9bf800640 1 Processor -- start 2024-04-17T19:48:47.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.328+0000 7fd9bf800640 1 -- start start 2024-04-17T19:48:47.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.332+0000 7fd9bf800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd9b81597e0 0x7fd9b8164630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:47.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.332+0000 7fd9bf800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9b815c6d0 con 0x7fd9b81593d0 2024-04-17T19:48:47.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.332+0000 7fd9bf800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9b815c9b0 con 0x7fd9b8071ce0 2024-04-17T19:48:47.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.332+0000 7fd9bf800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd9b80c7bd0 con 0x7fd9b81597e0 2024-04-17T19:48:47.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.332+0000 7fd9b7e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd9b81593d0 0x7fd9b80c9f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52306/0 (socket says 172.21.15.29:52306) 2024-04-17T19:48:47.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.332+0000 7fd9b7e00640 1 -- 172.21.15.29:0/2092263961 learned_addr learned my addr 172.21.15.29:0/2092263961 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:47.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.332+0000 7fd9bd800640 1 --2- 172.21.15.29:0/2092263961 >> v2:172.21.15.37:3300/0 conn(0x7fd9b81597e0 0x7fd9b8164630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:47.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.332+0000 7fd9b7400640 1 -- 172.21.15.29:0/2092263961 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1631091044 0 0) 0x7fd9b815c6d0 con 0x7fd9b81593d0 2024-04-17T19:48:47.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.332+0000 7fd9b7400640 1 -- 172.21.15.29:0/2092263961 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9a0003530 con 0x7fd9b81593d0 2024-04-17T19:48:47.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.332+0000 7fd9b7400640 1 -- 172.21.15.29:0/2092263961 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1447582121 0 0) 0x7fd9a0003530 con 0x7fd9b81593d0 2024-04-17T19:48:47.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.332+0000 7fd9bd800640 1 -- 172.21.15.29:0/2092263961 >> v1:172.21.15.37:6789/0 conn(0x7fd9b8071ce0 legacy=0x7fd9b8141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:47.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.332+0000 7fd9bd800640 1 -- 172.21.15.29:0/2092263961 >> v1:172.21.15.29:6789/0 conn(0x7fd9b81593d0 legacy=0x7fd9b80c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:47.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.332+0000 7fd9bd800640 1 -- 172.21.15.29:0/2092263961 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b815cc90 con 0x7fd9b81597e0 2024-04-17T19:48:47.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.336+0000 7fd9bd800640 1 --2- 172.21.15.29:0/2092263961 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9b81597e0 0x7fd9b8164630 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009d90 tx=0x7fd9b001c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0f687396e7897e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:47.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.336+0000 7fd9b7400640 1 -- 172.21.15.29:0/2092263961 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4001680 con 0x7fd9b81597e0 2024-04-17T19:48:47.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.336+0000 7fd9b7400640 1 -- 172.21.15.29:0/2092263961 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9b00a64e0 con 0x7fd9b81597e0 2024-04-17T19:48:47.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.336+0000 7fd9b7400640 1 -- 172.21.15.29:0/2092263961 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b00a6890 con 0x7fd9b81597e0 2024-04-17T19:48:47.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.336+0000 7fd9bf800640 1 -- 172.21.15.29:0/2092263961 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9b81597e0 msgr2=0x7fd9b8164630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:47.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.336+0000 7fd9bf800640 1 --2- 172.21.15.29:0/2092263961 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9b81597e0 0x7fd9b8164630 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009d90 tx=0x7fd9b001c890 comp rx=0 tx=0).stop 2024-04-17T19:48:47.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.336+0000 7fd9bf800640 1 -- 172.21.15.29:0/2092263961 shutdown_connections 2024-04-17T19:48:47.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.336+0000 7fd9bf800640 1 --2- 172.21.15.29:0/2092263961 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9b81597e0 0x7fd9b8164630 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.336+0000 7fd9bf800640 1 -- 172.21.15.29:0/2092263961 >> 172.21.15.29:0/2092263961 conn(0x7fd9b806bf90 msgr2=0x7fd9b806c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:47.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.336+0000 7fd9bf800640 1 -- 172.21.15.29:0/2092263961 shutdown_connections 2024-04-17T19:48:47.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.336+0000 7fd9bf800640 1 -- 172.21.15.29:0/2092263961 wait complete. 2024-04-17T19:48:47.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.336+0000 7fd9bf800640 1 Processor -- start 2024-04-17T19:48:47.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.336+0000 7fd9bf800640 1 -- start start 2024-04-17T19:48:47.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9bf800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9b8071ce0 0x7fd9b8157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:47.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9bf800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b81593d0 0x7fd9b8138d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:47.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9bf800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9b81597e0 0x7fd9b8151710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:47.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9bf800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b8072270 con 0x7fd9b81593d0 2024-04-17T19:48:47.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9bf800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b80720f0 con 0x7fd9b81597e0 2024-04-17T19:48:47.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9bf800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd9b80ca610 con 0x7fd9b8071ce0 2024-04-17T19:48:47.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9b7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b81593d0 0x7fd9b8138d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:47.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9b7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b81593d0 0x7fd9b8138d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33498/0 (socket says 172.21.15.29:33498) 2024-04-17T19:48:47.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9b7e00640 1 -- 172.21.15.29:0/121983291 learned_addr learned my addr 172.21.15.29:0/121983291 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:47.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9bd800640 1 --2- 172.21.15.29:0/121983291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9b81597e0 0x7fd9b8151710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:47.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9bce00640 1 --2- 172.21.15.29:0/121983291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9b8071ce0 0x7fd9b8157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:47.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9b7e00640 1 -- 172.21.15.29:0/121983291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9b8071ce0 msgr2=0x7fd9b8157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:47.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9b7e00640 1 --2- 172.21.15.29:0/121983291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9b8071ce0 0x7fd9b8157df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9b7e00640 1 -- 172.21.15.29:0/121983291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9b81597e0 msgr2=0x7fd9b8151710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:47.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9b7e00640 1 --2- 172.21.15.29:0/121983291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9b81597e0 0x7fd9b8151710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9b7e00640 1 -- 172.21.15.29:0/121983291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b8140390 con 0x7fd9b81593d0 2024-04-17T19:48:47.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9bd800640 1 --2- 172.21.15.29:0/121983291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9b81597e0 0x7fd9b8151710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:47.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9b7e00640 1 --2- 172.21.15.29:0/121983291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b81593d0 0x7fd9b8138d10 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd9a400c520 tx=0x7fd9a400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:47.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9b4c00640 1 -- 172.21.15.29:0/121983291 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a4017070 con 0x7fd9b81593d0 2024-04-17T19:48:47.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9b4c00640 1 -- 172.21.15.29:0/121983291 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9a4005d10 con 0x7fd9b81593d0 2024-04-17T19:48:47.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9bf800640 1 -- 172.21.15.29:0/121983291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9b8159fe0 con 0x7fd9b81593d0 2024-04-17T19:48:47.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9b4c00640 1 -- 172.21.15.29:0/121983291 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a40072b0 con 0x7fd9b81593d0 2024-04-17T19:48:47.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9bf800640 1 -- 172.21.15.29:0/121983291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9b8136ab0 con 0x7fd9b81593d0 2024-04-17T19:48:47.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.340+0000 7fd9bf800640 1 -- 172.21.15.29:0/121983291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9b815a1b0 con 0x7fd9b81593d0 2024-04-17T19:48:47.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.344+0000 7fd9b4c00640 1 -- 172.21.15.29:0/121983291 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd9a401a070 con 0x7fd9b81593d0 2024-04-17T19:48:47.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.344+0000 7fd9b4c00640 1 --2- 172.21.15.29:0/121983291 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd998068380 0x7fd99806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:47.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.344+0000 7fd9b4c00640 1 -- 172.21.15.29:0/121983291 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd9a4086250 con 0x7fd9b81593d0 2024-04-17T19:48:47.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.344+0000 7fd9bce00640 1 --2- 172.21.15.29:0/121983291 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd998068380 0x7fd99806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:47.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.344+0000 7fd9bce00640 1 --2- 172.21.15.29:0/121983291 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd998068380 0x7fd99806a890 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac007fd0 tx=0x7fd9ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:47.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.348+0000 7fd9b4c00640 1 -- 172.21.15.29:0/121983291 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd9a4055d20 con 0x7fd9b81593d0 2024-04-17T19:48:47.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.512+0000 7fd9bf800640 1 -- 172.21.15.29:0/121983291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd9b80cd2e0 con 0x7fd9b81593d0 2024-04-17T19:48:47.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.512+0000 7fd9b4c00640 1 -- 172.21.15.29:0/121983291 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fd9b80cd2e0 con 0x7fd9b81593d0 2024-04-17T19:48:47.502 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-04-17T19:48:47.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.516+0000 7fd9bf800640 1 -- 172.21.15.29:0/121983291 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd998068380 msgr2=0x7fd99806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:47.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.516+0000 7fd9bf800640 1 --2- 172.21.15.29:0/121983291 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd998068380 0x7fd99806a890 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac007fd0 tx=0x7fd9ac009040 comp rx=0 tx=0).stop 2024-04-17T19:48:47.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.516+0000 7fd9bf800640 1 -- 172.21.15.29:0/121983291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b81593d0 msgr2=0x7fd9b8138d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:47.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.516+0000 7fd9bf800640 1 --2- 172.21.15.29:0/121983291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b81593d0 0x7fd9b8138d10 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd9a400c520 tx=0x7fd9a400c9e0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.516+0000 7fd9bf800640 1 -- 172.21.15.29:0/121983291 shutdown_connections 2024-04-17T19:48:47.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.516+0000 7fd9bf800640 1 --2- 172.21.15.29:0/121983291 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd998068380 0x7fd99806a890 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.516+0000 7fd9bf800640 1 --2- 172.21.15.29:0/121983291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9b81597e0 0x7fd9b8151710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.516+0000 7fd9bf800640 1 --2- 172.21.15.29:0/121983291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9b81593d0 0x7fd9b8138d10 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.516+0000 7fd9bf800640 1 --2- 172.21.15.29:0/121983291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd9b8071ce0 0x7fd9b8157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.516+0000 7fd9bf800640 1 -- 172.21.15.29:0/121983291 >> 172.21.15.29:0/121983291 conn(0x7fd9b806bf90 msgr2=0x7fd9b8141de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:47.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.516+0000 7fd9bf800640 1 -- 172.21.15.29:0/121983291 shutdown_connections 2024-04-17T19:48:47.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.516+0000 7fd9bf800640 1 -- 172.21.15.29:0/121983291 wait complete. 2024-04-17T19:48:47.525 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2024-04-17T19:48:47.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:48:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.688+0000 7f33d8a00640 1 Processor -- start 2024-04-17T19:48:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.688+0000 7f33d8a00640 1 -- start start 2024-04-17T19:48:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.688+0000 7f33d8a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f33d415c540 0x7f33d41677c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.688+0000 7f33d8a00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33d40c55e0 con 0x7f33d40c9b20 2024-04-17T19:48:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.688+0000 7f33d8a00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33d40c9f30 con 0x7f33d41560d0 2024-04-17T19:48:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.688+0000 7f33d8a00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f33d40c8fa0 con 0x7f33d415c540 2024-04-17T19:48:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.688+0000 7f33d1e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f33d40c9b20 0x7f33d406e300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52310/0 (socket says 172.21.15.29:52310) 2024-04-17T19:48:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.688+0000 7f33d1e00640 1 -- 172.21.15.29:0/1552313977 learned_addr learned my addr 172.21.15.29:0/1552313977 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.688+0000 7f33d2800640 1 --2- 172.21.15.29:0/1552313977 >> v2:172.21.15.29:3300/0 conn(0x7f33d415c540 0x7f33d41677c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.692+0000 7f33d1400640 1 -- 172.21.15.29:0/1552313977 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3124984281 0 0) 0x7f33c4001120 con 0x7f33d40c9b20 2024-04-17T19:48:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.692+0000 7f33d1400640 1 -- 172.21.15.29:0/1552313977 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33b8003530 con 0x7f33d40c9b20 2024-04-17T19:48:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.692+0000 7f33d2800640 1 -- 172.21.15.29:0/1552313977 >> v1:172.21.15.37:6789/0 conn(0x7f33d41560d0 legacy=0x7f33d415be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.692+0000 7f33d2800640 1 -- 172.21.15.29:0/1552313977 >> v1:172.21.15.29:6789/0 conn(0x7f33d40c9b20 legacy=0x7f33d406e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.692+0000 7f33d2800640 1 -- 172.21.15.29:0/1552313977 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d40ca1d0 con 0x7f33d415c540 2024-04-17T19:48:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.692+0000 7f33d2800640 1 --2- 172.21.15.29:0/1552313977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33d415c540 0x7f33d41677c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009d90 tx=0x7f33cc01f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd286c9afd4c405 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.692+0000 7f33d1400640 1 -- 172.21.15.29:0/1552313977 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc0b4070 con 0x7f33d415c540 2024-04-17T19:48:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.692+0000 7f33d1400640 1 -- 172.21.15.29:0/1552313977 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33cc0af600 con 0x7f33d415c540 2024-04-17T19:48:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.692+0000 7f33d1400640 1 -- 172.21.15.29:0/1552313977 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc0af9d0 con 0x7f33d415c540 2024-04-17T19:48:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.696+0000 7f33d8a00640 1 -- 172.21.15.29:0/1552313977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33d415c540 msgr2=0x7f33d41677c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.696+0000 7f33d8a00640 1 --2- 172.21.15.29:0/1552313977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33d415c540 0x7f33d41677c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009d90 tx=0x7f33cc01f150 comp rx=0 tx=0).stop 2024-04-17T19:48:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.696+0000 7f33d8a00640 1 -- 172.21.15.29:0/1552313977 shutdown_connections 2024-04-17T19:48:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.696+0000 7f33d8a00640 1 --2- 172.21.15.29:0/1552313977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33d415c540 0x7f33d41677c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.696+0000 7f33d8a00640 1 -- 172.21.15.29:0/1552313977 >> 172.21.15.29:0/1552313977 conn(0x7f33d406c3c0 msgr2=0x7f33d40ca3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.696+0000 7f33d8a00640 1 -- 172.21.15.29:0/1552313977 shutdown_connections 2024-04-17T19:48:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.696+0000 7f33d8a00640 1 -- 172.21.15.29:0/1552313977 wait complete. 2024-04-17T19:48:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.696+0000 7f33d8a00640 1 Processor -- start 2024-04-17T19:48:47.686 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.696+0000 7f33d8a00640 1 -- start start 2024-04-17T19:48:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.696+0000 7f33d8a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33d40c9b20 0x7f33d41e7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.696+0000 7f33d8a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33d41560d0 0x7f33d41e7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.696+0000 7f33d8a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33d415c540 0x7f33d415f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33d8a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f33d40c8670 con 0x7f33d40c9b20 2024-04-17T19:48:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33d8a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f33d40c8390 con 0x7f33d41560d0 2024-04-17T19:48:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33d8a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f33d40c8950 con 0x7f33d415c540 2024-04-17T19:48:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33d1e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33d40c9b20 0x7f33d41e7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33d1e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33d40c9b20 0x7f33d41e7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33522/0 (socket says 172.21.15.29:33522) 2024-04-17T19:48:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33d1e00640 1 -- 172.21.15.29:0/440892291 learned_addr learned my addr 172.21.15.29:0/440892291 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33c9400640 1 --2- 172.21.15.29:0/440892291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33d41560d0 0x7f33d41e7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33d2800640 1 --2- 172.21.15.29:0/440892291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33d415c540 0x7f33d415f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33d1e00640 1 -- 172.21.15.29:0/440892291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33d415c540 msgr2=0x7f33d415f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33d1e00640 1 --2- 172.21.15.29:0/440892291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33d415c540 0x7f33d415f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33d1e00640 1 -- 172.21.15.29:0/440892291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33d41560d0 msgr2=0x7f33d41e7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33d1e00640 1 --2- 172.21.15.29:0/440892291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33d41560d0 0x7f33d41e7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33d1e00640 1 -- 172.21.15.29:0/440892291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d41e63f0 con 0x7f33d40c9b20 2024-04-17T19:48:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33d1e00640 1 --2- 172.21.15.29:0/440892291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33d40c9b20 0x7f33d41e7310 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f33c400c460 tx=0x7f33c400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33caa00640 1 -- 172.21.15.29:0/440892291 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c4017070 con 0x7f33d40c9b20 2024-04-17T19:48:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33caa00640 1 -- 172.21.15.29:0/440892291 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33c4005cf0 con 0x7f33d40c9b20 2024-04-17T19:48:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33d8a00640 1 -- 172.21.15.29:0/440892291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33d415bba0 con 0x7f33d40c9b20 2024-04-17T19:48:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33caa00640 1 -- 172.21.15.29:0/440892291 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c4007290 con 0x7f33d40c9b20 2024-04-17T19:48:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33d8a00640 1 -- 172.21.15.29:0/440892291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33d41e6110 con 0x7f33d40c9b20 2024-04-17T19:48:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33d8a00640 1 -- 172.21.15.29:0/440892291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33d4069b90 con 0x7f33d40c9b20 2024-04-17T19:48:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.700+0000 7f33caa00640 1 -- 172.21.15.29:0/440892291 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f33c401a070 con 0x7f33d40c9b20 2024-04-17T19:48:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.704+0000 7f33caa00640 1 --2- 172.21.15.29:0/440892291 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f33b0068380 0x7f33b006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.704+0000 7f33caa00640 1 -- 172.21.15.29:0/440892291 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f33c4086270 con 0x7f33d40c9b20 2024-04-17T19:48:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.704+0000 7f33c9400640 1 --2- 172.21.15.29:0/440892291 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f33b0068380 0x7f33b006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:47.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.704+0000 7f33c9400640 1 --2- 172.21.15.29:0/440892291 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f33b0068380 0x7f33b006a890 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f33d4150210 tx=0x7f33bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:47.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.708+0000 7f33caa00640 1 -- 172.21.15.29:0/440892291 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f33d40c55e0 con 0x7f33d40c9b20 2024-04-17T19:48:47.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.908+0000 7f33d8a00640 1 -- 172.21.15.29:0/440892291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f33d4159850 con 0x7f33d40c9b20 2024-04-17T19:48:47.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.908+0000 7f33caa00640 1 -- 172.21.15.29:0/440892291 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6902 (secure 0 0 0) 0x7f33c4056300 con 0x7f33d40c9b20 2024-04-17T19:48:47.896 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:47.896 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:48:34.874038+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4226":{"gid":4226,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.37:6838/332907250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":332907250},{"type":"v1","addr":"172.21.15.37:6838","nonce":332907250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4226,4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:48:47.896 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 19 2024-04-17T19:48:47.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.908+0000 7f33d8a00640 1 -- 172.21.15.29:0/440892291 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f33b0068380 msgr2=0x7f33b006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:47.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.908+0000 7f33d8a00640 1 --2- 172.21.15.29:0/440892291 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f33b0068380 0x7f33b006a890 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f33d4150210 tx=0x7f33bc009040 comp rx=0 tx=0).stop 2024-04-17T19:48:47.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.908+0000 7f33d8a00640 1 -- 172.21.15.29:0/440892291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33d40c9b20 msgr2=0x7f33d41e7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:47.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.908+0000 7f33d8a00640 1 --2- 172.21.15.29:0/440892291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33d40c9b20 0x7f33d41e7310 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f33c400c460 tx=0x7f33c400c920 comp rx=0 tx=0).stop 2024-04-17T19:48:47.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.908+0000 7f33d8a00640 1 -- 172.21.15.29:0/440892291 shutdown_connections 2024-04-17T19:48:47.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.908+0000 7f33d8a00640 1 --2- 172.21.15.29:0/440892291 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f33b0068380 0x7f33b006a890 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.908+0000 7f33d8a00640 1 --2- 172.21.15.29:0/440892291 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33d415c540 0x7f33d415f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.908+0000 7f33d8a00640 1 --2- 172.21.15.29:0/440892291 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f33d41560d0 0x7f33d41e7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.908+0000 7f33d8a00640 1 --2- 172.21.15.29:0/440892291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33d40c9b20 0x7f33d41e7310 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:47.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.908+0000 7f33d8a00640 1 -- 172.21.15.29:0/440892291 >> 172.21.15.29:0/440892291 conn(0x7f33d406c3c0 msgr2=0x7f33d414c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:47.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.908+0000 7f33d8a00640 1 -- 172.21.15.29:0/440892291 shutdown_connections 2024-04-17T19:48:47.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:47.908+0000 7f33d8a00640 1 -- 172.21.15.29:0/440892291 wait complete. 2024-04-17T19:48:47.918 INFO:tasks.mds_thrash:Ready to start thrashing 2024-04-17T19:48:47.918 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2024-04-17T19:48:47.918 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T19:48:48.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.080+0000 7f08c8c00640 1 Processor -- start 2024-04-17T19:48:48.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.080+0000 7f08c8c00640 1 -- start start 2024-04-17T19:48:48.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.084+0000 7f08c8c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f08c40cccd0 0x7f08c40ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.084+0000 7f08c8c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08c40ca870 0x7f08c41635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.084+0000 7f08c8c00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08c40cd0e0 con 0x7f08c40cc8c0 2024-04-17T19:48:48.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.084+0000 7f08c8c00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f08c40c88b0 con 0x7f08c40cccd0 2024-04-17T19:48:48.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.084+0000 7f08c8c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f08c40c6ce0 con 0x7f08c40ca870 2024-04-17T19:48:48.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.084+0000 7f08bbe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f08c40cccd0 0x7f08c40ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.084+0000 7f08bbe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f08c40cccd0 0x7f08c40ca330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33524/0 (socket says 172.21.15.29:33524) 2024-04-17T19:48:48.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.084+0000 7f08bbe00640 1 -- 172.21.15.29:0/3322165237 learned_addr learned my addr 172.21.15.29:0/3322165237 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:48.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.084+0000 7f08c2800640 1 --2- 172.21.15.29:0/3322165237 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08c40ca870 0x7f08c41635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.084+0000 7f08bbe00640 1 -- 172.21.15.29:0/3322165237 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08c40ca870 msgr2=0x7f08c41635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:48.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.084+0000 7f08bbe00640 1 --2- 172.21.15.29:0/3322165237 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08c40ca870 0x7f08c41635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.084+0000 7f08bbe00640 1 -- 172.21.15.29:0/3322165237 >> v1:172.21.15.37:6789/0 conn(0x7f08c40cc8c0 legacy=0x7f08c4159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:48.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.084+0000 7f08bbe00640 1 -- 172.21.15.29:0/3322165237 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c40c6fc0 con 0x7f08c40cccd0 2024-04-17T19:48:48.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.088+0000 7f08bbe00640 1 --2- 172.21.15.29:0/3322165237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08c40cccd0 0x7f08c40ca330 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009870 tx=0x7f08ac01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c66199df49fa074 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:48.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.088+0000 7f08c1400640 1 -- 172.21.15.29:0/3322165237 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac0b2070 con 0x7f08c40cccd0 2024-04-17T19:48:48.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.088+0000 7f08c1400640 1 -- 172.21.15.29:0/3322165237 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08ac0a9070 con 0x7f08c40cccd0 2024-04-17T19:48:48.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.088+0000 7f08c1400640 1 -- 172.21.15.29:0/3322165237 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08ac0ad880 con 0x7f08c40cccd0 2024-04-17T19:48:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.088+0000 7f08c8c00640 1 -- 172.21.15.29:0/3322165237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08c40cccd0 msgr2=0x7f08c40ca330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.088+0000 7f08c8c00640 1 --2- 172.21.15.29:0/3322165237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08c40cccd0 0x7f08c40ca330 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009870 tx=0x7f08ac01c560 comp rx=0 tx=0).stop 2024-04-17T19:48:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.088+0000 7f08c8c00640 1 -- 172.21.15.29:0/3322165237 shutdown_connections 2024-04-17T19:48:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.088+0000 7f08c8c00640 1 --2- 172.21.15.29:0/3322165237 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08c40ca870 0x7f08c41635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.088+0000 7f08c8c00640 1 --2- 172.21.15.29:0/3322165237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08c40cccd0 0x7f08c40ca330 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.088+0000 7f08c8c00640 1 -- 172.21.15.29:0/3322165237 >> 172.21.15.29:0/3322165237 conn(0x7f08c406fde0 msgr2=0x7f08c40701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.088+0000 7f08c8c00640 1 -- 172.21.15.29:0/3322165237 shutdown_connections 2024-04-17T19:48:48.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.088+0000 7f08c8c00640 1 -- 172.21.15.29:0/3322165237 wait complete. 2024-04-17T19:48:48.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.088+0000 7f08c8c00640 1 Processor -- start 2024-04-17T19:48:48.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c8c00640 1 -- start start 2024-04-17T19:48:48.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c8c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08c40ca870 0x7f08c4138fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c8c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f08c40cc8c0 0x7f08c413ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c8c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08c40cccd0 0x7f08c413b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c8c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f08c4159c20 con 0x7f08c40cccd0 2024-04-17T19:48:48.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c8c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f08c4159aa0 con 0x7f08c40cc8c0 2024-04-17T19:48:48.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c8c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f08c40c6880 con 0x7f08c40ca870 2024-04-17T19:48:48.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c2800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08c40cccd0 0x7f08c413b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c2800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08c40cccd0 0x7f08c413b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33536/0 (socket says 172.21.15.29:33536) 2024-04-17T19:48:48.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c2800640 1 -- 172.21.15.29:0/4197136722 learned_addr learned my addr 172.21.15.29:0/4197136722 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:48.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08bbe00640 1 --2- 172.21.15.29:0/4197136722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f08c40cc8c0 0x7f08c413ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c1e00640 1 --2- 172.21.15.29:0/4197136722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08c40ca870 0x7f08c4138fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c2800640 1 -- 172.21.15.29:0/4197136722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08c40ca870 msgr2=0x7f08c4138fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:48.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c2800640 1 --2- 172.21.15.29:0/4197136722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08c40ca870 0x7f08c4138fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c2800640 1 -- 172.21.15.29:0/4197136722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f08c40cc8c0 msgr2=0x7f08c413ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:48.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c2800640 1 --2- 172.21.15.29:0/4197136722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f08c40cc8c0 0x7f08c413ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c2800640 1 -- 172.21.15.29:0/4197136722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08c415cf50 con 0x7f08c40cccd0 2024-04-17T19:48:48.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c2800640 1 --2- 172.21.15.29:0/4197136722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08c40cccd0 0x7f08c413b190 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f08bc009fd0 tx=0x7f08bc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:48.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08ba000640 1 -- 172.21.15.29:0/4197136722 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08bc014420 con 0x7f08c40cccd0 2024-04-17T19:48:48.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08ba000640 1 -- 172.21.15.29:0/4197136722 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08bc019070 con 0x7f08c40cccd0 2024-04-17T19:48:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c8c00640 1 -- 172.21.15.29:0/4197136722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08c41f8a80 con 0x7f08c40cccd0 2024-04-17T19:48:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08ba000640 1 -- 172.21.15.29:0/4197136722 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08bc014800 con 0x7f08c40cccd0 2024-04-17T19:48:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c8c00640 1 -- 172.21.15.29:0/4197136722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08c41f5960 con 0x7f08c40cccd0 2024-04-17T19:48:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.092+0000 7f08c8c00640 1 -- 172.21.15.29:0/4197136722 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08c4072ac0 con 0x7f08c40cccd0 2024-04-17T19:48:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.096+0000 7f08ba000640 1 -- 172.21.15.29:0/4197136722 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f08bc0149a0 con 0x7f08c40cccd0 2024-04-17T19:48:48.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.096+0000 7f08ba000640 1 --2- 172.21.15.29:0/4197136722 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f08a0068380 0x7f08a006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.096+0000 7f08ba000640 1 -- 172.21.15.29:0/4197136722 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f08bc087580 con 0x7f08c40cccd0 2024-04-17T19:48:48.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.096+0000 7f08c1e00640 1 --2- 172.21.15.29:0/4197136722 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f08a0068380 0x7f08a006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.096+0000 7f08c1e00640 1 --2- 172.21.15.29:0/4197136722 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f08a0068380 0x7f08a006a890 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f08b4001770 tx=0x7f08b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:48.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.100+0000 7f08ba000640 1 -- 172.21.15.29:0/4197136722 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f08c4072ac0 con 0x7f08c40cccd0 2024-04-17T19:48:48.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.264+0000 7f08c8c00640 1 -- 172.21.15.29:0/4197136722 --> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f08c41f5470 con 0x7f08a0068380 2024-04-17T19:48:48.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.288+0000 7f08ba000640 1 -- 172.21.15.29:0/4197136722 <== mgr.4105 v2:172.21.15.37:6832/1588862286 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332656 (secure 0 0 0) 0x7f08c41f5470 con 0x7f08a0068380 2024-04-17T19:48:48.281 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:48.283 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-04-17T19:48:48.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.292+0000 7f08c8c00640 1 -- 172.21.15.29:0/4197136722 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f08a0068380 msgr2=0x7f08a006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:48.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.292+0000 7f08c8c00640 1 --2- 172.21.15.29:0/4197136722 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f08a0068380 0x7f08a006a890 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f08b4001770 tx=0x7f08b4009040 comp rx=0 tx=0).stop 2024-04-17T19:48:48.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.292+0000 7f08c8c00640 1 -- 172.21.15.29:0/4197136722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08c40cccd0 msgr2=0x7f08c413b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.292+0000 7f08c8c00640 1 --2- 172.21.15.29:0/4197136722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08c40cccd0 0x7f08c413b190 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f08bc009fd0 tx=0x7f08bc00e4b0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.292+0000 7f08c8c00640 1 -- 172.21.15.29:0/4197136722 shutdown_connections 2024-04-17T19:48:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.292+0000 7f08c8c00640 1 --2- 172.21.15.29:0/4197136722 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f08a0068380 0x7f08a006a890 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.292+0000 7f08c8c00640 1 --2- 172.21.15.29:0/4197136722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f08c40cccd0 0x7f08c413b190 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.292+0000 7f08c8c00640 1 --2- 172.21.15.29:0/4197136722 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f08c40cc8c0 0x7f08c413ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.292+0000 7f08c8c00640 1 --2- 172.21.15.29:0/4197136722 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f08c40ca870 0x7f08c4138fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.292+0000 7f08c8c00640 1 -- 172.21.15.29:0/4197136722 >> 172.21.15.29:0/4197136722 conn(0x7f08c406fde0 msgr2=0x7f08c40cbdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.292+0000 7f08c8c00640 1 -- 172.21.15.29:0/4197136722 shutdown_connections 2024-04-17T19:48:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.292+0000 7f08c8c00640 1 -- 172.21.15.29:0/4197136722 wait complete. 2024-04-17T19:48:48.303 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":55,"stamp":"2024-04-17T19:48:46.650532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600890,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":369,"num_write_kb":2484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":301,"ondisk_log_size":301,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240520,"kb_used_data":3592,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754734200,"statfs":{"total":773094113280,"available":772847820800,"internally_reserved":0,"allocated":3678208,"data_stored":2116732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1664,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005371"},"pg_stats":[{"pgid":"2.1f","version":"26'13","reported_seq":39,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.876148+0000","last_change":"2024-04-17T19:47:48.885235+0000","last_active":"2024-04-17T19:48:33.876148+0000","last_peered":"2024-04-17T19:48:33.876148+0000","last_clean":"2024-04-17T19:48:33.876148+0000","last_became_active":"2024-04-17T19:47:48.884844+0000","last_became_peered":"2024-04-17T19:47:48.884844+0000","last_unstale":"2024-04-17T19:48:33.876148+0000","last_undegraded":"2024-04-17T19:48:33.876148+0000","last_fullsized":"2024-04-17T19:48:33.876148+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:01:52.854763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834602+0000","last_change":"2024-04-17T19:47:49.872314+0000","last_active":"2024-04-17T19:48:29.834602+0000","last_peered":"2024-04-17T19:48:29.834602+0000","last_clean":"2024-04-17T19:48:29.834602+0000","last_became_active":"2024-04-17T19:47:49.871990+0000","last_became_peered":"2024-04-17T19:47:49.871990+0000","last_unstale":"2024-04-17T19:48:29.834602+0000","last_undegraded":"2024-04-17T19:48:29.834602+0000","last_fullsized":"2024-04-17T19:48:29.834602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:12:09.196610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.833967+0000","last_change":"2024-04-17T19:47:53.929840+0000","last_active":"2024-04-17T19:48:29.833967+0000","last_peered":"2024-04-17T19:48:29.833967+0000","last_clean":"2024-04-17T19:48:29.833967+0000","last_became_active":"2024-04-17T19:47:53.928272+0000","last_became_peered":"2024-04-17T19:47:53.928272+0000","last_unstale":"2024-04-17T19:48:29.833967+0000","last_undegraded":"2024-04-17T19:48:29.833967+0000","last_fullsized":"2024-04-17T19:48:29.833967+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:27:29.734327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"24'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.835623+0000","last_change":"2024-04-17T19:47:48.862315+0000","last_active":"2024-04-17T19:48:29.835623+0000","last_peered":"2024-04-17T19:48:29.835623+0000","last_clean":"2024-04-17T19:48:29.835623+0000","last_became_active":"2024-04-17T19:47:48.860408+0000","last_became_peered":"2024-04-17T19:47:48.860408+0000","last_unstale":"2024-04-17T19:48:29.835623+0000","last_undegraded":"2024-04-17T19:48:29.835623+0000","last_fullsized":"2024-04-17T19:48:29.835623+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:10:26.336063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162315+0000","last_change":"2024-04-17T19:47:49.902423+0000","last_active":"2024-04-17T19:48:30.162315+0000","last_peered":"2024-04-17T19:48:30.162315+0000","last_clean":"2024-04-17T19:48:30.162315+0000","last_became_active":"2024-04-17T19:47:49.902096+0000","last_became_peered":"2024-04-17T19:47:49.902096+0000","last_unstale":"2024-04-17T19:48:30.162315+0000","last_undegraded":"2024-04-17T19:48:30.162315+0000","last_fullsized":"2024-04-17T19:48:30.162315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:40.805247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.849677+0000","last_change":"2024-04-17T19:47:53.956851+0000","last_active":"2024-04-17T19:48:29.849677+0000","last_peered":"2024-04-17T19:48:29.849677+0000","last_clean":"2024-04-17T19:48:29.849677+0000","last_became_active":"2024-04-17T19:47:53.954197+0000","last_became_peered":"2024-04-17T19:47:53.954197+0000","last_unstale":"2024-04-17T19:48:29.849677+0000","last_undegraded":"2024-04-17T19:48:29.849677+0000","last_fullsized":"2024-04-17T19:48:29.849677+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:16:18.379831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"26'12","reported_seq":38,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.877776+0000","last_change":"2024-04-17T19:47:48.893969+0000","last_active":"2024-04-17T19:48:33.877776+0000","last_peered":"2024-04-17T19:48:33.877776+0000","last_clean":"2024-04-17T19:48:33.877776+0000","last_became_active":"2024-04-17T19:47:48.893229+0000","last_became_peered":"2024-04-17T19:47:48.893229+0000","last_unstale":"2024-04-17T19:48:33.877776+0000","last_undegraded":"2024-04-17T19:48:33.877776+0000","last_fullsized":"2024-04-17T19:48:33.877776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:38:42.472634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.848206+0000","last_change":"2024-04-17T19:47:49.897987+0000","last_active":"2024-04-17T19:48:29.848206+0000","last_peered":"2024-04-17T19:48:29.848206+0000","last_clean":"2024-04-17T19:48:29.848206+0000","last_became_active":"2024-04-17T19:47:49.897096+0000","last_became_peered":"2024-04-17T19:47:49.897096+0000","last_unstale":"2024-04-17T19:48:29.848206+0000","last_undegraded":"2024-04-17T19:48:29.848206+0000","last_fullsized":"2024-04-17T19:48:29.848206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:36:36.856454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.850365+0000","last_change":"2024-04-17T19:47:53.930784+0000","last_active":"2024-04-17T19:48:29.850365+0000","last_peered":"2024-04-17T19:48:29.850365+0000","last_clean":"2024-04-17T19:48:29.850365+0000","last_became_active":"2024-04-17T19:47:53.928281+0000","last_became_peered":"2024-04-17T19:47:53.928281+0000","last_unstale":"2024-04-17T19:48:29.850365+0000","last_undegraded":"2024-04-17T19:48:29.850365+0000","last_fullsized":"2024-04-17T19:48:29.850365+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:15:55.064622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.876428+0000","last_change":"2024-04-17T19:47:48.893184+0000","last_active":"2024-04-17T19:48:33.876428+0000","last_peered":"2024-04-17T19:48:33.876428+0000","last_clean":"2024-04-17T19:48:33.876428+0000","last_became_active":"2024-04-17T19:47:48.892375+0000","last_became_peered":"2024-04-17T19:47:48.892375+0000","last_unstale":"2024-04-17T19:48:33.876428+0000","last_undegraded":"2024-04-17T19:48:33.876428+0000","last_fullsized":"2024-04-17T19:48:33.876428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:56:01.211666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.848245+0000","last_change":"2024-04-17T19:47:49.896040+0000","last_active":"2024-04-17T19:48:29.848245+0000","last_peered":"2024-04-17T19:48:29.848245+0000","last_clean":"2024-04-17T19:48:29.848245+0000","last_became_active":"2024-04-17T19:47:49.894779+0000","last_became_peered":"2024-04-17T19:47:49.894779+0000","last_unstale":"2024-04-17T19:48:29.848245+0000","last_undegraded":"2024-04-17T19:48:29.848245+0000","last_fullsized":"2024-04-17T19:48:29.848245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:30:17.930203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.852223+0000","last_change":"2024-04-17T19:47:53.930402+0000","last_active":"2024-04-17T19:48:29.852223+0000","last_peered":"2024-04-17T19:48:29.852223+0000","last_clean":"2024-04-17T19:48:29.852223+0000","last_became_active":"2024-04-17T19:47:53.929166+0000","last_became_peered":"2024-04-17T19:47:53.929166+0000","last_unstale":"2024-04-17T19:48:29.852223+0000","last_undegraded":"2024-04-17T19:48:29.852223+0000","last_fullsized":"2024-04-17T19:48:29.852223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:34:08.282697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843243+0000","last_change":"2024-04-17T19:47:49.900142+0000","last_active":"2024-04-17T19:48:29.843243+0000","last_peered":"2024-04-17T19:48:29.843243+0000","last_clean":"2024-04-17T19:48:29.843243+0000","last_became_active":"2024-04-17T19:47:49.899365+0000","last_became_peered":"2024-04-17T19:47:49.899365+0000","last_unstale":"2024-04-17T19:48:29.843243+0000","last_undegraded":"2024-04-17T19:48:29.843243+0000","last_fullsized":"2024-04-17T19:48:29.843243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:34:49.616539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"24'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.844328+0000","last_change":"2024-04-17T19:47:48.891513+0000","last_active":"2024-04-17T19:48:29.844328+0000","last_peered":"2024-04-17T19:48:29.844328+0000","last_clean":"2024-04-17T19:48:29.844328+0000","last_became_active":"2024-04-17T19:47:48.886561+0000","last_became_peered":"2024-04-17T19:47:48.886561+0000","last_unstale":"2024-04-17T19:48:29.844328+0000","last_undegraded":"2024-04-17T19:48:29.844328+0000","last_fullsized":"2024-04-17T19:48:29.844328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:59:14.039700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.167531+0000","last_change":"2024-04-17T19:48:30.167531+0000","last_active":"2024-04-17T19:48:30.167531+0000","last_peered":"2024-04-17T19:48:30.167531+0000","last_clean":"2024-04-17T19:48:30.167531+0000","last_became_active":"2024-04-17T19:48:30.165421+0000","last_became_peered":"2024-04-17T19:48:30.165421+0000","last_unstale":"2024-04-17T19:48:30.167531+0000","last_undegraded":"2024-04-17T19:48:30.167531+0000","last_fullsized":"2024-04-17T19:48:30.167531+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:52:16.429658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,5],"acting":[4,0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834455+0000","last_change":"2024-04-17T19:47:49.852183+0000","last_active":"2024-04-17T19:48:29.834455+0000","last_peered":"2024-04-17T19:48:29.834455+0000","last_clean":"2024-04-17T19:48:29.834455+0000","last_became_active":"2024-04-17T19:47:49.851765+0000","last_became_peered":"2024-04-17T19:47:49.851765+0000","last_unstale":"2024-04-17T19:48:29.834455+0000","last_undegraded":"2024-04-17T19:48:29.834455+0000","last_fullsized":"2024-04-17T19:48:29.834455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:02:03.101317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.841041+0000","last_change":"2024-04-17T19:47:48.862865+0000","last_active":"2024-04-17T19:48:29.841041+0000","last_peered":"2024-04-17T19:48:29.841041+0000","last_clean":"2024-04-17T19:48:29.841041+0000","last_became_active":"2024-04-17T19:47:48.862524+0000","last_became_peered":"2024-04-17T19:47:48.862524+0000","last_unstale":"2024-04-17T19:48:29.841041+0000","last_undegraded":"2024-04-17T19:48:29.841041+0000","last_fullsized":"2024-04-17T19:48:29.841041+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:01:21.080029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.850650+0000","last_change":"2024-04-17T19:47:53.943674+0000","last_active":"2024-04-17T19:48:29.850650+0000","last_peered":"2024-04-17T19:48:29.850650+0000","last_clean":"2024-04-17T19:48:29.850650+0000","last_became_active":"2024-04-17T19:47:53.942549+0000","last_became_peered":"2024-04-17T19:47:53.942549+0000","last_unstale":"2024-04-17T19:48:29.850650+0000","last_undegraded":"2024-04-17T19:48:29.850650+0000","last_fullsized":"2024-04-17T19:48:29.850650+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:00:42.300738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834364+0000","last_change":"2024-04-17T19:47:49.862312+0000","last_active":"2024-04-17T19:48:29.834364+0000","last_peered":"2024-04-17T19:48:29.834364+0000","last_clean":"2024-04-17T19:48:29.834364+0000","last_became_active":"2024-04-17T19:47:49.860664+0000","last_became_peered":"2024-04-17T19:47:49.860664+0000","last_unstale":"2024-04-17T19:48:29.834364+0000","last_undegraded":"2024-04-17T19:48:29.834364+0000","last_fullsized":"2024-04-17T19:48:29.834364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:07:47.676973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.874890+0000","last_change":"2024-04-17T19:47:48.870638+0000","last_active":"2024-04-17T19:48:33.874890+0000","last_peered":"2024-04-17T19:48:33.874890+0000","last_clean":"2024-04-17T19:48:33.874890+0000","last_became_active":"2024-04-17T19:47:48.870353+0000","last_became_peered":"2024-04-17T19:47:48.870353+0000","last_unstale":"2024-04-17T19:48:33.874890+0000","last_undegraded":"2024-04-17T19:48:33.874890+0000","last_fullsized":"2024-04-17T19:48:33.874890+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:12:19.927824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.861156+0000","last_change":"2024-04-17T19:48:29.857425+0000","last_active":"2024-04-17T19:48:29.861156+0000","last_peered":"2024-04-17T19:48:29.861156+0000","last_clean":"2024-04-17T19:48:29.861156+0000","last_became_active":"2024-04-17T19:48:29.856886+0000","last_became_peered":"2024-04-17T19:48:29.856886+0000","last_unstale":"2024-04-17T19:48:29.861156+0000","last_undegraded":"2024-04-17T19:48:29.861156+0000","last_fullsized":"2024-04-17T19:48:29.861156+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:17:41.501101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.854191+0000","last_change":"2024-04-17T19:47:49.888118+0000","last_active":"2024-04-17T19:48:29.854191+0000","last_peered":"2024-04-17T19:48:29.854191+0000","last_clean":"2024-04-17T19:48:29.854191+0000","last_became_active":"2024-04-17T19:47:49.884037+0000","last_became_peered":"2024-04-17T19:47:49.884037+0000","last_unstale":"2024-04-17T19:48:29.854191+0000","last_undegraded":"2024-04-17T19:48:29.854191+0000","last_fullsized":"2024-04-17T19:48:29.854191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:44:27.180380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"24'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843109+0000","last_change":"2024-04-17T19:47:48.883686+0000","last_active":"2024-04-17T19:48:29.843109+0000","last_peered":"2024-04-17T19:48:29.843109+0000","last_clean":"2024-04-17T19:48:29.843109+0000","last_became_active":"2024-04-17T19:47:48.882375+0000","last_became_peered":"2024-04-17T19:47:48.882375+0000","last_unstale":"2024-04-17T19:48:29.843109+0000","last_undegraded":"2024-04-17T19:48:29.843109+0000","last_fullsized":"2024-04-17T19:48:29.843109+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:38:31.225374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.851300+0000","last_change":"2024-04-17T19:47:53.935396+0000","last_active":"2024-04-17T19:48:29.851300+0000","last_peered":"2024-04-17T19:48:29.851300+0000","last_clean":"2024-04-17T19:48:29.851300+0000","last_became_active":"2024-04-17T19:47:53.934094+0000","last_became_peered":"2024-04-17T19:47:53.934094+0000","last_unstale":"2024-04-17T19:48:29.851300+0000","last_undegraded":"2024-04-17T19:48:29.851300+0000","last_fullsized":"2024-04-17T19:48:29.851300+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:58.334357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162117+0000","last_change":"2024-04-17T19:47:49.870297+0000","last_active":"2024-04-17T19:48:30.162117+0000","last_peered":"2024-04-17T19:48:30.162117+0000","last_clean":"2024-04-17T19:48:30.162117+0000","last_became_active":"2024-04-17T19:47:49.870113+0000","last_became_peered":"2024-04-17T19:47:49.870113+0000","last_unstale":"2024-04-17T19:48:30.162117+0000","last_undegraded":"2024-04-17T19:48:30.162117+0000","last_fullsized":"2024-04-17T19:48:30.162117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:50:54.517273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.876067+0000","last_change":"2024-04-17T19:47:48.895905+0000","last_active":"2024-04-17T19:48:33.876067+0000","last_peered":"2024-04-17T19:48:33.876067+0000","last_clean":"2024-04-17T19:48:33.876067+0000","last_became_active":"2024-04-17T19:47:48.895263+0000","last_became_peered":"2024-04-17T19:47:48.895263+0000","last_unstale":"2024-04-17T19:48:33.876067+0000","last_undegraded":"2024-04-17T19:48:33.876067+0000","last_fullsized":"2024-04-17T19:48:33.876067+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:07:23.179031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.851454+0000","last_change":"2024-04-17T19:47:53.943286+0000","last_active":"2024-04-17T19:48:29.851454+0000","last_peered":"2024-04-17T19:48:29.851454+0000","last_clean":"2024-04-17T19:48:29.851454+0000","last_became_active":"2024-04-17T19:47:53.942325+0000","last_became_peered":"2024-04-17T19:47:53.942325+0000","last_unstale":"2024-04-17T19:48:29.851454+0000","last_undegraded":"2024-04-17T19:48:29.851454+0000","last_fullsized":"2024-04-17T19:48:29.851454+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:47:18.853991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.835175+0000","last_change":"2024-04-17T19:47:49.872925+0000","last_active":"2024-04-17T19:48:29.835175+0000","last_peered":"2024-04-17T19:48:29.835175+0000","last_clean":"2024-04-17T19:48:29.835175+0000","last_became_active":"2024-04-17T19:47:49.872668+0000","last_became_peered":"2024-04-17T19:47:49.872668+0000","last_unstale":"2024-04-17T19:48:29.835175+0000","last_undegraded":"2024-04-17T19:48:29.835175+0000","last_fullsized":"2024-04-17T19:48:29.835175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:16:21.209634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839696+0000","last_change":"2024-04-17T19:47:48.896669+0000","last_active":"2024-04-17T19:48:29.839696+0000","last_peered":"2024-04-17T19:48:29.839696+0000","last_clean":"2024-04-17T19:48:29.839696+0000","last_became_active":"2024-04-17T19:47:48.896397+0000","last_became_peered":"2024-04-17T19:47:48.896397+0000","last_unstale":"2024-04-17T19:48:29.839696+0000","last_undegraded":"2024-04-17T19:48:29.839696+0000","last_fullsized":"2024-04-17T19:48:29.839696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:07:23.179031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.833842+0000","last_change":"2024-04-17T19:47:53.906261+0000","last_active":"2024-04-17T19:48:29.833842+0000","last_peered":"2024-04-17T19:48:29.833842+0000","last_clean":"2024-04-17T19:48:29.833842+0000","last_became_active":"2024-04-17T19:47:53.903702+0000","last_became_peered":"2024-04-17T19:47:53.903702+0000","last_unstale":"2024-04-17T19:48:29.833842+0000","last_undegraded":"2024-04-17T19:48:29.833842+0000","last_fullsized":"2024-04-17T19:48:29.833842+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:35:53.892917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838432+0000","last_change":"2024-04-17T19:47:49.870272+0000","last_active":"2024-04-17T19:48:29.838432+0000","last_peered":"2024-04-17T19:48:29.838432+0000","last_clean":"2024-04-17T19:48:29.838432+0000","last_became_active":"2024-04-17T19:47:49.869914+0000","last_became_peered":"2024-04-17T19:47:49.869914+0000","last_unstale":"2024-04-17T19:48:29.838432+0000","last_undegraded":"2024-04-17T19:48:29.838432+0000","last_fullsized":"2024-04-17T19:48:29.838432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:44:30.553286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845870+0000","last_change":"2024-04-17T19:47:48.892675+0000","last_active":"2024-04-17T19:48:29.845870+0000","last_peered":"2024-04-17T19:48:29.845870+0000","last_clean":"2024-04-17T19:48:29.845870+0000","last_became_active":"2024-04-17T19:47:48.892057+0000","last_became_peered":"2024-04-17T19:47:48.892057+0000","last_unstale":"2024-04-17T19:48:29.845870+0000","last_undegraded":"2024-04-17T19:48:29.845870+0000","last_fullsized":"2024-04-17T19:48:29.845870+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:18:35.735350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.163029+0000","last_change":"2024-04-17T19:47:53.925135+0000","last_active":"2024-04-17T19:48:30.163029+0000","last_peered":"2024-04-17T19:48:30.163029+0000","last_clean":"2024-04-17T19:48:30.163029+0000","last_became_active":"2024-04-17T19:47:53.924617+0000","last_became_peered":"2024-04-17T19:47:53.924617+0000","last_unstale":"2024-04-17T19:48:30.163029+0000","last_undegraded":"2024-04-17T19:48:30.163029+0000","last_fullsized":"2024-04-17T19:48:30.163029+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:37:02.082648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.844178+0000","last_change":"2024-04-17T19:48:29.842544+0000","last_active":"2024-04-17T19:48:29.844178+0000","last_peered":"2024-04-17T19:48:29.844178+0000","last_clean":"2024-04-17T19:48:29.844178+0000","last_became_active":"2024-04-17T19:48:29.842198+0000","last_became_peered":"2024-04-17T19:48:29.842198+0000","last_unstale":"2024-04-17T19:48:29.844178+0000","last_undegraded":"2024-04-17T19:48:29.844178+0000","last_fullsized":"2024-04-17T19:48:29.844178+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:26:06.151001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"24'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838549+0000","last_change":"2024-04-17T19:47:48.859247+0000","last_active":"2024-04-17T19:48:29.838549+0000","last_peered":"2024-04-17T19:48:29.838549+0000","last_clean":"2024-04-17T19:48:29.838549+0000","last_became_active":"2024-04-17T19:47:48.858756+0000","last_became_peered":"2024-04-17T19:47:48.858756+0000","last_unstale":"2024-04-17T19:48:29.838549+0000","last_undegraded":"2024-04-17T19:48:29.838549+0000","last_fullsized":"2024-04-17T19:48:29.838549+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:19:56.987957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840601+0000","last_change":"2024-04-17T19:47:53.929952+0000","last_active":"2024-04-17T19:48:29.840601+0000","last_peered":"2024-04-17T19:48:29.840601+0000","last_clean":"2024-04-17T19:48:29.840601+0000","last_became_active":"2024-04-17T19:47:53.929528+0000","last_became_peered":"2024-04-17T19:47:53.929528+0000","last_unstale":"2024-04-17T19:48:29.840601+0000","last_undegraded":"2024-04-17T19:48:29.840601+0000","last_fullsized":"2024-04-17T19:48:29.840601+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:11:38.041301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.853204+0000","last_change":"2024-04-17T19:47:49.874150+0000","last_active":"2024-04-17T19:48:29.853204+0000","last_peered":"2024-04-17T19:48:29.853204+0000","last_clean":"2024-04-17T19:48:29.853204+0000","last_became_active":"2024-04-17T19:47:49.872859+0000","last_became_peered":"2024-04-17T19:47:49.872859+0000","last_unstale":"2024-04-17T19:48:29.853204+0000","last_undegraded":"2024-04-17T19:48:29.853204+0000","last_fullsized":"2024-04-17T19:48:29.853204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:57:54.666279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.876393+0000","last_change":"2024-04-17T19:47:48.876672+0000","last_active":"2024-04-17T19:48:33.876393+0000","last_peered":"2024-04-17T19:48:33.876393+0000","last_clean":"2024-04-17T19:48:33.876393+0000","last_became_active":"2024-04-17T19:47:48.875263+0000","last_became_peered":"2024-04-17T19:47:48.875263+0000","last_unstale":"2024-04-17T19:48:33.876393+0000","last_undegraded":"2024-04-17T19:48:33.876393+0000","last_fullsized":"2024-04-17T19:48:33.876393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:45:46.633540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.857338+0000","last_change":"2024-04-17T19:48:29.852166+0000","last_active":"2024-04-17T19:48:29.857338+0000","last_peered":"2024-04-17T19:48:29.857338+0000","last_clean":"2024-04-17T19:48:29.857338+0000","last_became_active":"2024-04-17T19:48:29.850441+0000","last_became_peered":"2024-04-17T19:48:29.850441+0000","last_unstale":"2024-04-17T19:48:29.857338+0000","last_undegraded":"2024-04-17T19:48:29.857338+0000","last_fullsized":"2024-04-17T19:48:29.857338+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:32:25.836924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3,6],"acting":[5,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838297+0000","last_change":"2024-04-17T19:47:49.865020+0000","last_active":"2024-04-17T19:48:29.838297+0000","last_peered":"2024-04-17T19:48:29.838297+0000","last_clean":"2024-04-17T19:48:29.838297+0000","last_became_active":"2024-04-17T19:47:49.864764+0000","last_became_peered":"2024-04-17T19:47:49.864764+0000","last_unstale":"2024-04-17T19:48:29.838297+0000","last_undegraded":"2024-04-17T19:48:29.838297+0000","last_fullsized":"2024-04-17T19:48:29.838297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:45:36.325765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838108+0000","last_change":"2024-04-17T19:47:48.846570+0000","last_active":"2024-04-17T19:48:29.838108+0000","last_peered":"2024-04-17T19:48:29.838108+0000","last_clean":"2024-04-17T19:48:29.838108+0000","last_became_active":"2024-04-17T19:47:48.846175+0000","last_became_peered":"2024-04-17T19:47:48.846175+0000","last_unstale":"2024-04-17T19:48:29.838108+0000","last_undegraded":"2024-04-17T19:48:29.838108+0000","last_fullsized":"2024-04-17T19:48:29.838108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:20:22.737016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839023+0000","last_change":"2024-04-17T19:47:53.904590+0000","last_active":"2024-04-17T19:48:29.839023+0000","last_peered":"2024-04-17T19:48:29.839023+0000","last_clean":"2024-04-17T19:48:29.839023+0000","last_became_active":"2024-04-17T19:47:53.904026+0000","last_became_peered":"2024-04-17T19:47:53.904026+0000","last_unstale":"2024-04-17T19:48:29.839023+0000","last_undegraded":"2024-04-17T19:48:29.839023+0000","last_fullsized":"2024-04-17T19:48:29.839023+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:16:22.837570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.161964+0000","last_change":"2024-04-17T19:47:49.860010+0000","last_active":"2024-04-17T19:48:30.161964+0000","last_peered":"2024-04-17T19:48:30.161964+0000","last_clean":"2024-04-17T19:48:30.161964+0000","last_became_active":"2024-04-17T19:47:49.859593+0000","last_became_peered":"2024-04-17T19:47:49.859593+0000","last_unstale":"2024-04-17T19:48:30.161964+0000","last_undegraded":"2024-04-17T19:48:30.161964+0000","last_fullsized":"2024-04-17T19:48:30.161964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:33:24.594397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843671+0000","last_change":"2024-04-17T19:47:48.883126+0000","last_active":"2024-04-17T19:48:29.843671+0000","last_peered":"2024-04-17T19:48:29.843671+0000","last_clean":"2024-04-17T19:48:29.843671+0000","last_became_active":"2024-04-17T19:47:48.882173+0000","last_became_peered":"2024-04-17T19:47:48.882173+0000","last_unstale":"2024-04-17T19:48:29.843671+0000","last_undegraded":"2024-04-17T19:48:29.843671+0000","last_fullsized":"2024-04-17T19:48:29.843671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:11:59.395316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.163046+0000","last_change":"2024-04-17T19:47:53.931258+0000","last_active":"2024-04-17T19:48:30.163046+0000","last_peered":"2024-04-17T19:48:30.163046+0000","last_clean":"2024-04-17T19:48:30.163046+0000","last_became_active":"2024-04-17T19:47:53.930773+0000","last_became_peered":"2024-04-17T19:47:53.930773+0000","last_unstale":"2024-04-17T19:48:30.163046+0000","last_undegraded":"2024-04-17T19:48:30.163046+0000","last_fullsized":"2024-04-17T19:48:30.163046+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:10:47.583714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.844178+0000","last_change":"2024-04-17T19:48:29.844178+0000","last_active":"2024-04-17T19:48:29.844178+0000","last_peered":"2024-04-17T19:48:29.844178+0000","last_clean":"2024-04-17T19:48:29.844178+0000","last_became_active":"2024-04-17T19:48:29.843807+0000","last_became_peered":"2024-04-17T19:48:29.843807+0000","last_unstale":"2024-04-17T19:48:29.844178+0000","last_undegraded":"2024-04-17T19:48:29.844178+0000","last_fullsized":"2024-04-17T19:48:29.844178+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:36:08.301346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"26'6","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.849667+0000","last_change":"2024-04-17T19:47:48.878592+0000","last_active":"2024-04-17T19:48:29.849667+0000","last_peered":"2024-04-17T19:48:29.849667+0000","last_clean":"2024-04-17T19:48:29.849667+0000","last_became_active":"2024-04-17T19:47:48.877617+0000","last_became_peered":"2024-04-17T19:47:48.877617+0000","last_unstale":"2024-04-17T19:48:29.849667+0000","last_undegraded":"2024-04-17T19:48:29.849667+0000","last_fullsized":"2024-04-17T19:48:29.849667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:30:17.013176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":182,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843673+0000","last_change":"2024-04-17T19:47:38.277798+0000","last_active":"2024-04-17T19:48:29.843673+0000","last_peered":"2024-04-17T19:48:29.843673+0000","last_clean":"2024-04-17T19:48:29.843673+0000","last_became_active":"2024-04-17T19:47:38.274697+0000","last_became_peered":"2024-04-17T19:47:38.274697+0000","last_unstale":"2024-04-17T19:48:29.843673+0000","last_undegraded":"2024-04-17T19:48:29.843673+0000","last_fullsized":"2024-04-17T19:48:29.843673+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:36.897071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:36.897071+0000","last_clean_scrub_stamp":"2024-04-17T19:47:36.897071+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:31:35.366600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.844472+0000","last_change":"2024-04-17T19:47:53.953415+0000","last_active":"2024-04-17T19:48:29.844472+0000","last_peered":"2024-04-17T19:48:29.844472+0000","last_clean":"2024-04-17T19:48:29.844472+0000","last_became_active":"2024-04-17T19:47:53.952644+0000","last_became_peered":"2024-04-17T19:47:53.952644+0000","last_unstale":"2024-04-17T19:48:29.844472+0000","last_undegraded":"2024-04-17T19:48:29.844472+0000","last_fullsized":"2024-04-17T19:48:29.844472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:00:25.923909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.844721+0000","last_change":"2024-04-17T19:47:49.901572+0000","last_active":"2024-04-17T19:48:29.844721+0000","last_peered":"2024-04-17T19:48:29.844721+0000","last_clean":"2024-04-17T19:48:29.844721+0000","last_became_active":"2024-04-17T19:47:49.900707+0000","last_became_peered":"2024-04-17T19:47:49.900707+0000","last_unstale":"2024-04-17T19:48:29.844721+0000","last_undegraded":"2024-04-17T19:48:29.844721+0000","last_fullsized":"2024-04-17T19:48:29.844721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:58:02.897564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"24'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838864+0000","last_change":"2024-04-17T19:47:48.874721+0000","last_active":"2024-04-17T19:48:29.838864+0000","last_peered":"2024-04-17T19:48:29.838864+0000","last_clean":"2024-04-17T19:48:29.838864+0000","last_became_active":"2024-04-17T19:47:48.874364+0000","last_became_peered":"2024-04-17T19:47:48.874364+0000","last_unstale":"2024-04-17T19:48:29.838864+0000","last_undegraded":"2024-04-17T19:48:29.838864+0000","last_fullsized":"2024-04-17T19:48:29.838864+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:47:08.856753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.842743+0000","last_change":"2024-04-17T19:47:53.942791+0000","last_active":"2024-04-17T19:48:29.842743+0000","last_peered":"2024-04-17T19:48:29.842743+0000","last_clean":"2024-04-17T19:48:29.842743+0000","last_became_active":"2024-04-17T19:47:53.941569+0000","last_became_peered":"2024-04-17T19:47:53.941569+0000","last_unstale":"2024-04-17T19:48:29.842743+0000","last_undegraded":"2024-04-17T19:48:29.842743+0000","last_fullsized":"2024-04-17T19:48:29.842743+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:23:35.452056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.848531+0000","last_change":"2024-04-17T19:47:49.895046+0000","last_active":"2024-04-17T19:48:29.848531+0000","last_peered":"2024-04-17T19:48:29.848531+0000","last_clean":"2024-04-17T19:48:29.848531+0000","last_became_active":"2024-04-17T19:47:49.894250+0000","last_became_peered":"2024-04-17T19:47:49.894250+0000","last_unstale":"2024-04-17T19:48:29.848531+0000","last_undegraded":"2024-04-17T19:48:29.848531+0000","last_fullsized":"2024-04-17T19:48:29.848531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:30:17.210135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839227+0000","last_change":"2024-04-17T19:47:48.861802+0000","last_active":"2024-04-17T19:48:29.839227+0000","last_peered":"2024-04-17T19:48:29.839227+0000","last_clean":"2024-04-17T19:48:29.839227+0000","last_became_active":"2024-04-17T19:47:48.860418+0000","last_became_peered":"2024-04-17T19:47:48.860418+0000","last_unstale":"2024-04-17T19:48:29.839227+0000","last_undegraded":"2024-04-17T19:48:29.839227+0000","last_fullsized":"2024-04-17T19:48:29.839227+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:54:41.949746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.163167+0000","last_change":"2024-04-17T19:47:53.917732+0000","last_active":"2024-04-17T19:48:30.163167+0000","last_peered":"2024-04-17T19:48:30.163167+0000","last_clean":"2024-04-17T19:48:30.163167+0000","last_became_active":"2024-04-17T19:47:53.917160+0000","last_became_peered":"2024-04-17T19:47:53.917160+0000","last_unstale":"2024-04-17T19:48:30.163167+0000","last_undegraded":"2024-04-17T19:48:30.163167+0000","last_fullsized":"2024-04-17T19:48:30.163167+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:19:20.394166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.847292+0000","last_change":"2024-04-17T19:47:49.893077+0000","last_active":"2024-04-17T19:48:29.847292+0000","last_peered":"2024-04-17T19:48:29.847292+0000","last_clean":"2024-04-17T19:48:29.847292+0000","last_became_active":"2024-04-17T19:47:49.892280+0000","last_became_peered":"2024-04-17T19:47:49.892280+0000","last_unstale":"2024-04-17T19:48:29.847292+0000","last_undegraded":"2024-04-17T19:48:29.847292+0000","last_fullsized":"2024-04-17T19:48:29.847292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:58:24.218821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.868436+0000","last_change":"2024-04-17T19:47:48.849027+0000","last_active":"2024-04-17T19:48:33.868436+0000","last_peered":"2024-04-17T19:48:33.868436+0000","last_clean":"2024-04-17T19:48:33.868436+0000","last_became_active":"2024-04-17T19:47:48.848485+0000","last_became_peered":"2024-04-17T19:47:48.848485+0000","last_unstale":"2024-04-17T19:48:33.868436+0000","last_undegraded":"2024-04-17T19:48:33.868436+0000","last_fullsized":"2024-04-17T19:48:33.868436+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:58:14.548891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839524+0000","last_change":"2024-04-17T19:47:53.946933+0000","last_active":"2024-04-17T19:48:29.839524+0000","last_peered":"2024-04-17T19:48:29.839524+0000","last_clean":"2024-04-17T19:48:29.839524+0000","last_became_active":"2024-04-17T19:47:53.943030+0000","last_became_peered":"2024-04-17T19:47:53.943030+0000","last_unstale":"2024-04-17T19:48:29.839524+0000","last_undegraded":"2024-04-17T19:48:29.839524+0000","last_fullsized":"2024-04-17T19:48:29.839524+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:37:03.489250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845055+0000","last_change":"2024-04-17T19:47:49.893137+0000","last_active":"2024-04-17T19:48:29.845055+0000","last_peered":"2024-04-17T19:48:29.845055+0000","last_clean":"2024-04-17T19:48:29.845055+0000","last_became_active":"2024-04-17T19:47:49.892379+0000","last_became_peered":"2024-04-17T19:47:49.892379+0000","last_unstale":"2024-04-17T19:48:29.845055+0000","last_undegraded":"2024-04-17T19:48:29.845055+0000","last_fullsized":"2024-04-17T19:48:29.845055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:55:08.562395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"24'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.853051+0000","last_change":"2024-04-17T19:47:48.861154+0000","last_active":"2024-04-17T19:48:29.853051+0000","last_peered":"2024-04-17T19:48:29.853051+0000","last_clean":"2024-04-17T19:48:29.853051+0000","last_became_active":"2024-04-17T19:47:48.859268+0000","last_became_peered":"2024-04-17T19:47:48.859268+0000","last_unstale":"2024-04-17T19:48:29.853051+0000","last_undegraded":"2024-04-17T19:48:29.853051+0000","last_fullsized":"2024-04-17T19:48:29.853051+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:04:19.867115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.849198+0000","last_change":"2024-04-17T19:47:53.950580+0000","last_active":"2024-04-17T19:48:29.849198+0000","last_peered":"2024-04-17T19:48:29.849198+0000","last_clean":"2024-04-17T19:48:29.849198+0000","last_became_active":"2024-04-17T19:47:53.943798+0000","last_became_peered":"2024-04-17T19:47:53.943798+0000","last_unstale":"2024-04-17T19:48:29.849198+0000","last_undegraded":"2024-04-17T19:48:29.849198+0000","last_fullsized":"2024-04-17T19:48:29.849198+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:52:14.763799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843479+0000","last_change":"2024-04-17T19:47:49.893774+0000","last_active":"2024-04-17T19:48:29.843479+0000","last_peered":"2024-04-17T19:48:29.843479+0000","last_clean":"2024-04-17T19:48:29.843479+0000","last_became_active":"2024-04-17T19:47:49.892780+0000","last_became_peered":"2024-04-17T19:47:49.892780+0000","last_unstale":"2024-04-17T19:48:29.843479+0000","last_undegraded":"2024-04-17T19:48:29.843479+0000","last_fullsized":"2024-04-17T19:48:29.843479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:50:47.658650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"26'17","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.837592+0000","last_change":"2024-04-17T19:47:48.852970+0000","last_active":"2024-04-17T19:48:29.837592+0000","last_peered":"2024-04-17T19:48:29.837592+0000","last_clean":"2024-04-17T19:48:29.837592+0000","last_became_active":"2024-04-17T19:47:48.852676+0000","last_became_peered":"2024-04-17T19:47:48.852676+0000","last_unstale":"2024-04-17T19:48:29.837592+0000","last_undegraded":"2024-04-17T19:48:29.837592+0000","last_fullsized":"2024-04-17T19:48:29.837592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:32:14.529596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162931+0000","last_change":"2024-04-17T19:47:53.915397+0000","last_active":"2024-04-17T19:48:30.162931+0000","last_peered":"2024-04-17T19:48:30.162931+0000","last_clean":"2024-04-17T19:48:30.162931+0000","last_became_active":"2024-04-17T19:47:53.914913+0000","last_became_peered":"2024-04-17T19:47:53.914913+0000","last_unstale":"2024-04-17T19:48:30.162931+0000","last_undegraded":"2024-04-17T19:48:30.162931+0000","last_fullsized":"2024-04-17T19:48:30.162931+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:02:01.175158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840177+0000","last_change":"2024-04-17T19:47:49.880484+0000","last_active":"2024-04-17T19:48:29.840177+0000","last_peered":"2024-04-17T19:48:29.840177+0000","last_clean":"2024-04-17T19:48:29.840177+0000","last_became_active":"2024-04-17T19:47:49.879625+0000","last_became_peered":"2024-04-17T19:47:49.879625+0000","last_unstale":"2024-04-17T19:48:29.840177+0000","last_undegraded":"2024-04-17T19:48:29.840177+0000","last_fullsized":"2024-04-17T19:48:29.840177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:50:00.611796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"24'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843272+0000","last_change":"2024-04-17T19:47:48.899160+0000","last_active":"2024-04-17T19:48:29.843272+0000","last_peered":"2024-04-17T19:48:29.843272+0000","last_clean":"2024-04-17T19:48:29.843272+0000","last_became_active":"2024-04-17T19:47:48.889831+0000","last_became_peered":"2024-04-17T19:47:48.889831+0000","last_unstale":"2024-04-17T19:48:29.843272+0000","last_undegraded":"2024-04-17T19:48:29.843272+0000","last_fullsized":"2024-04-17T19:48:29.843272+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:55:20.167510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.835052+0000","last_change":"2024-04-17T19:47:53.921093+0000","last_active":"2024-04-17T19:48:29.835052+0000","last_peered":"2024-04-17T19:48:29.835052+0000","last_clean":"2024-04-17T19:48:29.835052+0000","last_became_active":"2024-04-17T19:47:53.920479+0000","last_became_peered":"2024-04-17T19:47:53.920479+0000","last_unstale":"2024-04-17T19:48:29.835052+0000","last_undegraded":"2024-04-17T19:48:29.835052+0000","last_fullsized":"2024-04-17T19:48:29.835052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:08:09.569820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.844227+0000","last_change":"2024-04-17T19:47:49.895275+0000","last_active":"2024-04-17T19:48:29.844227+0000","last_peered":"2024-04-17T19:48:29.844227+0000","last_clean":"2024-04-17T19:48:29.844227+0000","last_became_active":"2024-04-17T19:47:49.894534+0000","last_became_peered":"2024-04-17T19:47:49.894534+0000","last_unstale":"2024-04-17T19:48:29.844227+0000","last_undegraded":"2024-04-17T19:48:29.844227+0000","last_fullsized":"2024-04-17T19:48:29.844227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:31:40.818807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.841327+0000","last_change":"2024-04-17T19:47:48.881715+0000","last_active":"2024-04-17T19:48:29.841327+0000","last_peered":"2024-04-17T19:48:29.841327+0000","last_clean":"2024-04-17T19:48:29.841327+0000","last_became_active":"2024-04-17T19:47:48.876203+0000","last_became_peered":"2024-04-17T19:47:48.876203+0000","last_unstale":"2024-04-17T19:48:29.841327+0000","last_undegraded":"2024-04-17T19:48:29.841327+0000","last_fullsized":"2024-04-17T19:48:29.841327+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:03:16.656528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834859+0000","last_change":"2024-04-17T19:47:53.905149+0000","last_active":"2024-04-17T19:48:29.834859+0000","last_peered":"2024-04-17T19:48:29.834859+0000","last_clean":"2024-04-17T19:48:29.834859+0000","last_became_active":"2024-04-17T19:47:53.904718+0000","last_became_peered":"2024-04-17T19:47:53.904718+0000","last_unstale":"2024-04-17T19:48:29.834859+0000","last_undegraded":"2024-04-17T19:48:29.834859+0000","last_fullsized":"2024-04-17T19:48:29.834859+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:09:31.908752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162277+0000","last_change":"2024-04-17T19:47:49.875819+0000","last_active":"2024-04-17T19:48:30.162277+0000","last_peered":"2024-04-17T19:48:30.162277+0000","last_clean":"2024-04-17T19:48:30.162277+0000","last_became_active":"2024-04-17T19:47:49.872075+0000","last_became_peered":"2024-04-17T19:47:49.872075+0000","last_unstale":"2024-04-17T19:48:30.162277+0000","last_undegraded":"2024-04-17T19:48:30.162277+0000","last_fullsized":"2024-04-17T19:48:30.162277+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:06:48.823135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"26'9","reported_seq":35,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.871452+0000","last_change":"2024-04-17T19:47:48.867633+0000","last_active":"2024-04-17T19:48:33.871452+0000","last_peered":"2024-04-17T19:48:33.871452+0000","last_clean":"2024-04-17T19:48:33.871452+0000","last_became_active":"2024-04-17T19:47:48.867150+0000","last_became_peered":"2024-04-17T19:47:48.867150+0000","last_unstale":"2024-04-17T19:48:33.871452+0000","last_undegraded":"2024-04-17T19:48:33.871452+0000","last_fullsized":"2024-04-17T19:48:33.871452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:33:34.275816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.847899+0000","last_change":"2024-04-17T19:47:53.925785+0000","last_active":"2024-04-17T19:48:29.847899+0000","last_peered":"2024-04-17T19:48:29.847899+0000","last_clean":"2024-04-17T19:48:29.847899+0000","last_became_active":"2024-04-17T19:47:53.924006+0000","last_became_peered":"2024-04-17T19:47:53.924006+0000","last_unstale":"2024-04-17T19:48:29.847899+0000","last_undegraded":"2024-04-17T19:48:29.847899+0000","last_fullsized":"2024-04-17T19:48:29.847899+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:46:31.965271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845415+0000","last_change":"2024-04-17T19:47:49.869829+0000","last_active":"2024-04-17T19:48:29.845415+0000","last_peered":"2024-04-17T19:48:29.845415+0000","last_clean":"2024-04-17T19:48:29.845415+0000","last_became_active":"2024-04-17T19:47:49.868933+0000","last_became_peered":"2024-04-17T19:47:49.868933+0000","last_unstale":"2024-04-17T19:48:29.845415+0000","last_undegraded":"2024-04-17T19:48:29.845415+0000","last_fullsized":"2024-04-17T19:48:29.845415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:36:08.866159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'15","reported_seq":41,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:43.269615+0000","last_change":"2024-04-17T19:47:48.887461+0000","last_active":"2024-04-17T19:48:43.269615+0000","last_peered":"2024-04-17T19:48:43.269615+0000","last_clean":"2024-04-17T19:48:43.269615+0000","last_became_active":"2024-04-17T19:47:48.886689+0000","last_became_peered":"2024-04-17T19:47:48.886689+0000","last_unstale":"2024-04-17T19:48:43.269615+0000","last_undegraded":"2024-04-17T19:48:43.269615+0000","last_fullsized":"2024-04-17T19:48:43.269615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:55:35.262995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4100,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840709+0000","last_change":"2024-04-17T19:47:53.941607+0000","last_active":"2024-04-17T19:48:29.840709+0000","last_peered":"2024-04-17T19:48:29.840709+0000","last_clean":"2024-04-17T19:48:29.840709+0000","last_became_active":"2024-04-17T19:47:53.941050+0000","last_became_peered":"2024-04-17T19:47:53.941050+0000","last_unstale":"2024-04-17T19:48:29.840709+0000","last_undegraded":"2024-04-17T19:48:29.840709+0000","last_fullsized":"2024-04-17T19:48:29.840709+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:26:08.964058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.851845+0000","last_change":"2024-04-17T19:47:49.879524+0000","last_active":"2024-04-17T19:48:29.851845+0000","last_peered":"2024-04-17T19:48:29.851845+0000","last_clean":"2024-04-17T19:48:29.851845+0000","last_became_active":"2024-04-17T19:47:49.877563+0000","last_became_peered":"2024-04-17T19:47:49.877563+0000","last_unstale":"2024-04-17T19:48:29.851845+0000","last_undegraded":"2024-04-17T19:48:29.851845+0000","last_fullsized":"2024-04-17T19:48:29.851845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:34:58.539940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"24'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839055+0000","last_change":"2024-04-17T19:47:48.870100+0000","last_active":"2024-04-17T19:48:29.839055+0000","last_peered":"2024-04-17T19:48:29.839055+0000","last_clean":"2024-04-17T19:48:29.839055+0000","last_became_active":"2024-04-17T19:47:48.869797+0000","last_became_peered":"2024-04-17T19:47:48.869797+0000","last_unstale":"2024-04-17T19:48:29.839055+0000","last_undegraded":"2024-04-17T19:48:29.839055+0000","last_fullsized":"2024-04-17T19:48:29.839055+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:51:21.463081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.850942+0000","last_change":"2024-04-17T19:47:53.944568+0000","last_active":"2024-04-17T19:48:29.850942+0000","last_peered":"2024-04-17T19:48:29.850942+0000","last_clean":"2024-04-17T19:48:29.850942+0000","last_became_active":"2024-04-17T19:47:53.943151+0000","last_became_peered":"2024-04-17T19:47:53.943151+0000","last_unstale":"2024-04-17T19:48:29.850942+0000","last_undegraded":"2024-04-17T19:48:29.850942+0000","last_fullsized":"2024-04-17T19:48:29.850942+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:19:00.150450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162170+0000","last_change":"2024-04-17T19:47:49.902401+0000","last_active":"2024-04-17T19:48:30.162170+0000","last_peered":"2024-04-17T19:48:30.162170+0000","last_clean":"2024-04-17T19:48:30.162170+0000","last_became_active":"2024-04-17T19:47:49.902166+0000","last_became_peered":"2024-04-17T19:47:49.902166+0000","last_unstale":"2024-04-17T19:48:30.162170+0000","last_undegraded":"2024-04-17T19:48:30.162170+0000","last_fullsized":"2024-04-17T19:48:30.162170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:54:13.129748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.876986+0000","last_change":"2024-04-17T19:47:48.900800+0000","last_active":"2024-04-17T19:48:33.876986+0000","last_peered":"2024-04-17T19:48:33.876986+0000","last_clean":"2024-04-17T19:48:33.876986+0000","last_became_active":"2024-04-17T19:47:48.900188+0000","last_became_peered":"2024-04-17T19:47:48.900188+0000","last_unstale":"2024-04-17T19:48:33.876986+0000","last_undegraded":"2024-04-17T19:48:33.876986+0000","last_fullsized":"2024-04-17T19:48:33.876986+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:05:04.305871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834112+0000","last_change":"2024-04-17T19:47:53.928721+0000","last_active":"2024-04-17T19:48:29.834112+0000","last_peered":"2024-04-17T19:48:29.834112+0000","last_clean":"2024-04-17T19:48:29.834112+0000","last_became_active":"2024-04-17T19:47:53.928311+0000","last_became_peered":"2024-04-17T19:47:53.928311+0000","last_unstale":"2024-04-17T19:48:29.834112+0000","last_undegraded":"2024-04-17T19:48:29.834112+0000","last_fullsized":"2024-04-17T19:48:29.834112+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:35:28.870803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.847789+0000","last_change":"2024-04-17T19:47:49.905421+0000","last_active":"2024-04-17T19:48:29.847789+0000","last_peered":"2024-04-17T19:48:29.847789+0000","last_clean":"2024-04-17T19:48:29.847789+0000","last_became_active":"2024-04-17T19:47:49.904646+0000","last_became_peered":"2024-04-17T19:47:49.904646+0000","last_unstale":"2024-04-17T19:48:29.847789+0000","last_undegraded":"2024-04-17T19:48:29.847789+0000","last_fullsized":"2024-04-17T19:48:29.847789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:02:23.936080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"24'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.846631+0000","last_change":"2024-04-17T19:47:48.895356+0000","last_active":"2024-04-17T19:48:29.846631+0000","last_peered":"2024-04-17T19:48:29.846631+0000","last_clean":"2024-04-17T19:48:29.846631+0000","last_became_active":"2024-04-17T19:47:48.889225+0000","last_became_peered":"2024-04-17T19:47:48.889225+0000","last_unstale":"2024-04-17T19:48:29.846631+0000","last_undegraded":"2024-04-17T19:48:29.846631+0000","last_fullsized":"2024-04-17T19:48:29.846631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:00:22.587917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.833996+0000","last_change":"2024-04-17T19:47:53.940206+0000","last_active":"2024-04-17T19:48:29.833996+0000","last_peered":"2024-04-17T19:48:29.833996+0000","last_clean":"2024-04-17T19:48:29.833996+0000","last_became_active":"2024-04-17T19:47:53.939843+0000","last_became_peered":"2024-04-17T19:47:53.939843+0000","last_unstale":"2024-04-17T19:48:29.833996+0000","last_undegraded":"2024-04-17T19:48:29.833996+0000","last_fullsized":"2024-04-17T19:48:29.833996+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:18:55.562573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162804+0000","last_change":"2024-04-17T19:47:53.920258+0000","last_active":"2024-04-17T19:48:30.162804+0000","last_peered":"2024-04-17T19:48:30.162804+0000","last_clean":"2024-04-17T19:48:30.162804+0000","last_became_active":"2024-04-17T19:47:53.918932+0000","last_became_peered":"2024-04-17T19:47:53.918932+0000","last_unstale":"2024-04-17T19:48:30.162804+0000","last_undegraded":"2024-04-17T19:48:30.162804+0000","last_fullsized":"2024-04-17T19:48:30.162804+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:45:55.456803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.842416+0000","last_change":"2024-04-17T19:48:29.842416+0000","last_active":"2024-04-17T19:48:29.842416+0000","last_peered":"2024-04-17T19:48:29.842416+0000","last_clean":"2024-04-17T19:48:29.842416+0000","last_became_active":"2024-04-17T19:48:29.841759+0000","last_became_peered":"2024-04-17T19:48:29.841759+0000","last_unstale":"2024-04-17T19:48:29.842416+0000","last_undegraded":"2024-04-17T19:48:29.842416+0000","last_fullsized":"2024-04-17T19:48:29.842416+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:57:17.664562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840971+0000","last_change":"2024-04-17T19:47:48.873355+0000","last_active":"2024-04-17T19:48:29.840971+0000","last_peered":"2024-04-17T19:48:29.840971+0000","last_clean":"2024-04-17T19:48:29.840971+0000","last_became_active":"2024-04-17T19:47:48.872427+0000","last_became_peered":"2024-04-17T19:47:48.872427+0000","last_unstale":"2024-04-17T19:48:29.840971+0000","last_undegraded":"2024-04-17T19:48:29.840971+0000","last_fullsized":"2024-04-17T19:48:29.840971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:32:41.792605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.846032+0000","last_change":"2024-04-17T19:47:53.952602+0000","last_active":"2024-04-17T19:48:29.846032+0000","last_peered":"2024-04-17T19:48:29.846032+0000","last_clean":"2024-04-17T19:48:29.846032+0000","last_became_active":"2024-04-17T19:47:53.951750+0000","last_became_peered":"2024-04-17T19:47:53.951750+0000","last_unstale":"2024-04-17T19:48:29.846032+0000","last_undegraded":"2024-04-17T19:48:29.846032+0000","last_fullsized":"2024-04-17T19:48:29.846032+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:13:08.485238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834245+0000","last_change":"2024-04-17T19:47:49.862071+0000","last_active":"2024-04-17T19:48:29.834245+0000","last_peered":"2024-04-17T19:48:29.834245+0000","last_clean":"2024-04-17T19:48:29.834245+0000","last_became_active":"2024-04-17T19:47:49.860809+0000","last_became_peered":"2024-04-17T19:47:49.860809+0000","last_unstale":"2024-04-17T19:48:29.834245+0000","last_undegraded":"2024-04-17T19:48:29.834245+0000","last_fullsized":"2024-04-17T19:48:29.834245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:42:34.451839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834368+0000","last_change":"2024-04-17T19:47:48.867855+0000","last_active":"2024-04-17T19:48:29.834368+0000","last_peered":"2024-04-17T19:48:29.834368+0000","last_clean":"2024-04-17T19:48:29.834368+0000","last_became_active":"2024-04-17T19:47:48.867532+0000","last_became_peered":"2024-04-17T19:47:48.867532+0000","last_unstale":"2024-04-17T19:48:29.834368+0000","last_undegraded":"2024-04-17T19:48:29.834368+0000","last_fullsized":"2024-04-17T19:48:29.834368+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:21:26.903245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"24'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843580+0000","last_change":"2024-04-17T19:47:48.894946+0000","last_active":"2024-04-17T19:48:29.843580+0000","last_peered":"2024-04-17T19:48:29.843580+0000","last_clean":"2024-04-17T19:48:29.843580+0000","last_became_active":"2024-04-17T19:47:48.893407+0000","last_became_peered":"2024-04-17T19:47:48.893407+0000","last_unstale":"2024-04-17T19:48:29.843580+0000","last_undegraded":"2024-04-17T19:48:29.843580+0000","last_fullsized":"2024-04-17T19:48:29.843580+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:09:56.284638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162336+0000","last_change":"2024-04-17T19:47:49.859985+0000","last_active":"2024-04-17T19:48:30.162336+0000","last_peered":"2024-04-17T19:48:30.162336+0000","last_clean":"2024-04-17T19:48:30.162336+0000","last_became_active":"2024-04-17T19:47:49.859635+0000","last_became_peered":"2024-04-17T19:47:49.859635+0000","last_unstale":"2024-04-17T19:48:30.162336+0000","last_undegraded":"2024-04-17T19:48:30.162336+0000","last_fullsized":"2024-04-17T19:48:30.162336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:04:14.834218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838003+0000","last_change":"2024-04-17T19:47:53.925444+0000","last_active":"2024-04-17T19:48:29.838003+0000","last_peered":"2024-04-17T19:48:29.838003+0000","last_clean":"2024-04-17T19:48:29.838003+0000","last_became_active":"2024-04-17T19:47:53.924805+0000","last_became_peered":"2024-04-17T19:47:53.924805+0000","last_unstale":"2024-04-17T19:48:29.838003+0000","last_undegraded":"2024-04-17T19:48:29.838003+0000","last_fullsized":"2024-04-17T19:48:29.838003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:19:24.022368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"24'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834567+0000","last_change":"2024-04-17T19:47:48.867176+0000","last_active":"2024-04-17T19:48:29.834567+0000","last_peered":"2024-04-17T19:48:29.834567+0000","last_clean":"2024-04-17T19:48:29.834567+0000","last_became_active":"2024-04-17T19:47:48.866817+0000","last_became_peered":"2024-04-17T19:47:48.866817+0000","last_unstale":"2024-04-17T19:48:29.834567+0000","last_undegraded":"2024-04-17T19:48:29.834567+0000","last_fullsized":"2024-04-17T19:48:29.834567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:44:30.842583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.850861+0000","last_change":"2024-04-17T19:47:49.858924+0000","last_active":"2024-04-17T19:48:29.850861+0000","last_peered":"2024-04-17T19:48:29.850861+0000","last_clean":"2024-04-17T19:48:29.850861+0000","last_became_active":"2024-04-17T19:47:49.857820+0000","last_became_peered":"2024-04-17T19:47:49.857820+0000","last_unstale":"2024-04-17T19:48:29.850861+0000","last_undegraded":"2024-04-17T19:48:29.850861+0000","last_fullsized":"2024-04-17T19:48:29.850861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:09:28.706054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.835377+0000","last_change":"2024-04-17T19:47:53.933337+0000","last_active":"2024-04-17T19:48:29.835377+0000","last_peered":"2024-04-17T19:48:29.835377+0000","last_clean":"2024-04-17T19:48:29.835377+0000","last_became_active":"2024-04-17T19:47:53.931623+0000","last_became_peered":"2024-04-17T19:47:53.931623+0000","last_unstale":"2024-04-17T19:48:29.835377+0000","last_undegraded":"2024-04-17T19:48:29.835377+0000","last_fullsized":"2024-04-17T19:48:29.835377+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:57:37.600957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839672+0000","last_change":"2024-04-17T19:47:49.881093+0000","last_active":"2024-04-17T19:48:29.839672+0000","last_peered":"2024-04-17T19:48:29.839672+0000","last_clean":"2024-04-17T19:48:29.839672+0000","last_became_active":"2024-04-17T19:47:49.880210+0000","last_became_peered":"2024-04-17T19:47:49.880210+0000","last_unstale":"2024-04-17T19:48:29.839672+0000","last_undegraded":"2024-04-17T19:48:29.839672+0000","last_fullsized":"2024-04-17T19:48:29.839672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:12:15.522651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.837813+0000","last_change":"2024-04-17T19:47:49.886936+0000","last_active":"2024-04-17T19:48:29.837813+0000","last_peered":"2024-04-17T19:48:29.837813+0000","last_clean":"2024-04-17T19:48:29.837813+0000","last_became_active":"2024-04-17T19:47:49.886580+0000","last_became_peered":"2024-04-17T19:47:49.886580+0000","last_unstale":"2024-04-17T19:48:29.837813+0000","last_undegraded":"2024-04-17T19:48:29.837813+0000","last_fullsized":"2024-04-17T19:48:29.837813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:00:44.599463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840923+0000","last_change":"2024-04-17T19:47:49.905445+0000","last_active":"2024-04-17T19:48:29.840923+0000","last_peered":"2024-04-17T19:48:29.840923+0000","last_clean":"2024-04-17T19:48:29.840923+0000","last_became_active":"2024-04-17T19:47:49.896510+0000","last_became_peered":"2024-04-17T19:47:49.896510+0000","last_unstale":"2024-04-17T19:48:29.840923+0000","last_undegraded":"2024-04-17T19:48:29.840923+0000","last_fullsized":"2024-04-17T19:48:29.840923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:23:27.401861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162472+0000","last_change":"2024-04-17T19:47:49.850490+0000","last_active":"2024-04-17T19:48:30.162472+0000","last_peered":"2024-04-17T19:48:30.162472+0000","last_clean":"2024-04-17T19:48:30.162472+0000","last_became_active":"2024-04-17T19:47:49.849344+0000","last_became_peered":"2024-04-17T19:47:49.849344+0000","last_unstale":"2024-04-17T19:48:30.162472+0000","last_undegraded":"2024-04-17T19:48:30.162472+0000","last_fullsized":"2024-04-17T19:48:30.162472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:44:32.231058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.835044+0000","last_change":"2024-04-17T19:47:49.857737+0000","last_active":"2024-04-17T19:48:29.835044+0000","last_peered":"2024-04-17T19:48:29.835044+0000","last_clean":"2024-04-17T19:48:29.835044+0000","last_became_active":"2024-04-17T19:47:49.857476+0000","last_became_peered":"2024-04-17T19:47:49.857476+0000","last_unstale":"2024-04-17T19:48:29.835044+0000","last_undegraded":"2024-04-17T19:48:29.835044+0000","last_fullsized":"2024-04-17T19:48:29.835044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:50:13.070628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838876+0000","last_change":"2024-04-17T19:47:49.864985+0000","last_active":"2024-04-17T19:48:29.838876+0000","last_peered":"2024-04-17T19:48:29.838876+0000","last_clean":"2024-04-17T19:48:29.838876+0000","last_became_active":"2024-04-17T19:47:49.864329+0000","last_became_peered":"2024-04-17T19:47:49.864329+0000","last_unstale":"2024-04-17T19:48:29.838876+0000","last_undegraded":"2024-04-17T19:48:29.838876+0000","last_fullsized":"2024-04-17T19:48:29.838876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:43:57.448269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845430+0000","last_change":"2024-04-17T19:47:49.881785+0000","last_active":"2024-04-17T19:48:29.845430+0000","last_peered":"2024-04-17T19:48:29.845430+0000","last_clean":"2024-04-17T19:48:29.845430+0000","last_became_active":"2024-04-17T19:47:49.880939+0000","last_became_peered":"2024-04-17T19:47:49.880939+0000","last_unstale":"2024-04-17T19:48:29.845430+0000","last_undegraded":"2024-04-17T19:48:29.845430+0000","last_fullsized":"2024-04-17T19:48:29.845430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:49:07.194643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.835152+0000","last_change":"2024-04-17T19:47:49.884944+0000","last_active":"2024-04-17T19:48:29.835152+0000","last_peered":"2024-04-17T19:48:29.835152+0000","last_clean":"2024-04-17T19:48:29.835152+0000","last_became_active":"2024-04-17T19:47:49.884559+0000","last_became_peered":"2024-04-17T19:47:49.884559+0000","last_unstale":"2024-04-17T19:48:29.835152+0000","last_undegraded":"2024-04-17T19:48:29.835152+0000","last_fullsized":"2024-04-17T19:48:29.835152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:00.485472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840468+0000","last_change":"2024-04-17T19:47:49.891230+0000","last_active":"2024-04-17T19:48:29.840468+0000","last_peered":"2024-04-17T19:48:29.840468+0000","last_clean":"2024-04-17T19:48:29.840468+0000","last_became_active":"2024-04-17T19:47:49.886001+0000","last_became_peered":"2024-04-17T19:47:49.886001+0000","last_unstale":"2024-04-17T19:48:29.840468+0000","last_undegraded":"2024-04-17T19:48:29.840468+0000","last_fullsized":"2024-04-17T19:48:29.840468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:00.485472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834129+0000","last_change":"2024-04-17T19:47:49.859382+0000","last_active":"2024-04-17T19:48:29.834129+0000","last_peered":"2024-04-17T19:48:29.834129+0000","last_clean":"2024-04-17T19:48:29.834129+0000","last_became_active":"2024-04-17T19:47:49.859170+0000","last_became_peered":"2024-04-17T19:47:49.859170+0000","last_unstale":"2024-04-17T19:48:29.834129+0000","last_undegraded":"2024-04-17T19:48:29.834129+0000","last_fullsized":"2024-04-17T19:48:29.834129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:58:51.111053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.854842+0000","last_change":"2024-04-17T19:48:29.853997+0000","last_active":"2024-04-17T19:48:29.854842+0000","last_peered":"2024-04-17T19:48:29.854842+0000","last_clean":"2024-04-17T19:48:29.854842+0000","last_became_active":"2024-04-17T19:48:29.853682+0000","last_became_peered":"2024-04-17T19:48:29.853682+0000","last_unstale":"2024-04-17T19:48:29.854842+0000","last_undegraded":"2024-04-17T19:48:29.854842+0000","last_fullsized":"2024-04-17T19:48:29.854842+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:29:20.526070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162602+0000","last_change":"2024-04-17T19:47:49.876406+0000","last_active":"2024-04-17T19:48:30.162602+0000","last_peered":"2024-04-17T19:48:30.162602+0000","last_clean":"2024-04-17T19:48:30.162602+0000","last_became_active":"2024-04-17T19:47:49.876132+0000","last_became_peered":"2024-04-17T19:47:49.876132+0000","last_unstale":"2024-04-17T19:48:30.162602+0000","last_undegraded":"2024-04-17T19:48:30.162602+0000","last_fullsized":"2024-04-17T19:48:30.162602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:51:20.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.846554+0000","last_change":"2024-04-17T19:47:49.896244+0000","last_active":"2024-04-17T19:48:29.846554+0000","last_peered":"2024-04-17T19:48:29.846554+0000","last_clean":"2024-04-17T19:48:29.846554+0000","last_became_active":"2024-04-17T19:47:49.895380+0000","last_became_peered":"2024-04-17T19:47:49.895380+0000","last_unstale":"2024-04-17T19:48:29.846554+0000","last_undegraded":"2024-04-17T19:48:29.846554+0000","last_fullsized":"2024-04-17T19:48:29.846554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:04:43.091307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845627+0000","last_change":"2024-04-17T19:47:49.870551+0000","last_active":"2024-04-17T19:48:29.845627+0000","last_peered":"2024-04-17T19:48:29.845627+0000","last_clean":"2024-04-17T19:48:29.845627+0000","last_became_active":"2024-04-17T19:47:49.869315+0000","last_became_peered":"2024-04-17T19:47:49.869315+0000","last_unstale":"2024-04-17T19:48:29.845627+0000","last_undegraded":"2024-04-17T19:48:29.845627+0000","last_fullsized":"2024-04-17T19:48:29.845627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:06.341123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839471+0000","last_change":"2024-04-17T19:47:49.880060+0000","last_active":"2024-04-17T19:48:29.839471+0000","last_peered":"2024-04-17T19:48:29.839471+0000","last_clean":"2024-04-17T19:48:29.839471+0000","last_became_active":"2024-04-17T19:47:49.879711+0000","last_became_peered":"2024-04-17T19:47:49.879711+0000","last_unstale":"2024-04-17T19:48:29.839471+0000","last_undegraded":"2024-04-17T19:48:29.839471+0000","last_fullsized":"2024-04-17T19:48:29.839471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:41:25.628870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.849283+0000","last_change":"2024-04-17T19:47:49.895452+0000","last_active":"2024-04-17T19:48:29.849283+0000","last_peered":"2024-04-17T19:48:29.849283+0000","last_clean":"2024-04-17T19:48:29.849283+0000","last_became_active":"2024-04-17T19:47:49.894684+0000","last_became_peered":"2024-04-17T19:47:49.894684+0000","last_unstale":"2024-04-17T19:48:29.849283+0000","last_undegraded":"2024-04-17T19:48:29.849283+0000","last_fullsized":"2024-04-17T19:48:29.849283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:15:55.224238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840156+0000","last_change":"2024-04-17T19:47:49.886902+0000","last_active":"2024-04-17T19:48:29.840156+0000","last_peered":"2024-04-17T19:48:29.840156+0000","last_clean":"2024-04-17T19:48:29.840156+0000","last_became_active":"2024-04-17T19:47:49.885451+0000","last_became_peered":"2024-04-17T19:47:49.885451+0000","last_unstale":"2024-04-17T19:48:29.840156+0000","last_undegraded":"2024-04-17T19:48:29.840156+0000","last_fullsized":"2024-04-17T19:48:29.840156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:12:09.196610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838401+0000","last_change":"2024-04-17T19:47:49.858908+0000","last_active":"2024-04-17T19:48:29.838401+0000","last_peered":"2024-04-17T19:48:29.838401+0000","last_clean":"2024-04-17T19:48:29.838401+0000","last_became_active":"2024-04-17T19:47:49.858457+0000","last_became_peered":"2024-04-17T19:47:49.858457+0000","last_unstale":"2024-04-17T19:48:29.838401+0000","last_undegraded":"2024-04-17T19:48:29.838401+0000","last_fullsized":"2024-04-17T19:48:29.838401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:09.467859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162134+0000","last_change":"2024-04-17T19:47:49.856371+0000","last_active":"2024-04-17T19:48:30.162134+0000","last_peered":"2024-04-17T19:48:30.162134+0000","last_clean":"2024-04-17T19:48:30.162134+0000","last_became_active":"2024-04-17T19:47:49.856062+0000","last_became_peered":"2024-04-17T19:47:49.856062+0000","last_unstale":"2024-04-17T19:48:30.162134+0000","last_undegraded":"2024-04-17T19:48:30.162134+0000","last_fullsized":"2024-04-17T19:48:30.162134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:39:54.502619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834338+0000","last_change":"2024-04-17T19:47:49.873350+0000","last_active":"2024-04-17T19:48:29.834338+0000","last_peered":"2024-04-17T19:48:29.834338+0000","last_clean":"2024-04-17T19:48:29.834338+0000","last_became_active":"2024-04-17T19:47:49.873019+0000","last_became_peered":"2024-04-17T19:47:49.873019+0000","last_unstale":"2024-04-17T19:48:29.834338+0000","last_undegraded":"2024-04-17T19:48:29.834338+0000","last_fullsized":"2024-04-17T19:48:29.834338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:06:58.641727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.851454+0000","last_change":"2024-04-17T19:47:49.891551+0000","last_active":"2024-04-17T19:48:29.851454+0000","last_peered":"2024-04-17T19:48:29.851454+0000","last_clean":"2024-04-17T19:48:29.851454+0000","last_became_active":"2024-04-17T19:47:49.890132+0000","last_became_peered":"2024-04-17T19:47:49.890132+0000","last_unstale":"2024-04-17T19:48:29.851454+0000","last_undegraded":"2024-04-17T19:48:29.851454+0000","last_fullsized":"2024-04-17T19:48:29.851454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:26:22.971489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845078+0000","last_change":"2024-04-17T19:47:49.903854+0000","last_active":"2024-04-17T19:48:29.845078+0000","last_peered":"2024-04-17T19:48:29.845078+0000","last_clean":"2024-04-17T19:48:29.845078+0000","last_became_active":"2024-04-17T19:47:49.903178+0000","last_became_peered":"2024-04-17T19:47:49.903178+0000","last_unstale":"2024-04-17T19:48:29.845078+0000","last_undegraded":"2024-04-17T19:48:29.845078+0000","last_fullsized":"2024-04-17T19:48:29.845078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:24:52.699532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845192+0000","last_change":"2024-04-17T19:47:49.900848+0000","last_active":"2024-04-17T19:48:29.845192+0000","last_peered":"2024-04-17T19:48:29.845192+0000","last_clean":"2024-04-17T19:48:29.845192+0000","last_became_active":"2024-04-17T19:47:49.900052+0000","last_became_peered":"2024-04-17T19:47:49.900052+0000","last_unstale":"2024-04-17T19:48:29.845192+0000","last_undegraded":"2024-04-17T19:48:29.845192+0000","last_fullsized":"2024-04-17T19:48:29.845192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:28:41.115173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.846967+0000","last_change":"2024-04-17T19:47:49.906531+0000","last_active":"2024-04-17T19:48:29.846967+0000","last_peered":"2024-04-17T19:48:29.846967+0000","last_clean":"2024-04-17T19:48:29.846967+0000","last_became_active":"2024-04-17T19:47:49.905217+0000","last_became_peered":"2024-04-17T19:47:49.905217+0000","last_unstale":"2024-04-17T19:48:29.846967+0000","last_undegraded":"2024-04-17T19:48:29.846967+0000","last_fullsized":"2024-04-17T19:48:29.846967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:22.078473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.833993+0000","last_change":"2024-04-17T19:47:49.864577+0000","last_active":"2024-04-17T19:48:29.833993+0000","last_peered":"2024-04-17T19:48:29.833993+0000","last_clean":"2024-04-17T19:48:29.833993+0000","last_became_active":"2024-04-17T19:47:49.864251+0000","last_became_peered":"2024-04-17T19:47:49.864251+0000","last_unstale":"2024-04-17T19:48:29.833993+0000","last_undegraded":"2024-04-17T19:48:29.833993+0000","last_fullsized":"2024-04-17T19:48:29.833993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:32:10.329543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843153+0000","last_change":"2024-04-17T19:47:49.869245+0000","last_active":"2024-04-17T19:48:29.843153+0000","last_peered":"2024-04-17T19:48:29.843153+0000","last_clean":"2024-04-17T19:48:29.843153+0000","last_became_active":"2024-04-17T19:47:49.868156+0000","last_became_peered":"2024-04-17T19:47:49.868156+0000","last_unstale":"2024-04-17T19:48:29.843153+0000","last_undegraded":"2024-04-17T19:48:29.843153+0000","last_fullsized":"2024-04-17T19:48:29.843153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:58:39.814933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840460+0000","last_change":"2024-04-17T19:47:49.907573+0000","last_active":"2024-04-17T19:48:29.840460+0000","last_peered":"2024-04-17T19:48:29.840460+0000","last_clean":"2024-04-17T19:48:29.840460+0000","last_became_active":"2024-04-17T19:47:49.897308+0000","last_became_peered":"2024-04-17T19:47:49.897308+0000","last_unstale":"2024-04-17T19:48:29.840460+0000","last_undegraded":"2024-04-17T19:48:29.840460+0000","last_fullsized":"2024-04-17T19:48:29.840460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:02:03.101317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843871+0000","last_change":"2024-04-17T19:47:49.876052+0000","last_active":"2024-04-17T19:48:29.843871+0000","last_peered":"2024-04-17T19:48:29.843871+0000","last_clean":"2024-04-17T19:48:29.843871+0000","last_became_active":"2024-04-17T19:47:49.875008+0000","last_became_peered":"2024-04-17T19:47:49.875008+0000","last_unstale":"2024-04-17T19:48:29.843871+0000","last_undegraded":"2024-04-17T19:48:29.843871+0000","last_fullsized":"2024-04-17T19:48:29.843871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:36:14.486150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.847790+0000","last_change":"2024-04-17T19:47:49.906840+0000","last_active":"2024-04-17T19:48:29.847790+0000","last_peered":"2024-04-17T19:48:29.847790+0000","last_clean":"2024-04-17T19:48:29.847790+0000","last_became_active":"2024-04-17T19:47:49.905703+0000","last_became_peered":"2024-04-17T19:47:49.905703+0000","last_unstale":"2024-04-17T19:48:29.847790+0000","last_undegraded":"2024-04-17T19:48:29.847790+0000","last_fullsized":"2024-04-17T19:48:29.847790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:28:26.686472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162905+0000","last_change":"2024-04-17T19:47:49.869425+0000","last_active":"2024-04-17T19:48:30.162905+0000","last_peered":"2024-04-17T19:48:30.162905+0000","last_clean":"2024-04-17T19:48:30.162905+0000","last_became_active":"2024-04-17T19:47:49.868758+0000","last_became_peered":"2024-04-17T19:47:49.868758+0000","last_unstale":"2024-04-17T19:48:30.162905+0000","last_undegraded":"2024-04-17T19:48:30.162905+0000","last_fullsized":"2024-04-17T19:48:30.162905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:57:27.214150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839586+0000","last_change":"2024-04-17T19:47:49.856309+0000","last_active":"2024-04-17T19:48:29.839586+0000","last_peered":"2024-04-17T19:48:29.839586+0000","last_clean":"2024-04-17T19:48:29.839586+0000","last_became_active":"2024-04-17T19:47:49.856026+0000","last_became_peered":"2024-04-17T19:47:49.856026+0000","last_unstale":"2024-04-17T19:48:29.839586+0000","last_undegraded":"2024-04-17T19:48:29.839586+0000","last_fullsized":"2024-04-17T19:48:29.839586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:51:57.963637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845128+0000","last_change":"2024-04-17T19:47:49.890343+0000","last_active":"2024-04-17T19:48:29.845128+0000","last_peered":"2024-04-17T19:48:29.845128+0000","last_clean":"2024-04-17T19:48:29.845128+0000","last_became_active":"2024-04-17T19:47:49.880566+0000","last_became_peered":"2024-04-17T19:47:49.880566+0000","last_unstale":"2024-04-17T19:48:29.845128+0000","last_undegraded":"2024-04-17T19:48:29.845128+0000","last_fullsized":"2024-04-17T19:48:29.845128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:19:29.359024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10522,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":210,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":21044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":184,"ondisk_log_size":184,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":909312,"data_stored":706837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":307200,"data_stored":119259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":327680,"data_stored":118301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":303104,"data_stored":115699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31480,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340360,"statfs":{"total":96636764160,"available":96604528640,"internally_reserved":0,"allocated":311296,"data_stored":119293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31480,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340360,"statfs":{"total":96636764160,"available":96604528640,"internally_reserved":0,"allocated":311296,"data_stored":115831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31476,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340364,"statfs":{"total":96636764160,"available":96604532736,"internally_reserved":0,"allocated":307200,"data_stored":115901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":901120,"data_stored":705611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T19:48:48.308 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T19:48:48.324 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:48:48.351 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:48:48.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.508+0000 7fbb5bc00640 1 Processor -- start 2024-04-17T19:48:48.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.508+0000 7fbb5bc00640 1 -- start start 2024-04-17T19:48:48.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.512+0000 7fbb5bc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb540c6f70 0x7fbb540c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.512+0000 7fbb5bc00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbb5406c660 0x7fbb5406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.512+0000 7fbb5bc00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb5415a0a0 con 0x7fbb54159c90 2024-04-17T19:48:48.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.512+0000 7fbb5bc00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fbb5415abe0 con 0x7fbb5406c660 2024-04-17T19:48:48.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.512+0000 7fbb5bc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbb5415a340 con 0x7fbb540c6f70 2024-04-17T19:48:48.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.512+0000 7fbb59c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbb5406c660 0x7fbb5406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.512+0000 7fbb59c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbb5406c660 0x7fbb5406ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33544/0 (socket says 172.21.15.29:33544) 2024-04-17T19:48:48.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.512+0000 7fbb59c00640 1 -- 172.21.15.29:0/298259179 learned_addr learned my addr 172.21.15.29:0/298259179 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:48.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.512+0000 7fbb4be00640 1 --2- 172.21.15.29:0/298259179 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb540c6f70 0x7fbb540c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.512+0000 7fbb59c00640 1 -- 172.21.15.29:0/298259179 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb540c6f70 msgr2=0x7fbb540c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:48.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.512+0000 7fbb59c00640 1 --2- 172.21.15.29:0/298259179 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb540c6f70 0x7fbb540c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.512+0000 7fbb59c00640 1 -- 172.21.15.29:0/298259179 >> v1:172.21.15.37:6789/0 conn(0x7fbb54159c90 legacy=0x7fbb540ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:48.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.512+0000 7fbb59c00640 1 -- 172.21.15.29:0/298259179 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5415a620 con 0x7fbb5406c660 2024-04-17T19:48:48.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.516+0000 7fbb59c00640 1 --2- 172.21.15.29:0/298259179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb5406c660 0x7fbb5406ca70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c009e50 tx=0x7fbb4c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45acd74d962432f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:48.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.516+0000 7fbb53e00640 1 -- 172.21.15.29:0/298259179 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c0a5700 con 0x7fbb5406c660 2024-04-17T19:48:48.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.516+0000 7fbb53e00640 1 -- 172.21.15.29:0/298259179 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb4c0a58a0 con 0x7fbb5406c660 2024-04-17T19:48:48.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.516+0000 7fbb53e00640 1 -- 172.21.15.29:0/298259179 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c0a5c70 con 0x7fbb5406c660 2024-04-17T19:48:48.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.516+0000 7fbb5bc00640 1 -- 172.21.15.29:0/298259179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb5406c660 msgr2=0x7fbb5406ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:48.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.516+0000 7fbb5bc00640 1 --2- 172.21.15.29:0/298259179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb5406c660 0x7fbb5406ca70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c009e50 tx=0x7fbb4c01ca30 comp rx=0 tx=0).stop 2024-04-17T19:48:48.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.516+0000 7fbb5bc00640 1 -- 172.21.15.29:0/298259179 shutdown_connections 2024-04-17T19:48:48.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.516+0000 7fbb5bc00640 1 --2- 172.21.15.29:0/298259179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb5406c660 0x7fbb5406ca70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.516+0000 7fbb5bc00640 1 --2- 172.21.15.29:0/298259179 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb540c6f70 0x7fbb540c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.516+0000 7fbb5bc00640 1 -- 172.21.15.29:0/298259179 >> 172.21.15.29:0/298259179 conn(0x7fbb5406fde0 msgr2=0x7fbb540701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:48.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.516+0000 7fbb5bc00640 1 -- 172.21.15.29:0/298259179 shutdown_connections 2024-04-17T19:48:48.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.516+0000 7fbb5bc00640 1 -- 172.21.15.29:0/298259179 wait complete. 2024-04-17T19:48:48.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.516+0000 7fbb5bc00640 1 Processor -- start 2024-04-17T19:48:48.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.516+0000 7fbb5bc00640 1 -- start start 2024-04-17T19:48:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb5bc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb5406c660 0x7fbb54145840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb5bc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb540c6f70 0x7fbb54143a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb5bc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb54159c90 0x7fbb541f85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb5bc00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbb540ca9c0 con 0x7fbb540c6f70 2024-04-17T19:48:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb5bc00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5406cfb0 con 0x7fbb5406c660 2024-04-17T19:48:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb5bc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbb540cab40 con 0x7fbb54159c90 2024-04-17T19:48:48.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb4be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb540c6f70 0x7fbb54143a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb4be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb540c6f70 0x7fbb54143a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33546/0 (socket says 172.21.15.29:33546) 2024-04-17T19:48:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb4be00640 1 -- 172.21.15.29:0/3295765584 learned_addr learned my addr 172.21.15.29:0/3295765584 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb59c00640 1 --2- 172.21.15.29:0/3295765584 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb54159c90 0x7fbb541f85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb59200640 1 --2- 172.21.15.29:0/3295765584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb5406c660 0x7fbb54145840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb4be00640 1 -- 172.21.15.29:0/3295765584 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb54159c90 msgr2=0x7fbb541f85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb4be00640 1 --2- 172.21.15.29:0/3295765584 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb54159c90 0x7fbb541f85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb4be00640 1 -- 172.21.15.29:0/3295765584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb5406c660 msgr2=0x7fbb54145840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb4be00640 1 --2- 172.21.15.29:0/3295765584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb5406c660 0x7fbb54145840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb4be00640 1 -- 172.21.15.29:0/3295765584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5414ee60 con 0x7fbb540c6f70 2024-04-17T19:48:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb4be00640 1 --2- 172.21.15.29:0/3295765584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb540c6f70 0x7fbb54143a90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c00d800 tx=0x7fbb3c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb51600640 1 -- 172.21.15.29:0/3295765584 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c00b6c0 con 0x7fbb540c6f70 2024-04-17T19:48:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb51600640 1 -- 172.21.15.29:0/3295765584 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb3c019070 con 0x7fbb540c6f70 2024-04-17T19:48:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb51600640 1 -- 172.21.15.29:0/3295765584 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c00ba80 con 0x7fbb540c6f70 2024-04-17T19:48:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb5bc00640 1 -- 172.21.15.29:0/3295765584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb54145d80 con 0x7fbb540c6f70 2024-04-17T19:48:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb5bc00640 1 -- 172.21.15.29:0/3295765584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb5414eb80 con 0x7fbb540c6f70 2024-04-17T19:48:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.520+0000 7fbb5bc00640 1 -- 172.21.15.29:0/3295765584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb540c66b0 con 0x7fbb540c6f70 2024-04-17T19:48:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.524+0000 7fbb51600640 1 -- 172.21.15.29:0/3295765584 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbb3c00bc20 con 0x7fbb540c6f70 2024-04-17T19:48:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.524+0000 7fbb51600640 1 --2- 172.21.15.29:0/3295765584 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbb30068380 0x7fbb3006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.524+0000 7fbb51600640 1 -- 172.21.15.29:0/3295765584 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fbb3c087210 con 0x7fbb540c6f70 2024-04-17T19:48:48.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.524+0000 7fbb59200640 1 --2- 172.21.15.29:0/3295765584 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbb30068380 0x7fbb3006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.524+0000 7fbb59200640 1 --2- 172.21.15.29:0/3295765584 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbb30068380 0x7fbb3006a890 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbb44000f70 tx=0x7fbb44009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:48.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.528+0000 7fbb51600640 1 -- 172.21.15.29:0/3295765584 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbb3c057330 con 0x7fbb540c6f70 2024-04-17T19:48:48.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.692+0000 7fbb5bc00640 1 -- 172.21.15.29:0/3295765584 --> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbb5414fcf0 con 0x7fbb30068380 2024-04-17T19:48:48.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.716+0000 7fbb51600640 1 -- 172.21.15.29:0/3295765584 <== mgr.4105 v2:172.21.15.37:6832/1588862286 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332649 (secure 0 0 0) 0x7fbb440050d0 con 0x7fbb30068380 2024-04-17T19:48:48.708 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:48.710 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-04-17T19:48:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.720+0000 7fbb5bc00640 1 -- 172.21.15.29:0/3295765584 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbb30068380 msgr2=0x7fbb3006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.720+0000 7fbb5bc00640 1 --2- 172.21.15.29:0/3295765584 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbb30068380 0x7fbb3006a890 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbb44000f70 tx=0x7fbb44009040 comp rx=0 tx=0).stop 2024-04-17T19:48:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.720+0000 7fbb5bc00640 1 -- 172.21.15.29:0/3295765584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb540c6f70 msgr2=0x7fbb54143a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.720+0000 7fbb5bc00640 1 --2- 172.21.15.29:0/3295765584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb540c6f70 0x7fbb54143a90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c00d800 tx=0x7fbb3c00dcc0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.720+0000 7fbb5bc00640 1 -- 172.21.15.29:0/3295765584 shutdown_connections 2024-04-17T19:48:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.720+0000 7fbb5bc00640 1 --2- 172.21.15.29:0/3295765584 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbb30068380 0x7fbb3006a890 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.720+0000 7fbb5bc00640 1 --2- 172.21.15.29:0/3295765584 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb54159c90 0x7fbb541f85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.720+0000 7fbb5bc00640 1 --2- 172.21.15.29:0/3295765584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb540c6f70 0x7fbb54143a90 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.720+0000 7fbb5bc00640 1 --2- 172.21.15.29:0/3295765584 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb5406c660 0x7fbb54145840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.720+0000 7fbb5bc00640 1 -- 172.21.15.29:0/3295765584 >> 172.21.15.29:0/3295765584 conn(0x7fbb5406fde0 msgr2=0x7fbb54072650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:48.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.720+0000 7fbb5bc00640 1 -- 172.21.15.29:0/3295765584 shutdown_connections 2024-04-17T19:48:48.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.720+0000 7fbb5bc00640 1 -- 172.21.15.29:0/3295765584 wait complete. 2024-04-17T19:48:48.730 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":56,"stamp":"2024-04-17T19:48:48.651809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600890,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":369,"num_write_kb":2484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":301,"ondisk_log_size":301,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240520,"kb_used_data":3592,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754734200,"statfs":{"total":773094113280,"available":772847820800,"internally_reserved":0,"allocated":3678208,"data_stored":2116732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":258,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005969"},"pg_stats":[{"pgid":"2.1f","version":"26'13","reported_seq":39,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.876148+0000","last_change":"2024-04-17T19:47:48.885235+0000","last_active":"2024-04-17T19:48:33.876148+0000","last_peered":"2024-04-17T19:48:33.876148+0000","last_clean":"2024-04-17T19:48:33.876148+0000","last_became_active":"2024-04-17T19:47:48.884844+0000","last_became_peered":"2024-04-17T19:47:48.884844+0000","last_unstale":"2024-04-17T19:48:33.876148+0000","last_undegraded":"2024-04-17T19:48:33.876148+0000","last_fullsized":"2024-04-17T19:48:33.876148+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:01:52.854763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834602+0000","last_change":"2024-04-17T19:47:49.872314+0000","last_active":"2024-04-17T19:48:29.834602+0000","last_peered":"2024-04-17T19:48:29.834602+0000","last_clean":"2024-04-17T19:48:29.834602+0000","last_became_active":"2024-04-17T19:47:49.871990+0000","last_became_peered":"2024-04-17T19:47:49.871990+0000","last_unstale":"2024-04-17T19:48:29.834602+0000","last_undegraded":"2024-04-17T19:48:29.834602+0000","last_fullsized":"2024-04-17T19:48:29.834602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:12:09.196610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.833967+0000","last_change":"2024-04-17T19:47:53.929840+0000","last_active":"2024-04-17T19:48:29.833967+0000","last_peered":"2024-04-17T19:48:29.833967+0000","last_clean":"2024-04-17T19:48:29.833967+0000","last_became_active":"2024-04-17T19:47:53.928272+0000","last_became_peered":"2024-04-17T19:47:53.928272+0000","last_unstale":"2024-04-17T19:48:29.833967+0000","last_undegraded":"2024-04-17T19:48:29.833967+0000","last_fullsized":"2024-04-17T19:48:29.833967+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:27:29.734327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"24'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.835623+0000","last_change":"2024-04-17T19:47:48.862315+0000","last_active":"2024-04-17T19:48:29.835623+0000","last_peered":"2024-04-17T19:48:29.835623+0000","last_clean":"2024-04-17T19:48:29.835623+0000","last_became_active":"2024-04-17T19:47:48.860408+0000","last_became_peered":"2024-04-17T19:47:48.860408+0000","last_unstale":"2024-04-17T19:48:29.835623+0000","last_undegraded":"2024-04-17T19:48:29.835623+0000","last_fullsized":"2024-04-17T19:48:29.835623+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:10:26.336063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162315+0000","last_change":"2024-04-17T19:47:49.902423+0000","last_active":"2024-04-17T19:48:30.162315+0000","last_peered":"2024-04-17T19:48:30.162315+0000","last_clean":"2024-04-17T19:48:30.162315+0000","last_became_active":"2024-04-17T19:47:49.902096+0000","last_became_peered":"2024-04-17T19:47:49.902096+0000","last_unstale":"2024-04-17T19:48:30.162315+0000","last_undegraded":"2024-04-17T19:48:30.162315+0000","last_fullsized":"2024-04-17T19:48:30.162315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:40.805247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.849677+0000","last_change":"2024-04-17T19:47:53.956851+0000","last_active":"2024-04-17T19:48:29.849677+0000","last_peered":"2024-04-17T19:48:29.849677+0000","last_clean":"2024-04-17T19:48:29.849677+0000","last_became_active":"2024-04-17T19:47:53.954197+0000","last_became_peered":"2024-04-17T19:47:53.954197+0000","last_unstale":"2024-04-17T19:48:29.849677+0000","last_undegraded":"2024-04-17T19:48:29.849677+0000","last_fullsized":"2024-04-17T19:48:29.849677+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:16:18.379831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"26'12","reported_seq":38,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.877776+0000","last_change":"2024-04-17T19:47:48.893969+0000","last_active":"2024-04-17T19:48:33.877776+0000","last_peered":"2024-04-17T19:48:33.877776+0000","last_clean":"2024-04-17T19:48:33.877776+0000","last_became_active":"2024-04-17T19:47:48.893229+0000","last_became_peered":"2024-04-17T19:47:48.893229+0000","last_unstale":"2024-04-17T19:48:33.877776+0000","last_undegraded":"2024-04-17T19:48:33.877776+0000","last_fullsized":"2024-04-17T19:48:33.877776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:38:42.472634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.848206+0000","last_change":"2024-04-17T19:47:49.897987+0000","last_active":"2024-04-17T19:48:29.848206+0000","last_peered":"2024-04-17T19:48:29.848206+0000","last_clean":"2024-04-17T19:48:29.848206+0000","last_became_active":"2024-04-17T19:47:49.897096+0000","last_became_peered":"2024-04-17T19:47:49.897096+0000","last_unstale":"2024-04-17T19:48:29.848206+0000","last_undegraded":"2024-04-17T19:48:29.848206+0000","last_fullsized":"2024-04-17T19:48:29.848206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:36:36.856454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.850365+0000","last_change":"2024-04-17T19:47:53.930784+0000","last_active":"2024-04-17T19:48:29.850365+0000","last_peered":"2024-04-17T19:48:29.850365+0000","last_clean":"2024-04-17T19:48:29.850365+0000","last_became_active":"2024-04-17T19:47:53.928281+0000","last_became_peered":"2024-04-17T19:47:53.928281+0000","last_unstale":"2024-04-17T19:48:29.850365+0000","last_undegraded":"2024-04-17T19:48:29.850365+0000","last_fullsized":"2024-04-17T19:48:29.850365+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:15:55.064622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.876428+0000","last_change":"2024-04-17T19:47:48.893184+0000","last_active":"2024-04-17T19:48:33.876428+0000","last_peered":"2024-04-17T19:48:33.876428+0000","last_clean":"2024-04-17T19:48:33.876428+0000","last_became_active":"2024-04-17T19:47:48.892375+0000","last_became_peered":"2024-04-17T19:47:48.892375+0000","last_unstale":"2024-04-17T19:48:33.876428+0000","last_undegraded":"2024-04-17T19:48:33.876428+0000","last_fullsized":"2024-04-17T19:48:33.876428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:56:01.211666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.848245+0000","last_change":"2024-04-17T19:47:49.896040+0000","last_active":"2024-04-17T19:48:29.848245+0000","last_peered":"2024-04-17T19:48:29.848245+0000","last_clean":"2024-04-17T19:48:29.848245+0000","last_became_active":"2024-04-17T19:47:49.894779+0000","last_became_peered":"2024-04-17T19:47:49.894779+0000","last_unstale":"2024-04-17T19:48:29.848245+0000","last_undegraded":"2024-04-17T19:48:29.848245+0000","last_fullsized":"2024-04-17T19:48:29.848245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:30:17.930203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.852223+0000","last_change":"2024-04-17T19:47:53.930402+0000","last_active":"2024-04-17T19:48:29.852223+0000","last_peered":"2024-04-17T19:48:29.852223+0000","last_clean":"2024-04-17T19:48:29.852223+0000","last_became_active":"2024-04-17T19:47:53.929166+0000","last_became_peered":"2024-04-17T19:47:53.929166+0000","last_unstale":"2024-04-17T19:48:29.852223+0000","last_undegraded":"2024-04-17T19:48:29.852223+0000","last_fullsized":"2024-04-17T19:48:29.852223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:34:08.282697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843243+0000","last_change":"2024-04-17T19:47:49.900142+0000","last_active":"2024-04-17T19:48:29.843243+0000","last_peered":"2024-04-17T19:48:29.843243+0000","last_clean":"2024-04-17T19:48:29.843243+0000","last_became_active":"2024-04-17T19:47:49.899365+0000","last_became_peered":"2024-04-17T19:47:49.899365+0000","last_unstale":"2024-04-17T19:48:29.843243+0000","last_undegraded":"2024-04-17T19:48:29.843243+0000","last_fullsized":"2024-04-17T19:48:29.843243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:34:49.616539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"24'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.844328+0000","last_change":"2024-04-17T19:47:48.891513+0000","last_active":"2024-04-17T19:48:29.844328+0000","last_peered":"2024-04-17T19:48:29.844328+0000","last_clean":"2024-04-17T19:48:29.844328+0000","last_became_active":"2024-04-17T19:47:48.886561+0000","last_became_peered":"2024-04-17T19:47:48.886561+0000","last_unstale":"2024-04-17T19:48:29.844328+0000","last_undegraded":"2024-04-17T19:48:29.844328+0000","last_fullsized":"2024-04-17T19:48:29.844328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:59:14.039700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.167531+0000","last_change":"2024-04-17T19:48:30.167531+0000","last_active":"2024-04-17T19:48:30.167531+0000","last_peered":"2024-04-17T19:48:30.167531+0000","last_clean":"2024-04-17T19:48:30.167531+0000","last_became_active":"2024-04-17T19:48:30.165421+0000","last_became_peered":"2024-04-17T19:48:30.165421+0000","last_unstale":"2024-04-17T19:48:30.167531+0000","last_undegraded":"2024-04-17T19:48:30.167531+0000","last_fullsized":"2024-04-17T19:48:30.167531+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:52:16.429658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,5],"acting":[4,0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834455+0000","last_change":"2024-04-17T19:47:49.852183+0000","last_active":"2024-04-17T19:48:29.834455+0000","last_peered":"2024-04-17T19:48:29.834455+0000","last_clean":"2024-04-17T19:48:29.834455+0000","last_became_active":"2024-04-17T19:47:49.851765+0000","last_became_peered":"2024-04-17T19:47:49.851765+0000","last_unstale":"2024-04-17T19:48:29.834455+0000","last_undegraded":"2024-04-17T19:48:29.834455+0000","last_fullsized":"2024-04-17T19:48:29.834455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:02:03.101317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.841041+0000","last_change":"2024-04-17T19:47:48.862865+0000","last_active":"2024-04-17T19:48:29.841041+0000","last_peered":"2024-04-17T19:48:29.841041+0000","last_clean":"2024-04-17T19:48:29.841041+0000","last_became_active":"2024-04-17T19:47:48.862524+0000","last_became_peered":"2024-04-17T19:47:48.862524+0000","last_unstale":"2024-04-17T19:48:29.841041+0000","last_undegraded":"2024-04-17T19:48:29.841041+0000","last_fullsized":"2024-04-17T19:48:29.841041+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:01:21.080029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.850650+0000","last_change":"2024-04-17T19:47:53.943674+0000","last_active":"2024-04-17T19:48:29.850650+0000","last_peered":"2024-04-17T19:48:29.850650+0000","last_clean":"2024-04-17T19:48:29.850650+0000","last_became_active":"2024-04-17T19:47:53.942549+0000","last_became_peered":"2024-04-17T19:47:53.942549+0000","last_unstale":"2024-04-17T19:48:29.850650+0000","last_undegraded":"2024-04-17T19:48:29.850650+0000","last_fullsized":"2024-04-17T19:48:29.850650+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:00:42.300738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834364+0000","last_change":"2024-04-17T19:47:49.862312+0000","last_active":"2024-04-17T19:48:29.834364+0000","last_peered":"2024-04-17T19:48:29.834364+0000","last_clean":"2024-04-17T19:48:29.834364+0000","last_became_active":"2024-04-17T19:47:49.860664+0000","last_became_peered":"2024-04-17T19:47:49.860664+0000","last_unstale":"2024-04-17T19:48:29.834364+0000","last_undegraded":"2024-04-17T19:48:29.834364+0000","last_fullsized":"2024-04-17T19:48:29.834364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:07:47.676973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.874890+0000","last_change":"2024-04-17T19:47:48.870638+0000","last_active":"2024-04-17T19:48:33.874890+0000","last_peered":"2024-04-17T19:48:33.874890+0000","last_clean":"2024-04-17T19:48:33.874890+0000","last_became_active":"2024-04-17T19:47:48.870353+0000","last_became_peered":"2024-04-17T19:47:48.870353+0000","last_unstale":"2024-04-17T19:48:33.874890+0000","last_undegraded":"2024-04-17T19:48:33.874890+0000","last_fullsized":"2024-04-17T19:48:33.874890+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:12:19.927824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.861156+0000","last_change":"2024-04-17T19:48:29.857425+0000","last_active":"2024-04-17T19:48:29.861156+0000","last_peered":"2024-04-17T19:48:29.861156+0000","last_clean":"2024-04-17T19:48:29.861156+0000","last_became_active":"2024-04-17T19:48:29.856886+0000","last_became_peered":"2024-04-17T19:48:29.856886+0000","last_unstale":"2024-04-17T19:48:29.861156+0000","last_undegraded":"2024-04-17T19:48:29.861156+0000","last_fullsized":"2024-04-17T19:48:29.861156+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:17:41.501101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.854191+0000","last_change":"2024-04-17T19:47:49.888118+0000","last_active":"2024-04-17T19:48:29.854191+0000","last_peered":"2024-04-17T19:48:29.854191+0000","last_clean":"2024-04-17T19:48:29.854191+0000","last_became_active":"2024-04-17T19:47:49.884037+0000","last_became_peered":"2024-04-17T19:47:49.884037+0000","last_unstale":"2024-04-17T19:48:29.854191+0000","last_undegraded":"2024-04-17T19:48:29.854191+0000","last_fullsized":"2024-04-17T19:48:29.854191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:44:27.180380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"24'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843109+0000","last_change":"2024-04-17T19:47:48.883686+0000","last_active":"2024-04-17T19:48:29.843109+0000","last_peered":"2024-04-17T19:48:29.843109+0000","last_clean":"2024-04-17T19:48:29.843109+0000","last_became_active":"2024-04-17T19:47:48.882375+0000","last_became_peered":"2024-04-17T19:47:48.882375+0000","last_unstale":"2024-04-17T19:48:29.843109+0000","last_undegraded":"2024-04-17T19:48:29.843109+0000","last_fullsized":"2024-04-17T19:48:29.843109+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:38:31.225374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.851300+0000","last_change":"2024-04-17T19:47:53.935396+0000","last_active":"2024-04-17T19:48:29.851300+0000","last_peered":"2024-04-17T19:48:29.851300+0000","last_clean":"2024-04-17T19:48:29.851300+0000","last_became_active":"2024-04-17T19:47:53.934094+0000","last_became_peered":"2024-04-17T19:47:53.934094+0000","last_unstale":"2024-04-17T19:48:29.851300+0000","last_undegraded":"2024-04-17T19:48:29.851300+0000","last_fullsized":"2024-04-17T19:48:29.851300+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:58.334357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162117+0000","last_change":"2024-04-17T19:47:49.870297+0000","last_active":"2024-04-17T19:48:30.162117+0000","last_peered":"2024-04-17T19:48:30.162117+0000","last_clean":"2024-04-17T19:48:30.162117+0000","last_became_active":"2024-04-17T19:47:49.870113+0000","last_became_peered":"2024-04-17T19:47:49.870113+0000","last_unstale":"2024-04-17T19:48:30.162117+0000","last_undegraded":"2024-04-17T19:48:30.162117+0000","last_fullsized":"2024-04-17T19:48:30.162117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:50:54.517273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.876067+0000","last_change":"2024-04-17T19:47:48.895905+0000","last_active":"2024-04-17T19:48:33.876067+0000","last_peered":"2024-04-17T19:48:33.876067+0000","last_clean":"2024-04-17T19:48:33.876067+0000","last_became_active":"2024-04-17T19:47:48.895263+0000","last_became_peered":"2024-04-17T19:47:48.895263+0000","last_unstale":"2024-04-17T19:48:33.876067+0000","last_undegraded":"2024-04-17T19:48:33.876067+0000","last_fullsized":"2024-04-17T19:48:33.876067+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:07:23.179031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.851454+0000","last_change":"2024-04-17T19:47:53.943286+0000","last_active":"2024-04-17T19:48:29.851454+0000","last_peered":"2024-04-17T19:48:29.851454+0000","last_clean":"2024-04-17T19:48:29.851454+0000","last_became_active":"2024-04-17T19:47:53.942325+0000","last_became_peered":"2024-04-17T19:47:53.942325+0000","last_unstale":"2024-04-17T19:48:29.851454+0000","last_undegraded":"2024-04-17T19:48:29.851454+0000","last_fullsized":"2024-04-17T19:48:29.851454+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:47:18.853991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.835175+0000","last_change":"2024-04-17T19:47:49.872925+0000","last_active":"2024-04-17T19:48:29.835175+0000","last_peered":"2024-04-17T19:48:29.835175+0000","last_clean":"2024-04-17T19:48:29.835175+0000","last_became_active":"2024-04-17T19:47:49.872668+0000","last_became_peered":"2024-04-17T19:47:49.872668+0000","last_unstale":"2024-04-17T19:48:29.835175+0000","last_undegraded":"2024-04-17T19:48:29.835175+0000","last_fullsized":"2024-04-17T19:48:29.835175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:16:21.209634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839696+0000","last_change":"2024-04-17T19:47:48.896669+0000","last_active":"2024-04-17T19:48:29.839696+0000","last_peered":"2024-04-17T19:48:29.839696+0000","last_clean":"2024-04-17T19:48:29.839696+0000","last_became_active":"2024-04-17T19:47:48.896397+0000","last_became_peered":"2024-04-17T19:47:48.896397+0000","last_unstale":"2024-04-17T19:48:29.839696+0000","last_undegraded":"2024-04-17T19:48:29.839696+0000","last_fullsized":"2024-04-17T19:48:29.839696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:07:23.179031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.833842+0000","last_change":"2024-04-17T19:47:53.906261+0000","last_active":"2024-04-17T19:48:29.833842+0000","last_peered":"2024-04-17T19:48:29.833842+0000","last_clean":"2024-04-17T19:48:29.833842+0000","last_became_active":"2024-04-17T19:47:53.903702+0000","last_became_peered":"2024-04-17T19:47:53.903702+0000","last_unstale":"2024-04-17T19:48:29.833842+0000","last_undegraded":"2024-04-17T19:48:29.833842+0000","last_fullsized":"2024-04-17T19:48:29.833842+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:35:53.892917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838432+0000","last_change":"2024-04-17T19:47:49.870272+0000","last_active":"2024-04-17T19:48:29.838432+0000","last_peered":"2024-04-17T19:48:29.838432+0000","last_clean":"2024-04-17T19:48:29.838432+0000","last_became_active":"2024-04-17T19:47:49.869914+0000","last_became_peered":"2024-04-17T19:47:49.869914+0000","last_unstale":"2024-04-17T19:48:29.838432+0000","last_undegraded":"2024-04-17T19:48:29.838432+0000","last_fullsized":"2024-04-17T19:48:29.838432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:44:30.553286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845870+0000","last_change":"2024-04-17T19:47:48.892675+0000","last_active":"2024-04-17T19:48:29.845870+0000","last_peered":"2024-04-17T19:48:29.845870+0000","last_clean":"2024-04-17T19:48:29.845870+0000","last_became_active":"2024-04-17T19:47:48.892057+0000","last_became_peered":"2024-04-17T19:47:48.892057+0000","last_unstale":"2024-04-17T19:48:29.845870+0000","last_undegraded":"2024-04-17T19:48:29.845870+0000","last_fullsized":"2024-04-17T19:48:29.845870+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:18:35.735350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.163029+0000","last_change":"2024-04-17T19:47:53.925135+0000","last_active":"2024-04-17T19:48:30.163029+0000","last_peered":"2024-04-17T19:48:30.163029+0000","last_clean":"2024-04-17T19:48:30.163029+0000","last_became_active":"2024-04-17T19:47:53.924617+0000","last_became_peered":"2024-04-17T19:47:53.924617+0000","last_unstale":"2024-04-17T19:48:30.163029+0000","last_undegraded":"2024-04-17T19:48:30.163029+0000","last_fullsized":"2024-04-17T19:48:30.163029+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:37:02.082648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.844178+0000","last_change":"2024-04-17T19:48:29.842544+0000","last_active":"2024-04-17T19:48:29.844178+0000","last_peered":"2024-04-17T19:48:29.844178+0000","last_clean":"2024-04-17T19:48:29.844178+0000","last_became_active":"2024-04-17T19:48:29.842198+0000","last_became_peered":"2024-04-17T19:48:29.842198+0000","last_unstale":"2024-04-17T19:48:29.844178+0000","last_undegraded":"2024-04-17T19:48:29.844178+0000","last_fullsized":"2024-04-17T19:48:29.844178+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:26:06.151001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"24'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838549+0000","last_change":"2024-04-17T19:47:48.859247+0000","last_active":"2024-04-17T19:48:29.838549+0000","last_peered":"2024-04-17T19:48:29.838549+0000","last_clean":"2024-04-17T19:48:29.838549+0000","last_became_active":"2024-04-17T19:47:48.858756+0000","last_became_peered":"2024-04-17T19:47:48.858756+0000","last_unstale":"2024-04-17T19:48:29.838549+0000","last_undegraded":"2024-04-17T19:48:29.838549+0000","last_fullsized":"2024-04-17T19:48:29.838549+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:19:56.987957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840601+0000","last_change":"2024-04-17T19:47:53.929952+0000","last_active":"2024-04-17T19:48:29.840601+0000","last_peered":"2024-04-17T19:48:29.840601+0000","last_clean":"2024-04-17T19:48:29.840601+0000","last_became_active":"2024-04-17T19:47:53.929528+0000","last_became_peered":"2024-04-17T19:47:53.929528+0000","last_unstale":"2024-04-17T19:48:29.840601+0000","last_undegraded":"2024-04-17T19:48:29.840601+0000","last_fullsized":"2024-04-17T19:48:29.840601+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:11:38.041301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.853204+0000","last_change":"2024-04-17T19:47:49.874150+0000","last_active":"2024-04-17T19:48:29.853204+0000","last_peered":"2024-04-17T19:48:29.853204+0000","last_clean":"2024-04-17T19:48:29.853204+0000","last_became_active":"2024-04-17T19:47:49.872859+0000","last_became_peered":"2024-04-17T19:47:49.872859+0000","last_unstale":"2024-04-17T19:48:29.853204+0000","last_undegraded":"2024-04-17T19:48:29.853204+0000","last_fullsized":"2024-04-17T19:48:29.853204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:57:54.666279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.876393+0000","last_change":"2024-04-17T19:47:48.876672+0000","last_active":"2024-04-17T19:48:33.876393+0000","last_peered":"2024-04-17T19:48:33.876393+0000","last_clean":"2024-04-17T19:48:33.876393+0000","last_became_active":"2024-04-17T19:47:48.875263+0000","last_became_peered":"2024-04-17T19:47:48.875263+0000","last_unstale":"2024-04-17T19:48:33.876393+0000","last_undegraded":"2024-04-17T19:48:33.876393+0000","last_fullsized":"2024-04-17T19:48:33.876393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:45:46.633540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.857338+0000","last_change":"2024-04-17T19:48:29.852166+0000","last_active":"2024-04-17T19:48:29.857338+0000","last_peered":"2024-04-17T19:48:29.857338+0000","last_clean":"2024-04-17T19:48:29.857338+0000","last_became_active":"2024-04-17T19:48:29.850441+0000","last_became_peered":"2024-04-17T19:48:29.850441+0000","last_unstale":"2024-04-17T19:48:29.857338+0000","last_undegraded":"2024-04-17T19:48:29.857338+0000","last_fullsized":"2024-04-17T19:48:29.857338+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:32:25.836924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3,6],"acting":[5,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838297+0000","last_change":"2024-04-17T19:47:49.865020+0000","last_active":"2024-04-17T19:48:29.838297+0000","last_peered":"2024-04-17T19:48:29.838297+0000","last_clean":"2024-04-17T19:48:29.838297+0000","last_became_active":"2024-04-17T19:47:49.864764+0000","last_became_peered":"2024-04-17T19:47:49.864764+0000","last_unstale":"2024-04-17T19:48:29.838297+0000","last_undegraded":"2024-04-17T19:48:29.838297+0000","last_fullsized":"2024-04-17T19:48:29.838297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:45:36.325765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838108+0000","last_change":"2024-04-17T19:47:48.846570+0000","last_active":"2024-04-17T19:48:29.838108+0000","last_peered":"2024-04-17T19:48:29.838108+0000","last_clean":"2024-04-17T19:48:29.838108+0000","last_became_active":"2024-04-17T19:47:48.846175+0000","last_became_peered":"2024-04-17T19:47:48.846175+0000","last_unstale":"2024-04-17T19:48:29.838108+0000","last_undegraded":"2024-04-17T19:48:29.838108+0000","last_fullsized":"2024-04-17T19:48:29.838108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:20:22.737016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839023+0000","last_change":"2024-04-17T19:47:53.904590+0000","last_active":"2024-04-17T19:48:29.839023+0000","last_peered":"2024-04-17T19:48:29.839023+0000","last_clean":"2024-04-17T19:48:29.839023+0000","last_became_active":"2024-04-17T19:47:53.904026+0000","last_became_peered":"2024-04-17T19:47:53.904026+0000","last_unstale":"2024-04-17T19:48:29.839023+0000","last_undegraded":"2024-04-17T19:48:29.839023+0000","last_fullsized":"2024-04-17T19:48:29.839023+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:16:22.837570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.161964+0000","last_change":"2024-04-17T19:47:49.860010+0000","last_active":"2024-04-17T19:48:30.161964+0000","last_peered":"2024-04-17T19:48:30.161964+0000","last_clean":"2024-04-17T19:48:30.161964+0000","last_became_active":"2024-04-17T19:47:49.859593+0000","last_became_peered":"2024-04-17T19:47:49.859593+0000","last_unstale":"2024-04-17T19:48:30.161964+0000","last_undegraded":"2024-04-17T19:48:30.161964+0000","last_fullsized":"2024-04-17T19:48:30.161964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:33:24.594397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843671+0000","last_change":"2024-04-17T19:47:48.883126+0000","last_active":"2024-04-17T19:48:29.843671+0000","last_peered":"2024-04-17T19:48:29.843671+0000","last_clean":"2024-04-17T19:48:29.843671+0000","last_became_active":"2024-04-17T19:47:48.882173+0000","last_became_peered":"2024-04-17T19:47:48.882173+0000","last_unstale":"2024-04-17T19:48:29.843671+0000","last_undegraded":"2024-04-17T19:48:29.843671+0000","last_fullsized":"2024-04-17T19:48:29.843671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:11:59.395316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.163046+0000","last_change":"2024-04-17T19:47:53.931258+0000","last_active":"2024-04-17T19:48:30.163046+0000","last_peered":"2024-04-17T19:48:30.163046+0000","last_clean":"2024-04-17T19:48:30.163046+0000","last_became_active":"2024-04-17T19:47:53.930773+0000","last_became_peered":"2024-04-17T19:47:53.930773+0000","last_unstale":"2024-04-17T19:48:30.163046+0000","last_undegraded":"2024-04-17T19:48:30.163046+0000","last_fullsized":"2024-04-17T19:48:30.163046+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:10:47.583714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.844178+0000","last_change":"2024-04-17T19:48:29.844178+0000","last_active":"2024-04-17T19:48:29.844178+0000","last_peered":"2024-04-17T19:48:29.844178+0000","last_clean":"2024-04-17T19:48:29.844178+0000","last_became_active":"2024-04-17T19:48:29.843807+0000","last_became_peered":"2024-04-17T19:48:29.843807+0000","last_unstale":"2024-04-17T19:48:29.844178+0000","last_undegraded":"2024-04-17T19:48:29.844178+0000","last_fullsized":"2024-04-17T19:48:29.844178+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:36:08.301346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"26'6","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.849667+0000","last_change":"2024-04-17T19:47:48.878592+0000","last_active":"2024-04-17T19:48:29.849667+0000","last_peered":"2024-04-17T19:48:29.849667+0000","last_clean":"2024-04-17T19:48:29.849667+0000","last_became_active":"2024-04-17T19:47:48.877617+0000","last_became_peered":"2024-04-17T19:47:48.877617+0000","last_unstale":"2024-04-17T19:48:29.849667+0000","last_undegraded":"2024-04-17T19:48:29.849667+0000","last_fullsized":"2024-04-17T19:48:29.849667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:30:17.013176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":182,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843673+0000","last_change":"2024-04-17T19:47:38.277798+0000","last_active":"2024-04-17T19:48:29.843673+0000","last_peered":"2024-04-17T19:48:29.843673+0000","last_clean":"2024-04-17T19:48:29.843673+0000","last_became_active":"2024-04-17T19:47:38.274697+0000","last_became_peered":"2024-04-17T19:47:38.274697+0000","last_unstale":"2024-04-17T19:48:29.843673+0000","last_undegraded":"2024-04-17T19:48:29.843673+0000","last_fullsized":"2024-04-17T19:48:29.843673+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:36.897071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:36.897071+0000","last_clean_scrub_stamp":"2024-04-17T19:47:36.897071+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:31:35.366600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.844472+0000","last_change":"2024-04-17T19:47:53.953415+0000","last_active":"2024-04-17T19:48:29.844472+0000","last_peered":"2024-04-17T19:48:29.844472+0000","last_clean":"2024-04-17T19:48:29.844472+0000","last_became_active":"2024-04-17T19:47:53.952644+0000","last_became_peered":"2024-04-17T19:47:53.952644+0000","last_unstale":"2024-04-17T19:48:29.844472+0000","last_undegraded":"2024-04-17T19:48:29.844472+0000","last_fullsized":"2024-04-17T19:48:29.844472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:00:25.923909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.844721+0000","last_change":"2024-04-17T19:47:49.901572+0000","last_active":"2024-04-17T19:48:29.844721+0000","last_peered":"2024-04-17T19:48:29.844721+0000","last_clean":"2024-04-17T19:48:29.844721+0000","last_became_active":"2024-04-17T19:47:49.900707+0000","last_became_peered":"2024-04-17T19:47:49.900707+0000","last_unstale":"2024-04-17T19:48:29.844721+0000","last_undegraded":"2024-04-17T19:48:29.844721+0000","last_fullsized":"2024-04-17T19:48:29.844721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:58:02.897564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"24'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838864+0000","last_change":"2024-04-17T19:47:48.874721+0000","last_active":"2024-04-17T19:48:29.838864+0000","last_peered":"2024-04-17T19:48:29.838864+0000","last_clean":"2024-04-17T19:48:29.838864+0000","last_became_active":"2024-04-17T19:47:48.874364+0000","last_became_peered":"2024-04-17T19:47:48.874364+0000","last_unstale":"2024-04-17T19:48:29.838864+0000","last_undegraded":"2024-04-17T19:48:29.838864+0000","last_fullsized":"2024-04-17T19:48:29.838864+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:47:08.856753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.842743+0000","last_change":"2024-04-17T19:47:53.942791+0000","last_active":"2024-04-17T19:48:29.842743+0000","last_peered":"2024-04-17T19:48:29.842743+0000","last_clean":"2024-04-17T19:48:29.842743+0000","last_became_active":"2024-04-17T19:47:53.941569+0000","last_became_peered":"2024-04-17T19:47:53.941569+0000","last_unstale":"2024-04-17T19:48:29.842743+0000","last_undegraded":"2024-04-17T19:48:29.842743+0000","last_fullsized":"2024-04-17T19:48:29.842743+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:23:35.452056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.848531+0000","last_change":"2024-04-17T19:47:49.895046+0000","last_active":"2024-04-17T19:48:29.848531+0000","last_peered":"2024-04-17T19:48:29.848531+0000","last_clean":"2024-04-17T19:48:29.848531+0000","last_became_active":"2024-04-17T19:47:49.894250+0000","last_became_peered":"2024-04-17T19:47:49.894250+0000","last_unstale":"2024-04-17T19:48:29.848531+0000","last_undegraded":"2024-04-17T19:48:29.848531+0000","last_fullsized":"2024-04-17T19:48:29.848531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:30:17.210135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839227+0000","last_change":"2024-04-17T19:47:48.861802+0000","last_active":"2024-04-17T19:48:29.839227+0000","last_peered":"2024-04-17T19:48:29.839227+0000","last_clean":"2024-04-17T19:48:29.839227+0000","last_became_active":"2024-04-17T19:47:48.860418+0000","last_became_peered":"2024-04-17T19:47:48.860418+0000","last_unstale":"2024-04-17T19:48:29.839227+0000","last_undegraded":"2024-04-17T19:48:29.839227+0000","last_fullsized":"2024-04-17T19:48:29.839227+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:54:41.949746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.163167+0000","last_change":"2024-04-17T19:47:53.917732+0000","last_active":"2024-04-17T19:48:30.163167+0000","last_peered":"2024-04-17T19:48:30.163167+0000","last_clean":"2024-04-17T19:48:30.163167+0000","last_became_active":"2024-04-17T19:47:53.917160+0000","last_became_peered":"2024-04-17T19:47:53.917160+0000","last_unstale":"2024-04-17T19:48:30.163167+0000","last_undegraded":"2024-04-17T19:48:30.163167+0000","last_fullsized":"2024-04-17T19:48:30.163167+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:19:20.394166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.847292+0000","last_change":"2024-04-17T19:47:49.893077+0000","last_active":"2024-04-17T19:48:29.847292+0000","last_peered":"2024-04-17T19:48:29.847292+0000","last_clean":"2024-04-17T19:48:29.847292+0000","last_became_active":"2024-04-17T19:47:49.892280+0000","last_became_peered":"2024-04-17T19:47:49.892280+0000","last_unstale":"2024-04-17T19:48:29.847292+0000","last_undegraded":"2024-04-17T19:48:29.847292+0000","last_fullsized":"2024-04-17T19:48:29.847292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:58:24.218821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.868436+0000","last_change":"2024-04-17T19:47:48.849027+0000","last_active":"2024-04-17T19:48:33.868436+0000","last_peered":"2024-04-17T19:48:33.868436+0000","last_clean":"2024-04-17T19:48:33.868436+0000","last_became_active":"2024-04-17T19:47:48.848485+0000","last_became_peered":"2024-04-17T19:47:48.848485+0000","last_unstale":"2024-04-17T19:48:33.868436+0000","last_undegraded":"2024-04-17T19:48:33.868436+0000","last_fullsized":"2024-04-17T19:48:33.868436+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:58:14.548891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839524+0000","last_change":"2024-04-17T19:47:53.946933+0000","last_active":"2024-04-17T19:48:29.839524+0000","last_peered":"2024-04-17T19:48:29.839524+0000","last_clean":"2024-04-17T19:48:29.839524+0000","last_became_active":"2024-04-17T19:47:53.943030+0000","last_became_peered":"2024-04-17T19:47:53.943030+0000","last_unstale":"2024-04-17T19:48:29.839524+0000","last_undegraded":"2024-04-17T19:48:29.839524+0000","last_fullsized":"2024-04-17T19:48:29.839524+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:37:03.489250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845055+0000","last_change":"2024-04-17T19:47:49.893137+0000","last_active":"2024-04-17T19:48:29.845055+0000","last_peered":"2024-04-17T19:48:29.845055+0000","last_clean":"2024-04-17T19:48:29.845055+0000","last_became_active":"2024-04-17T19:47:49.892379+0000","last_became_peered":"2024-04-17T19:47:49.892379+0000","last_unstale":"2024-04-17T19:48:29.845055+0000","last_undegraded":"2024-04-17T19:48:29.845055+0000","last_fullsized":"2024-04-17T19:48:29.845055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:55:08.562395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"24'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.853051+0000","last_change":"2024-04-17T19:47:48.861154+0000","last_active":"2024-04-17T19:48:29.853051+0000","last_peered":"2024-04-17T19:48:29.853051+0000","last_clean":"2024-04-17T19:48:29.853051+0000","last_became_active":"2024-04-17T19:47:48.859268+0000","last_became_peered":"2024-04-17T19:47:48.859268+0000","last_unstale":"2024-04-17T19:48:29.853051+0000","last_undegraded":"2024-04-17T19:48:29.853051+0000","last_fullsized":"2024-04-17T19:48:29.853051+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:04:19.867115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.849198+0000","last_change":"2024-04-17T19:47:53.950580+0000","last_active":"2024-04-17T19:48:29.849198+0000","last_peered":"2024-04-17T19:48:29.849198+0000","last_clean":"2024-04-17T19:48:29.849198+0000","last_became_active":"2024-04-17T19:47:53.943798+0000","last_became_peered":"2024-04-17T19:47:53.943798+0000","last_unstale":"2024-04-17T19:48:29.849198+0000","last_undegraded":"2024-04-17T19:48:29.849198+0000","last_fullsized":"2024-04-17T19:48:29.849198+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:52:14.763799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843479+0000","last_change":"2024-04-17T19:47:49.893774+0000","last_active":"2024-04-17T19:48:29.843479+0000","last_peered":"2024-04-17T19:48:29.843479+0000","last_clean":"2024-04-17T19:48:29.843479+0000","last_became_active":"2024-04-17T19:47:49.892780+0000","last_became_peered":"2024-04-17T19:47:49.892780+0000","last_unstale":"2024-04-17T19:48:29.843479+0000","last_undegraded":"2024-04-17T19:48:29.843479+0000","last_fullsized":"2024-04-17T19:48:29.843479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:50:47.658650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"26'17","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.837592+0000","last_change":"2024-04-17T19:47:48.852970+0000","last_active":"2024-04-17T19:48:29.837592+0000","last_peered":"2024-04-17T19:48:29.837592+0000","last_clean":"2024-04-17T19:48:29.837592+0000","last_became_active":"2024-04-17T19:47:48.852676+0000","last_became_peered":"2024-04-17T19:47:48.852676+0000","last_unstale":"2024-04-17T19:48:29.837592+0000","last_undegraded":"2024-04-17T19:48:29.837592+0000","last_fullsized":"2024-04-17T19:48:29.837592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:32:14.529596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162931+0000","last_change":"2024-04-17T19:47:53.915397+0000","last_active":"2024-04-17T19:48:30.162931+0000","last_peered":"2024-04-17T19:48:30.162931+0000","last_clean":"2024-04-17T19:48:30.162931+0000","last_became_active":"2024-04-17T19:47:53.914913+0000","last_became_peered":"2024-04-17T19:47:53.914913+0000","last_unstale":"2024-04-17T19:48:30.162931+0000","last_undegraded":"2024-04-17T19:48:30.162931+0000","last_fullsized":"2024-04-17T19:48:30.162931+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:02:01.175158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840177+0000","last_change":"2024-04-17T19:47:49.880484+0000","last_active":"2024-04-17T19:48:29.840177+0000","last_peered":"2024-04-17T19:48:29.840177+0000","last_clean":"2024-04-17T19:48:29.840177+0000","last_became_active":"2024-04-17T19:47:49.879625+0000","last_became_peered":"2024-04-17T19:47:49.879625+0000","last_unstale":"2024-04-17T19:48:29.840177+0000","last_undegraded":"2024-04-17T19:48:29.840177+0000","last_fullsized":"2024-04-17T19:48:29.840177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:50:00.611796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"24'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843272+0000","last_change":"2024-04-17T19:47:48.899160+0000","last_active":"2024-04-17T19:48:29.843272+0000","last_peered":"2024-04-17T19:48:29.843272+0000","last_clean":"2024-04-17T19:48:29.843272+0000","last_became_active":"2024-04-17T19:47:48.889831+0000","last_became_peered":"2024-04-17T19:47:48.889831+0000","last_unstale":"2024-04-17T19:48:29.843272+0000","last_undegraded":"2024-04-17T19:48:29.843272+0000","last_fullsized":"2024-04-17T19:48:29.843272+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:55:20.167510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.835052+0000","last_change":"2024-04-17T19:47:53.921093+0000","last_active":"2024-04-17T19:48:29.835052+0000","last_peered":"2024-04-17T19:48:29.835052+0000","last_clean":"2024-04-17T19:48:29.835052+0000","last_became_active":"2024-04-17T19:47:53.920479+0000","last_became_peered":"2024-04-17T19:47:53.920479+0000","last_unstale":"2024-04-17T19:48:29.835052+0000","last_undegraded":"2024-04-17T19:48:29.835052+0000","last_fullsized":"2024-04-17T19:48:29.835052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:08:09.569820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.844227+0000","last_change":"2024-04-17T19:47:49.895275+0000","last_active":"2024-04-17T19:48:29.844227+0000","last_peered":"2024-04-17T19:48:29.844227+0000","last_clean":"2024-04-17T19:48:29.844227+0000","last_became_active":"2024-04-17T19:47:49.894534+0000","last_became_peered":"2024-04-17T19:47:49.894534+0000","last_unstale":"2024-04-17T19:48:29.844227+0000","last_undegraded":"2024-04-17T19:48:29.844227+0000","last_fullsized":"2024-04-17T19:48:29.844227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:31:40.818807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.841327+0000","last_change":"2024-04-17T19:47:48.881715+0000","last_active":"2024-04-17T19:48:29.841327+0000","last_peered":"2024-04-17T19:48:29.841327+0000","last_clean":"2024-04-17T19:48:29.841327+0000","last_became_active":"2024-04-17T19:47:48.876203+0000","last_became_peered":"2024-04-17T19:47:48.876203+0000","last_unstale":"2024-04-17T19:48:29.841327+0000","last_undegraded":"2024-04-17T19:48:29.841327+0000","last_fullsized":"2024-04-17T19:48:29.841327+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:03:16.656528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834859+0000","last_change":"2024-04-17T19:47:53.905149+0000","last_active":"2024-04-17T19:48:29.834859+0000","last_peered":"2024-04-17T19:48:29.834859+0000","last_clean":"2024-04-17T19:48:29.834859+0000","last_became_active":"2024-04-17T19:47:53.904718+0000","last_became_peered":"2024-04-17T19:47:53.904718+0000","last_unstale":"2024-04-17T19:48:29.834859+0000","last_undegraded":"2024-04-17T19:48:29.834859+0000","last_fullsized":"2024-04-17T19:48:29.834859+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:09:31.908752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162277+0000","last_change":"2024-04-17T19:47:49.875819+0000","last_active":"2024-04-17T19:48:30.162277+0000","last_peered":"2024-04-17T19:48:30.162277+0000","last_clean":"2024-04-17T19:48:30.162277+0000","last_became_active":"2024-04-17T19:47:49.872075+0000","last_became_peered":"2024-04-17T19:47:49.872075+0000","last_unstale":"2024-04-17T19:48:30.162277+0000","last_undegraded":"2024-04-17T19:48:30.162277+0000","last_fullsized":"2024-04-17T19:48:30.162277+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:06:48.823135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"26'9","reported_seq":35,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.871452+0000","last_change":"2024-04-17T19:47:48.867633+0000","last_active":"2024-04-17T19:48:33.871452+0000","last_peered":"2024-04-17T19:48:33.871452+0000","last_clean":"2024-04-17T19:48:33.871452+0000","last_became_active":"2024-04-17T19:47:48.867150+0000","last_became_peered":"2024-04-17T19:47:48.867150+0000","last_unstale":"2024-04-17T19:48:33.871452+0000","last_undegraded":"2024-04-17T19:48:33.871452+0000","last_fullsized":"2024-04-17T19:48:33.871452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:33:34.275816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.847899+0000","last_change":"2024-04-17T19:47:53.925785+0000","last_active":"2024-04-17T19:48:29.847899+0000","last_peered":"2024-04-17T19:48:29.847899+0000","last_clean":"2024-04-17T19:48:29.847899+0000","last_became_active":"2024-04-17T19:47:53.924006+0000","last_became_peered":"2024-04-17T19:47:53.924006+0000","last_unstale":"2024-04-17T19:48:29.847899+0000","last_undegraded":"2024-04-17T19:48:29.847899+0000","last_fullsized":"2024-04-17T19:48:29.847899+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:46:31.965271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845415+0000","last_change":"2024-04-17T19:47:49.869829+0000","last_active":"2024-04-17T19:48:29.845415+0000","last_peered":"2024-04-17T19:48:29.845415+0000","last_clean":"2024-04-17T19:48:29.845415+0000","last_became_active":"2024-04-17T19:47:49.868933+0000","last_became_peered":"2024-04-17T19:47:49.868933+0000","last_unstale":"2024-04-17T19:48:29.845415+0000","last_undegraded":"2024-04-17T19:48:29.845415+0000","last_fullsized":"2024-04-17T19:48:29.845415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:36:08.866159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'15","reported_seq":41,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:43.269615+0000","last_change":"2024-04-17T19:47:48.887461+0000","last_active":"2024-04-17T19:48:43.269615+0000","last_peered":"2024-04-17T19:48:43.269615+0000","last_clean":"2024-04-17T19:48:43.269615+0000","last_became_active":"2024-04-17T19:47:48.886689+0000","last_became_peered":"2024-04-17T19:47:48.886689+0000","last_unstale":"2024-04-17T19:48:43.269615+0000","last_undegraded":"2024-04-17T19:48:43.269615+0000","last_fullsized":"2024-04-17T19:48:43.269615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:55:35.262995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4100,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840709+0000","last_change":"2024-04-17T19:47:53.941607+0000","last_active":"2024-04-17T19:48:29.840709+0000","last_peered":"2024-04-17T19:48:29.840709+0000","last_clean":"2024-04-17T19:48:29.840709+0000","last_became_active":"2024-04-17T19:47:53.941050+0000","last_became_peered":"2024-04-17T19:47:53.941050+0000","last_unstale":"2024-04-17T19:48:29.840709+0000","last_undegraded":"2024-04-17T19:48:29.840709+0000","last_fullsized":"2024-04-17T19:48:29.840709+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:26:08.964058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.851845+0000","last_change":"2024-04-17T19:47:49.879524+0000","last_active":"2024-04-17T19:48:29.851845+0000","last_peered":"2024-04-17T19:48:29.851845+0000","last_clean":"2024-04-17T19:48:29.851845+0000","last_became_active":"2024-04-17T19:47:49.877563+0000","last_became_peered":"2024-04-17T19:47:49.877563+0000","last_unstale":"2024-04-17T19:48:29.851845+0000","last_undegraded":"2024-04-17T19:48:29.851845+0000","last_fullsized":"2024-04-17T19:48:29.851845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:34:58.539940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"24'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839055+0000","last_change":"2024-04-17T19:47:48.870100+0000","last_active":"2024-04-17T19:48:29.839055+0000","last_peered":"2024-04-17T19:48:29.839055+0000","last_clean":"2024-04-17T19:48:29.839055+0000","last_became_active":"2024-04-17T19:47:48.869797+0000","last_became_peered":"2024-04-17T19:47:48.869797+0000","last_unstale":"2024-04-17T19:48:29.839055+0000","last_undegraded":"2024-04-17T19:48:29.839055+0000","last_fullsized":"2024-04-17T19:48:29.839055+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:51:21.463081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.850942+0000","last_change":"2024-04-17T19:47:53.944568+0000","last_active":"2024-04-17T19:48:29.850942+0000","last_peered":"2024-04-17T19:48:29.850942+0000","last_clean":"2024-04-17T19:48:29.850942+0000","last_became_active":"2024-04-17T19:47:53.943151+0000","last_became_peered":"2024-04-17T19:47:53.943151+0000","last_unstale":"2024-04-17T19:48:29.850942+0000","last_undegraded":"2024-04-17T19:48:29.850942+0000","last_fullsized":"2024-04-17T19:48:29.850942+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:19:00.150450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162170+0000","last_change":"2024-04-17T19:47:49.902401+0000","last_active":"2024-04-17T19:48:30.162170+0000","last_peered":"2024-04-17T19:48:30.162170+0000","last_clean":"2024-04-17T19:48:30.162170+0000","last_became_active":"2024-04-17T19:47:49.902166+0000","last_became_peered":"2024-04-17T19:47:49.902166+0000","last_unstale":"2024-04-17T19:48:30.162170+0000","last_undegraded":"2024-04-17T19:48:30.162170+0000","last_fullsized":"2024-04-17T19:48:30.162170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:54:13.129748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.876986+0000","last_change":"2024-04-17T19:47:48.900800+0000","last_active":"2024-04-17T19:48:33.876986+0000","last_peered":"2024-04-17T19:48:33.876986+0000","last_clean":"2024-04-17T19:48:33.876986+0000","last_became_active":"2024-04-17T19:47:48.900188+0000","last_became_peered":"2024-04-17T19:47:48.900188+0000","last_unstale":"2024-04-17T19:48:33.876986+0000","last_undegraded":"2024-04-17T19:48:33.876986+0000","last_fullsized":"2024-04-17T19:48:33.876986+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:05:04.305871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834112+0000","last_change":"2024-04-17T19:47:53.928721+0000","last_active":"2024-04-17T19:48:29.834112+0000","last_peered":"2024-04-17T19:48:29.834112+0000","last_clean":"2024-04-17T19:48:29.834112+0000","last_became_active":"2024-04-17T19:47:53.928311+0000","last_became_peered":"2024-04-17T19:47:53.928311+0000","last_unstale":"2024-04-17T19:48:29.834112+0000","last_undegraded":"2024-04-17T19:48:29.834112+0000","last_fullsized":"2024-04-17T19:48:29.834112+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:35:28.870803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.847789+0000","last_change":"2024-04-17T19:47:49.905421+0000","last_active":"2024-04-17T19:48:29.847789+0000","last_peered":"2024-04-17T19:48:29.847789+0000","last_clean":"2024-04-17T19:48:29.847789+0000","last_became_active":"2024-04-17T19:47:49.904646+0000","last_became_peered":"2024-04-17T19:47:49.904646+0000","last_unstale":"2024-04-17T19:48:29.847789+0000","last_undegraded":"2024-04-17T19:48:29.847789+0000","last_fullsized":"2024-04-17T19:48:29.847789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:02:23.936080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"24'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.846631+0000","last_change":"2024-04-17T19:47:48.895356+0000","last_active":"2024-04-17T19:48:29.846631+0000","last_peered":"2024-04-17T19:48:29.846631+0000","last_clean":"2024-04-17T19:48:29.846631+0000","last_became_active":"2024-04-17T19:47:48.889225+0000","last_became_peered":"2024-04-17T19:47:48.889225+0000","last_unstale":"2024-04-17T19:48:29.846631+0000","last_undegraded":"2024-04-17T19:48:29.846631+0000","last_fullsized":"2024-04-17T19:48:29.846631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:00:22.587917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.833996+0000","last_change":"2024-04-17T19:47:53.940206+0000","last_active":"2024-04-17T19:48:29.833996+0000","last_peered":"2024-04-17T19:48:29.833996+0000","last_clean":"2024-04-17T19:48:29.833996+0000","last_became_active":"2024-04-17T19:47:53.939843+0000","last_became_peered":"2024-04-17T19:47:53.939843+0000","last_unstale":"2024-04-17T19:48:29.833996+0000","last_undegraded":"2024-04-17T19:48:29.833996+0000","last_fullsized":"2024-04-17T19:48:29.833996+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:18:55.562573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162804+0000","last_change":"2024-04-17T19:47:53.920258+0000","last_active":"2024-04-17T19:48:30.162804+0000","last_peered":"2024-04-17T19:48:30.162804+0000","last_clean":"2024-04-17T19:48:30.162804+0000","last_became_active":"2024-04-17T19:47:53.918932+0000","last_became_peered":"2024-04-17T19:47:53.918932+0000","last_unstale":"2024-04-17T19:48:30.162804+0000","last_undegraded":"2024-04-17T19:48:30.162804+0000","last_fullsized":"2024-04-17T19:48:30.162804+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:45:55.456803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.842416+0000","last_change":"2024-04-17T19:48:29.842416+0000","last_active":"2024-04-17T19:48:29.842416+0000","last_peered":"2024-04-17T19:48:29.842416+0000","last_clean":"2024-04-17T19:48:29.842416+0000","last_became_active":"2024-04-17T19:48:29.841759+0000","last_became_peered":"2024-04-17T19:48:29.841759+0000","last_unstale":"2024-04-17T19:48:29.842416+0000","last_undegraded":"2024-04-17T19:48:29.842416+0000","last_fullsized":"2024-04-17T19:48:29.842416+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:57:17.664562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840971+0000","last_change":"2024-04-17T19:47:48.873355+0000","last_active":"2024-04-17T19:48:29.840971+0000","last_peered":"2024-04-17T19:48:29.840971+0000","last_clean":"2024-04-17T19:48:29.840971+0000","last_became_active":"2024-04-17T19:47:48.872427+0000","last_became_peered":"2024-04-17T19:47:48.872427+0000","last_unstale":"2024-04-17T19:48:29.840971+0000","last_undegraded":"2024-04-17T19:48:29.840971+0000","last_fullsized":"2024-04-17T19:48:29.840971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:32:41.792605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.846032+0000","last_change":"2024-04-17T19:47:53.952602+0000","last_active":"2024-04-17T19:48:29.846032+0000","last_peered":"2024-04-17T19:48:29.846032+0000","last_clean":"2024-04-17T19:48:29.846032+0000","last_became_active":"2024-04-17T19:47:53.951750+0000","last_became_peered":"2024-04-17T19:47:53.951750+0000","last_unstale":"2024-04-17T19:48:29.846032+0000","last_undegraded":"2024-04-17T19:48:29.846032+0000","last_fullsized":"2024-04-17T19:48:29.846032+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:13:08.485238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834245+0000","last_change":"2024-04-17T19:47:49.862071+0000","last_active":"2024-04-17T19:48:29.834245+0000","last_peered":"2024-04-17T19:48:29.834245+0000","last_clean":"2024-04-17T19:48:29.834245+0000","last_became_active":"2024-04-17T19:47:49.860809+0000","last_became_peered":"2024-04-17T19:47:49.860809+0000","last_unstale":"2024-04-17T19:48:29.834245+0000","last_undegraded":"2024-04-17T19:48:29.834245+0000","last_fullsized":"2024-04-17T19:48:29.834245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:42:34.451839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834368+0000","last_change":"2024-04-17T19:47:48.867855+0000","last_active":"2024-04-17T19:48:29.834368+0000","last_peered":"2024-04-17T19:48:29.834368+0000","last_clean":"2024-04-17T19:48:29.834368+0000","last_became_active":"2024-04-17T19:47:48.867532+0000","last_became_peered":"2024-04-17T19:47:48.867532+0000","last_unstale":"2024-04-17T19:48:29.834368+0000","last_undegraded":"2024-04-17T19:48:29.834368+0000","last_fullsized":"2024-04-17T19:48:29.834368+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:21:26.903245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"24'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843580+0000","last_change":"2024-04-17T19:47:48.894946+0000","last_active":"2024-04-17T19:48:29.843580+0000","last_peered":"2024-04-17T19:48:29.843580+0000","last_clean":"2024-04-17T19:48:29.843580+0000","last_became_active":"2024-04-17T19:47:48.893407+0000","last_became_peered":"2024-04-17T19:47:48.893407+0000","last_unstale":"2024-04-17T19:48:29.843580+0000","last_undegraded":"2024-04-17T19:48:29.843580+0000","last_fullsized":"2024-04-17T19:48:29.843580+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:09:56.284638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162336+0000","last_change":"2024-04-17T19:47:49.859985+0000","last_active":"2024-04-17T19:48:30.162336+0000","last_peered":"2024-04-17T19:48:30.162336+0000","last_clean":"2024-04-17T19:48:30.162336+0000","last_became_active":"2024-04-17T19:47:49.859635+0000","last_became_peered":"2024-04-17T19:47:49.859635+0000","last_unstale":"2024-04-17T19:48:30.162336+0000","last_undegraded":"2024-04-17T19:48:30.162336+0000","last_fullsized":"2024-04-17T19:48:30.162336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:04:14.834218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838003+0000","last_change":"2024-04-17T19:47:53.925444+0000","last_active":"2024-04-17T19:48:29.838003+0000","last_peered":"2024-04-17T19:48:29.838003+0000","last_clean":"2024-04-17T19:48:29.838003+0000","last_became_active":"2024-04-17T19:47:53.924805+0000","last_became_peered":"2024-04-17T19:47:53.924805+0000","last_unstale":"2024-04-17T19:48:29.838003+0000","last_undegraded":"2024-04-17T19:48:29.838003+0000","last_fullsized":"2024-04-17T19:48:29.838003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:19:24.022368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"24'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834567+0000","last_change":"2024-04-17T19:47:48.867176+0000","last_active":"2024-04-17T19:48:29.834567+0000","last_peered":"2024-04-17T19:48:29.834567+0000","last_clean":"2024-04-17T19:48:29.834567+0000","last_became_active":"2024-04-17T19:47:48.866817+0000","last_became_peered":"2024-04-17T19:47:48.866817+0000","last_unstale":"2024-04-17T19:48:29.834567+0000","last_undegraded":"2024-04-17T19:48:29.834567+0000","last_fullsized":"2024-04-17T19:48:29.834567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:44:30.842583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.850861+0000","last_change":"2024-04-17T19:47:49.858924+0000","last_active":"2024-04-17T19:48:29.850861+0000","last_peered":"2024-04-17T19:48:29.850861+0000","last_clean":"2024-04-17T19:48:29.850861+0000","last_became_active":"2024-04-17T19:47:49.857820+0000","last_became_peered":"2024-04-17T19:47:49.857820+0000","last_unstale":"2024-04-17T19:48:29.850861+0000","last_undegraded":"2024-04-17T19:48:29.850861+0000","last_fullsized":"2024-04-17T19:48:29.850861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:09:28.706054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.835377+0000","last_change":"2024-04-17T19:47:53.933337+0000","last_active":"2024-04-17T19:48:29.835377+0000","last_peered":"2024-04-17T19:48:29.835377+0000","last_clean":"2024-04-17T19:48:29.835377+0000","last_became_active":"2024-04-17T19:47:53.931623+0000","last_became_peered":"2024-04-17T19:47:53.931623+0000","last_unstale":"2024-04-17T19:48:29.835377+0000","last_undegraded":"2024-04-17T19:48:29.835377+0000","last_fullsized":"2024-04-17T19:48:29.835377+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:57:37.600957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839672+0000","last_change":"2024-04-17T19:47:49.881093+0000","last_active":"2024-04-17T19:48:29.839672+0000","last_peered":"2024-04-17T19:48:29.839672+0000","last_clean":"2024-04-17T19:48:29.839672+0000","last_became_active":"2024-04-17T19:47:49.880210+0000","last_became_peered":"2024-04-17T19:47:49.880210+0000","last_unstale":"2024-04-17T19:48:29.839672+0000","last_undegraded":"2024-04-17T19:48:29.839672+0000","last_fullsized":"2024-04-17T19:48:29.839672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:12:15.522651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.837813+0000","last_change":"2024-04-17T19:47:49.886936+0000","last_active":"2024-04-17T19:48:29.837813+0000","last_peered":"2024-04-17T19:48:29.837813+0000","last_clean":"2024-04-17T19:48:29.837813+0000","last_became_active":"2024-04-17T19:47:49.886580+0000","last_became_peered":"2024-04-17T19:47:49.886580+0000","last_unstale":"2024-04-17T19:48:29.837813+0000","last_undegraded":"2024-04-17T19:48:29.837813+0000","last_fullsized":"2024-04-17T19:48:29.837813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:00:44.599463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840923+0000","last_change":"2024-04-17T19:47:49.905445+0000","last_active":"2024-04-17T19:48:29.840923+0000","last_peered":"2024-04-17T19:48:29.840923+0000","last_clean":"2024-04-17T19:48:29.840923+0000","last_became_active":"2024-04-17T19:47:49.896510+0000","last_became_peered":"2024-04-17T19:47:49.896510+0000","last_unstale":"2024-04-17T19:48:29.840923+0000","last_undegraded":"2024-04-17T19:48:29.840923+0000","last_fullsized":"2024-04-17T19:48:29.840923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:23:27.401861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162472+0000","last_change":"2024-04-17T19:47:49.850490+0000","last_active":"2024-04-17T19:48:30.162472+0000","last_peered":"2024-04-17T19:48:30.162472+0000","last_clean":"2024-04-17T19:48:30.162472+0000","last_became_active":"2024-04-17T19:47:49.849344+0000","last_became_peered":"2024-04-17T19:47:49.849344+0000","last_unstale":"2024-04-17T19:48:30.162472+0000","last_undegraded":"2024-04-17T19:48:30.162472+0000","last_fullsized":"2024-04-17T19:48:30.162472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:44:32.231058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.835044+0000","last_change":"2024-04-17T19:47:49.857737+0000","last_active":"2024-04-17T19:48:29.835044+0000","last_peered":"2024-04-17T19:48:29.835044+0000","last_clean":"2024-04-17T19:48:29.835044+0000","last_became_active":"2024-04-17T19:47:49.857476+0000","last_became_peered":"2024-04-17T19:47:49.857476+0000","last_unstale":"2024-04-17T19:48:29.835044+0000","last_undegraded":"2024-04-17T19:48:29.835044+0000","last_fullsized":"2024-04-17T19:48:29.835044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:50:13.070628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838876+0000","last_change":"2024-04-17T19:47:49.864985+0000","last_active":"2024-04-17T19:48:29.838876+0000","last_peered":"2024-04-17T19:48:29.838876+0000","last_clean":"2024-04-17T19:48:29.838876+0000","last_became_active":"2024-04-17T19:47:49.864329+0000","last_became_peered":"2024-04-17T19:47:49.864329+0000","last_unstale":"2024-04-17T19:48:29.838876+0000","last_undegraded":"2024-04-17T19:48:29.838876+0000","last_fullsized":"2024-04-17T19:48:29.838876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:43:57.448269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845430+0000","last_change":"2024-04-17T19:47:49.881785+0000","last_active":"2024-04-17T19:48:29.845430+0000","last_peered":"2024-04-17T19:48:29.845430+0000","last_clean":"2024-04-17T19:48:29.845430+0000","last_became_active":"2024-04-17T19:47:49.880939+0000","last_became_peered":"2024-04-17T19:47:49.880939+0000","last_unstale":"2024-04-17T19:48:29.845430+0000","last_undegraded":"2024-04-17T19:48:29.845430+0000","last_fullsized":"2024-04-17T19:48:29.845430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:49:07.194643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.835152+0000","last_change":"2024-04-17T19:47:49.884944+0000","last_active":"2024-04-17T19:48:29.835152+0000","last_peered":"2024-04-17T19:48:29.835152+0000","last_clean":"2024-04-17T19:48:29.835152+0000","last_became_active":"2024-04-17T19:47:49.884559+0000","last_became_peered":"2024-04-17T19:47:49.884559+0000","last_unstale":"2024-04-17T19:48:29.835152+0000","last_undegraded":"2024-04-17T19:48:29.835152+0000","last_fullsized":"2024-04-17T19:48:29.835152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:00.485472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840468+0000","last_change":"2024-04-17T19:47:49.891230+0000","last_active":"2024-04-17T19:48:29.840468+0000","last_peered":"2024-04-17T19:48:29.840468+0000","last_clean":"2024-04-17T19:48:29.840468+0000","last_became_active":"2024-04-17T19:47:49.886001+0000","last_became_peered":"2024-04-17T19:47:49.886001+0000","last_unstale":"2024-04-17T19:48:29.840468+0000","last_undegraded":"2024-04-17T19:48:29.840468+0000","last_fullsized":"2024-04-17T19:48:29.840468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:00.485472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834129+0000","last_change":"2024-04-17T19:47:49.859382+0000","last_active":"2024-04-17T19:48:29.834129+0000","last_peered":"2024-04-17T19:48:29.834129+0000","last_clean":"2024-04-17T19:48:29.834129+0000","last_became_active":"2024-04-17T19:47:49.859170+0000","last_became_peered":"2024-04-17T19:47:49.859170+0000","last_unstale":"2024-04-17T19:48:29.834129+0000","last_undegraded":"2024-04-17T19:48:29.834129+0000","last_fullsized":"2024-04-17T19:48:29.834129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:58:51.111053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.854842+0000","last_change":"2024-04-17T19:48:29.853997+0000","last_active":"2024-04-17T19:48:29.854842+0000","last_peered":"2024-04-17T19:48:29.854842+0000","last_clean":"2024-04-17T19:48:29.854842+0000","last_became_active":"2024-04-17T19:48:29.853682+0000","last_became_peered":"2024-04-17T19:48:29.853682+0000","last_unstale":"2024-04-17T19:48:29.854842+0000","last_undegraded":"2024-04-17T19:48:29.854842+0000","last_fullsized":"2024-04-17T19:48:29.854842+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:29:20.526070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162602+0000","last_change":"2024-04-17T19:47:49.876406+0000","last_active":"2024-04-17T19:48:30.162602+0000","last_peered":"2024-04-17T19:48:30.162602+0000","last_clean":"2024-04-17T19:48:30.162602+0000","last_became_active":"2024-04-17T19:47:49.876132+0000","last_became_peered":"2024-04-17T19:47:49.876132+0000","last_unstale":"2024-04-17T19:48:30.162602+0000","last_undegraded":"2024-04-17T19:48:30.162602+0000","last_fullsized":"2024-04-17T19:48:30.162602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:51:20.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.846554+0000","last_change":"2024-04-17T19:47:49.896244+0000","last_active":"2024-04-17T19:48:29.846554+0000","last_peered":"2024-04-17T19:48:29.846554+0000","last_clean":"2024-04-17T19:48:29.846554+0000","last_became_active":"2024-04-17T19:47:49.895380+0000","last_became_peered":"2024-04-17T19:47:49.895380+0000","last_unstale":"2024-04-17T19:48:29.846554+0000","last_undegraded":"2024-04-17T19:48:29.846554+0000","last_fullsized":"2024-04-17T19:48:29.846554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:04:43.091307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845627+0000","last_change":"2024-04-17T19:47:49.870551+0000","last_active":"2024-04-17T19:48:29.845627+0000","last_peered":"2024-04-17T19:48:29.845627+0000","last_clean":"2024-04-17T19:48:29.845627+0000","last_became_active":"2024-04-17T19:47:49.869315+0000","last_became_peered":"2024-04-17T19:47:49.869315+0000","last_unstale":"2024-04-17T19:48:29.845627+0000","last_undegraded":"2024-04-17T19:48:29.845627+0000","last_fullsized":"2024-04-17T19:48:29.845627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:06.341123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839471+0000","last_change":"2024-04-17T19:47:49.880060+0000","last_active":"2024-04-17T19:48:29.839471+0000","last_peered":"2024-04-17T19:48:29.839471+0000","last_clean":"2024-04-17T19:48:29.839471+0000","last_became_active":"2024-04-17T19:47:49.879711+0000","last_became_peered":"2024-04-17T19:47:49.879711+0000","last_unstale":"2024-04-17T19:48:29.839471+0000","last_undegraded":"2024-04-17T19:48:29.839471+0000","last_fullsized":"2024-04-17T19:48:29.839471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:41:25.628870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.849283+0000","last_change":"2024-04-17T19:47:49.895452+0000","last_active":"2024-04-17T19:48:29.849283+0000","last_peered":"2024-04-17T19:48:29.849283+0000","last_clean":"2024-04-17T19:48:29.849283+0000","last_became_active":"2024-04-17T19:47:49.894684+0000","last_became_peered":"2024-04-17T19:47:49.894684+0000","last_unstale":"2024-04-17T19:48:29.849283+0000","last_undegraded":"2024-04-17T19:48:29.849283+0000","last_fullsized":"2024-04-17T19:48:29.849283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:15:55.224238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840156+0000","last_change":"2024-04-17T19:47:49.886902+0000","last_active":"2024-04-17T19:48:29.840156+0000","last_peered":"2024-04-17T19:48:29.840156+0000","last_clean":"2024-04-17T19:48:29.840156+0000","last_became_active":"2024-04-17T19:47:49.885451+0000","last_became_peered":"2024-04-17T19:47:49.885451+0000","last_unstale":"2024-04-17T19:48:29.840156+0000","last_undegraded":"2024-04-17T19:48:29.840156+0000","last_fullsized":"2024-04-17T19:48:29.840156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:12:09.196610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838401+0000","last_change":"2024-04-17T19:47:49.858908+0000","last_active":"2024-04-17T19:48:29.838401+0000","last_peered":"2024-04-17T19:48:29.838401+0000","last_clean":"2024-04-17T19:48:29.838401+0000","last_became_active":"2024-04-17T19:47:49.858457+0000","last_became_peered":"2024-04-17T19:47:49.858457+0000","last_unstale":"2024-04-17T19:48:29.838401+0000","last_undegraded":"2024-04-17T19:48:29.838401+0000","last_fullsized":"2024-04-17T19:48:29.838401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:09.467859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162134+0000","last_change":"2024-04-17T19:47:49.856371+0000","last_active":"2024-04-17T19:48:30.162134+0000","last_peered":"2024-04-17T19:48:30.162134+0000","last_clean":"2024-04-17T19:48:30.162134+0000","last_became_active":"2024-04-17T19:47:49.856062+0000","last_became_peered":"2024-04-17T19:47:49.856062+0000","last_unstale":"2024-04-17T19:48:30.162134+0000","last_undegraded":"2024-04-17T19:48:30.162134+0000","last_fullsized":"2024-04-17T19:48:30.162134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:39:54.502619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834338+0000","last_change":"2024-04-17T19:47:49.873350+0000","last_active":"2024-04-17T19:48:29.834338+0000","last_peered":"2024-04-17T19:48:29.834338+0000","last_clean":"2024-04-17T19:48:29.834338+0000","last_became_active":"2024-04-17T19:47:49.873019+0000","last_became_peered":"2024-04-17T19:47:49.873019+0000","last_unstale":"2024-04-17T19:48:29.834338+0000","last_undegraded":"2024-04-17T19:48:29.834338+0000","last_fullsized":"2024-04-17T19:48:29.834338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:06:58.641727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.851454+0000","last_change":"2024-04-17T19:47:49.891551+0000","last_active":"2024-04-17T19:48:29.851454+0000","last_peered":"2024-04-17T19:48:29.851454+0000","last_clean":"2024-04-17T19:48:29.851454+0000","last_became_active":"2024-04-17T19:47:49.890132+0000","last_became_peered":"2024-04-17T19:47:49.890132+0000","last_unstale":"2024-04-17T19:48:29.851454+0000","last_undegraded":"2024-04-17T19:48:29.851454+0000","last_fullsized":"2024-04-17T19:48:29.851454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:26:22.971489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845078+0000","last_change":"2024-04-17T19:47:49.903854+0000","last_active":"2024-04-17T19:48:29.845078+0000","last_peered":"2024-04-17T19:48:29.845078+0000","last_clean":"2024-04-17T19:48:29.845078+0000","last_became_active":"2024-04-17T19:47:49.903178+0000","last_became_peered":"2024-04-17T19:47:49.903178+0000","last_unstale":"2024-04-17T19:48:29.845078+0000","last_undegraded":"2024-04-17T19:48:29.845078+0000","last_fullsized":"2024-04-17T19:48:29.845078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:24:52.699532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845192+0000","last_change":"2024-04-17T19:47:49.900848+0000","last_active":"2024-04-17T19:48:29.845192+0000","last_peered":"2024-04-17T19:48:29.845192+0000","last_clean":"2024-04-17T19:48:29.845192+0000","last_became_active":"2024-04-17T19:47:49.900052+0000","last_became_peered":"2024-04-17T19:47:49.900052+0000","last_unstale":"2024-04-17T19:48:29.845192+0000","last_undegraded":"2024-04-17T19:48:29.845192+0000","last_fullsized":"2024-04-17T19:48:29.845192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:28:41.115173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.846967+0000","last_change":"2024-04-17T19:47:49.906531+0000","last_active":"2024-04-17T19:48:29.846967+0000","last_peered":"2024-04-17T19:48:29.846967+0000","last_clean":"2024-04-17T19:48:29.846967+0000","last_became_active":"2024-04-17T19:47:49.905217+0000","last_became_peered":"2024-04-17T19:47:49.905217+0000","last_unstale":"2024-04-17T19:48:29.846967+0000","last_undegraded":"2024-04-17T19:48:29.846967+0000","last_fullsized":"2024-04-17T19:48:29.846967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:22.078473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.833993+0000","last_change":"2024-04-17T19:47:49.864577+0000","last_active":"2024-04-17T19:48:29.833993+0000","last_peered":"2024-04-17T19:48:29.833993+0000","last_clean":"2024-04-17T19:48:29.833993+0000","last_became_active":"2024-04-17T19:47:49.864251+0000","last_became_peered":"2024-04-17T19:47:49.864251+0000","last_unstale":"2024-04-17T19:48:29.833993+0000","last_undegraded":"2024-04-17T19:48:29.833993+0000","last_fullsized":"2024-04-17T19:48:29.833993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:32:10.329543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843153+0000","last_change":"2024-04-17T19:47:49.869245+0000","last_active":"2024-04-17T19:48:29.843153+0000","last_peered":"2024-04-17T19:48:29.843153+0000","last_clean":"2024-04-17T19:48:29.843153+0000","last_became_active":"2024-04-17T19:47:49.868156+0000","last_became_peered":"2024-04-17T19:47:49.868156+0000","last_unstale":"2024-04-17T19:48:29.843153+0000","last_undegraded":"2024-04-17T19:48:29.843153+0000","last_fullsized":"2024-04-17T19:48:29.843153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:58:39.814933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840460+0000","last_change":"2024-04-17T19:47:49.907573+0000","last_active":"2024-04-17T19:48:29.840460+0000","last_peered":"2024-04-17T19:48:29.840460+0000","last_clean":"2024-04-17T19:48:29.840460+0000","last_became_active":"2024-04-17T19:47:49.897308+0000","last_became_peered":"2024-04-17T19:47:49.897308+0000","last_unstale":"2024-04-17T19:48:29.840460+0000","last_undegraded":"2024-04-17T19:48:29.840460+0000","last_fullsized":"2024-04-17T19:48:29.840460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:02:03.101317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843871+0000","last_change":"2024-04-17T19:47:49.876052+0000","last_active":"2024-04-17T19:48:29.843871+0000","last_peered":"2024-04-17T19:48:29.843871+0000","last_clean":"2024-04-17T19:48:29.843871+0000","last_became_active":"2024-04-17T19:47:49.875008+0000","last_became_peered":"2024-04-17T19:47:49.875008+0000","last_unstale":"2024-04-17T19:48:29.843871+0000","last_undegraded":"2024-04-17T19:48:29.843871+0000","last_fullsized":"2024-04-17T19:48:29.843871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:36:14.486150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.847790+0000","last_change":"2024-04-17T19:47:49.906840+0000","last_active":"2024-04-17T19:48:29.847790+0000","last_peered":"2024-04-17T19:48:29.847790+0000","last_clean":"2024-04-17T19:48:29.847790+0000","last_became_active":"2024-04-17T19:47:49.905703+0000","last_became_peered":"2024-04-17T19:47:49.905703+0000","last_unstale":"2024-04-17T19:48:29.847790+0000","last_undegraded":"2024-04-17T19:48:29.847790+0000","last_fullsized":"2024-04-17T19:48:29.847790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:28:26.686472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162905+0000","last_change":"2024-04-17T19:47:49.869425+0000","last_active":"2024-04-17T19:48:30.162905+0000","last_peered":"2024-04-17T19:48:30.162905+0000","last_clean":"2024-04-17T19:48:30.162905+0000","last_became_active":"2024-04-17T19:47:49.868758+0000","last_became_peered":"2024-04-17T19:47:49.868758+0000","last_unstale":"2024-04-17T19:48:30.162905+0000","last_undegraded":"2024-04-17T19:48:30.162905+0000","last_fullsized":"2024-04-17T19:48:30.162905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:57:27.214150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839586+0000","last_change":"2024-04-17T19:47:49.856309+0000","last_active":"2024-04-17T19:48:29.839586+0000","last_peered":"2024-04-17T19:48:29.839586+0000","last_clean":"2024-04-17T19:48:29.839586+0000","last_became_active":"2024-04-17T19:47:49.856026+0000","last_became_peered":"2024-04-17T19:47:49.856026+0000","last_unstale":"2024-04-17T19:48:29.839586+0000","last_undegraded":"2024-04-17T19:48:29.839586+0000","last_fullsized":"2024-04-17T19:48:29.839586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:51:57.963637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845128+0000","last_change":"2024-04-17T19:47:49.890343+0000","last_active":"2024-04-17T19:48:29.845128+0000","last_peered":"2024-04-17T19:48:29.845128+0000","last_clean":"2024-04-17T19:48:29.845128+0000","last_became_active":"2024-04-17T19:47:49.880566+0000","last_became_peered":"2024-04-17T19:47:49.880566+0000","last_unstale":"2024-04-17T19:48:29.845128+0000","last_undegraded":"2024-04-17T19:48:29.845128+0000","last_fullsized":"2024-04-17T19:48:29.845128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:19:29.359024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10522,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":210,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":21044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":184,"ondisk_log_size":184,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":909312,"data_stored":706837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":307200,"data_stored":119259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":327680,"data_stored":118301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":303104,"data_stored":115699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31480,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340360,"statfs":{"total":96636764160,"available":96604528640,"internally_reserved":0,"allocated":311296,"data_stored":119293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31480,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340360,"statfs":{"total":96636764160,"available":96604528640,"internally_reserved":0,"allocated":311296,"data_stored":115831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31476,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340364,"statfs":{"total":96636764160,"available":96604532736,"internally_reserved":0,"allocated":307200,"data_stored":115901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":901120,"data_stored":705611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T19:48:48.734 INFO:tasks.mds_thrash.ceph_manager:clean! 2024-04-17T19:48:48.734 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T19:48:48.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.932+0000 7efd1ba00640 1 Processor -- start 2024-04-17T19:48:48.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.932+0000 7efd1ba00640 1 -- start start 2024-04-17T19:48:48.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.936+0000 7efd1ba00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efd140cc8c0 0x7efd140cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.936+0000 7efd1ba00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7efd14159390 0x7efd141597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.936+0000 7efd1ba00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efd140ca330 0x7efd140ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.936+0000 7efd1ba00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7efd140c88b0 con 0x7efd140ca330 2024-04-17T19:48:48.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.936+0000 7efd1ba00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7efd140c6ce0 con 0x7efd14159390 2024-04-17T19:48:48.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.936+0000 7efd1ba00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7efd140c6fc0 con 0x7efd140cc8c0 2024-04-17T19:48:48.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.936+0000 7efd19a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efd140ca330 0x7efd140ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.936+0000 7efd19a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efd140ca330 0x7efd140ca740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48452/0 (socket says 172.21.15.29:48452) 2024-04-17T19:48:48.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.936+0000 7efd19a00640 1 -- 172.21.15.29:0/2604179456 learned_addr learned my addr 172.21.15.29:0/2604179456 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:48.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.936+0000 7efd13e00640 1 --2- 172.21.15.29:0/2604179456 >> v2:172.21.15.37:3300/0 conn(0x7efd14159390 0x7efd141597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.936+0000 7efd19000640 1 --2- 172.21.15.29:0/2604179456 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efd140cc8c0 0x7efd140cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.936+0000 7efd19a00640 1 -- 172.21.15.29:0/2604179456 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efd140cc8c0 msgr2=0x7efd140cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:48.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.936+0000 7efd19a00640 1 --2- 172.21.15.29:0/2604179456 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efd140cc8c0 0x7efd140cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.936+0000 7efd19a00640 1 -- 172.21.15.29:0/2604179456 >> v2:172.21.15.37:3300/0 conn(0x7efd14159390 msgr2=0x7efd141597a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:48.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.936+0000 7efd19a00640 1 --2- 172.21.15.29:0/2604179456 >> v2:172.21.15.37:3300/0 conn(0x7efd14159390 0x7efd141597a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.936+0000 7efd19a00640 1 -- 172.21.15.29:0/2604179456 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd140c72a0 con 0x7efd140ca330 2024-04-17T19:48:48.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.936+0000 7efd19a00640 1 --2- 172.21.15.29:0/2604179456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd140ca330 0x7efd140ca740 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7efd140ce400 tx=0x7efd0c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dcc2f29192a4be7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:48.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.940+0000 7efd13400640 1 -- 172.21.15.29:0/2604179456 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd0c0a5670 con 0x7efd140ca330 2024-04-17T19:48:48.927 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.940+0000 7efd13400640 1 -- 172.21.15.29:0/2604179456 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd0c0a5810 con 0x7efd140ca330 2024-04-17T19:48:48.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.940+0000 7efd13400640 1 -- 172.21.15.29:0/2604179456 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd0c0a5be0 con 0x7efd140ca330 2024-04-17T19:48:48.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.940+0000 7efd1ba00640 1 -- 172.21.15.29:0/2604179456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd140ca330 msgr2=0x7efd140ca740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:48.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.940+0000 7efd1ba00640 1 --2- 172.21.15.29:0/2604179456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd140ca330 0x7efd140ca740 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7efd140ce400 tx=0x7efd0c01c9f0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.940+0000 7efd1ba00640 1 -- 172.21.15.29:0/2604179456 shutdown_connections 2024-04-17T19:48:48.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.940+0000 7efd1ba00640 1 --2- 172.21.15.29:0/2604179456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd140ca330 0x7efd140ca740 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.940+0000 7efd1ba00640 1 --2- 172.21.15.29:0/2604179456 >> v2:172.21.15.37:3300/0 conn(0x7efd14159390 0x7efd141597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.928 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.940+0000 7efd1ba00640 1 --2- 172.21.15.29:0/2604179456 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efd140cc8c0 0x7efd140cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.940+0000 7efd1ba00640 1 -- 172.21.15.29:0/2604179456 >> 172.21.15.29:0/2604179456 conn(0x7efd1406fde0 msgr2=0x7efd140701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:48.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.940+0000 7efd1ba00640 1 -- 172.21.15.29:0/2604179456 shutdown_connections 2024-04-17T19:48:48.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.940+0000 7efd1ba00640 1 -- 172.21.15.29:0/2604179456 wait complete. 2024-04-17T19:48:48.930 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.940+0000 7efd1ba00640 1 Processor -- start 2024-04-17T19:48:48.931 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.940+0000 7efd1ba00640 1 -- start start 2024-04-17T19:48:48.932 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd1ba00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efd140ca330 0x7efd1415a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd1ba00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd140cc8c0 0x7efd1413b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd1ba00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efd14159390 0x7efd14151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd1ba00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efd1415ca60 con 0x7efd140cc8c0 2024-04-17T19:48:48.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd1ba00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7efd140c6a90 con 0x7efd140ca330 2024-04-17T19:48:48.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd1ba00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7efd1415cbe0 con 0x7efd14159390 2024-04-17T19:48:48.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd13e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd140cc8c0 0x7efd1413b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd13e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd140cc8c0 0x7efd1413b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48458/0 (socket says 172.21.15.29:48458) 2024-04-17T19:48:48.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd13e00640 1 -- 172.21.15.29:0/2992208611 learned_addr learned my addr 172.21.15.29:0/2992208611 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:48.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd19000640 1 --2- 172.21.15.29:0/2992208611 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efd140ca330 0x7efd1415a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd19a00640 1 --2- 172.21.15.29:0/2992208611 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efd14159390 0x7efd14151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd13e00640 1 -- 172.21.15.29:0/2992208611 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efd14159390 msgr2=0x7efd14151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:48.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd13e00640 1 --2- 172.21.15.29:0/2992208611 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efd14159390 0x7efd14151bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd13e00640 1 -- 172.21.15.29:0/2992208611 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efd140ca330 msgr2=0x7efd1415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:48.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd13e00640 1 --2- 172.21.15.29:0/2992208611 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efd140ca330 0x7efd1415a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:48.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd13e00640 1 -- 172.21.15.29:0/2992208611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd1415aa20 con 0x7efd140cc8c0 2024-04-17T19:48:48.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd13e00640 1 --2- 172.21.15.29:0/2992208611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd140cc8c0 0x7efd1413b680 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efd0000d5d0 tx=0x7efd0000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:48.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd10c00640 1 -- 172.21.15.29:0/2992208611 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd00004240 con 0x7efd140cc8c0 2024-04-17T19:48:48.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd10c00640 1 -- 172.21.15.29:0/2992208611 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd000043e0 con 0x7efd140cc8c0 2024-04-17T19:48:48.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd10c00640 1 -- 172.21.15.29:0/2992208611 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd00003840 con 0x7efd140cc8c0 2024-04-17T19:48:48.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd1ba00640 1 -- 172.21.15.29:0/2992208611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd141f4ef0 con 0x7efd140cc8c0 2024-04-17T19:48:48.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd1ba00640 1 -- 172.21.15.29:0/2992208611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd14140da0 con 0x7efd140cc8c0 2024-04-17T19:48:48.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.944+0000 7efd1ba00640 1 -- 172.21.15.29:0/2992208611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd14069b90 con 0x7efd140cc8c0 2024-04-17T19:48:48.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.948+0000 7efd10c00640 1 -- 172.21.15.29:0/2992208611 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efd000039e0 con 0x7efd140cc8c0 2024-04-17T19:48:48.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.948+0000 7efd10c00640 1 --2- 172.21.15.29:0/2992208611 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efcf4068380 0x7efcf406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:48.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.948+0000 7efd10c00640 1 -- 172.21.15.29:0/2992208611 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7efd00087210 con 0x7efd140cc8c0 2024-04-17T19:48:48.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.948+0000 7efd19000640 1 --2- 172.21.15.29:0/2992208611 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efcf4068380 0x7efcf406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:48.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.948+0000 7efd19000640 1 --2- 172.21.15.29:0/2992208611 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efcf4068380 0x7efcf406a890 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7efd0800b990 tx=0x7efd08008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:48.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:48.952+0000 7efd10c00640 1 -- 172.21.15.29:0/2992208611 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7efd0001c360 con 0x7efd140cc8c0 2024-04-17T19:48:49.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.112+0000 7efd1ba00640 1 -- 172.21.15.29:0/2992208611 --> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efd14159e20 con 0x7efcf4068380 2024-04-17T19:48:49.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.136+0000 7efd10c00640 1 -- 172.21.15.29:0/2992208611 <== mgr.4105 v2:172.21.15.37:6832/1588862286 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332649 (secure 0 0 0) 0x7efd14159e20 con 0x7efcf4068380 2024-04-17T19:48:49.129 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:49.131 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-04-17T19:48:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.140+0000 7efd1ba00640 1 -- 172.21.15.29:0/2992208611 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efcf4068380 msgr2=0x7efcf406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.140+0000 7efd1ba00640 1 --2- 172.21.15.29:0/2992208611 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efcf4068380 0x7efcf406a890 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7efd0800b990 tx=0x7efd08008040 comp rx=0 tx=0).stop 2024-04-17T19:48:49.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.140+0000 7efd1ba00640 1 -- 172.21.15.29:0/2992208611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd140cc8c0 msgr2=0x7efd1413b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.140+0000 7efd1ba00640 1 --2- 172.21.15.29:0/2992208611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd140cc8c0 0x7efd1413b680 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7efd0000d5d0 tx=0x7efd0000da90 comp rx=0 tx=0).stop 2024-04-17T19:48:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.140+0000 7efd1ba00640 1 -- 172.21.15.29:0/2992208611 shutdown_connections 2024-04-17T19:48:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.140+0000 7efd1ba00640 1 --2- 172.21.15.29:0/2992208611 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efcf4068380 0x7efcf406a890 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.140+0000 7efd1ba00640 1 --2- 172.21.15.29:0/2992208611 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efd14159390 0x7efd14151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.140+0000 7efd1ba00640 1 --2- 172.21.15.29:0/2992208611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efd140cc8c0 0x7efd1413b680 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.140+0000 7efd1ba00640 1 --2- 172.21.15.29:0/2992208611 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efd140ca330 0x7efd1415a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.140+0000 7efd1ba00640 1 -- 172.21.15.29:0/2992208611 >> 172.21.15.29:0/2992208611 conn(0x7efd1406fde0 msgr2=0x7efd140cb920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.140+0000 7efd1ba00640 1 -- 172.21.15.29:0/2992208611 shutdown_connections 2024-04-17T19:48:49.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.140+0000 7efd1ba00640 1 -- 172.21.15.29:0/2992208611 wait complete. 2024-04-17T19:48:49.150 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":56,"stamp":"2024-04-17T19:48:48.651809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600890,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":369,"num_write_kb":2484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":301,"ondisk_log_size":301,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240520,"kb_used_data":3592,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754734200,"statfs":{"total":773094113280,"available":772847820800,"internally_reserved":0,"allocated":3678208,"data_stored":2116732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":258,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005969"},"pg_stats":[{"pgid":"2.1f","version":"26'13","reported_seq":39,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.876148+0000","last_change":"2024-04-17T19:47:48.885235+0000","last_active":"2024-04-17T19:48:33.876148+0000","last_peered":"2024-04-17T19:48:33.876148+0000","last_clean":"2024-04-17T19:48:33.876148+0000","last_became_active":"2024-04-17T19:47:48.884844+0000","last_became_peered":"2024-04-17T19:47:48.884844+0000","last_unstale":"2024-04-17T19:48:33.876148+0000","last_undegraded":"2024-04-17T19:48:33.876148+0000","last_fullsized":"2024-04-17T19:48:33.876148+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:01:52.854763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834602+0000","last_change":"2024-04-17T19:47:49.872314+0000","last_active":"2024-04-17T19:48:29.834602+0000","last_peered":"2024-04-17T19:48:29.834602+0000","last_clean":"2024-04-17T19:48:29.834602+0000","last_became_active":"2024-04-17T19:47:49.871990+0000","last_became_peered":"2024-04-17T19:47:49.871990+0000","last_unstale":"2024-04-17T19:48:29.834602+0000","last_undegraded":"2024-04-17T19:48:29.834602+0000","last_fullsized":"2024-04-17T19:48:29.834602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:12:09.196610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.833967+0000","last_change":"2024-04-17T19:47:53.929840+0000","last_active":"2024-04-17T19:48:29.833967+0000","last_peered":"2024-04-17T19:48:29.833967+0000","last_clean":"2024-04-17T19:48:29.833967+0000","last_became_active":"2024-04-17T19:47:53.928272+0000","last_became_peered":"2024-04-17T19:47:53.928272+0000","last_unstale":"2024-04-17T19:48:29.833967+0000","last_undegraded":"2024-04-17T19:48:29.833967+0000","last_fullsized":"2024-04-17T19:48:29.833967+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:27:29.734327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"24'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.835623+0000","last_change":"2024-04-17T19:47:48.862315+0000","last_active":"2024-04-17T19:48:29.835623+0000","last_peered":"2024-04-17T19:48:29.835623+0000","last_clean":"2024-04-17T19:48:29.835623+0000","last_became_active":"2024-04-17T19:47:48.860408+0000","last_became_peered":"2024-04-17T19:47:48.860408+0000","last_unstale":"2024-04-17T19:48:29.835623+0000","last_undegraded":"2024-04-17T19:48:29.835623+0000","last_fullsized":"2024-04-17T19:48:29.835623+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:10:26.336063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162315+0000","last_change":"2024-04-17T19:47:49.902423+0000","last_active":"2024-04-17T19:48:30.162315+0000","last_peered":"2024-04-17T19:48:30.162315+0000","last_clean":"2024-04-17T19:48:30.162315+0000","last_became_active":"2024-04-17T19:47:49.902096+0000","last_became_peered":"2024-04-17T19:47:49.902096+0000","last_unstale":"2024-04-17T19:48:30.162315+0000","last_undegraded":"2024-04-17T19:48:30.162315+0000","last_fullsized":"2024-04-17T19:48:30.162315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:40.805247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.849677+0000","last_change":"2024-04-17T19:47:53.956851+0000","last_active":"2024-04-17T19:48:29.849677+0000","last_peered":"2024-04-17T19:48:29.849677+0000","last_clean":"2024-04-17T19:48:29.849677+0000","last_became_active":"2024-04-17T19:47:53.954197+0000","last_became_peered":"2024-04-17T19:47:53.954197+0000","last_unstale":"2024-04-17T19:48:29.849677+0000","last_undegraded":"2024-04-17T19:48:29.849677+0000","last_fullsized":"2024-04-17T19:48:29.849677+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:16:18.379831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"26'12","reported_seq":38,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.877776+0000","last_change":"2024-04-17T19:47:48.893969+0000","last_active":"2024-04-17T19:48:33.877776+0000","last_peered":"2024-04-17T19:48:33.877776+0000","last_clean":"2024-04-17T19:48:33.877776+0000","last_became_active":"2024-04-17T19:47:48.893229+0000","last_became_peered":"2024-04-17T19:47:48.893229+0000","last_unstale":"2024-04-17T19:48:33.877776+0000","last_undegraded":"2024-04-17T19:48:33.877776+0000","last_fullsized":"2024-04-17T19:48:33.877776+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:38:42.472634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.848206+0000","last_change":"2024-04-17T19:47:49.897987+0000","last_active":"2024-04-17T19:48:29.848206+0000","last_peered":"2024-04-17T19:48:29.848206+0000","last_clean":"2024-04-17T19:48:29.848206+0000","last_became_active":"2024-04-17T19:47:49.897096+0000","last_became_peered":"2024-04-17T19:47:49.897096+0000","last_unstale":"2024-04-17T19:48:29.848206+0000","last_undegraded":"2024-04-17T19:48:29.848206+0000","last_fullsized":"2024-04-17T19:48:29.848206+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:36:36.856454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.850365+0000","last_change":"2024-04-17T19:47:53.930784+0000","last_active":"2024-04-17T19:48:29.850365+0000","last_peered":"2024-04-17T19:48:29.850365+0000","last_clean":"2024-04-17T19:48:29.850365+0000","last_became_active":"2024-04-17T19:47:53.928281+0000","last_became_peered":"2024-04-17T19:47:53.928281+0000","last_unstale":"2024-04-17T19:48:29.850365+0000","last_undegraded":"2024-04-17T19:48:29.850365+0000","last_fullsized":"2024-04-17T19:48:29.850365+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:15:55.064622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.876428+0000","last_change":"2024-04-17T19:47:48.893184+0000","last_active":"2024-04-17T19:48:33.876428+0000","last_peered":"2024-04-17T19:48:33.876428+0000","last_clean":"2024-04-17T19:48:33.876428+0000","last_became_active":"2024-04-17T19:47:48.892375+0000","last_became_peered":"2024-04-17T19:47:48.892375+0000","last_unstale":"2024-04-17T19:48:33.876428+0000","last_undegraded":"2024-04-17T19:48:33.876428+0000","last_fullsized":"2024-04-17T19:48:33.876428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:56:01.211666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.848245+0000","last_change":"2024-04-17T19:47:49.896040+0000","last_active":"2024-04-17T19:48:29.848245+0000","last_peered":"2024-04-17T19:48:29.848245+0000","last_clean":"2024-04-17T19:48:29.848245+0000","last_became_active":"2024-04-17T19:47:49.894779+0000","last_became_peered":"2024-04-17T19:47:49.894779+0000","last_unstale":"2024-04-17T19:48:29.848245+0000","last_undegraded":"2024-04-17T19:48:29.848245+0000","last_fullsized":"2024-04-17T19:48:29.848245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:30:17.930203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.852223+0000","last_change":"2024-04-17T19:47:53.930402+0000","last_active":"2024-04-17T19:48:29.852223+0000","last_peered":"2024-04-17T19:48:29.852223+0000","last_clean":"2024-04-17T19:48:29.852223+0000","last_became_active":"2024-04-17T19:47:53.929166+0000","last_became_peered":"2024-04-17T19:47:53.929166+0000","last_unstale":"2024-04-17T19:48:29.852223+0000","last_undegraded":"2024-04-17T19:48:29.852223+0000","last_fullsized":"2024-04-17T19:48:29.852223+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:34:08.282697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843243+0000","last_change":"2024-04-17T19:47:49.900142+0000","last_active":"2024-04-17T19:48:29.843243+0000","last_peered":"2024-04-17T19:48:29.843243+0000","last_clean":"2024-04-17T19:48:29.843243+0000","last_became_active":"2024-04-17T19:47:49.899365+0000","last_became_peered":"2024-04-17T19:47:49.899365+0000","last_unstale":"2024-04-17T19:48:29.843243+0000","last_undegraded":"2024-04-17T19:48:29.843243+0000","last_fullsized":"2024-04-17T19:48:29.843243+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:34:49.616539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"24'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.844328+0000","last_change":"2024-04-17T19:47:48.891513+0000","last_active":"2024-04-17T19:48:29.844328+0000","last_peered":"2024-04-17T19:48:29.844328+0000","last_clean":"2024-04-17T19:48:29.844328+0000","last_became_active":"2024-04-17T19:47:48.886561+0000","last_became_peered":"2024-04-17T19:47:48.886561+0000","last_unstale":"2024-04-17T19:48:29.844328+0000","last_undegraded":"2024-04-17T19:48:29.844328+0000","last_fullsized":"2024-04-17T19:48:29.844328+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:59:14.039700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.167531+0000","last_change":"2024-04-17T19:48:30.167531+0000","last_active":"2024-04-17T19:48:30.167531+0000","last_peered":"2024-04-17T19:48:30.167531+0000","last_clean":"2024-04-17T19:48:30.167531+0000","last_became_active":"2024-04-17T19:48:30.165421+0000","last_became_peered":"2024-04-17T19:48:30.165421+0000","last_unstale":"2024-04-17T19:48:30.167531+0000","last_undegraded":"2024-04-17T19:48:30.167531+0000","last_fullsized":"2024-04-17T19:48:30.167531+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:52:16.429658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,5],"acting":[4,0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834455+0000","last_change":"2024-04-17T19:47:49.852183+0000","last_active":"2024-04-17T19:48:29.834455+0000","last_peered":"2024-04-17T19:48:29.834455+0000","last_clean":"2024-04-17T19:48:29.834455+0000","last_became_active":"2024-04-17T19:47:49.851765+0000","last_became_peered":"2024-04-17T19:47:49.851765+0000","last_unstale":"2024-04-17T19:48:29.834455+0000","last_undegraded":"2024-04-17T19:48:29.834455+0000","last_fullsized":"2024-04-17T19:48:29.834455+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:02:03.101317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.841041+0000","last_change":"2024-04-17T19:47:48.862865+0000","last_active":"2024-04-17T19:48:29.841041+0000","last_peered":"2024-04-17T19:48:29.841041+0000","last_clean":"2024-04-17T19:48:29.841041+0000","last_became_active":"2024-04-17T19:47:48.862524+0000","last_became_peered":"2024-04-17T19:47:48.862524+0000","last_unstale":"2024-04-17T19:48:29.841041+0000","last_undegraded":"2024-04-17T19:48:29.841041+0000","last_fullsized":"2024-04-17T19:48:29.841041+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:01:21.080029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.850650+0000","last_change":"2024-04-17T19:47:53.943674+0000","last_active":"2024-04-17T19:48:29.850650+0000","last_peered":"2024-04-17T19:48:29.850650+0000","last_clean":"2024-04-17T19:48:29.850650+0000","last_became_active":"2024-04-17T19:47:53.942549+0000","last_became_peered":"2024-04-17T19:47:53.942549+0000","last_unstale":"2024-04-17T19:48:29.850650+0000","last_undegraded":"2024-04-17T19:48:29.850650+0000","last_fullsized":"2024-04-17T19:48:29.850650+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:00:42.300738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834364+0000","last_change":"2024-04-17T19:47:49.862312+0000","last_active":"2024-04-17T19:48:29.834364+0000","last_peered":"2024-04-17T19:48:29.834364+0000","last_clean":"2024-04-17T19:48:29.834364+0000","last_became_active":"2024-04-17T19:47:49.860664+0000","last_became_peered":"2024-04-17T19:47:49.860664+0000","last_unstale":"2024-04-17T19:48:29.834364+0000","last_undegraded":"2024-04-17T19:48:29.834364+0000","last_fullsized":"2024-04-17T19:48:29.834364+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:07:47.676973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.874890+0000","last_change":"2024-04-17T19:47:48.870638+0000","last_active":"2024-04-17T19:48:33.874890+0000","last_peered":"2024-04-17T19:48:33.874890+0000","last_clean":"2024-04-17T19:48:33.874890+0000","last_became_active":"2024-04-17T19:47:48.870353+0000","last_became_peered":"2024-04-17T19:47:48.870353+0000","last_unstale":"2024-04-17T19:48:33.874890+0000","last_undegraded":"2024-04-17T19:48:33.874890+0000","last_fullsized":"2024-04-17T19:48:33.874890+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:12:19.927824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.861156+0000","last_change":"2024-04-17T19:48:29.857425+0000","last_active":"2024-04-17T19:48:29.861156+0000","last_peered":"2024-04-17T19:48:29.861156+0000","last_clean":"2024-04-17T19:48:29.861156+0000","last_became_active":"2024-04-17T19:48:29.856886+0000","last_became_peered":"2024-04-17T19:48:29.856886+0000","last_unstale":"2024-04-17T19:48:29.861156+0000","last_undegraded":"2024-04-17T19:48:29.861156+0000","last_fullsized":"2024-04-17T19:48:29.861156+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:17:41.501101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4,7],"acting":[2,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.854191+0000","last_change":"2024-04-17T19:47:49.888118+0000","last_active":"2024-04-17T19:48:29.854191+0000","last_peered":"2024-04-17T19:48:29.854191+0000","last_clean":"2024-04-17T19:48:29.854191+0000","last_became_active":"2024-04-17T19:47:49.884037+0000","last_became_peered":"2024-04-17T19:47:49.884037+0000","last_unstale":"2024-04-17T19:48:29.854191+0000","last_undegraded":"2024-04-17T19:48:29.854191+0000","last_fullsized":"2024-04-17T19:48:29.854191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:44:27.180380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"24'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843109+0000","last_change":"2024-04-17T19:47:48.883686+0000","last_active":"2024-04-17T19:48:29.843109+0000","last_peered":"2024-04-17T19:48:29.843109+0000","last_clean":"2024-04-17T19:48:29.843109+0000","last_became_active":"2024-04-17T19:47:48.882375+0000","last_became_peered":"2024-04-17T19:47:48.882375+0000","last_unstale":"2024-04-17T19:48:29.843109+0000","last_undegraded":"2024-04-17T19:48:29.843109+0000","last_fullsized":"2024-04-17T19:48:29.843109+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:38:31.225374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.851300+0000","last_change":"2024-04-17T19:47:53.935396+0000","last_active":"2024-04-17T19:48:29.851300+0000","last_peered":"2024-04-17T19:48:29.851300+0000","last_clean":"2024-04-17T19:48:29.851300+0000","last_became_active":"2024-04-17T19:47:53.934094+0000","last_became_peered":"2024-04-17T19:47:53.934094+0000","last_unstale":"2024-04-17T19:48:29.851300+0000","last_undegraded":"2024-04-17T19:48:29.851300+0000","last_fullsized":"2024-04-17T19:48:29.851300+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:58.334357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162117+0000","last_change":"2024-04-17T19:47:49.870297+0000","last_active":"2024-04-17T19:48:30.162117+0000","last_peered":"2024-04-17T19:48:30.162117+0000","last_clean":"2024-04-17T19:48:30.162117+0000","last_became_active":"2024-04-17T19:47:49.870113+0000","last_became_peered":"2024-04-17T19:47:49.870113+0000","last_unstale":"2024-04-17T19:48:30.162117+0000","last_undegraded":"2024-04-17T19:48:30.162117+0000","last_fullsized":"2024-04-17T19:48:30.162117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:50:54.517273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"26'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.876067+0000","last_change":"2024-04-17T19:47:48.895905+0000","last_active":"2024-04-17T19:48:33.876067+0000","last_peered":"2024-04-17T19:48:33.876067+0000","last_clean":"2024-04-17T19:48:33.876067+0000","last_became_active":"2024-04-17T19:47:48.895263+0000","last_became_peered":"2024-04-17T19:47:48.895263+0000","last_unstale":"2024-04-17T19:48:33.876067+0000","last_undegraded":"2024-04-17T19:48:33.876067+0000","last_fullsized":"2024-04-17T19:48:33.876067+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:07:23.179031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.851454+0000","last_change":"2024-04-17T19:47:53.943286+0000","last_active":"2024-04-17T19:48:29.851454+0000","last_peered":"2024-04-17T19:48:29.851454+0000","last_clean":"2024-04-17T19:48:29.851454+0000","last_became_active":"2024-04-17T19:47:53.942325+0000","last_became_peered":"2024-04-17T19:47:53.942325+0000","last_unstale":"2024-04-17T19:48:29.851454+0000","last_undegraded":"2024-04-17T19:48:29.851454+0000","last_fullsized":"2024-04-17T19:48:29.851454+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:47:18.853991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.835175+0000","last_change":"2024-04-17T19:47:49.872925+0000","last_active":"2024-04-17T19:48:29.835175+0000","last_peered":"2024-04-17T19:48:29.835175+0000","last_clean":"2024-04-17T19:48:29.835175+0000","last_became_active":"2024-04-17T19:47:49.872668+0000","last_became_peered":"2024-04-17T19:47:49.872668+0000","last_unstale":"2024-04-17T19:48:29.835175+0000","last_undegraded":"2024-04-17T19:48:29.835175+0000","last_fullsized":"2024-04-17T19:48:29.835175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:16:21.209634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839696+0000","last_change":"2024-04-17T19:47:48.896669+0000","last_active":"2024-04-17T19:48:29.839696+0000","last_peered":"2024-04-17T19:48:29.839696+0000","last_clean":"2024-04-17T19:48:29.839696+0000","last_became_active":"2024-04-17T19:47:48.896397+0000","last_became_peered":"2024-04-17T19:47:48.896397+0000","last_unstale":"2024-04-17T19:48:29.839696+0000","last_undegraded":"2024-04-17T19:48:29.839696+0000","last_fullsized":"2024-04-17T19:48:29.839696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:07:23.179031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.833842+0000","last_change":"2024-04-17T19:47:53.906261+0000","last_active":"2024-04-17T19:48:29.833842+0000","last_peered":"2024-04-17T19:48:29.833842+0000","last_clean":"2024-04-17T19:48:29.833842+0000","last_became_active":"2024-04-17T19:47:53.903702+0000","last_became_peered":"2024-04-17T19:47:53.903702+0000","last_unstale":"2024-04-17T19:48:29.833842+0000","last_undegraded":"2024-04-17T19:48:29.833842+0000","last_fullsized":"2024-04-17T19:48:29.833842+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:35:53.892917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838432+0000","last_change":"2024-04-17T19:47:49.870272+0000","last_active":"2024-04-17T19:48:29.838432+0000","last_peered":"2024-04-17T19:48:29.838432+0000","last_clean":"2024-04-17T19:48:29.838432+0000","last_became_active":"2024-04-17T19:47:49.869914+0000","last_became_peered":"2024-04-17T19:47:49.869914+0000","last_unstale":"2024-04-17T19:48:29.838432+0000","last_undegraded":"2024-04-17T19:48:29.838432+0000","last_fullsized":"2024-04-17T19:48:29.838432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:44:30.553286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845870+0000","last_change":"2024-04-17T19:47:48.892675+0000","last_active":"2024-04-17T19:48:29.845870+0000","last_peered":"2024-04-17T19:48:29.845870+0000","last_clean":"2024-04-17T19:48:29.845870+0000","last_became_active":"2024-04-17T19:47:48.892057+0000","last_became_peered":"2024-04-17T19:47:48.892057+0000","last_unstale":"2024-04-17T19:48:29.845870+0000","last_undegraded":"2024-04-17T19:48:29.845870+0000","last_fullsized":"2024-04-17T19:48:29.845870+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:18:35.735350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.163029+0000","last_change":"2024-04-17T19:47:53.925135+0000","last_active":"2024-04-17T19:48:30.163029+0000","last_peered":"2024-04-17T19:48:30.163029+0000","last_clean":"2024-04-17T19:48:30.163029+0000","last_became_active":"2024-04-17T19:47:53.924617+0000","last_became_peered":"2024-04-17T19:47:53.924617+0000","last_unstale":"2024-04-17T19:48:30.163029+0000","last_undegraded":"2024-04-17T19:48:30.163029+0000","last_fullsized":"2024-04-17T19:48:30.163029+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:37:02.082648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.844178+0000","last_change":"2024-04-17T19:48:29.842544+0000","last_active":"2024-04-17T19:48:29.844178+0000","last_peered":"2024-04-17T19:48:29.844178+0000","last_clean":"2024-04-17T19:48:29.844178+0000","last_became_active":"2024-04-17T19:48:29.842198+0000","last_became_peered":"2024-04-17T19:48:29.842198+0000","last_unstale":"2024-04-17T19:48:29.844178+0000","last_undegraded":"2024-04-17T19:48:29.844178+0000","last_fullsized":"2024-04-17T19:48:29.844178+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:26:06.151001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"24'1","reported_seq":27,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838549+0000","last_change":"2024-04-17T19:47:48.859247+0000","last_active":"2024-04-17T19:48:29.838549+0000","last_peered":"2024-04-17T19:48:29.838549+0000","last_clean":"2024-04-17T19:48:29.838549+0000","last_became_active":"2024-04-17T19:47:48.858756+0000","last_became_peered":"2024-04-17T19:47:48.858756+0000","last_unstale":"2024-04-17T19:48:29.838549+0000","last_undegraded":"2024-04-17T19:48:29.838549+0000","last_fullsized":"2024-04-17T19:48:29.838549+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:19:56.987957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840601+0000","last_change":"2024-04-17T19:47:53.929952+0000","last_active":"2024-04-17T19:48:29.840601+0000","last_peered":"2024-04-17T19:48:29.840601+0000","last_clean":"2024-04-17T19:48:29.840601+0000","last_became_active":"2024-04-17T19:47:53.929528+0000","last_became_peered":"2024-04-17T19:47:53.929528+0000","last_unstale":"2024-04-17T19:48:29.840601+0000","last_undegraded":"2024-04-17T19:48:29.840601+0000","last_fullsized":"2024-04-17T19:48:29.840601+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:11:38.041301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.853204+0000","last_change":"2024-04-17T19:47:49.874150+0000","last_active":"2024-04-17T19:48:29.853204+0000","last_peered":"2024-04-17T19:48:29.853204+0000","last_clean":"2024-04-17T19:48:29.853204+0000","last_became_active":"2024-04-17T19:47:49.872859+0000","last_became_peered":"2024-04-17T19:47:49.872859+0000","last_unstale":"2024-04-17T19:48:29.853204+0000","last_undegraded":"2024-04-17T19:48:29.853204+0000","last_fullsized":"2024-04-17T19:48:29.853204+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:57:54.666279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.876393+0000","last_change":"2024-04-17T19:47:48.876672+0000","last_active":"2024-04-17T19:48:33.876393+0000","last_peered":"2024-04-17T19:48:33.876393+0000","last_clean":"2024-04-17T19:48:33.876393+0000","last_became_active":"2024-04-17T19:47:48.875263+0000","last_became_peered":"2024-04-17T19:47:48.875263+0000","last_unstale":"2024-04-17T19:48:33.876393+0000","last_undegraded":"2024-04-17T19:48:33.876393+0000","last_fullsized":"2024-04-17T19:48:33.876393+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:45:46.633540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.857338+0000","last_change":"2024-04-17T19:48:29.852166+0000","last_active":"2024-04-17T19:48:29.857338+0000","last_peered":"2024-04-17T19:48:29.857338+0000","last_clean":"2024-04-17T19:48:29.857338+0000","last_became_active":"2024-04-17T19:48:29.850441+0000","last_became_peered":"2024-04-17T19:48:29.850441+0000","last_unstale":"2024-04-17T19:48:29.857338+0000","last_undegraded":"2024-04-17T19:48:29.857338+0000","last_fullsized":"2024-04-17T19:48:29.857338+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:32:25.836924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3,6],"acting":[5,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838297+0000","last_change":"2024-04-17T19:47:49.865020+0000","last_active":"2024-04-17T19:48:29.838297+0000","last_peered":"2024-04-17T19:48:29.838297+0000","last_clean":"2024-04-17T19:48:29.838297+0000","last_became_active":"2024-04-17T19:47:49.864764+0000","last_became_peered":"2024-04-17T19:47:49.864764+0000","last_unstale":"2024-04-17T19:48:29.838297+0000","last_undegraded":"2024-04-17T19:48:29.838297+0000","last_fullsized":"2024-04-17T19:48:29.838297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:45:36.325765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838108+0000","last_change":"2024-04-17T19:47:48.846570+0000","last_active":"2024-04-17T19:48:29.838108+0000","last_peered":"2024-04-17T19:48:29.838108+0000","last_clean":"2024-04-17T19:48:29.838108+0000","last_became_active":"2024-04-17T19:47:48.846175+0000","last_became_peered":"2024-04-17T19:47:48.846175+0000","last_unstale":"2024-04-17T19:48:29.838108+0000","last_undegraded":"2024-04-17T19:48:29.838108+0000","last_fullsized":"2024-04-17T19:48:29.838108+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:20:22.737016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839023+0000","last_change":"2024-04-17T19:47:53.904590+0000","last_active":"2024-04-17T19:48:29.839023+0000","last_peered":"2024-04-17T19:48:29.839023+0000","last_clean":"2024-04-17T19:48:29.839023+0000","last_became_active":"2024-04-17T19:47:53.904026+0000","last_became_peered":"2024-04-17T19:47:53.904026+0000","last_unstale":"2024-04-17T19:48:29.839023+0000","last_undegraded":"2024-04-17T19:48:29.839023+0000","last_fullsized":"2024-04-17T19:48:29.839023+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:16:22.837570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.161964+0000","last_change":"2024-04-17T19:47:49.860010+0000","last_active":"2024-04-17T19:48:30.161964+0000","last_peered":"2024-04-17T19:48:30.161964+0000","last_clean":"2024-04-17T19:48:30.161964+0000","last_became_active":"2024-04-17T19:47:49.859593+0000","last_became_peered":"2024-04-17T19:47:49.859593+0000","last_unstale":"2024-04-17T19:48:30.161964+0000","last_undegraded":"2024-04-17T19:48:30.161964+0000","last_fullsized":"2024-04-17T19:48:30.161964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:33:24.594397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843671+0000","last_change":"2024-04-17T19:47:48.883126+0000","last_active":"2024-04-17T19:48:29.843671+0000","last_peered":"2024-04-17T19:48:29.843671+0000","last_clean":"2024-04-17T19:48:29.843671+0000","last_became_active":"2024-04-17T19:47:48.882173+0000","last_became_peered":"2024-04-17T19:47:48.882173+0000","last_unstale":"2024-04-17T19:48:29.843671+0000","last_undegraded":"2024-04-17T19:48:29.843671+0000","last_fullsized":"2024-04-17T19:48:29.843671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:11:59.395316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.163046+0000","last_change":"2024-04-17T19:47:53.931258+0000","last_active":"2024-04-17T19:48:30.163046+0000","last_peered":"2024-04-17T19:48:30.163046+0000","last_clean":"2024-04-17T19:48:30.163046+0000","last_became_active":"2024-04-17T19:47:53.930773+0000","last_became_peered":"2024-04-17T19:47:53.930773+0000","last_unstale":"2024-04-17T19:48:30.163046+0000","last_undegraded":"2024-04-17T19:48:30.163046+0000","last_fullsized":"2024-04-17T19:48:30.163046+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:10:47.583714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.844178+0000","last_change":"2024-04-17T19:48:29.844178+0000","last_active":"2024-04-17T19:48:29.844178+0000","last_peered":"2024-04-17T19:48:29.844178+0000","last_clean":"2024-04-17T19:48:29.844178+0000","last_became_active":"2024-04-17T19:48:29.843807+0000","last_became_peered":"2024-04-17T19:48:29.843807+0000","last_unstale":"2024-04-17T19:48:29.844178+0000","last_undegraded":"2024-04-17T19:48:29.844178+0000","last_fullsized":"2024-04-17T19:48:29.844178+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:36:08.301346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"26'6","reported_seq":32,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.849667+0000","last_change":"2024-04-17T19:47:48.878592+0000","last_active":"2024-04-17T19:48:29.849667+0000","last_peered":"2024-04-17T19:48:29.849667+0000","last_clean":"2024-04-17T19:48:29.849667+0000","last_became_active":"2024-04-17T19:47:48.877617+0000","last_became_peered":"2024-04-17T19:47:48.877617+0000","last_unstale":"2024-04-17T19:48:29.849667+0000","last_undegraded":"2024-04-17T19:48:29.849667+0000","last_fullsized":"2024-04-17T19:48:29.849667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:30:17.013176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":182,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843673+0000","last_change":"2024-04-17T19:47:38.277798+0000","last_active":"2024-04-17T19:48:29.843673+0000","last_peered":"2024-04-17T19:48:29.843673+0000","last_clean":"2024-04-17T19:48:29.843673+0000","last_became_active":"2024-04-17T19:47:38.274697+0000","last_became_peered":"2024-04-17T19:47:38.274697+0000","last_unstale":"2024-04-17T19:48:29.843673+0000","last_undegraded":"2024-04-17T19:48:29.843673+0000","last_fullsized":"2024-04-17T19:48:29.843673+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:36.897071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:36.897071+0000","last_clean_scrub_stamp":"2024-04-17T19:47:36.897071+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:31:35.366600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.844472+0000","last_change":"2024-04-17T19:47:53.953415+0000","last_active":"2024-04-17T19:48:29.844472+0000","last_peered":"2024-04-17T19:48:29.844472+0000","last_clean":"2024-04-17T19:48:29.844472+0000","last_became_active":"2024-04-17T19:47:53.952644+0000","last_became_peered":"2024-04-17T19:47:53.952644+0000","last_unstale":"2024-04-17T19:48:29.844472+0000","last_undegraded":"2024-04-17T19:48:29.844472+0000","last_fullsized":"2024-04-17T19:48:29.844472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:00:25.923909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.844721+0000","last_change":"2024-04-17T19:47:49.901572+0000","last_active":"2024-04-17T19:48:29.844721+0000","last_peered":"2024-04-17T19:48:29.844721+0000","last_clean":"2024-04-17T19:48:29.844721+0000","last_became_active":"2024-04-17T19:47:49.900707+0000","last_became_peered":"2024-04-17T19:47:49.900707+0000","last_unstale":"2024-04-17T19:48:29.844721+0000","last_undegraded":"2024-04-17T19:48:29.844721+0000","last_fullsized":"2024-04-17T19:48:29.844721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:58:02.897564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"24'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838864+0000","last_change":"2024-04-17T19:47:48.874721+0000","last_active":"2024-04-17T19:48:29.838864+0000","last_peered":"2024-04-17T19:48:29.838864+0000","last_clean":"2024-04-17T19:48:29.838864+0000","last_became_active":"2024-04-17T19:47:48.874364+0000","last_became_peered":"2024-04-17T19:47:48.874364+0000","last_unstale":"2024-04-17T19:48:29.838864+0000","last_undegraded":"2024-04-17T19:48:29.838864+0000","last_fullsized":"2024-04-17T19:48:29.838864+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:47:08.856753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.842743+0000","last_change":"2024-04-17T19:47:53.942791+0000","last_active":"2024-04-17T19:48:29.842743+0000","last_peered":"2024-04-17T19:48:29.842743+0000","last_clean":"2024-04-17T19:48:29.842743+0000","last_became_active":"2024-04-17T19:47:53.941569+0000","last_became_peered":"2024-04-17T19:47:53.941569+0000","last_unstale":"2024-04-17T19:48:29.842743+0000","last_undegraded":"2024-04-17T19:48:29.842743+0000","last_fullsized":"2024-04-17T19:48:29.842743+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:23:35.452056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.848531+0000","last_change":"2024-04-17T19:47:49.895046+0000","last_active":"2024-04-17T19:48:29.848531+0000","last_peered":"2024-04-17T19:48:29.848531+0000","last_clean":"2024-04-17T19:48:29.848531+0000","last_became_active":"2024-04-17T19:47:49.894250+0000","last_became_peered":"2024-04-17T19:47:49.894250+0000","last_unstale":"2024-04-17T19:48:29.848531+0000","last_undegraded":"2024-04-17T19:48:29.848531+0000","last_fullsized":"2024-04-17T19:48:29.848531+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:30:17.210135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839227+0000","last_change":"2024-04-17T19:47:48.861802+0000","last_active":"2024-04-17T19:48:29.839227+0000","last_peered":"2024-04-17T19:48:29.839227+0000","last_clean":"2024-04-17T19:48:29.839227+0000","last_became_active":"2024-04-17T19:47:48.860418+0000","last_became_peered":"2024-04-17T19:47:48.860418+0000","last_unstale":"2024-04-17T19:48:29.839227+0000","last_undegraded":"2024-04-17T19:48:29.839227+0000","last_fullsized":"2024-04-17T19:48:29.839227+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:54:41.949746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.163167+0000","last_change":"2024-04-17T19:47:53.917732+0000","last_active":"2024-04-17T19:48:30.163167+0000","last_peered":"2024-04-17T19:48:30.163167+0000","last_clean":"2024-04-17T19:48:30.163167+0000","last_became_active":"2024-04-17T19:47:53.917160+0000","last_became_peered":"2024-04-17T19:47:53.917160+0000","last_unstale":"2024-04-17T19:48:30.163167+0000","last_undegraded":"2024-04-17T19:48:30.163167+0000","last_fullsized":"2024-04-17T19:48:30.163167+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:19:20.394166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.847292+0000","last_change":"2024-04-17T19:47:49.893077+0000","last_active":"2024-04-17T19:48:29.847292+0000","last_peered":"2024-04-17T19:48:29.847292+0000","last_clean":"2024-04-17T19:48:29.847292+0000","last_became_active":"2024-04-17T19:47:49.892280+0000","last_became_peered":"2024-04-17T19:47:49.892280+0000","last_unstale":"2024-04-17T19:48:29.847292+0000","last_undegraded":"2024-04-17T19:48:29.847292+0000","last_fullsized":"2024-04-17T19:48:29.847292+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:58:24.218821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.868436+0000","last_change":"2024-04-17T19:47:48.849027+0000","last_active":"2024-04-17T19:48:33.868436+0000","last_peered":"2024-04-17T19:48:33.868436+0000","last_clean":"2024-04-17T19:48:33.868436+0000","last_became_active":"2024-04-17T19:47:48.848485+0000","last_became_peered":"2024-04-17T19:47:48.848485+0000","last_unstale":"2024-04-17T19:48:33.868436+0000","last_undegraded":"2024-04-17T19:48:33.868436+0000","last_fullsized":"2024-04-17T19:48:33.868436+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:58:14.548891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839524+0000","last_change":"2024-04-17T19:47:53.946933+0000","last_active":"2024-04-17T19:48:29.839524+0000","last_peered":"2024-04-17T19:48:29.839524+0000","last_clean":"2024-04-17T19:48:29.839524+0000","last_became_active":"2024-04-17T19:47:53.943030+0000","last_became_peered":"2024-04-17T19:47:53.943030+0000","last_unstale":"2024-04-17T19:48:29.839524+0000","last_undegraded":"2024-04-17T19:48:29.839524+0000","last_fullsized":"2024-04-17T19:48:29.839524+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:37:03.489250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845055+0000","last_change":"2024-04-17T19:47:49.893137+0000","last_active":"2024-04-17T19:48:29.845055+0000","last_peered":"2024-04-17T19:48:29.845055+0000","last_clean":"2024-04-17T19:48:29.845055+0000","last_became_active":"2024-04-17T19:47:49.892379+0000","last_became_peered":"2024-04-17T19:47:49.892379+0000","last_unstale":"2024-04-17T19:48:29.845055+0000","last_undegraded":"2024-04-17T19:48:29.845055+0000","last_fullsized":"2024-04-17T19:48:29.845055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:55:08.562395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"24'2","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.853051+0000","last_change":"2024-04-17T19:47:48.861154+0000","last_active":"2024-04-17T19:48:29.853051+0000","last_peered":"2024-04-17T19:48:29.853051+0000","last_clean":"2024-04-17T19:48:29.853051+0000","last_became_active":"2024-04-17T19:47:48.859268+0000","last_became_peered":"2024-04-17T19:47:48.859268+0000","last_unstale":"2024-04-17T19:48:29.853051+0000","last_undegraded":"2024-04-17T19:48:29.853051+0000","last_fullsized":"2024-04-17T19:48:29.853051+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:04:19.867115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.849198+0000","last_change":"2024-04-17T19:47:53.950580+0000","last_active":"2024-04-17T19:48:29.849198+0000","last_peered":"2024-04-17T19:48:29.849198+0000","last_clean":"2024-04-17T19:48:29.849198+0000","last_became_active":"2024-04-17T19:47:53.943798+0000","last_became_peered":"2024-04-17T19:47:53.943798+0000","last_unstale":"2024-04-17T19:48:29.849198+0000","last_undegraded":"2024-04-17T19:48:29.849198+0000","last_fullsized":"2024-04-17T19:48:29.849198+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:52:14.763799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843479+0000","last_change":"2024-04-17T19:47:49.893774+0000","last_active":"2024-04-17T19:48:29.843479+0000","last_peered":"2024-04-17T19:48:29.843479+0000","last_clean":"2024-04-17T19:48:29.843479+0000","last_became_active":"2024-04-17T19:47:49.892780+0000","last_became_peered":"2024-04-17T19:47:49.892780+0000","last_unstale":"2024-04-17T19:48:29.843479+0000","last_undegraded":"2024-04-17T19:48:29.843479+0000","last_fullsized":"2024-04-17T19:48:29.843479+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:50:47.658650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"26'17","reported_seq":43,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.837592+0000","last_change":"2024-04-17T19:47:48.852970+0000","last_active":"2024-04-17T19:48:29.837592+0000","last_peered":"2024-04-17T19:48:29.837592+0000","last_clean":"2024-04-17T19:48:29.837592+0000","last_became_active":"2024-04-17T19:47:48.852676+0000","last_became_peered":"2024-04-17T19:47:48.852676+0000","last_unstale":"2024-04-17T19:48:29.837592+0000","last_undegraded":"2024-04-17T19:48:29.837592+0000","last_fullsized":"2024-04-17T19:48:29.837592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:32:14.529596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162931+0000","last_change":"2024-04-17T19:47:53.915397+0000","last_active":"2024-04-17T19:48:30.162931+0000","last_peered":"2024-04-17T19:48:30.162931+0000","last_clean":"2024-04-17T19:48:30.162931+0000","last_became_active":"2024-04-17T19:47:53.914913+0000","last_became_peered":"2024-04-17T19:47:53.914913+0000","last_unstale":"2024-04-17T19:48:30.162931+0000","last_undegraded":"2024-04-17T19:48:30.162931+0000","last_fullsized":"2024-04-17T19:48:30.162931+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:02:01.175158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840177+0000","last_change":"2024-04-17T19:47:49.880484+0000","last_active":"2024-04-17T19:48:29.840177+0000","last_peered":"2024-04-17T19:48:29.840177+0000","last_clean":"2024-04-17T19:48:29.840177+0000","last_became_active":"2024-04-17T19:47:49.879625+0000","last_became_peered":"2024-04-17T19:47:49.879625+0000","last_unstale":"2024-04-17T19:48:29.840177+0000","last_undegraded":"2024-04-17T19:48:29.840177+0000","last_fullsized":"2024-04-17T19:48:29.840177+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:50:00.611796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"24'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843272+0000","last_change":"2024-04-17T19:47:48.899160+0000","last_active":"2024-04-17T19:48:29.843272+0000","last_peered":"2024-04-17T19:48:29.843272+0000","last_clean":"2024-04-17T19:48:29.843272+0000","last_became_active":"2024-04-17T19:47:48.889831+0000","last_became_peered":"2024-04-17T19:47:48.889831+0000","last_unstale":"2024-04-17T19:48:29.843272+0000","last_undegraded":"2024-04-17T19:48:29.843272+0000","last_fullsized":"2024-04-17T19:48:29.843272+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:55:20.167510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.835052+0000","last_change":"2024-04-17T19:47:53.921093+0000","last_active":"2024-04-17T19:48:29.835052+0000","last_peered":"2024-04-17T19:48:29.835052+0000","last_clean":"2024-04-17T19:48:29.835052+0000","last_became_active":"2024-04-17T19:47:53.920479+0000","last_became_peered":"2024-04-17T19:47:53.920479+0000","last_unstale":"2024-04-17T19:48:29.835052+0000","last_undegraded":"2024-04-17T19:48:29.835052+0000","last_fullsized":"2024-04-17T19:48:29.835052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:08:09.569820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.844227+0000","last_change":"2024-04-17T19:47:49.895275+0000","last_active":"2024-04-17T19:48:29.844227+0000","last_peered":"2024-04-17T19:48:29.844227+0000","last_clean":"2024-04-17T19:48:29.844227+0000","last_became_active":"2024-04-17T19:47:49.894534+0000","last_became_peered":"2024-04-17T19:47:49.894534+0000","last_unstale":"2024-04-17T19:48:29.844227+0000","last_undegraded":"2024-04-17T19:48:29.844227+0000","last_fullsized":"2024-04-17T19:48:29.844227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:31:40.818807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"26'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.841327+0000","last_change":"2024-04-17T19:47:48.881715+0000","last_active":"2024-04-17T19:48:29.841327+0000","last_peered":"2024-04-17T19:48:29.841327+0000","last_clean":"2024-04-17T19:48:29.841327+0000","last_became_active":"2024-04-17T19:47:48.876203+0000","last_became_peered":"2024-04-17T19:47:48.876203+0000","last_unstale":"2024-04-17T19:48:29.841327+0000","last_undegraded":"2024-04-17T19:48:29.841327+0000","last_fullsized":"2024-04-17T19:48:29.841327+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:03:16.656528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834859+0000","last_change":"2024-04-17T19:47:53.905149+0000","last_active":"2024-04-17T19:48:29.834859+0000","last_peered":"2024-04-17T19:48:29.834859+0000","last_clean":"2024-04-17T19:48:29.834859+0000","last_became_active":"2024-04-17T19:47:53.904718+0000","last_became_peered":"2024-04-17T19:47:53.904718+0000","last_unstale":"2024-04-17T19:48:29.834859+0000","last_undegraded":"2024-04-17T19:48:29.834859+0000","last_fullsized":"2024-04-17T19:48:29.834859+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:09:31.908752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162277+0000","last_change":"2024-04-17T19:47:49.875819+0000","last_active":"2024-04-17T19:48:30.162277+0000","last_peered":"2024-04-17T19:48:30.162277+0000","last_clean":"2024-04-17T19:48:30.162277+0000","last_became_active":"2024-04-17T19:47:49.872075+0000","last_became_peered":"2024-04-17T19:47:49.872075+0000","last_unstale":"2024-04-17T19:48:30.162277+0000","last_undegraded":"2024-04-17T19:48:30.162277+0000","last_fullsized":"2024-04-17T19:48:30.162277+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:06:48.823135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"26'9","reported_seq":35,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.871452+0000","last_change":"2024-04-17T19:47:48.867633+0000","last_active":"2024-04-17T19:48:33.871452+0000","last_peered":"2024-04-17T19:48:33.871452+0000","last_clean":"2024-04-17T19:48:33.871452+0000","last_became_active":"2024-04-17T19:47:48.867150+0000","last_became_peered":"2024-04-17T19:47:48.867150+0000","last_unstale":"2024-04-17T19:48:33.871452+0000","last_undegraded":"2024-04-17T19:48:33.871452+0000","last_fullsized":"2024-04-17T19:48:33.871452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:33:34.275816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.847899+0000","last_change":"2024-04-17T19:47:53.925785+0000","last_active":"2024-04-17T19:48:29.847899+0000","last_peered":"2024-04-17T19:48:29.847899+0000","last_clean":"2024-04-17T19:48:29.847899+0000","last_became_active":"2024-04-17T19:47:53.924006+0000","last_became_peered":"2024-04-17T19:47:53.924006+0000","last_unstale":"2024-04-17T19:48:29.847899+0000","last_undegraded":"2024-04-17T19:48:29.847899+0000","last_fullsized":"2024-04-17T19:48:29.847899+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:46:31.965271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845415+0000","last_change":"2024-04-17T19:47:49.869829+0000","last_active":"2024-04-17T19:48:29.845415+0000","last_peered":"2024-04-17T19:48:29.845415+0000","last_clean":"2024-04-17T19:48:29.845415+0000","last_became_active":"2024-04-17T19:47:49.868933+0000","last_became_peered":"2024-04-17T19:47:49.868933+0000","last_unstale":"2024-04-17T19:48:29.845415+0000","last_undegraded":"2024-04-17T19:48:29.845415+0000","last_fullsized":"2024-04-17T19:48:29.845415+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:36:08.866159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'15","reported_seq":41,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:43.269615+0000","last_change":"2024-04-17T19:47:48.887461+0000","last_active":"2024-04-17T19:48:43.269615+0000","last_peered":"2024-04-17T19:48:43.269615+0000","last_clean":"2024-04-17T19:48:43.269615+0000","last_became_active":"2024-04-17T19:47:48.886689+0000","last_became_peered":"2024-04-17T19:47:48.886689+0000","last_unstale":"2024-04-17T19:48:43.269615+0000","last_undegraded":"2024-04-17T19:48:43.269615+0000","last_fullsized":"2024-04-17T19:48:43.269615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:55:35.262995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4100,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840709+0000","last_change":"2024-04-17T19:47:53.941607+0000","last_active":"2024-04-17T19:48:29.840709+0000","last_peered":"2024-04-17T19:48:29.840709+0000","last_clean":"2024-04-17T19:48:29.840709+0000","last_became_active":"2024-04-17T19:47:53.941050+0000","last_became_peered":"2024-04-17T19:47:53.941050+0000","last_unstale":"2024-04-17T19:48:29.840709+0000","last_undegraded":"2024-04-17T19:48:29.840709+0000","last_fullsized":"2024-04-17T19:48:29.840709+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:26:08.964058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.851845+0000","last_change":"2024-04-17T19:47:49.879524+0000","last_active":"2024-04-17T19:48:29.851845+0000","last_peered":"2024-04-17T19:48:29.851845+0000","last_clean":"2024-04-17T19:48:29.851845+0000","last_became_active":"2024-04-17T19:47:49.877563+0000","last_became_peered":"2024-04-17T19:47:49.877563+0000","last_unstale":"2024-04-17T19:48:29.851845+0000","last_undegraded":"2024-04-17T19:48:29.851845+0000","last_fullsized":"2024-04-17T19:48:29.851845+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:34:58.539940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"24'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839055+0000","last_change":"2024-04-17T19:47:48.870100+0000","last_active":"2024-04-17T19:48:29.839055+0000","last_peered":"2024-04-17T19:48:29.839055+0000","last_clean":"2024-04-17T19:48:29.839055+0000","last_became_active":"2024-04-17T19:47:48.869797+0000","last_became_peered":"2024-04-17T19:47:48.869797+0000","last_unstale":"2024-04-17T19:48:29.839055+0000","last_undegraded":"2024-04-17T19:48:29.839055+0000","last_fullsized":"2024-04-17T19:48:29.839055+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:51:21.463081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.850942+0000","last_change":"2024-04-17T19:47:53.944568+0000","last_active":"2024-04-17T19:48:29.850942+0000","last_peered":"2024-04-17T19:48:29.850942+0000","last_clean":"2024-04-17T19:48:29.850942+0000","last_became_active":"2024-04-17T19:47:53.943151+0000","last_became_peered":"2024-04-17T19:47:53.943151+0000","last_unstale":"2024-04-17T19:48:29.850942+0000","last_undegraded":"2024-04-17T19:48:29.850942+0000","last_fullsized":"2024-04-17T19:48:29.850942+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:19:00.150450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162170+0000","last_change":"2024-04-17T19:47:49.902401+0000","last_active":"2024-04-17T19:48:30.162170+0000","last_peered":"2024-04-17T19:48:30.162170+0000","last_clean":"2024-04-17T19:48:30.162170+0000","last_became_active":"2024-04-17T19:47:49.902166+0000","last_became_peered":"2024-04-17T19:47:49.902166+0000","last_unstale":"2024-04-17T19:48:30.162170+0000","last_undegraded":"2024-04-17T19:48:30.162170+0000","last_fullsized":"2024-04-17T19:48:30.162170+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:54:13.129748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"26'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:33.876986+0000","last_change":"2024-04-17T19:47:48.900800+0000","last_active":"2024-04-17T19:48:33.876986+0000","last_peered":"2024-04-17T19:48:33.876986+0000","last_clean":"2024-04-17T19:48:33.876986+0000","last_became_active":"2024-04-17T19:47:48.900188+0000","last_became_peered":"2024-04-17T19:47:48.900188+0000","last_unstale":"2024-04-17T19:48:33.876986+0000","last_undegraded":"2024-04-17T19:48:33.876986+0000","last_fullsized":"2024-04-17T19:48:33.876986+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:05:04.305871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834112+0000","last_change":"2024-04-17T19:47:53.928721+0000","last_active":"2024-04-17T19:48:29.834112+0000","last_peered":"2024-04-17T19:48:29.834112+0000","last_clean":"2024-04-17T19:48:29.834112+0000","last_became_active":"2024-04-17T19:47:53.928311+0000","last_became_peered":"2024-04-17T19:47:53.928311+0000","last_unstale":"2024-04-17T19:48:29.834112+0000","last_undegraded":"2024-04-17T19:48:29.834112+0000","last_fullsized":"2024-04-17T19:48:29.834112+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:35:28.870803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.847789+0000","last_change":"2024-04-17T19:47:49.905421+0000","last_active":"2024-04-17T19:48:29.847789+0000","last_peered":"2024-04-17T19:48:29.847789+0000","last_clean":"2024-04-17T19:48:29.847789+0000","last_became_active":"2024-04-17T19:47:49.904646+0000","last_became_peered":"2024-04-17T19:47:49.904646+0000","last_unstale":"2024-04-17T19:48:29.847789+0000","last_undegraded":"2024-04-17T19:48:29.847789+0000","last_fullsized":"2024-04-17T19:48:29.847789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:02:23.936080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"24'4","reported_seq":30,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.846631+0000","last_change":"2024-04-17T19:47:48.895356+0000","last_active":"2024-04-17T19:48:29.846631+0000","last_peered":"2024-04-17T19:48:29.846631+0000","last_clean":"2024-04-17T19:48:29.846631+0000","last_became_active":"2024-04-17T19:47:48.889225+0000","last_became_peered":"2024-04-17T19:47:48.889225+0000","last_unstale":"2024-04-17T19:48:29.846631+0000","last_undegraded":"2024-04-17T19:48:29.846631+0000","last_fullsized":"2024-04-17T19:48:29.846631+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:00:22.587917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.833996+0000","last_change":"2024-04-17T19:47:53.940206+0000","last_active":"2024-04-17T19:48:29.833996+0000","last_peered":"2024-04-17T19:48:29.833996+0000","last_clean":"2024-04-17T19:48:29.833996+0000","last_became_active":"2024-04-17T19:47:53.939843+0000","last_became_peered":"2024-04-17T19:47:53.939843+0000","last_unstale":"2024-04-17T19:48:29.833996+0000","last_undegraded":"2024-04-17T19:48:29.833996+0000","last_fullsized":"2024-04-17T19:48:29.833996+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:18:55.562573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162804+0000","last_change":"2024-04-17T19:47:53.920258+0000","last_active":"2024-04-17T19:48:30.162804+0000","last_peered":"2024-04-17T19:48:30.162804+0000","last_clean":"2024-04-17T19:48:30.162804+0000","last_became_active":"2024-04-17T19:47:53.918932+0000","last_became_peered":"2024-04-17T19:47:53.918932+0000","last_unstale":"2024-04-17T19:48:30.162804+0000","last_undegraded":"2024-04-17T19:48:30.162804+0000","last_fullsized":"2024-04-17T19:48:30.162804+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:45:55.456803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.842416+0000","last_change":"2024-04-17T19:48:29.842416+0000","last_active":"2024-04-17T19:48:29.842416+0000","last_peered":"2024-04-17T19:48:29.842416+0000","last_clean":"2024-04-17T19:48:29.842416+0000","last_became_active":"2024-04-17T19:48:29.841759+0000","last_became_peered":"2024-04-17T19:48:29.841759+0000","last_unstale":"2024-04-17T19:48:29.842416+0000","last_undegraded":"2024-04-17T19:48:29.842416+0000","last_fullsized":"2024-04-17T19:48:29.842416+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:57:17.664562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840971+0000","last_change":"2024-04-17T19:47:48.873355+0000","last_active":"2024-04-17T19:48:29.840971+0000","last_peered":"2024-04-17T19:48:29.840971+0000","last_clean":"2024-04-17T19:48:29.840971+0000","last_became_active":"2024-04-17T19:47:48.872427+0000","last_became_peered":"2024-04-17T19:47:48.872427+0000","last_unstale":"2024-04-17T19:48:29.840971+0000","last_undegraded":"2024-04-17T19:48:29.840971+0000","last_fullsized":"2024-04-17T19:48:29.840971+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:32:41.792605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.846032+0000","last_change":"2024-04-17T19:47:53.952602+0000","last_active":"2024-04-17T19:48:29.846032+0000","last_peered":"2024-04-17T19:48:29.846032+0000","last_clean":"2024-04-17T19:48:29.846032+0000","last_became_active":"2024-04-17T19:47:53.951750+0000","last_became_peered":"2024-04-17T19:47:53.951750+0000","last_unstale":"2024-04-17T19:48:29.846032+0000","last_undegraded":"2024-04-17T19:48:29.846032+0000","last_fullsized":"2024-04-17T19:48:29.846032+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:13:08.485238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834245+0000","last_change":"2024-04-17T19:47:49.862071+0000","last_active":"2024-04-17T19:48:29.834245+0000","last_peered":"2024-04-17T19:48:29.834245+0000","last_clean":"2024-04-17T19:48:29.834245+0000","last_became_active":"2024-04-17T19:47:49.860809+0000","last_became_peered":"2024-04-17T19:47:49.860809+0000","last_unstale":"2024-04-17T19:48:29.834245+0000","last_undegraded":"2024-04-17T19:48:29.834245+0000","last_fullsized":"2024-04-17T19:48:29.834245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:42:34.451839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"26'3","reported_seq":29,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834368+0000","last_change":"2024-04-17T19:47:48.867855+0000","last_active":"2024-04-17T19:48:29.834368+0000","last_peered":"2024-04-17T19:48:29.834368+0000","last_clean":"2024-04-17T19:48:29.834368+0000","last_became_active":"2024-04-17T19:47:48.867532+0000","last_became_peered":"2024-04-17T19:47:48.867532+0000","last_unstale":"2024-04-17T19:48:29.834368+0000","last_undegraded":"2024-04-17T19:48:29.834368+0000","last_fullsized":"2024-04-17T19:48:29.834368+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:21:26.903245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"24'7","reported_seq":33,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843580+0000","last_change":"2024-04-17T19:47:48.894946+0000","last_active":"2024-04-17T19:48:29.843580+0000","last_peered":"2024-04-17T19:48:29.843580+0000","last_clean":"2024-04-17T19:48:29.843580+0000","last_became_active":"2024-04-17T19:47:48.893407+0000","last_became_peered":"2024-04-17T19:47:48.893407+0000","last_unstale":"2024-04-17T19:48:29.843580+0000","last_undegraded":"2024-04-17T19:48:29.843580+0000","last_fullsized":"2024-04-17T19:48:29.843580+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:09:56.284638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162336+0000","last_change":"2024-04-17T19:47:49.859985+0000","last_active":"2024-04-17T19:48:30.162336+0000","last_peered":"2024-04-17T19:48:30.162336+0000","last_clean":"2024-04-17T19:48:30.162336+0000","last_became_active":"2024-04-17T19:47:49.859635+0000","last_became_peered":"2024-04-17T19:47:49.859635+0000","last_unstale":"2024-04-17T19:48:30.162336+0000","last_undegraded":"2024-04-17T19:48:30.162336+0000","last_fullsized":"2024-04-17T19:48:30.162336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:04:14.834218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838003+0000","last_change":"2024-04-17T19:47:53.925444+0000","last_active":"2024-04-17T19:48:29.838003+0000","last_peered":"2024-04-17T19:48:29.838003+0000","last_clean":"2024-04-17T19:48:29.838003+0000","last_became_active":"2024-04-17T19:47:53.924805+0000","last_became_peered":"2024-04-17T19:47:53.924805+0000","last_unstale":"2024-04-17T19:48:29.838003+0000","last_undegraded":"2024-04-17T19:48:29.838003+0000","last_fullsized":"2024-04-17T19:48:29.838003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:19:24.022368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"24'5","reported_seq":31,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834567+0000","last_change":"2024-04-17T19:47:48.867176+0000","last_active":"2024-04-17T19:48:29.834567+0000","last_peered":"2024-04-17T19:48:29.834567+0000","last_clean":"2024-04-17T19:48:29.834567+0000","last_became_active":"2024-04-17T19:47:48.866817+0000","last_became_peered":"2024-04-17T19:47:48.866817+0000","last_unstale":"2024-04-17T19:48:29.834567+0000","last_undegraded":"2024-04-17T19:48:29.834567+0000","last_fullsized":"2024-04-17T19:48:29.834567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:47.816910+0000","last_clean_scrub_stamp":"2024-04-17T19:47:47.816910+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:44:30.842583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.850861+0000","last_change":"2024-04-17T19:47:49.858924+0000","last_active":"2024-04-17T19:48:29.850861+0000","last_peered":"2024-04-17T19:48:29.850861+0000","last_clean":"2024-04-17T19:48:29.850861+0000","last_became_active":"2024-04-17T19:47:49.857820+0000","last_became_peered":"2024-04-17T19:47:49.857820+0000","last_unstale":"2024-04-17T19:48:29.850861+0000","last_undegraded":"2024-04-17T19:48:29.850861+0000","last_fullsized":"2024-04-17T19:48:29.850861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:09:28.706054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.835377+0000","last_change":"2024-04-17T19:47:53.933337+0000","last_active":"2024-04-17T19:48:29.835377+0000","last_peered":"2024-04-17T19:48:29.835377+0000","last_clean":"2024-04-17T19:48:29.835377+0000","last_became_active":"2024-04-17T19:47:53.931623+0000","last_became_peered":"2024-04-17T19:47:53.931623+0000","last_unstale":"2024-04-17T19:48:29.835377+0000","last_undegraded":"2024-04-17T19:48:29.835377+0000","last_fullsized":"2024-04-17T19:48:29.835377+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:52.874711+0000","last_clean_scrub_stamp":"2024-04-17T19:47:52.874711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T03:57:37.600957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839672+0000","last_change":"2024-04-17T19:47:49.881093+0000","last_active":"2024-04-17T19:48:29.839672+0000","last_peered":"2024-04-17T19:48:29.839672+0000","last_clean":"2024-04-17T19:48:29.839672+0000","last_became_active":"2024-04-17T19:47:49.880210+0000","last_became_peered":"2024-04-17T19:47:49.880210+0000","last_unstale":"2024-04-17T19:48:29.839672+0000","last_undegraded":"2024-04-17T19:48:29.839672+0000","last_fullsized":"2024-04-17T19:48:29.839672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:12:15.522651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.837813+0000","last_change":"2024-04-17T19:47:49.886936+0000","last_active":"2024-04-17T19:48:29.837813+0000","last_peered":"2024-04-17T19:48:29.837813+0000","last_clean":"2024-04-17T19:48:29.837813+0000","last_became_active":"2024-04-17T19:47:49.886580+0000","last_became_peered":"2024-04-17T19:47:49.886580+0000","last_unstale":"2024-04-17T19:48:29.837813+0000","last_undegraded":"2024-04-17T19:48:29.837813+0000","last_fullsized":"2024-04-17T19:48:29.837813+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:00:44.599463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840923+0000","last_change":"2024-04-17T19:47:49.905445+0000","last_active":"2024-04-17T19:48:29.840923+0000","last_peered":"2024-04-17T19:48:29.840923+0000","last_clean":"2024-04-17T19:48:29.840923+0000","last_became_active":"2024-04-17T19:47:49.896510+0000","last_became_peered":"2024-04-17T19:47:49.896510+0000","last_unstale":"2024-04-17T19:48:29.840923+0000","last_undegraded":"2024-04-17T19:48:29.840923+0000","last_fullsized":"2024-04-17T19:48:29.840923+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:23:27.401861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162472+0000","last_change":"2024-04-17T19:47:49.850490+0000","last_active":"2024-04-17T19:48:30.162472+0000","last_peered":"2024-04-17T19:48:30.162472+0000","last_clean":"2024-04-17T19:48:30.162472+0000","last_became_active":"2024-04-17T19:47:49.849344+0000","last_became_peered":"2024-04-17T19:47:49.849344+0000","last_unstale":"2024-04-17T19:48:30.162472+0000","last_undegraded":"2024-04-17T19:48:30.162472+0000","last_fullsized":"2024-04-17T19:48:30.162472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:44:32.231058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.835044+0000","last_change":"2024-04-17T19:47:49.857737+0000","last_active":"2024-04-17T19:48:29.835044+0000","last_peered":"2024-04-17T19:48:29.835044+0000","last_clean":"2024-04-17T19:48:29.835044+0000","last_became_active":"2024-04-17T19:47:49.857476+0000","last_became_peered":"2024-04-17T19:47:49.857476+0000","last_unstale":"2024-04-17T19:48:29.835044+0000","last_undegraded":"2024-04-17T19:48:29.835044+0000","last_fullsized":"2024-04-17T19:48:29.835044+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:50:13.070628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838876+0000","last_change":"2024-04-17T19:47:49.864985+0000","last_active":"2024-04-17T19:48:29.838876+0000","last_peered":"2024-04-17T19:48:29.838876+0000","last_clean":"2024-04-17T19:48:29.838876+0000","last_became_active":"2024-04-17T19:47:49.864329+0000","last_became_peered":"2024-04-17T19:47:49.864329+0000","last_unstale":"2024-04-17T19:48:29.838876+0000","last_undegraded":"2024-04-17T19:48:29.838876+0000","last_fullsized":"2024-04-17T19:48:29.838876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:43:57.448269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845430+0000","last_change":"2024-04-17T19:47:49.881785+0000","last_active":"2024-04-17T19:48:29.845430+0000","last_peered":"2024-04-17T19:48:29.845430+0000","last_clean":"2024-04-17T19:48:29.845430+0000","last_became_active":"2024-04-17T19:47:49.880939+0000","last_became_peered":"2024-04-17T19:47:49.880939+0000","last_unstale":"2024-04-17T19:48:29.845430+0000","last_undegraded":"2024-04-17T19:48:29.845430+0000","last_fullsized":"2024-04-17T19:48:29.845430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:49:07.194643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.835152+0000","last_change":"2024-04-17T19:47:49.884944+0000","last_active":"2024-04-17T19:48:29.835152+0000","last_peered":"2024-04-17T19:48:29.835152+0000","last_clean":"2024-04-17T19:48:29.835152+0000","last_became_active":"2024-04-17T19:47:49.884559+0000","last_became_peered":"2024-04-17T19:47:49.884559+0000","last_unstale":"2024-04-17T19:48:29.835152+0000","last_undegraded":"2024-04-17T19:48:29.835152+0000","last_fullsized":"2024-04-17T19:48:29.835152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:00.485472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840468+0000","last_change":"2024-04-17T19:47:49.891230+0000","last_active":"2024-04-17T19:48:29.840468+0000","last_peered":"2024-04-17T19:48:29.840468+0000","last_clean":"2024-04-17T19:48:29.840468+0000","last_became_active":"2024-04-17T19:47:49.886001+0000","last_became_peered":"2024-04-17T19:47:49.886001+0000","last_unstale":"2024-04-17T19:48:29.840468+0000","last_undegraded":"2024-04-17T19:48:29.840468+0000","last_fullsized":"2024-04-17T19:48:29.840468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:00.485472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834129+0000","last_change":"2024-04-17T19:47:49.859382+0000","last_active":"2024-04-17T19:48:29.834129+0000","last_peered":"2024-04-17T19:48:29.834129+0000","last_clean":"2024-04-17T19:48:29.834129+0000","last_became_active":"2024-04-17T19:47:49.859170+0000","last_became_peered":"2024-04-17T19:47:49.859170+0000","last_unstale":"2024-04-17T19:48:29.834129+0000","last_undegraded":"2024-04-17T19:48:29.834129+0000","last_fullsized":"2024-04-17T19:48:29.834129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:58:51.111053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.854842+0000","last_change":"2024-04-17T19:48:29.853997+0000","last_active":"2024-04-17T19:48:29.854842+0000","last_peered":"2024-04-17T19:48:29.854842+0000","last_clean":"2024-04-17T19:48:29.854842+0000","last_became_active":"2024-04-17T19:48:29.853682+0000","last_became_peered":"2024-04-17T19:48:29.853682+0000","last_unstale":"2024-04-17T19:48:29.854842+0000","last_undegraded":"2024-04-17T19:48:29.854842+0000","last_fullsized":"2024-04-17T19:48:29.854842+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T00:29:20.526070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162602+0000","last_change":"2024-04-17T19:47:49.876406+0000","last_active":"2024-04-17T19:48:30.162602+0000","last_peered":"2024-04-17T19:48:30.162602+0000","last_clean":"2024-04-17T19:48:30.162602+0000","last_became_active":"2024-04-17T19:47:49.876132+0000","last_became_peered":"2024-04-17T19:47:49.876132+0000","last_unstale":"2024-04-17T19:48:30.162602+0000","last_undegraded":"2024-04-17T19:48:30.162602+0000","last_fullsized":"2024-04-17T19:48:30.162602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:51:20.791740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.846554+0000","last_change":"2024-04-17T19:47:49.896244+0000","last_active":"2024-04-17T19:48:29.846554+0000","last_peered":"2024-04-17T19:48:29.846554+0000","last_clean":"2024-04-17T19:48:29.846554+0000","last_became_active":"2024-04-17T19:47:49.895380+0000","last_became_peered":"2024-04-17T19:47:49.895380+0000","last_unstale":"2024-04-17T19:48:29.846554+0000","last_undegraded":"2024-04-17T19:48:29.846554+0000","last_fullsized":"2024-04-17T19:48:29.846554+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:04:43.091307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845627+0000","last_change":"2024-04-17T19:47:49.870551+0000","last_active":"2024-04-17T19:48:29.845627+0000","last_peered":"2024-04-17T19:48:29.845627+0000","last_clean":"2024-04-17T19:48:29.845627+0000","last_became_active":"2024-04-17T19:47:49.869315+0000","last_became_peered":"2024-04-17T19:47:49.869315+0000","last_unstale":"2024-04-17T19:48:29.845627+0000","last_undegraded":"2024-04-17T19:48:29.845627+0000","last_fullsized":"2024-04-17T19:48:29.845627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:50:06.341123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839471+0000","last_change":"2024-04-17T19:47:49.880060+0000","last_active":"2024-04-17T19:48:29.839471+0000","last_peered":"2024-04-17T19:48:29.839471+0000","last_clean":"2024-04-17T19:48:29.839471+0000","last_became_active":"2024-04-17T19:47:49.879711+0000","last_became_peered":"2024-04-17T19:47:49.879711+0000","last_unstale":"2024-04-17T19:48:29.839471+0000","last_undegraded":"2024-04-17T19:48:29.839471+0000","last_fullsized":"2024-04-17T19:48:29.839471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:41:25.628870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.849283+0000","last_change":"2024-04-17T19:47:49.895452+0000","last_active":"2024-04-17T19:48:29.849283+0000","last_peered":"2024-04-17T19:48:29.849283+0000","last_clean":"2024-04-17T19:48:29.849283+0000","last_became_active":"2024-04-17T19:47:49.894684+0000","last_became_peered":"2024-04-17T19:47:49.894684+0000","last_unstale":"2024-04-17T19:48:29.849283+0000","last_undegraded":"2024-04-17T19:48:29.849283+0000","last_fullsized":"2024-04-17T19:48:29.849283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T02:15:55.224238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840156+0000","last_change":"2024-04-17T19:47:49.886902+0000","last_active":"2024-04-17T19:48:29.840156+0000","last_peered":"2024-04-17T19:48:29.840156+0000","last_clean":"2024-04-17T19:48:29.840156+0000","last_became_active":"2024-04-17T19:47:49.885451+0000","last_became_peered":"2024-04-17T19:47:49.885451+0000","last_unstale":"2024-04-17T19:48:29.840156+0000","last_undegraded":"2024-04-17T19:48:29.840156+0000","last_fullsized":"2024-04-17T19:48:29.840156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:12:09.196610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.838401+0000","last_change":"2024-04-17T19:47:49.858908+0000","last_active":"2024-04-17T19:48:29.838401+0000","last_peered":"2024-04-17T19:48:29.838401+0000","last_clean":"2024-04-17T19:48:29.838401+0000","last_became_active":"2024-04-17T19:47:49.858457+0000","last_became_peered":"2024-04-17T19:47:49.858457+0000","last_unstale":"2024-04-17T19:48:29.838401+0000","last_undegraded":"2024-04-17T19:48:29.838401+0000","last_fullsized":"2024-04-17T19:48:29.838401+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:09.467859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162134+0000","last_change":"2024-04-17T19:47:49.856371+0000","last_active":"2024-04-17T19:48:30.162134+0000","last_peered":"2024-04-17T19:48:30.162134+0000","last_clean":"2024-04-17T19:48:30.162134+0000","last_became_active":"2024-04-17T19:47:49.856062+0000","last_became_peered":"2024-04-17T19:47:49.856062+0000","last_unstale":"2024-04-17T19:48:30.162134+0000","last_undegraded":"2024-04-17T19:48:30.162134+0000","last_fullsized":"2024-04-17T19:48:30.162134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:39:54.502619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.834338+0000","last_change":"2024-04-17T19:47:49.873350+0000","last_active":"2024-04-17T19:48:29.834338+0000","last_peered":"2024-04-17T19:48:29.834338+0000","last_clean":"2024-04-17T19:48:29.834338+0000","last_became_active":"2024-04-17T19:47:49.873019+0000","last_became_peered":"2024-04-17T19:47:49.873019+0000","last_unstale":"2024-04-17T19:48:29.834338+0000","last_undegraded":"2024-04-17T19:48:29.834338+0000","last_fullsized":"2024-04-17T19:48:29.834338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T06:06:58.641727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.851454+0000","last_change":"2024-04-17T19:47:49.891551+0000","last_active":"2024-04-17T19:48:29.851454+0000","last_peered":"2024-04-17T19:48:29.851454+0000","last_clean":"2024-04-17T19:48:29.851454+0000","last_became_active":"2024-04-17T19:47:49.890132+0000","last_became_peered":"2024-04-17T19:47:49.890132+0000","last_unstale":"2024-04-17T19:48:29.851454+0000","last_undegraded":"2024-04-17T19:48:29.851454+0000","last_fullsized":"2024-04-17T19:48:29.851454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T07:26:22.971489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845078+0000","last_change":"2024-04-17T19:47:49.903854+0000","last_active":"2024-04-17T19:48:29.845078+0000","last_peered":"2024-04-17T19:48:29.845078+0000","last_clean":"2024-04-17T19:48:29.845078+0000","last_became_active":"2024-04-17T19:47:49.903178+0000","last_became_peered":"2024-04-17T19:47:49.903178+0000","last_unstale":"2024-04-17T19:48:29.845078+0000","last_undegraded":"2024-04-17T19:48:29.845078+0000","last_fullsized":"2024-04-17T19:48:29.845078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:24:52.699532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845192+0000","last_change":"2024-04-17T19:47:49.900848+0000","last_active":"2024-04-17T19:48:29.845192+0000","last_peered":"2024-04-17T19:48:29.845192+0000","last_clean":"2024-04-17T19:48:29.845192+0000","last_became_active":"2024-04-17T19:47:49.900052+0000","last_became_peered":"2024-04-17T19:47:49.900052+0000","last_unstale":"2024-04-17T19:48:29.845192+0000","last_undegraded":"2024-04-17T19:48:29.845192+0000","last_fullsized":"2024-04-17T19:48:29.845192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:28:41.115173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.846967+0000","last_change":"2024-04-17T19:47:49.906531+0000","last_active":"2024-04-17T19:48:29.846967+0000","last_peered":"2024-04-17T19:48:29.846967+0000","last_clean":"2024-04-17T19:48:29.846967+0000","last_became_active":"2024-04-17T19:47:49.905217+0000","last_became_peered":"2024-04-17T19:47:49.905217+0000","last_unstale":"2024-04-17T19:48:29.846967+0000","last_undegraded":"2024-04-17T19:48:29.846967+0000","last_fullsized":"2024-04-17T19:48:29.846967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T23:36:22.078473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.833993+0000","last_change":"2024-04-17T19:47:49.864577+0000","last_active":"2024-04-17T19:48:29.833993+0000","last_peered":"2024-04-17T19:48:29.833993+0000","last_clean":"2024-04-17T19:48:29.833993+0000","last_became_active":"2024-04-17T19:47:49.864251+0000","last_became_peered":"2024-04-17T19:47:49.864251+0000","last_unstale":"2024-04-17T19:48:29.833993+0000","last_undegraded":"2024-04-17T19:48:29.833993+0000","last_fullsized":"2024-04-17T19:48:29.833993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:32:10.329543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843153+0000","last_change":"2024-04-17T19:47:49.869245+0000","last_active":"2024-04-17T19:48:29.843153+0000","last_peered":"2024-04-17T19:48:29.843153+0000","last_clean":"2024-04-17T19:48:29.843153+0000","last_became_active":"2024-04-17T19:47:49.868156+0000","last_became_peered":"2024-04-17T19:47:49.868156+0000","last_unstale":"2024-04-17T19:48:29.843153+0000","last_undegraded":"2024-04-17T19:48:29.843153+0000","last_fullsized":"2024-04-17T19:48:29.843153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T04:58:39.814933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.840460+0000","last_change":"2024-04-17T19:47:49.907573+0000","last_active":"2024-04-17T19:48:29.840460+0000","last_peered":"2024-04-17T19:48:29.840460+0000","last_clean":"2024-04-17T19:48:29.840460+0000","last_became_active":"2024-04-17T19:47:49.897308+0000","last_became_peered":"2024-04-17T19:47:49.897308+0000","last_unstale":"2024-04-17T19:48:29.840460+0000","last_undegraded":"2024-04-17T19:48:29.840460+0000","last_fullsized":"2024-04-17T19:48:29.840460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:02:03.101317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.843871+0000","last_change":"2024-04-17T19:47:49.876052+0000","last_active":"2024-04-17T19:48:29.843871+0000","last_peered":"2024-04-17T19:48:29.843871+0000","last_clean":"2024-04-17T19:48:29.843871+0000","last_became_active":"2024-04-17T19:47:49.875008+0000","last_became_peered":"2024-04-17T19:47:49.875008+0000","last_unstale":"2024-04-17T19:48:29.843871+0000","last_undegraded":"2024-04-17T19:48:29.843871+0000","last_fullsized":"2024-04-17T19:48:29.843871+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:36:14.486150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.847790+0000","last_change":"2024-04-17T19:47:49.906840+0000","last_active":"2024-04-17T19:48:29.847790+0000","last_peered":"2024-04-17T19:48:29.847790+0000","last_clean":"2024-04-17T19:48:29.847790+0000","last_became_active":"2024-04-17T19:47:49.905703+0000","last_became_peered":"2024-04-17T19:47:49.905703+0000","last_unstale":"2024-04-17T19:48:29.847790+0000","last_undegraded":"2024-04-17T19:48:29.847790+0000","last_fullsized":"2024-04-17T19:48:29.847790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T05:28:26.686472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:30.162905+0000","last_change":"2024-04-17T19:47:49.869425+0000","last_active":"2024-04-17T19:48:30.162905+0000","last_peered":"2024-04-17T19:48:30.162905+0000","last_clean":"2024-04-17T19:48:30.162905+0000","last_became_active":"2024-04-17T19:47:49.868758+0000","last_became_peered":"2024-04-17T19:47:49.868758+0000","last_unstale":"2024-04-17T19:48:30.162905+0000","last_undegraded":"2024-04-17T19:48:30.162905+0000","last_fullsized":"2024-04-17T19:48:30.162905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:57:27.214150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.839586+0000","last_change":"2024-04-17T19:47:49.856309+0000","last_active":"2024-04-17T19:48:29.839586+0000","last_peered":"2024-04-17T19:48:29.839586+0000","last_clean":"2024-04-17T19:48:29.839586+0000","last_became_active":"2024-04-17T19:47:49.856026+0000","last_became_peered":"2024-04-17T19:47:49.856026+0000","last_unstale":"2024-04-17T19:48:29.839586+0000","last_undegraded":"2024-04-17T19:48:29.839586+0000","last_fullsized":"2024-04-17T19:48:29.839586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-19T01:51:57.963637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-17T19:48:29.845128+0000","last_change":"2024-04-17T19:47:49.890343+0000","last_active":"2024-04-17T19:48:29.845128+0000","last_peered":"2024-04-17T19:48:29.845128+0000","last_clean":"2024-04-17T19:48:29.845128+0000","last_became_active":"2024-04-17T19:47:49.880566+0000","last_became_peered":"2024-04-17T19:47:49.880566+0000","last_unstale":"2024-04-17T19:48:29.845128+0000","last_undegraded":"2024-04-17T19:48:29.845128+0000","last_fullsized":"2024-04-17T19:48:29.845128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T19:47:48.825871+0000","last_clean_scrub_stamp":"2024-04-17T19:47:48.825871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:19:29.359024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10522,"num_objects":98,"num_object_clones":0,"num_object_copies":196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":210,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":21044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":184,"ondisk_log_size":184,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32064,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339776,"statfs":{"total":96636764160,"available":96603930624,"internally_reserved":0,"allocated":909312,"data_stored":706837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574864,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":307200,"data_stored":119259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":327680,"data_stored":118301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27312,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344528,"statfs":{"total":96636764160,"available":96608796672,"internally_reserved":0,"allocated":303104,"data_stored":115699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31480,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340360,"statfs":{"total":96636764160,"available":96604528640,"internally_reserved":0,"allocated":311296,"data_stored":119293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31480,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340360,"statfs":{"total":96636764160,"available":96604528640,"internally_reserved":0,"allocated":311296,"data_stored":115831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31476,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340364,"statfs":{"total":96636764160,"available":96604532736,"internally_reserved":0,"allocated":307200,"data_stored":115901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32056,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339784,"statfs":{"total":96636764160,"available":96603938816,"internally_reserved":0,"allocated":901120,"data_stored":705611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T19:48:49.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:48:49.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.356+0000 7f29cde00640 1 Processor -- start 2024-04-17T19:48:49.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.356+0000 7f29cde00640 1 -- start start 2024-04-17T19:48:49.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.356+0000 7f29cde00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29c80c71b0 0x7f29c815c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:49.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.356+0000 7f29cde00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29c80c60e0 con 0x7f29c80c75c0 2024-04-17T19:48:49.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.356+0000 7f29cde00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29c80c6350 con 0x7f29c81523d0 2024-04-17T19:48:49.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.356+0000 7f29cde00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f29c815be90 con 0x7f29c80c71b0 2024-04-17T19:48:49.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.356+0000 7f29bfe00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f29c80c75c0 0x7f29c806bf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55842/0 (socket says 172.21.15.29:55842) 2024-04-17T19:48:49.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.356+0000 7f29bfe00640 1 -- 172.21.15.29:0/1321696878 learned_addr learned my addr 172.21.15.29:0/1321696878 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:49.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6a00640 1 --2- 172.21.15.29:0/1321696878 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29c80c71b0 0x7f29c815c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:49.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6000640 1 -- 172.21.15.29:0/1321696878 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3520565627 0 0) 0x7f29c80c60e0 con 0x7f29c80c75c0 2024-04-17T19:48:49.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6000640 1 -- 172.21.15.29:0/1321696878 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29ac003530 con 0x7f29c80c75c0 2024-04-17T19:48:49.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6000640 1 -- 172.21.15.29:0/1321696878 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 887062100 0 0) 0x7f29c80c6350 con 0x7f29c81523d0 2024-04-17T19:48:49.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6000640 1 -- 172.21.15.29:0/1321696878 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29c80c60e0 con 0x7f29c81523d0 2024-04-17T19:48:49.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6000640 1 -- 172.21.15.29:0/1321696878 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1536825570 0 0) 0x7f29ac003530 con 0x7f29c80c75c0 2024-04-17T19:48:49.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6000640 1 -- 172.21.15.29:0/1321696878 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29c80c71b0 msgr2=0x7f29c815c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:49.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6000640 1 --2- 172.21.15.29:0/1321696878 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29c80c71b0 0x7f29c815c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6000640 1 -- 172.21.15.29:0/1321696878 >> v1:172.21.15.37:6789/0 conn(0x7f29c81523d0 legacy=0x7f29c8162e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:49.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6000640 1 -- 172.21.15.29:0/1321696878 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c80c6630 con 0x7f29c80c75c0 2024-04-17T19:48:49.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6000640 1 -- 172.21.15.29:0/1321696878 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f29b00016a0 con 0x7f29c80c75c0 2024-04-17T19:48:49.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6a00640 1 --2- 172.21.15.29:0/1321696878 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29c80c71b0 0x7f29c815c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:48:49.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6000640 1 -- 172.21.15.29:0/1321696878 >> v1:172.21.15.29:6789/0 conn(0x7f29c80c75c0 legacy=0x7f29c806bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:49.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6000640 1 --2- 172.21.15.29:0/1321696878 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29c80c71b0 0x7f29ac003bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:49.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6000640 1 --2- 172.21.15.29:0/1321696878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29ac004150 0x7f29ac004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:49.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6000640 1 --2- 172.21.15.29:0/1321696878 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29ac004d50 0x7f29ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:49.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6000640 1 -- 172.21.15.29:0/1321696878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f29c815be90 con 0x7f29ac004150 2024-04-17T19:48:49.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6000640 1 -- 172.21.15.29:0/1321696878 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f29ac00ddd0 con 0x7f29ac004d50 2024-04-17T19:48:49.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6000640 1 -- 172.21.15.29:0/1321696878 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f29ac00dfd0 con 0x7f29c80c71b0 2024-04-17T19:48:49.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29bfe00640 1 --2- 172.21.15.29:0/1321696878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29ac004150 0x7f29ac004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:49.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c7400640 1 --2- 172.21.15.29:0/1321696878 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29ac004d50 0x7f29ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:49.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29c6a00640 1 --2- 172.21.15.29:0/1321696878 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29c80c71b0 0x7f29ac003bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:49.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29bfe00640 1 -- 172.21.15.29:0/1321696878 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29c80c71b0 msgr2=0x7f29ac003bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:49.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29bfe00640 1 --2- 172.21.15.29:0/1321696878 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29c80c71b0 0x7f29ac003bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29bfe00640 1 -- 172.21.15.29:0/1321696878 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29ac004d50 msgr2=0x7f29ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:49.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29bfe00640 1 --2- 172.21.15.29:0/1321696878 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29ac004d50 0x7f29ac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.360+0000 7f29bfe00640 1 -- 172.21.15.29:0/1321696878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ac003530 con 0x7f29ac004150 2024-04-17T19:48:49.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.364+0000 7f29bfe00640 1 --2- 172.21.15.29:0/1321696878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29ac004150 0x7f29ac004810 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f29c80c7a00 tx=0x7f29b001f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b470f4ca75ed2850 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:49.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.364+0000 7f29c6000640 1 -- 172.21.15.29:0/1321696878 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b0007cf0 con 0x7f29ac004150 2024-04-17T19:48:49.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.364+0000 7f29c6000640 1 -- 172.21.15.29:0/1321696878 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29b0005b00 con 0x7f29ac004150 2024-04-17T19:48:49.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.364+0000 7f29c6000640 1 -- 172.21.15.29:0/1321696878 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b0007070 con 0x7f29ac004150 2024-04-17T19:48:49.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.364+0000 7f29cde00640 1 -- 172.21.15.29:0/1321696878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29ac004150 msgr2=0x7f29ac004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:49.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.364+0000 7f29cde00640 1 --2- 172.21.15.29:0/1321696878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29ac004150 0x7f29ac004810 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f29c80c7a00 tx=0x7f29b001f330 comp rx=0 tx=0).stop 2024-04-17T19:48:49.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.364+0000 7f29c7400640 1 -- 172.21.15.29:0/1321696878 reap_dead start 2024-04-17T19:48:49.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.364+0000 7f29cde00640 1 -- 172.21.15.29:0/1321696878 shutdown_connections 2024-04-17T19:48:49.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.364+0000 7f29cde00640 1 -- 172.21.15.29:0/1321696878 >> 172.21.15.29:0/1321696878 conn(0x7f29c806fde0 msgr2=0x7f29c80701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:49.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.364+0000 7f29cde00640 1 -- 172.21.15.29:0/1321696878 shutdown_connections 2024-04-17T19:48:49.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.364+0000 7f29cde00640 1 -- 172.21.15.29:0/1321696878 wait complete. 2024-04-17T19:48:49.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.364+0000 7f29cde00640 1 Processor -- start 2024-04-17T19:48:49.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.364+0000 7f29cde00640 1 -- start start 2024-04-17T19:48:49.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29cde00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29ac004150 0x7f29c81f6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:49.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29cde00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c81f34a0 0x7f29c814e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:49.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29cde00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29c81f2a30 0x7f29c8073760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:49.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29cde00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f29c80cb140 con 0x7f29c81f34a0 2024-04-17T19:48:49.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29cde00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f29c806c6a0 con 0x7f29c81f2a30 2024-04-17T19:48:49.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29cde00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f29c80c6910 con 0x7f29ac004150 2024-04-17T19:48:49.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29bfe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c81f34a0 0x7f29c814e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:49.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29bfe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c81f34a0 0x7f29c814e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48482/0 (socket says 172.21.15.29:48482) 2024-04-17T19:48:49.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29bfe00640 1 -- 172.21.15.29:0/3365821814 learned_addr learned my addr 172.21.15.29:0/3365821814 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:49.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29c6a00640 1 --2- 172.21.15.29:0/3365821814 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29ac004150 0x7f29c81f6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:49.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29c7400640 1 --2- 172.21.15.29:0/3365821814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29c81f2a30 0x7f29c8073760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:49.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29bfe00640 1 -- 172.21.15.29:0/3365821814 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29ac004150 msgr2=0x7f29c81f6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:49.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29bfe00640 1 --2- 172.21.15.29:0/3365821814 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29ac004150 0x7f29c81f6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29bfe00640 1 -- 172.21.15.29:0/3365821814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29c81f2a30 msgr2=0x7f29c8073760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:49.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29bfe00640 1 --2- 172.21.15.29:0/3365821814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29c81f2a30 0x7f29c8073760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29bfe00640 1 -- 172.21.15.29:0/3365821814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c814f480 con 0x7f29c81f34a0 2024-04-17T19:48:49.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29bfe00640 1 --2- 172.21.15.29:0/3365821814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c81f34a0 0x7f29c814e830 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f29b0000c10 tx=0x7f29ac005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:49.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29bf400640 1 -- 172.21.15.29:0/3365821814 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b0001c30 con 0x7f29c81f34a0 2024-04-17T19:48:49.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29bf400640 1 -- 172.21.15.29:0/3365821814 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29b0005cb0 con 0x7f29c81f34a0 2024-04-17T19:48:49.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29cde00640 1 -- 172.21.15.29:0/3365821814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29c814f1a0 con 0x7f29c81f34a0 2024-04-17T19:48:49.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29bf400640 1 -- 172.21.15.29:0/3365821814 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b0005e60 con 0x7f29c81f34a0 2024-04-17T19:48:49.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29cde00640 1 -- 172.21.15.29:0/3365821814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29c81f5920 con 0x7f29c81f34a0 2024-04-17T19:48:49.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.368+0000 7f29cde00640 1 -- 172.21.15.29:0/3365821814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c8069b90 con 0x7f29c81f34a0 2024-04-17T19:48:49.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.372+0000 7f29bf400640 1 -- 172.21.15.29:0/3365821814 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f29b00ba030 con 0x7f29c81f34a0 2024-04-17T19:48:49.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.372+0000 7f29bf400640 1 --2- 172.21.15.29:0/3365821814 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f29a4068380 0x7f29a406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:49.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.372+0000 7f29bf400640 1 -- 172.21.15.29:0/3365821814 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f29b0123a60 con 0x7f29c81f34a0 2024-04-17T19:48:49.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.372+0000 7f29c6a00640 1 --2- 172.21.15.29:0/3365821814 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f29a4068380 0x7f29a406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:49.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.372+0000 7f29c6a00640 1 --2- 172.21.15.29:0/3365821814 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f29a4068380 0x7f29a406a890 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009160 tx=0x7f29b8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:49.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.376+0000 7f29bf400640 1 -- 172.21.15.29:0/3365821814 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f29b0019040 con 0x7f29c81f34a0 2024-04-17T19:48:49.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.540+0000 7f29cde00640 1 -- 172.21.15.29:0/3365821814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f29c8075700 con 0x7f29c81f34a0 2024-04-17T19:48:49.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.544+0000 7f29bf400640 1 -- 172.21.15.29:0/3365821814 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18780 (secure 0 0 0) 0x7f29b00f3af0 con 0x7f29c81f34a0 2024-04-17T19:48:49.532 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:49.532 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":26,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:48:29.829433+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T19:47:47.658449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T19:47:48.209047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T19:47:51.887909+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:48:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.544+0000 7f29cde00640 1 -- 172.21.15.29:0/3365821814 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f29a4068380 msgr2=0x7f29a406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.544+0000 7f29cde00640 1 --2- 172.21.15.29:0/3365821814 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f29a4068380 0x7f29a406a890 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009160 tx=0x7f29b8008040 comp rx=0 tx=0).stop 2024-04-17T19:48:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.544+0000 7f29cde00640 1 -- 172.21.15.29:0/3365821814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c81f34a0 msgr2=0x7f29c814e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.544+0000 7f29cde00640 1 --2- 172.21.15.29:0/3365821814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c81f34a0 0x7f29c814e830 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f29b0000c10 tx=0x7f29ac005840 comp rx=0 tx=0).stop 2024-04-17T19:48:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.544+0000 7f29cde00640 1 -- 172.21.15.29:0/3365821814 shutdown_connections 2024-04-17T19:48:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.544+0000 7f29cde00640 1 --2- 172.21.15.29:0/3365821814 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f29a4068380 0x7f29a406a890 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.544+0000 7f29cde00640 1 --2- 172.21.15.29:0/3365821814 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f29c81f2a30 0x7f29c8073760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.544+0000 7f29cde00640 1 --2- 172.21.15.29:0/3365821814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29c81f34a0 0x7f29c814e830 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.544+0000 7f29cde00640 1 --2- 172.21.15.29:0/3365821814 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f29ac004150 0x7f29c81f6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.544+0000 7f29cde00640 1 -- 172.21.15.29:0/3365821814 >> 172.21.15.29:0/3365821814 conn(0x7f29c806fde0 msgr2=0x7f29c80cc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.544+0000 7f29cde00640 1 -- 172.21.15.29:0/3365821814 shutdown_connections 2024-04-17T19:48:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.544+0000 7f29cde00640 1 -- 172.21.15.29:0/3365821814 wait complete. 2024-04-17T19:48:49.554 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T19:47:36.674676+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T19:47:47.658449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T19:47:48.209047+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T19:47:51.887909+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T19:48:49.554 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T19:48:49.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.720+0000 7f8d10400640 1 Processor -- start 2024-04-17T19:48:49.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.720+0000 7f8d10400640 1 -- start start 2024-04-17T19:48:49.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.720+0000 7f8d10400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8d081593d0 0x7f8d081597e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:49.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.720+0000 7f8d10400640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d0815c6d0 con 0x7f8d080c9f00 2024-04-17T19:48:49.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.720+0000 7f8d10400640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d0815c9b0 con 0x7f8d08071ce0 2024-04-17T19:48:49.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.720+0000 7f8d10400640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f8d080c7bd0 con 0x7f8d081593d0 2024-04-17T19:48:49.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.720+0000 7f8d0d000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8d081593d0 0x7f8d081597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:49.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.720+0000 7f8d0e400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8d080c9f00 0x7f8d08163620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55852/0 (socket says 172.21.15.29:55852) 2024-04-17T19:48:49.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.720+0000 7f8d0e400640 1 -- 172.21.15.29:0/625258076 learned_addr learned my addr 172.21.15.29:0/625258076 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:49.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.724+0000 7f8d07e00640 1 -- 172.21.15.29:0/625258076 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4060324236 0 0) 0x7f8d0815c6d0 con 0x7f8d080c9f00 2024-04-17T19:48:49.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.724+0000 7f8d07e00640 1 -- 172.21.15.29:0/625258076 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cf0005740 con 0x7f8d080c9f00 2024-04-17T19:48:49.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.724+0000 7f8d0d000640 1 -- 172.21.15.29:0/625258076 >> v1:172.21.15.37:6789/0 conn(0x7f8d08071ce0 legacy=0x7f8d08141de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:49.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.724+0000 7f8d0d000640 1 -- 172.21.15.29:0/625258076 >> v1:172.21.15.29:6789/0 conn(0x7f8d080c9f00 legacy=0x7f8d08163620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:49.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.724+0000 7f8d0d000640 1 -- 172.21.15.29:0/625258076 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d0815cc90 con 0x7f8d081593d0 2024-04-17T19:48:49.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.724+0000 7f8d0d000640 1 --2- 172.21.15.29:0/625258076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d081593d0 0x7f8d081597e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8cf40097c0 tx=0x7f8cf401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8ff04ff5949afd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:49.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.724+0000 7f8d07e00640 1 -- 172.21.15.29:0/625258076 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf40b0070 con 0x7f8d081593d0 2024-04-17T19:48:49.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.724+0000 7f8d07e00640 1 -- 172.21.15.29:0/625258076 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cf40ab600 con 0x7f8d081593d0 2024-04-17T19:48:49.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.724+0000 7f8d07e00640 1 -- 172.21.15.29:0/625258076 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf40ab9b0 con 0x7f8d081593d0 2024-04-17T19:48:49.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.724+0000 7f8d10400640 1 -- 172.21.15.29:0/625258076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d081593d0 msgr2=0x7f8d081597e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:49.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.724+0000 7f8d10400640 1 --2- 172.21.15.29:0/625258076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d081593d0 0x7f8d081597e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8cf40097c0 tx=0x7f8cf401c4a0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.724+0000 7f8d10400640 1 -- 172.21.15.29:0/625258076 shutdown_connections 2024-04-17T19:48:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.724+0000 7f8d10400640 1 --2- 172.21.15.29:0/625258076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d081593d0 0x7f8d081597e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.724+0000 7f8d10400640 1 -- 172.21.15.29:0/625258076 >> 172.21.15.29:0/625258076 conn(0x7f8d0806bf90 msgr2=0x7f8d0806c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.728+0000 7f8d10400640 1 -- 172.21.15.29:0/625258076 shutdown_connections 2024-04-17T19:48:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.728+0000 7f8d10400640 1 -- 172.21.15.29:0/625258076 wait complete. 2024-04-17T19:48:49.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.728+0000 7f8d10400640 1 Processor -- start 2024-04-17T19:48:49.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.728+0000 7f8d10400640 1 -- start start 2024-04-17T19:48:49.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.728+0000 7f8d10400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d08071ce0 0x7f8d081f5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.728+0000 7f8d10400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d080c9f00 0x7f8d08139c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.728+0000 7f8d10400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d081593d0 0x7f8d081ebc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.728+0000 7f8d10400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8d08072270 con 0x7f8d080c9f00 2024-04-17T19:48:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.728+0000 7f8d10400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8d080720f0 con 0x7f8d081593d0 2024-04-17T19:48:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.728+0000 7f8d10400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8d0815c240 con 0x7f8d08071ce0 2024-04-17T19:48:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.728+0000 7f8d0d000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d080c9f00 0x7f8d08139c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.728+0000 7f8d0d000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d080c9f00 0x7f8d08139c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48494/0 (socket says 172.21.15.29:48494) 2024-04-17T19:48:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.732+0000 7f8d0d000640 1 -- 172.21.15.29:0/842999135 learned_addr learned my addr 172.21.15.29:0/842999135 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.732+0000 7f8d0da00640 1 --2- 172.21.15.29:0/842999135 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d08071ce0 0x7f8d081f5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.732+0000 7f8d0e400640 1 --2- 172.21.15.29:0/842999135 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d081593d0 0x7f8d081ebc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.732+0000 7f8d0d000640 1 -- 172.21.15.29:0/842999135 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d08071ce0 msgr2=0x7f8d081f5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.732+0000 7f8d0d000640 1 --2- 172.21.15.29:0/842999135 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d08071ce0 0x7f8d081f5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.732+0000 7f8d0d000640 1 -- 172.21.15.29:0/842999135 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d081593d0 msgr2=0x7f8d081ebc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.732+0000 7f8d0d000640 1 --2- 172.21.15.29:0/842999135 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d081593d0 0x7f8d081ebc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.732+0000 7f8d0d000640 1 -- 172.21.15.29:0/842999135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d081f5b60 con 0x7f8d080c9f00 2024-04-17T19:48:49.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.732+0000 7f8d0e400640 1 --2- 172.21.15.29:0/842999135 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d081593d0 0x7f8d081ebc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:49.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.732+0000 7f8d0d000640 1 --2- 172.21.15.29:0/842999135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d080c9f00 0x7f8d08139c00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8cf40ae040 tx=0x7f8cf401ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:49.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.732+0000 7f8d05600640 1 -- 172.21.15.29:0/842999135 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf40bc070 con 0x7f8d080c9f00 2024-04-17T19:48:49.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.732+0000 7f8d05600640 1 -- 172.21.15.29:0/842999135 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cf40b1040 con 0x7f8d080c9f00 2024-04-17T19:48:49.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.732+0000 7f8d10400640 1 -- 172.21.15.29:0/842999135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d0815aa10 con 0x7f8d080c9f00 2024-04-17T19:48:49.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.732+0000 7f8d05600640 1 -- 172.21.15.29:0/842999135 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf40b0050 con 0x7f8d080c9f00 2024-04-17T19:48:49.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.732+0000 7f8d10400640 1 -- 172.21.15.29:0/842999135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d0815afd0 con 0x7f8d080c9f00 2024-04-17T19:48:49.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.732+0000 7f8d10400640 1 -- 172.21.15.29:0/842999135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d08069b90 con 0x7f8d080c9f00 2024-04-17T19:48:49.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.732+0000 7f8d05600640 1 -- 172.21.15.29:0/842999135 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8cf40bf070 con 0x7f8d080c9f00 2024-04-17T19:48:49.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.736+0000 7f8d05600640 1 --2- 172.21.15.29:0/842999135 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8ce8068380 0x7f8ce806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:49.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.736+0000 7f8d05600640 1 -- 172.21.15.29:0/842999135 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8cf41207e0 con 0x7f8d080c9f00 2024-04-17T19:48:49.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.736+0000 7f8d0da00640 1 --2- 172.21.15.29:0/842999135 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8ce8068380 0x7f8ce806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:49.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.736+0000 7f8d0da00640 1 --2- 172.21.15.29:0/842999135 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8ce8068380 0x7f8ce806a890 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc007fd0 tx=0x7f8cfc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:49.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.740+0000 7f8d05600640 1 -- 172.21.15.29:0/842999135 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8cf4016090 con 0x7f8d080c9f00 2024-04-17T19:48:49.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.904+0000 7f8d10400640 1 -- 172.21.15.29:0/842999135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8d0813a140 con 0x7f8d080c9f00 2024-04-17T19:48:49.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.908+0000 7f8d05600640 1 -- 172.21.15.29:0/842999135 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f8cf40f0220 con 0x7f8d080c9f00 2024-04-17T19:48:49.895 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-04-17T19:48:49.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.908+0000 7f8d10400640 1 -- 172.21.15.29:0/842999135 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8ce8068380 msgr2=0x7f8ce806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:49.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.908+0000 7f8d10400640 1 --2- 172.21.15.29:0/842999135 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8ce8068380 0x7f8ce806a890 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc007fd0 tx=0x7f8cfc009040 comp rx=0 tx=0).stop 2024-04-17T19:48:49.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.908+0000 7f8d10400640 1 -- 172.21.15.29:0/842999135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d080c9f00 msgr2=0x7f8d08139c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:49.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.908+0000 7f8d10400640 1 --2- 172.21.15.29:0/842999135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d080c9f00 0x7f8d08139c00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8cf40ae040 tx=0x7f8cf401ace0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.908+0000 7f8d10400640 1 -- 172.21.15.29:0/842999135 shutdown_connections 2024-04-17T19:48:49.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.908+0000 7f8d10400640 1 --2- 172.21.15.29:0/842999135 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8ce8068380 0x7f8ce806a890 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.908+0000 7f8d10400640 1 --2- 172.21.15.29:0/842999135 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8d081593d0 0x7f8d081ebc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.908+0000 7f8d10400640 1 --2- 172.21.15.29:0/842999135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d080c9f00 0x7f8d08139c00 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.908+0000 7f8d10400640 1 --2- 172.21.15.29:0/842999135 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8d08071ce0 0x7f8d081f5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:49.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.908+0000 7f8d10400640 1 -- 172.21.15.29:0/842999135 >> 172.21.15.29:0/842999135 conn(0x7f8d0806bf90 msgr2=0x7f8d08141de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:49.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.908+0000 7f8d10400640 1 -- 172.21.15.29:0/842999135 shutdown_connections 2024-04-17T19:48:49.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:49.908+0000 7f8d10400640 1 -- 172.21.15.29:0/842999135 wait complete. 2024-04-17T19:48:49.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T19:48:50.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.088+0000 7fb07ec00640 1 Processor -- start 2024-04-17T19:48:50.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.088+0000 7fb07ec00640 1 -- start start 2024-04-17T19:48:50.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.088+0000 7fb07ec00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb07815a7f0 0x7fb0780c4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.092+0000 7fb07ec00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0780ca7e0 0x7fb078154020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.092+0000 7fb07ec00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0780c84c0 con 0x7fb078153c10 2024-04-17T19:48:50.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.092+0000 7fb07ec00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fb078154560 con 0x7fb0780ca7e0 2024-04-17T19:48:50.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.092+0000 7fb07ec00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb0780c87a0 con 0x7fb07815a7f0 2024-04-17T19:48:50.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.092+0000 7fb07cc00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0780ca7e0 0x7fb078154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.092+0000 7fb06fe00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb078153c10 0x7fb0780ca0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55858/0 (socket says 172.21.15.29:55858) 2024-04-17T19:48:50.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.092+0000 7fb06fe00640 1 -- 172.21.15.29:0/3038077045 learned_addr learned my addr 172.21.15.29:0/3038077045 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:50.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.092+0000 7fb077e00640 1 --2- 172.21.15.29:0/3038077045 >> v2:172.21.15.37:3300/0 conn(0x7fb07815a7f0 0x7fb0780c4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.092+0000 7fb07cc00640 1 -- 172.21.15.29:0/3038077045 >> v2:172.21.15.37:3300/0 conn(0x7fb07815a7f0 msgr2=0x7fb0780c4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:50.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.092+0000 7fb07cc00640 1 --2- 172.21.15.29:0/3038077045 >> v2:172.21.15.37:3300/0 conn(0x7fb07815a7f0 0x7fb0780c4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.092+0000 7fb07cc00640 1 -- 172.21.15.29:0/3038077045 >> v1:172.21.15.29:6789/0 conn(0x7fb078153c10 legacy=0x7fb0780ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.092+0000 7fb07cc00640 1 -- 172.21.15.29:0/3038077045 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb07815d390 con 0x7fb0780ca7e0 2024-04-17T19:48:50.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.092+0000 7fb077400640 1 -- 172.21.15.29:0/3038077045 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 493628062 0 0) 0x7fb0780c84c0 con 0x7fb078153c10 2024-04-17T19:48:50.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.092+0000 7fb07cc00640 1 --2- 172.21.15.29:0/3038077045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0780ca7e0 0x7fb078154020 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb078070820 tx=0x7fb07001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47161016569d37df server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:50.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.096+0000 7fb077400640 1 -- 172.21.15.29:0/3038077045 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0700b2070 con 0x7fb0780ca7e0 2024-04-17T19:48:50.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.096+0000 7fb077400640 1 -- 172.21.15.29:0/3038077045 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0700a58e0 con 0x7fb0780ca7e0 2024-04-17T19:48:50.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.096+0000 7fb077400640 1 -- 172.21.15.29:0/3038077045 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0700a5cb0 con 0x7fb0780ca7e0 2024-04-17T19:48:50.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.096+0000 7fb07ec00640 1 -- 172.21.15.29:0/3038077045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0780ca7e0 msgr2=0x7fb078154020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.096+0000 7fb07ec00640 1 --2- 172.21.15.29:0/3038077045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0780ca7e0 0x7fb078154020 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb078070820 tx=0x7fb07001c930 comp rx=0 tx=0).stop 2024-04-17T19:48:50.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.096+0000 7fb07ec00640 1 -- 172.21.15.29:0/3038077045 shutdown_connections 2024-04-17T19:48:50.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.096+0000 7fb07ec00640 1 --2- 172.21.15.29:0/3038077045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0780ca7e0 0x7fb078154020 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.096+0000 7fb07ec00640 1 --2- 172.21.15.29:0/3038077045 >> v2:172.21.15.37:3300/0 conn(0x7fb07815a7f0 0x7fb0780c4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.096+0000 7fb07ec00640 1 -- 172.21.15.29:0/3038077045 >> 172.21.15.29:0/3038077045 conn(0x7fb07806bf90 msgr2=0x7fb07806c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:50.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.096+0000 7fb07ec00640 1 -- 172.21.15.29:0/3038077045 shutdown_connections 2024-04-17T19:48:50.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.096+0000 7fb07ec00640 1 -- 172.21.15.29:0/3038077045 wait complete. 2024-04-17T19:48:50.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.096+0000 7fb07ec00640 1 Processor -- start 2024-04-17T19:48:50.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.096+0000 7fb07ec00640 1 -- start start 2024-04-17T19:48:50.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb07ec00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0780ca7e0 0x7fb078163810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb07ec00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb078153c10 0x7fb0780cc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb07ec00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb07815a7f0 0x7fb0780cc560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb07ec00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb07815e150 con 0x7fb0780ca7e0 2024-04-17T19:48:50.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb07ec00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb07815ac00 con 0x7fb078153c10 2024-04-17T19:48:50.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb07ec00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb0780c4f40 con 0x7fb07815a7f0 2024-04-17T19:48:50.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb077e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0780ca7e0 0x7fb078163810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb077e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0780ca7e0 0x7fb078163810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48516/0 (socket says 172.21.15.29:48516) 2024-04-17T19:48:50.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb077e00640 1 -- 172.21.15.29:0/2376146920 learned_addr learned my addr 172.21.15.29:0/2376146920 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:50.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb07cc00640 1 --2- 172.21.15.29:0/2376146920 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb07815a7f0 0x7fb0780cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb06fe00640 1 --2- 172.21.15.29:0/2376146920 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb078153c10 0x7fb0780cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb077e00640 1 -- 172.21.15.29:0/2376146920 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb07815a7f0 msgr2=0x7fb0780cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb077e00640 1 --2- 172.21.15.29:0/2376146920 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb07815a7f0 0x7fb0780cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb077e00640 1 -- 172.21.15.29:0/2376146920 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb078153c10 msgr2=0x7fb0780cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.090 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb077e00640 1 --2- 172.21.15.29:0/2376146920 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb078153c10 0x7fb0780cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb077e00640 1 -- 172.21.15.29:0/2376146920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb078071e80 con 0x7fb0780ca7e0 2024-04-17T19:48:50.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb077e00640 1 --2- 172.21.15.29:0/2376146920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0780ca7e0 0x7fb078163810 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb06800e870 tx=0x7fb06800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:50.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb074c00640 1 -- 172.21.15.29:0/2376146920 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb068017070 con 0x7fb0780ca7e0 2024-04-17T19:48:50.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb074c00640 1 -- 172.21.15.29:0/2376146920 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb06800c8e0 con 0x7fb0780ca7e0 2024-04-17T19:48:50.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb074c00640 1 -- 172.21.15.29:0/2376146920 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb06800cc90 con 0x7fb0780ca7e0 2024-04-17T19:48:50.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb07ec00640 1 -- 172.21.15.29:0/2376146920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb078072160 con 0x7fb0780ca7e0 2024-04-17T19:48:50.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb07ec00640 1 -- 172.21.15.29:0/2376146920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb078144ce0 con 0x7fb0780ca7e0 2024-04-17T19:48:50.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.100+0000 7fb07ec00640 1 -- 172.21.15.29:0/2376146920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb078069b90 con 0x7fb0780ca7e0 2024-04-17T19:48:50.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.104+0000 7fb074c00640 1 -- 172.21.15.29:0/2376146920 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb06801a070 con 0x7fb0780ca7e0 2024-04-17T19:48:50.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.104+0000 7fb074c00640 1 --2- 172.21.15.29:0/2376146920 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb054068380 0x7fb05406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.104+0000 7fb074c00640 1 -- 172.21.15.29:0/2376146920 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fb068086c50 con 0x7fb0780ca7e0 2024-04-17T19:48:50.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.104+0000 7fb06fe00640 1 --2- 172.21.15.29:0/2376146920 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb054068380 0x7fb05406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.104+0000 7fb06fe00640 1 --2- 172.21.15.29:0/2376146920 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb054068380 0x7fb05406a890 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb0780c40f0 tx=0x7fb060007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:50.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.108+0000 7fb074c00640 1 -- 172.21.15.29:0/2376146920 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb068056d80 con 0x7fb0780ca7e0 2024-04-17T19:48:50.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.272+0000 7fb07ec00640 1 -- 172.21.15.29:0/2376146920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb0780c43b0 con 0x7fb0780ca7e0 2024-04-17T19:48:50.260 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.272+0000 7fb074c00640 1 -- 172.21.15.29:0/2376146920 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fb068056680 con 0x7fb0780ca7e0 2024-04-17T19:48:50.260 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-04-17T19:48:50.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.272+0000 7fb07ec00640 1 -- 172.21.15.29:0/2376146920 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb054068380 msgr2=0x7fb05406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.272+0000 7fb07ec00640 1 --2- 172.21.15.29:0/2376146920 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb054068380 0x7fb05406a890 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fb0780c40f0 tx=0x7fb060007c10 comp rx=0 tx=0).stop 2024-04-17T19:48:50.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.272+0000 7fb07ec00640 1 -- 172.21.15.29:0/2376146920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0780ca7e0 msgr2=0x7fb078163810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.272+0000 7fb07ec00640 1 --2- 172.21.15.29:0/2376146920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0780ca7e0 0x7fb078163810 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb06800e870 tx=0x7fb06800ec70 comp rx=0 tx=0).stop 2024-04-17T19:48:50.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.272+0000 7fb07ec00640 1 -- 172.21.15.29:0/2376146920 shutdown_connections 2024-04-17T19:48:50.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.272+0000 7fb07ec00640 1 --2- 172.21.15.29:0/2376146920 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb054068380 0x7fb05406a890 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.272+0000 7fb07ec00640 1 --2- 172.21.15.29:0/2376146920 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb07815a7f0 0x7fb0780cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.272+0000 7fb07ec00640 1 --2- 172.21.15.29:0/2376146920 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb078153c10 0x7fb0780cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.272+0000 7fb07ec00640 1 --2- 172.21.15.29:0/2376146920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0780ca7e0 0x7fb078163810 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.272+0000 7fb07ec00640 1 -- 172.21.15.29:0/2376146920 >> 172.21.15.29:0/2376146920 conn(0x7fb07806bf90 msgr2=0x7fb0780c5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:50.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.272+0000 7fb07ec00640 1 -- 172.21.15.29:0/2376146920 shutdown_connections 2024-04-17T19:48:50.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.272+0000 7fb07ec00640 1 -- 172.21.15.29:0/2376146920 wait complete. 2024-04-17T19:48:50.282 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T19:48:50.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.448+0000 7f5328600640 1 Processor -- start 2024-04-17T19:48:50.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.448+0000 7f5328600640 1 -- start start 2024-04-17T19:48:50.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f5328600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53200ca7e0 0x7f5320154020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f5328600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53200c5110 con 0x7f532015a7f0 2024-04-17T19:48:50.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f5328600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53200c84c0 con 0x7f5320153c10 2024-04-17T19:48:50.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f5328600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5320154560 con 0x7f53200ca7e0 2024-04-17T19:48:50.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f5325c00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f532015a7f0 0x7f53200c4a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55874/0 (socket says 172.21.15.29:55874) 2024-04-17T19:48:50.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f5325c00640 1 -- 172.21.15.29:0/525285032 learned_addr learned my addr 172.21.15.29:0/525285032 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:50.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f5326600640 1 --2- 172.21.15.29:0/525285032 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53200ca7e0 0x7f5320154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f531fe00640 1 -- 172.21.15.29:0/525285032 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2011930907 0 0) 0x7f53200c5110 con 0x7f532015a7f0 2024-04-17T19:48:50.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f531fe00640 1 -- 172.21.15.29:0/525285032 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5308003530 con 0x7f532015a7f0 2024-04-17T19:48:50.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f531fe00640 1 -- 172.21.15.29:0/525285032 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 96881895 0 0) 0x7f5308003530 con 0x7f532015a7f0 2024-04-17T19:48:50.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f531fe00640 1 -- 172.21.15.29:0/525285032 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53200ca7e0 msgr2=0x7f5320154020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:50.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f531fe00640 1 --2- 172.21.15.29:0/525285032 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53200ca7e0 0x7f5320154020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f531fe00640 1 -- 172.21.15.29:0/525285032 >> v1:172.21.15.37:6789/0 conn(0x7f5320153c10 legacy=0x7f53200ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f531fe00640 1 -- 172.21.15.29:0/525285032 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53200c87a0 con 0x7f532015a7f0 2024-04-17T19:48:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f531fe00640 1 -- 172.21.15.29:0/525285032 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f5314001680 con 0x7f532015a7f0 2024-04-17T19:48:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f531fe00640 1 -- 172.21.15.29:0/525285032 >> v1:172.21.15.29:6789/0 conn(0x7f532015a7f0 legacy=0x7f53200c4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f531fe00640 1 --2- 172.21.15.29:0/525285032 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53080039f0 0x7f5308003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f531fe00640 1 --2- 172.21.15.29:0/525285032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5308004810 0x7f5308004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f531fe00640 1 --2- 172.21.15.29:0/525285032 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53200ca7e0 0x7f5308005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f531fe00640 1 -- 172.21.15.29:0/525285032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5320154560 con 0x7f5308004810 2024-04-17T19:48:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f531fe00640 1 -- 172.21.15.29:0/525285032 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f530800ba80 con 0x7f53080039f0 2024-04-17T19:48:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f531fe00640 1 -- 172.21.15.29:0/525285032 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f530800bc80 con 0x7f53200ca7e0 2024-04-17T19:48:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f5325200640 1 --2- 172.21.15.29:0/525285032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5308004810 0x7f5308004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f5325c00640 1 --2- 172.21.15.29:0/525285032 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53080039f0 0x7f5308003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f5326600640 1 --2- 172.21.15.29:0/525285032 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53200ca7e0 0x7f5308005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f5325200640 1 -- 172.21.15.29:0/525285032 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53200ca7e0 msgr2=0x7f5308005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f5325200640 1 --2- 172.21.15.29:0/525285032 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f53200ca7e0 0x7f5308005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f5325200640 1 -- 172.21.15.29:0/525285032 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53080039f0 msgr2=0x7f5308003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:50.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f5325200640 1 --2- 172.21.15.29:0/525285032 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f53080039f0 0x7f5308003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.444 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.452+0000 7f5325200640 1 -- 172.21.15.29:0/525285032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5308003530 con 0x7f5308004810 2024-04-17T19:48:50.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.456+0000 7f5325200640 1 --2- 172.21.15.29:0/525285032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5308004810 0x7f5308004c20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f530c001910 tx=0x7f530c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26a36559c818c34d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:50.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.456+0000 7f531fe00640 1 -- 172.21.15.29:0/525285032 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c0b5070 con 0x7f5308004810 2024-04-17T19:48:50.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.456+0000 7f531fe00640 1 -- 172.21.15.29:0/525285032 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f530c0a8930 con 0x7f5308004810 2024-04-17T19:48:50.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.456+0000 7f531fe00640 1 -- 172.21.15.29:0/525285032 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c001510 con 0x7f5308004810 2024-04-17T19:48:50.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.456+0000 7f5328600640 1 -- 172.21.15.29:0/525285032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5308004810 msgr2=0x7f5308004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.456+0000 7f5328600640 1 --2- 172.21.15.29:0/525285032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5308004810 0x7f5308004c20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f530c001910 tx=0x7f530c01ecf0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.456+0000 7f5326600640 1 -- 172.21.15.29:0/525285032 reap_dead start 2024-04-17T19:48:50.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.456+0000 7f5328600640 1 -- 172.21.15.29:0/525285032 shutdown_connections 2024-04-17T19:48:50.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.456+0000 7f5328600640 1 -- 172.21.15.29:0/525285032 >> 172.21.15.29:0/525285032 conn(0x7f532006bf90 msgr2=0x7f532006c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:50.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.456+0000 7f5328600640 1 -- 172.21.15.29:0/525285032 shutdown_connections 2024-04-17T19:48:50.446 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.456+0000 7f5328600640 1 -- 172.21.15.29:0/525285032 wait complete. 2024-04-17T19:48:50.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5328600640 1 Processor -- start 2024-04-17T19:48:50.448 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5328600640 1 -- start start 2024-04-17T19:48:50.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5328600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5308004810 0x7f532014f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5328600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f532015c9b0 0x7f5320151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5328600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5320073030 0x7f53201f5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5328600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f532015d950 con 0x7f5308004810 2024-04-17T19:48:50.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5328600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f532015d670 con 0x7f5320073030 2024-04-17T19:48:50.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5328600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f532015dc30 con 0x7f532015c9b0 2024-04-17T19:48:50.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5325c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5308004810 0x7f532014f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5325c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5308004810 0x7f532014f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48534/0 (socket says 172.21.15.29:48534) 2024-04-17T19:48:50.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5325c00640 1 -- 172.21.15.29:0/628214386 learned_addr learned my addr 172.21.15.29:0/628214386 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:50.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5325200640 1 --2- 172.21.15.29:0/628214386 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f532015c9b0 0x7f5320151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5326600640 1 --2- 172.21.15.29:0/628214386 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5320073030 0x7f53201f5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5325c00640 1 -- 172.21.15.29:0/628214386 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f532015c9b0 msgr2=0x7f5320151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5325c00640 1 --2- 172.21.15.29:0/628214386 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f532015c9b0 0x7f5320151d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5325c00640 1 -- 172.21.15.29:0/628214386 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5320073030 msgr2=0x7f53201f5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5325c00640 1 --2- 172.21.15.29:0/628214386 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5320073030 0x7f53201f5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5325c00640 1 -- 172.21.15.29:0/628214386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5320152e00 con 0x7f5308004810 2024-04-17T19:48:50.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5325c00640 1 --2- 172.21.15.29:0/628214386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5308004810 0x7f532014f200 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5314004ed0 tx=0x7f5314007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:50.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f531d600640 1 -- 172.21.15.29:0/628214386 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531400f750 con 0x7f5308004810 2024-04-17T19:48:50.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f531d600640 1 -- 172.21.15.29:0/628214386 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5314018070 con 0x7f5308004810 2024-04-17T19:48:50.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5328600640 1 -- 172.21.15.29:0/628214386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53201530c0 con 0x7f5308004810 2024-04-17T19:48:50.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f531d600640 1 -- 172.21.15.29:0/628214386 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531400faf0 con 0x7f5308004810 2024-04-17T19:48:50.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.460+0000 7f5328600640 1 -- 172.21.15.29:0/628214386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5320153680 con 0x7f5308004810 2024-04-17T19:48:50.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.464+0000 7f5328600640 1 -- 172.21.15.29:0/628214386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5320069b90 con 0x7f5308004810 2024-04-17T19:48:50.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.464+0000 7f531d600640 1 -- 172.21.15.29:0/628214386 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f531400fc90 con 0x7f5308004810 2024-04-17T19:48:50.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.464+0000 7f531d600640 1 --2- 172.21.15.29:0/628214386 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f53000683b0 0x7f530006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.464+0000 7f531d600640 1 -- 172.21.15.29:0/628214386 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5314086d90 con 0x7f5308004810 2024-04-17T19:48:50.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.464+0000 7f5325200640 1 --2- 172.21.15.29:0/628214386 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f53000683b0 0x7f530006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.468+0000 7f5325200640 1 --2- 172.21.15.29:0/628214386 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f53000683b0 0x7f530006a8c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f530c001910 tx=0x7f530c001330 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:50.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.468+0000 7f531d600640 1 -- 172.21.15.29:0/628214386 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5314056eb0 con 0x7f5308004810 2024-04-17T19:48:50.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.632+0000 7f5328600640 1 -- 172.21.15.29:0/628214386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f53200c51a0 con 0x7f5308004810 2024-04-17T19:48:50.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.632+0000 7f531d600640 1 -- 172.21.15.29:0/628214386 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f53140567b0 con 0x7f5308004810 2024-04-17T19:48:50.622 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2024-04-17T19:48:50.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.636+0000 7f5328600640 1 -- 172.21.15.29:0/628214386 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f53000683b0 msgr2=0x7f530006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.636+0000 7f5328600640 1 --2- 172.21.15.29:0/628214386 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f53000683b0 0x7f530006a8c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f530c001910 tx=0x7f530c001330 comp rx=0 tx=0).stop 2024-04-17T19:48:50.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.636+0000 7f5328600640 1 -- 172.21.15.29:0/628214386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5308004810 msgr2=0x7f532014f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.636+0000 7f5328600640 1 --2- 172.21.15.29:0/628214386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5308004810 0x7f532014f200 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5314004ed0 tx=0x7f5314007430 comp rx=0 tx=0).stop 2024-04-17T19:48:50.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.636+0000 7f5328600640 1 -- 172.21.15.29:0/628214386 shutdown_connections 2024-04-17T19:48:50.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.636+0000 7f5328600640 1 --2- 172.21.15.29:0/628214386 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f53000683b0 0x7f530006a8c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.636+0000 7f5328600640 1 --2- 172.21.15.29:0/628214386 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5320073030 0x7f53201f5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.636+0000 7f5328600640 1 --2- 172.21.15.29:0/628214386 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f532015c9b0 0x7f5320151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.636+0000 7f5328600640 1 --2- 172.21.15.29:0/628214386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5308004810 0x7f532014f200 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.636+0000 7f5328600640 1 -- 172.21.15.29:0/628214386 >> 172.21.15.29:0/628214386 conn(0x7f532006bf90 msgr2=0x7f53200ca0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:50.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.636+0000 7f5328600640 1 -- 172.21.15.29:0/628214386 shutdown_connections 2024-04-17T19:48:50.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.636+0000 7f5328600640 1 -- 172.21.15.29:0/628214386 wait complete. 2024-04-17T19:48:50.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T19:48:50.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.808+0000 7f1d96600640 1 Processor -- start 2024-04-17T19:48:50.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.808+0000 7f1d96600640 1 -- start start 2024-04-17T19:48:50.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.812+0000 7f1d96600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1d9015cef0 0x7f1d90067eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.812+0000 7f1d96600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d9006c940 0x7f1d9006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.812+0000 7f1d96600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d90071240 con 0x7f1d901562c0 2024-04-17T19:48:50.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.812+0000 7f1d96600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f1d900741c0 con 0x7f1d9015cef0 2024-04-17T19:48:50.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.812+0000 7f1d96600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1d900714e0 con 0x7f1d9006c940 2024-04-17T19:48:50.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.812+0000 7f1d8fe00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1d901562c0 0x7f1d901566d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55886/0 (socket says 172.21.15.29:55886) 2024-04-17T19:48:50.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.812+0000 7f1d8fe00640 1 -- 172.21.15.29:0/3308273617 learned_addr learned my addr 172.21.15.29:0/3308273617 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:50.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.812+0000 7f1d8f400640 1 --2- 172.21.15.29:0/3308273617 >> v2:172.21.15.29:3300/0 conn(0x7f1d9015cef0 0x7f1d90067eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.812+0000 7f1d87e00640 1 --2- 172.21.15.29:0/3308273617 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d9006c940 0x7f1d9006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.812+0000 7f1d8ea00640 1 -- 172.21.15.29:0/3308273617 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 803639363 0 0) 0x7f1d90071240 con 0x7f1d901562c0 2024-04-17T19:48:50.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.812+0000 7f1d8ea00640 1 -- 172.21.15.29:0/3308273617 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d74003530 con 0x7f1d901562c0 2024-04-17T19:48:50.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.812+0000 7f1d8f400640 1 -- 172.21.15.29:0/3308273617 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d9006c940 msgr2=0x7f1d9006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:50.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.812+0000 7f1d8f400640 1 --2- 172.21.15.29:0/3308273617 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d9006c940 0x7f1d9006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.812+0000 7f1d8f400640 1 -- 172.21.15.29:0/3308273617 >> v1:172.21.15.29:6789/0 conn(0x7f1d901562c0 legacy=0x7f1d901566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.812+0000 7f1d8f400640 1 -- 172.21.15.29:0/3308273617 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d900717c0 con 0x7f1d9015cef0 2024-04-17T19:48:50.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.812+0000 7f1d87e00640 1 --2- 172.21.15.29:0/3308273617 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d9006c940 0x7f1d9006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:48:50.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.816+0000 7f1d8f400640 1 --2- 172.21.15.29:0/3308273617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d9015cef0 0x7f1d90067eb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1d800097c0 tx=0x7f1d8001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f78bd5fc1cff438 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:50.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.816+0000 7f1d8ea00640 1 -- 172.21.15.29:0/3308273617 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d800b5070 con 0x7f1d9015cef0 2024-04-17T19:48:50.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.816+0000 7f1d8ea00640 1 -- 172.21.15.29:0/3308273617 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d800a8990 con 0x7f1d9015cef0 2024-04-17T19:48:50.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.816+0000 7f1d8ea00640 1 -- 172.21.15.29:0/3308273617 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d800a8d40 con 0x7f1d9015cef0 2024-04-17T19:48:50.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.816+0000 7f1d96600640 1 -- 172.21.15.29:0/3308273617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d9015cef0 msgr2=0x7f1d90067eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.816+0000 7f1d96600640 1 --2- 172.21.15.29:0/3308273617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d9015cef0 0x7f1d90067eb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1d800097c0 tx=0x7f1d8001ec90 comp rx=0 tx=0).stop 2024-04-17T19:48:50.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.816+0000 7f1d96600640 1 -- 172.21.15.29:0/3308273617 shutdown_connections 2024-04-17T19:48:50.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.816+0000 7f1d96600640 1 --2- 172.21.15.29:0/3308273617 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d9006c940 0x7f1d9006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.816+0000 7f1d96600640 1 --2- 172.21.15.29:0/3308273617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d9015cef0 0x7f1d90067eb0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.816+0000 7f1d96600640 1 -- 172.21.15.29:0/3308273617 >> 172.21.15.29:0/3308273617 conn(0x7f1d90072e80 msgr2=0x7f1d90073290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:50.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.816+0000 7f1d96600640 1 -- 172.21.15.29:0/3308273617 shutdown_connections 2024-04-17T19:48:50.806 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.816+0000 7f1d96600640 1 -- 172.21.15.29:0/3308273617 wait complete. 2024-04-17T19:48:50.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d96600640 1 Processor -- start 2024-04-17T19:48:50.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d96600640 1 -- start start 2024-04-17T19:48:50.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d96600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d9006c940 0x7f1d900bfa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d96600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d901562c0 0x7f1d901e5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d96600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d9015cef0 0x7f1d900cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d96600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9006c420 con 0x7f1d9015cef0 2024-04-17T19:48:50.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d96600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9006c140 con 0x7f1d901562c0 2024-04-17T19:48:50.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d96600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1d900c1d40 con 0x7f1d9006c940 2024-04-17T19:48:50.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d8fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d9015cef0 0x7f1d900cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d8fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d9015cef0 0x7f1d900cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48552/0 (socket says 172.21.15.29:48552) 2024-04-17T19:48:50.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d8fe00640 1 -- 172.21.15.29:0/510608001 learned_addr learned my addr 172.21.15.29:0/510608001 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:50.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d8f400640 1 --2- 172.21.15.29:0/510608001 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d9006c940 0x7f1d900bfa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d87e00640 1 --2- 172.21.15.29:0/510608001 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d901562c0 0x7f1d901e5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d8fe00640 1 -- 172.21.15.29:0/510608001 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d9006c940 msgr2=0x7f1d900bfa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d8fe00640 1 --2- 172.21.15.29:0/510608001 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d9006c940 0x7f1d900bfa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d8fe00640 1 -- 172.21.15.29:0/510608001 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d901562c0 msgr2=0x7f1d901e5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d8fe00640 1 --2- 172.21.15.29:0/510608001 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d901562c0 0x7f1d901e5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d8fe00640 1 -- 172.21.15.29:0/510608001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d900c4170 con 0x7f1d9015cef0 2024-04-17T19:48:50.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d8fe00640 1 --2- 172.21.15.29:0/510608001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d9015cef0 0x7f1d900cd6a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1d8800cad0 tx=0x7f1d8800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:50.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.820+0000 7f1d87400640 1 -- 172.21.15.29:0/510608001 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d88017070 con 0x7f1d9015cef0 2024-04-17T19:48:50.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.824+0000 7f1d87400640 1 -- 172.21.15.29:0/510608001 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d8800abd0 con 0x7f1d9015cef0 2024-04-17T19:48:50.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.824+0000 7f1d96600640 1 -- 172.21.15.29:0/510608001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d900c4450 con 0x7f1d9015cef0 2024-04-17T19:48:50.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.824+0000 7f1d87400640 1 -- 172.21.15.29:0/510608001 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d88012400 con 0x7f1d9015cef0 2024-04-17T19:48:50.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.824+0000 7f1d96600640 1 -- 172.21.15.29:0/510608001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d900bf220 con 0x7f1d9015cef0 2024-04-17T19:48:50.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.824+0000 7f1d96600640 1 -- 172.21.15.29:0/510608001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d90069b90 con 0x7f1d9015cef0 2024-04-17T19:48:50.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.824+0000 7f1d87400640 1 -- 172.21.15.29:0/510608001 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1d8801a070 con 0x7f1d9015cef0 2024-04-17T19:48:50.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.824+0000 7f1d87400640 1 --2- 172.21.15.29:0/510608001 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1d6c0683b0 0x7f1d6c06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:50.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.824+0000 7f1d87400640 1 -- 172.21.15.29:0/510608001 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f1d88086a30 con 0x7f1d9015cef0 2024-04-17T19:48:50.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.824+0000 7f1d8f400640 1 --2- 172.21.15.29:0/510608001 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1d6c0683b0 0x7f1d6c06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:50.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.828+0000 7f1d8f400640 1 --2- 172.21.15.29:0/510608001 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1d6c0683b0 0x7f1d6c06a8c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1d800b2000 tx=0x7f1d800aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:50.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.828+0000 7f1d87400640 1 -- 172.21.15.29:0/510608001 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1d88056590 con 0x7f1d9015cef0 2024-04-17T19:48:50.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.996+0000 7f1d96600640 1 -- 172.21.15.29:0/510608001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1d900c8530 con 0x7f1d9015cef0 2024-04-17T19:48:50.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:50.996+0000 7f1d87400640 1 -- 172.21.15.29:0/510608001 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f1d8805a890 con 0x7f1d9015cef0 2024-04-17T19:48:50.987 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2024-04-17T19:48:50.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.000+0000 7f1d96600640 1 -- 172.21.15.29:0/510608001 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1d6c0683b0 msgr2=0x7f1d6c06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.000+0000 7f1d96600640 1 --2- 172.21.15.29:0/510608001 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1d6c0683b0 0x7f1d6c06a8c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1d800b2000 tx=0x7f1d800aa070 comp rx=0 tx=0).stop 2024-04-17T19:48:50.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.000+0000 7f1d96600640 1 -- 172.21.15.29:0/510608001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d9015cef0 msgr2=0x7f1d900cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:50.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.000+0000 7f1d96600640 1 --2- 172.21.15.29:0/510608001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d9015cef0 0x7f1d900cd6a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1d8800cad0 tx=0x7f1d8800cf90 comp rx=0 tx=0).stop 2024-04-17T19:48:50.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.000+0000 7f1d96600640 1 -- 172.21.15.29:0/510608001 shutdown_connections 2024-04-17T19:48:50.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.000+0000 7f1d96600640 1 --2- 172.21.15.29:0/510608001 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1d6c0683b0 0x7f1d6c06a8c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.000+0000 7f1d96600640 1 --2- 172.21.15.29:0/510608001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d9015cef0 0x7f1d900cd6a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.000+0000 7f1d96600640 1 --2- 172.21.15.29:0/510608001 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d901562c0 0x7f1d901e5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.000+0000 7f1d96600640 1 --2- 172.21.15.29:0/510608001 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d9006c940 0x7f1d900bfa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:50.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.000+0000 7f1d96600640 1 -- 172.21.15.29:0/510608001 >> 172.21.15.29:0/510608001 conn(0x7f1d90072e80 msgr2=0x7f1d900c77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:50.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.000+0000 7f1d96600640 1 -- 172.21.15.29:0/510608001 shutdown_connections 2024-04-17T19:48:50.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.000+0000 7f1d96600640 1 -- 172.21.15.29:0/510608001 wait complete. 2024-04-17T19:48:51.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:48:51.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.172+0000 7fa36a000640 1 Processor -- start 2024-04-17T19:48:51.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.172+0000 7fa36a000640 1 -- start start 2024-04-17T19:48:51.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.176+0000 7fa36a000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa364153ea0 0x7fa3640725b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:51.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.176+0000 7fa36a000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa36415d460 0x7fa36415d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:51.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.176+0000 7fa36a000640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa36415ddb0 con 0x7fa364072af0 2024-04-17T19:48:51.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.176+0000 7fa36a000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fa364073060 con 0x7fa36415d460 2024-04-17T19:48:51.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.176+0000 7fa36a000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa36406e430 con 0x7fa364153ea0 2024-04-17T19:48:51.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.176+0000 7fa35be00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa364072af0 0x7fa3640c7000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55890/0 (socket says 172.21.15.29:55890) 2024-04-17T19:48:51.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.176+0000 7fa35be00640 1 -- 172.21.15.29:0/1225061757 learned_addr learned my addr 172.21.15.29:0/1225061757 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:48:51.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.176+0000 7fa363e00640 1 --2- 172.21.15.29:0/1225061757 >> v2:172.21.15.29:3300/0 conn(0x7fa36415d460 0x7fa36415d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:51.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.176+0000 7fa363400640 1 --2- 172.21.15.29:0/1225061757 >> v2:172.21.15.37:3300/0 conn(0x7fa364153ea0 0x7fa3640725b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.176+0000 7fa363e00640 1 -- 172.21.15.29:0/1225061757 >> v2:172.21.15.37:3300/0 conn(0x7fa364153ea0 msgr2=0x7fa3640725b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.176+0000 7fa363e00640 1 --2- 172.21.15.29:0/1225061757 >> v2:172.21.15.37:3300/0 conn(0x7fa364153ea0 0x7fa3640725b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.176+0000 7fa363e00640 1 -- 172.21.15.29:0/1225061757 >> v1:172.21.15.29:6789/0 conn(0x7fa364072af0 legacy=0x7fa3640c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.176+0000 7fa363e00640 1 -- 172.21.15.29:0/1225061757 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa36406e710 con 0x7fa36415d460 2024-04-17T19:48:51.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.176+0000 7fa362a00640 1 -- 172.21.15.29:0/1225061757 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 678331952 0 0) 0x7fa36415ddb0 con 0x7fa364072af0 2024-04-17T19:48:51.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.176+0000 7fa363e00640 1 --2- 172.21.15.29:0/1225061757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa36415d460 0x7fa36415d870 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa36406f490 tx=0x7fa35c01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d19a4636889c4f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:51.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.180+0000 7fa362a00640 1 -- 172.21.15.29:0/1225061757 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c0b2070 con 0x7fa36415d460 2024-04-17T19:48:51.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.180+0000 7fa362a00640 1 -- 172.21.15.29:0/1225061757 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa35c0a59f0 con 0x7fa36415d460 2024-04-17T19:48:51.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.180+0000 7fa362a00640 1 -- 172.21.15.29:0/1225061757 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35c0a5dc0 con 0x7fa36415d460 2024-04-17T19:48:51.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.180+0000 7fa36a000640 1 -- 172.21.15.29:0/1225061757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa36415d460 msgr2=0x7fa36415d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:51.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.180+0000 7fa36a000640 1 --2- 172.21.15.29:0/1225061757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa36415d460 0x7fa36415d870 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa36406f490 tx=0x7fa35c01c930 comp rx=0 tx=0).stop 2024-04-17T19:48:51.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.180+0000 7fa36a000640 1 -- 172.21.15.29:0/1225061757 shutdown_connections 2024-04-17T19:48:51.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.180+0000 7fa36a000640 1 --2- 172.21.15.29:0/1225061757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa36415d460 0x7fa36415d870 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.180+0000 7fa36a000640 1 --2- 172.21.15.29:0/1225061757 >> v2:172.21.15.37:3300/0 conn(0x7fa364153ea0 0x7fa3640725b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.180+0000 7fa36a000640 1 -- 172.21.15.29:0/1225061757 >> 172.21.15.29:0/1225061757 conn(0x7fa36406c3c0 msgr2=0x7fa3640ca3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:51.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.180+0000 7fa36a000640 1 -- 172.21.15.29:0/1225061757 shutdown_connections 2024-04-17T19:48:51.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.180+0000 7fa36a000640 1 -- 172.21.15.29:0/1225061757 wait complete. 2024-04-17T19:48:51.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.180+0000 7fa36a000640 1 Processor -- start 2024-04-17T19:48:51.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.180+0000 7fa36a000640 1 -- start start 2024-04-17T19:48:51.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa36a000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa364072af0 0x7fa364135fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:51.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa36a000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa364153ea0 0x7fa3640c6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:51.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa36a000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa36415d460 0x7fa364133f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:51.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa36a000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa3640c7890 con 0x7fa364072af0 2024-04-17T19:48:51.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa36a000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa3640c7710 con 0x7fa364153ea0 2024-04-17T19:48:51.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa36a000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa3641542b0 con 0x7fa36415d460 2024-04-17T19:48:51.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa363400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa364072af0 0x7fa364135fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:51.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa363400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa364072af0 0x7fa364135fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48568/0 (socket says 172.21.15.29:48568) 2024-04-17T19:48:51.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa363400640 1 -- 172.21.15.29:0/108974669 learned_addr learned my addr 172.21.15.29:0/108974669 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:51.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa35be00640 1 --2- 172.21.15.29:0/108974669 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa364153ea0 0x7fa3640c6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa363e00640 1 --2- 172.21.15.29:0/108974669 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa36415d460 0x7fa364133f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa363400640 1 -- 172.21.15.29:0/108974669 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa36415d460 msgr2=0x7fa364133f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa363400640 1 --2- 172.21.15.29:0/108974669 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa36415d460 0x7fa364133f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa363400640 1 -- 172.21.15.29:0/108974669 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa364153ea0 msgr2=0x7fa3640c6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa363400640 1 --2- 172.21.15.29:0/108974669 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa364153ea0 0x7fa3640c6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa363400640 1 -- 172.21.15.29:0/108974669 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3640c4cd0 con 0x7fa364072af0 2024-04-17T19:48:51.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa363400640 1 --2- 172.21.15.29:0/108974669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa364072af0 0x7fa364135fa0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa35400e870 tx=0x7fa35400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:51.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa35b400640 1 -- 172.21.15.29:0/108974669 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa354017070 con 0x7fa364072af0 2024-04-17T19:48:51.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa35b400640 1 -- 172.21.15.29:0/108974669 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa35400c940 con 0x7fa364072af0 2024-04-17T19:48:51.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa35b400640 1 -- 172.21.15.29:0/108974669 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa35400ccf0 con 0x7fa364072af0 2024-04-17T19:48:51.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa36a000640 1 -- 172.21.15.29:0/108974669 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3640c4f90 con 0x7fa364072af0 2024-04-17T19:48:51.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa36a000640 1 -- 172.21.15.29:0/108974669 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3640c5550 con 0x7fa364072af0 2024-04-17T19:48:51.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.184+0000 7fa36a000640 1 -- 172.21.15.29:0/108974669 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa364069b90 con 0x7fa364072af0 2024-04-17T19:48:51.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.188+0000 7fa35b400640 1 -- 172.21.15.29:0/108974669 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa35401a070 con 0x7fa364072af0 2024-04-17T19:48:51.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.188+0000 7fa35b400640 1 --2- 172.21.15.29:0/108974669 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa3400683b0 0x7fa34006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:51.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.188+0000 7fa35b400640 1 -- 172.21.15.29:0/108974669 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa354086c90 con 0x7fa364072af0 2024-04-17T19:48:51.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.188+0000 7fa35be00640 1 --2- 172.21.15.29:0/108974669 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa3400683b0 0x7fa34006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:51.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.188+0000 7fa35be00640 1 --2- 172.21.15.29:0/108974669 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa3400683b0 0x7fa34006a8c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa36413f830 tx=0x7fa34c007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:51.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.192+0000 7fa35b400640 1 -- 172.21.15.29:0/108974669 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa354056db0 con 0x7fa364072af0 2024-04-17T19:48:51.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.392+0000 7fa36a000640 1 -- 172.21.15.29:0/108974669 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3640c8430 con 0x7fa364072af0 2024-04-17T19:48:51.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.392+0000 7fa35b400640 1 -- 172.21.15.29:0/108974669 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6902 (secure 0 0 0) 0x7fa3540566b0 con 0x7fa364072af0 2024-04-17T19:48:51.383 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:51.383 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:48:34.874038+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4226":{"gid":4226,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.37:6838/332907250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":332907250},{"type":"v1","addr":"172.21.15.37:6838","nonce":332907250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4226,4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:48:51.383 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 19 2024-04-17T19:48:51.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.396+0000 7fa36a000640 1 -- 172.21.15.29:0/108974669 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa3400683b0 msgr2=0x7fa34006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:51.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.396+0000 7fa36a000640 1 --2- 172.21.15.29:0/108974669 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa3400683b0 0x7fa34006a8c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa36413f830 tx=0x7fa34c007c10 comp rx=0 tx=0).stop 2024-04-17T19:48:51.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.396+0000 7fa36a000640 1 -- 172.21.15.29:0/108974669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa364072af0 msgr2=0x7fa364135fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:51.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.396+0000 7fa36a000640 1 --2- 172.21.15.29:0/108974669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa364072af0 0x7fa364135fa0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa35400e870 tx=0x7fa35400ec70 comp rx=0 tx=0).stop 2024-04-17T19:48:51.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.396+0000 7fa36a000640 1 -- 172.21.15.29:0/108974669 shutdown_connections 2024-04-17T19:48:51.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.396+0000 7fa36a000640 1 --2- 172.21.15.29:0/108974669 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa3400683b0 0x7fa34006a8c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.396+0000 7fa36a000640 1 --2- 172.21.15.29:0/108974669 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa36415d460 0x7fa364133f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.396+0000 7fa36a000640 1 --2- 172.21.15.29:0/108974669 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa364153ea0 0x7fa3640c6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.396+0000 7fa36a000640 1 --2- 172.21.15.29:0/108974669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa364072af0 0x7fa364135fa0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.396+0000 7fa36a000640 1 -- 172.21.15.29:0/108974669 >> 172.21.15.29:0/108974669 conn(0x7fa36406c3c0 msgr2=0x7fa364152e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:51.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.396+0000 7fa36a000640 1 -- 172.21.15.29:0/108974669 shutdown_connections 2024-04-17T19:48:51.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.396+0000 7fa36a000640 1 -- 172.21.15.29:0/108974669 wait complete. 2024-04-17T19:48:51.405 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T19:48:51.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.568+0000 7f8dea200640 1 Processor -- start 2024-04-17T19:48:51.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.568+0000 7f8dea200640 1 -- start start 2024-04-17T19:48:51.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.568+0000 7f8dea200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8de40cc8c0 0x7f8de40cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:51.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.568+0000 7f8dea200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f8de41597a0 0x7f8de41635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:51.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.568+0000 7f8dea200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8de40c88b0 con 0x7f8de4159390 2024-04-17T19:48:51.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.568+0000 7f8dea200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f8de40c6ce0 con 0x7f8de40cc8c0 2024-04-17T19:48:51.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.568+0000 7f8dea200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f8de40c6fc0 con 0x7f8de41597a0 2024-04-17T19:48:51.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.572+0000 7f8de3400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8de40cc8c0 0x7f8de40cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:51.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.572+0000 7f8de3400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8de40cc8c0 0x7f8de40cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48584/0 (socket says 172.21.15.29:48584) 2024-04-17T19:48:51.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.572+0000 7f8de3400640 1 -- 172.21.15.29:0/942568142 learned_addr learned my addr 172.21.15.29:0/942568142 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:51.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.572+0000 7f8de3e00640 1 --2- 172.21.15.29:0/942568142 >> v2:172.21.15.37:3300/0 conn(0x7f8de41597a0 0x7f8de41635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:51.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.572+0000 7f8de3400640 1 -- 172.21.15.29:0/942568142 >> v2:172.21.15.37:3300/0 conn(0x7f8de41597a0 msgr2=0x7f8de41635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:48:51.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.572+0000 7f8de3400640 1 --2- 172.21.15.29:0/942568142 >> v2:172.21.15.37:3300/0 conn(0x7f8de41597a0 0x7f8de41635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.572+0000 7f8de3400640 1 -- 172.21.15.29:0/942568142 >> v1:172.21.15.37:6789/0 conn(0x7f8de4159390 legacy=0x7f8de40ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:51.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.572+0000 7f8de3400640 1 -- 172.21.15.29:0/942568142 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de40c72a0 con 0x7f8de40cc8c0 2024-04-17T19:48:51.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.572+0000 7f8de3400640 1 --2- 172.21.15.29:0/942568142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8de40cc8c0 0x7f8de40cccd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8de40722d0 tx=0x7f8dd401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b815150e0344d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:51.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.572+0000 7f8de2a00640 1 -- 172.21.15.29:0/942568142 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd40a56c0 con 0x7f8de40cc8c0 2024-04-17T19:48:51.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.572+0000 7f8de2a00640 1 -- 172.21.15.29:0/942568142 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dd40a5860 con 0x7f8de40cc8c0 2024-04-17T19:48:51.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.572+0000 7f8de2a00640 1 -- 172.21.15.29:0/942568142 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd40a5c30 con 0x7f8de40cc8c0 2024-04-17T19:48:51.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.576+0000 7f8dea200640 1 -- 172.21.15.29:0/942568142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8de40cc8c0 msgr2=0x7f8de40cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:51.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.576+0000 7f8dea200640 1 --2- 172.21.15.29:0/942568142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8de40cc8c0 0x7f8de40cccd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8de40722d0 tx=0x7f8dd401c9f0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.576+0000 7f8dea200640 1 -- 172.21.15.29:0/942568142 shutdown_connections 2024-04-17T19:48:51.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.576+0000 7f8dea200640 1 --2- 172.21.15.29:0/942568142 >> v2:172.21.15.37:3300/0 conn(0x7f8de41597a0 0x7f8de41635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.576+0000 7f8dea200640 1 --2- 172.21.15.29:0/942568142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8de40cc8c0 0x7f8de40cccd0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.576+0000 7f8dea200640 1 -- 172.21.15.29:0/942568142 >> 172.21.15.29:0/942568142 conn(0x7f8de406fde0 msgr2=0x7f8de40701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:51.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.576+0000 7f8dea200640 1 -- 172.21.15.29:0/942568142 shutdown_connections 2024-04-17T19:48:51.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.576+0000 7f8dea200640 1 -- 172.21.15.29:0/942568142 wait complete. 2024-04-17T19:48:51.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.576+0000 7f8dea200640 1 Processor -- start 2024-04-17T19:48:51.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.576+0000 7f8dea200640 1 -- start start 2024-04-17T19:48:51.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8dea200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8de40cc8c0 0x7f8de41f49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:51.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8dea200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8de4159390 0x7f8de4157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:51.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8dea200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8de41597a0 0x7f8de415dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:51.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8dea200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8de40cabc0 con 0x7f8de40cc8c0 2024-04-17T19:48:51.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8dea200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8de40caa40 con 0x7f8de4159390 2024-04-17T19:48:51.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8dea200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8de40c6880 con 0x7f8de41597a0 2024-04-17T19:48:51.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8de3400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8de40cc8c0 0x7f8de41f49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:51.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8de3400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8de40cc8c0 0x7f8de41f49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48586/0 (socket says 172.21.15.29:48586) 2024-04-17T19:48:51.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8de3400640 1 -- 172.21.15.29:0/625252955 learned_addr learned my addr 172.21.15.29:0/625252955 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:48:51.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8ddaa00640 1 --2- 172.21.15.29:0/625252955 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8de4159390 0x7f8de4157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:51.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8de3e00640 1 --2- 172.21.15.29:0/625252955 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8de41597a0 0x7f8de415dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:51.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8de3400640 1 -- 172.21.15.29:0/625252955 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8de41597a0 msgr2=0x7f8de415dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:51.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8de3400640 1 --2- 172.21.15.29:0/625252955 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8de41597a0 0x7f8de415dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8de3400640 1 -- 172.21.15.29:0/625252955 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8de4159390 msgr2=0x7f8de4157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:51.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8de3400640 1 --2- 172.21.15.29:0/625252955 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8de4159390 0x7f8de4157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8de3400640 1 -- 172.21.15.29:0/625252955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de41f4ef0 con 0x7f8de40cc8c0 2024-04-17T19:48:51.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8de3400640 1 --2- 172.21.15.29:0/625252955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8de40cc8c0 0x7f8de41f49b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40af040 tx=0x7f8dd4019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:51.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8ddbe00640 1 -- 172.21.15.29:0/625252955 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd40b8470 con 0x7f8de40cc8c0 2024-04-17T19:48:51.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8ddbe00640 1 -- 172.21.15.29:0/625252955 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dd40b8610 con 0x7f8de40cc8c0 2024-04-17T19:48:51.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8ddbe00640 1 -- 172.21.15.29:0/625252955 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd40b8a00 con 0x7f8de40cc8c0 2024-04-17T19:48:51.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8dea200640 1 -- 172.21.15.29:0/625252955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8de413bfe0 con 0x7f8de40cc8c0 2024-04-17T19:48:51.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8dea200640 1 -- 172.21.15.29:0/625252955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8de413c580 con 0x7f8de40cc8c0 2024-04-17T19:48:51.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8dea200640 1 -- 172.21.15.29:0/625252955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8de4069b90 con 0x7f8de40cc8c0 2024-04-17T19:48:51.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.580+0000 7f8ddbe00640 1 -- 172.21.15.29:0/625252955 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8dd40c0030 con 0x7f8de40cc8c0 2024-04-17T19:48:51.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.584+0000 7f8ddbe00640 1 --2- 172.21.15.29:0/625252955 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8dc0068380 0x7f8dc006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:48:51.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.584+0000 7f8ddbe00640 1 -- 172.21.15.29:0/625252955 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8dd4121a50 con 0x7f8de40cc8c0 2024-04-17T19:48:51.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.584+0000 7f8ddaa00640 1 --2- 172.21.15.29:0/625252955 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8dc0068380 0x7f8dc006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:48:51.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.584+0000 7f8ddaa00640 1 --2- 172.21.15.29:0/625252955 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8dc0068380 0x7f8dc006a890 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8de40c40f0 tx=0x7f8dcc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:48:51.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.588+0000 7f8ddbe00640 1 -- 172.21.15.29:0/625252955 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8dd40c7020 con 0x7f8de40cc8c0 2024-04-17T19:48:51.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.756+0000 7f8dea200640 1 -- 172.21.15.29:0/625252955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8de40c7e10 con 0x7f8de40cc8c0 2024-04-17T19:48:51.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.756+0000 7f8ddbe00640 1 -- 172.21.15.29:0/625252955 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18780 (secure 0 0 0) 0x7f8dd40165e0 con 0x7f8de40cc8c0 2024-04-17T19:48:51.747 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:48:51.747 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":26,"fsid":"a680ee90-875c-4283-a901-86d7638393c3","created":"2024-04-17T19:47:23.788079+0000","modified":"2024-04-17T19:48:29.829433+0000","last_up_change":"2024-04-17T19:47:35.619415+0000","last_in_change":"2024-04-17T19:47:26.537699+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T19:47:36.674676+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T19:47:47.658449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T19:47:48.209047+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T19:47:51.887909+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a63774a5-4a49-4bae-a850-18ecff670ccb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6801","nonce":515534509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6803","nonce":515534509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6807","nonce":515534509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":515534509},{"type":"v1","addr":"172.21.15.29:6805","nonce":515534509}]},"public_addr":"172.21.15.29:6801/515534509","cluster_addr":"172.21.15.29:6803/515534509","heartbeat_back_addr":"172.21.15.29:6807/515534509","heartbeat_front_addr":"172.21.15.29:6805/515534509","state":["exists","up"]},{"osd":1,"uuid":"a567ac96-9ee4-495f-a20c-2686d59f1fe2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6825","nonce":843118748}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6826","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6827","nonce":843118748}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6830","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6831","nonce":843118748}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":843118748},{"type":"v1","addr":"172.21.15.29:6829","nonce":843118748}]},"public_addr":"172.21.15.29:6825/843118748","cluster_addr":"172.21.15.29:6827/843118748","heartbeat_back_addr":"172.21.15.29:6831/843118748","heartbeat_front_addr":"172.21.15.29:6829/843118748","state":["exists","up"]},{"osd":2,"uuid":"eb2aca46-5f62-4cd2-979f-906958ea2cc9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6809","nonce":868032589}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6811","nonce":868032589}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6815","nonce":868032589}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":868032589},{"type":"v1","addr":"172.21.15.29:6813","nonce":868032589}]},"public_addr":"172.21.15.29:6809/868032589","cluster_addr":"172.21.15.29:6811/868032589","heartbeat_back_addr":"172.21.15.29:6815/868032589","heartbeat_front_addr":"172.21.15.29:6813/868032589","state":["exists","up"]},{"osd":3,"uuid":"667781ea-7172-42ab-9a97-d03fa0b273ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6817","nonce":155534919}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6819","nonce":155534919}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6822","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6823","nonce":155534919}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6820","nonce":155534919},{"type":"v1","addr":"172.21.15.29:6821","nonce":155534919}]},"public_addr":"172.21.15.29:6817/155534919","cluster_addr":"172.21.15.29:6819/155534919","heartbeat_back_addr":"172.21.15.29:6823/155534919","heartbeat_front_addr":"172.21.15.29:6821/155534919","state":["exists","up"]},{"osd":4,"uuid":"1e88ed79-c62d-4257-ac5e-51dcefaa1f94","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6800","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6801","nonce":811671695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6802","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6803","nonce":811671695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6806","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6807","nonce":811671695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6804","nonce":811671695},{"type":"v1","addr":"172.21.15.37:6805","nonce":811671695}]},"public_addr":"172.21.15.37:6801/811671695","cluster_addr":"172.21.15.37:6803/811671695","heartbeat_back_addr":"172.21.15.37:6807/811671695","heartbeat_front_addr":"172.21.15.37:6805/811671695","state":["exists","up"]},{"osd":5,"uuid":"0857a0aa-c148-4ce9-baa5-96c519989c8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6824","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6825","nonce":255655587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6826","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6827","nonce":255655587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6830","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6831","nonce":255655587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6828","nonce":255655587},{"type":"v1","addr":"172.21.15.37:6829","nonce":255655587}]},"public_addr":"172.21.15.37:6825/255655587","cluster_addr":"172.21.15.37:6827/255655587","heartbeat_back_addr":"172.21.15.37:6831/255655587","heartbeat_front_addr":"172.21.15.37:6829/255655587","state":["exists","up"]},{"osd":6,"uuid":"6ce4a0ee-7778-4238-b1e5-3efaf90c9b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6808","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6809","nonce":264607202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6810","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6811","nonce":264607202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6814","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6815","nonce":264607202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6812","nonce":264607202},{"type":"v1","addr":"172.21.15.37:6813","nonce":264607202}]},"public_addr":"172.21.15.37:6809/264607202","cluster_addr":"172.21.15.37:6811/264607202","heartbeat_back_addr":"172.21.15.37:6815/264607202","heartbeat_front_addr":"172.21.15.37:6813/264607202","state":["exists","up"]},{"osd":7,"uuid":"8c4a07d2-5528-4995-a660-b551a61f1289","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6816","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6817","nonce":3883152709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6818","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6819","nonce":3883152709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6822","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6823","nonce":3883152709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6820","nonce":3883152709},{"type":"v1","addr":"172.21.15.37:6821","nonce":3883152709}]},"public_addr":"172.21.15.37:6817/3883152709","cluster_addr":"172.21.15.37:6819/3883152709","heartbeat_back_addr":"172.21.15.37:6823/3883152709","heartbeat_front_addr":"172.21.15.37:6821/3883152709","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.230107+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.263779+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.112433+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.305360+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.916654+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.089407+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:31.893182+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T19:47:32.007320+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":5,"to":1}]},{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T19:48:51.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.760+0000 7f8dea200640 1 -- 172.21.15.29:0/625252955 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8dc0068380 msgr2=0x7f8dc006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:51.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.760+0000 7f8dea200640 1 --2- 172.21.15.29:0/625252955 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8dc0068380 0x7f8dc006a890 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8de40c40f0 tx=0x7f8dcc009040 comp rx=0 tx=0).stop 2024-04-17T19:48:51.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.760+0000 7f8dea200640 1 -- 172.21.15.29:0/625252955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8de40cc8c0 msgr2=0x7f8de41f49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:48:51.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.760+0000 7f8dea200640 1 --2- 172.21.15.29:0/625252955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8de40cc8c0 0x7f8de41f49b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40af040 tx=0x7f8dd4019790 comp rx=0 tx=0).stop 2024-04-17T19:48:51.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.760+0000 7f8dea200640 1 -- 172.21.15.29:0/625252955 shutdown_connections 2024-04-17T19:48:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.760+0000 7f8dea200640 1 --2- 172.21.15.29:0/625252955 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8dc0068380 0x7f8dc006a890 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.760+0000 7f8dea200640 1 --2- 172.21.15.29:0/625252955 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8de41597a0 0x7f8de415dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.760+0000 7f8dea200640 1 --2- 172.21.15.29:0/625252955 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8de4159390 0x7f8de4157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.760+0000 7f8dea200640 1 --2- 172.21.15.29:0/625252955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8de40cc8c0 0x7f8de41f49b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:48:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.760+0000 7f8dea200640 1 -- 172.21.15.29:0/625252955 >> 172.21.15.29:0/625252955 conn(0x7f8de406fde0 msgr2=0x7f8de406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:48:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.760+0000 7f8dea200640 1 -- 172.21.15.29:0/625252955 shutdown_connections 2024-04-17T19:48:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:48:51.760+0000 7f8dea200640 1 -- 172.21.15.29:0/625252955 wait complete. 2024-04-17T19:48:51.770 DEBUG:tasks.mds_thrash:Yielding 2024-04-17T19:48:51.770 INFO:teuthology.run_tasks:Running task workunit... 2024-04-17T19:48:51.784 INFO:tasks.workunit:Pulling workunits from ref f3a98d3f110319c567ec86bf70955c141495ccf6 2024-04-17T19:48:51.784 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-17T19:48:51.784 INFO:tasks.workunit:timeout=6h 2024-04-17T19:48:51.784 INFO:tasks.workunit:cleanup=True 2024-04-17T19:48:51.784 DEBUG:teuthology.orchestra.run.smithi029:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-17T19:48:51.785 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2024-04-17T19:48:51.785 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 67 secs before thrashing 2024-04-17T19:48:51.793 INFO:teuthology.orchestra.run.smithi029.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-17T19:48:51.793 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-17T19:48:51.793 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 2ch/44d Inode: 1 Links: 2 2024-04-17T19:48:51.793 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T19:48:51.793 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-17T19:48:51.793 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-04-17 19:47:54.839302247 +0000 2024-04-17T19:48:51.793 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-04-17 19:48:03.658528348 +0000 2024-04-17T19:48:51.794 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-04-17 19:47:54.839302247 +0000 2024-04-17T19:48:51.794 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-17T19:48:51.794 DEBUG:teuthology.orchestra.run.smithi029:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-17T19:48:51.862 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout f3a98d3f110319c567ec86bf70955c141495ccf6 2024-04-17T19:48:51.913 INFO:tasks.workunit.client.0.smithi029.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-17T19:49:18.403 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:49:18.407 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:49:48.465 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:49:48.470 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:49:58.786 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:49:59.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.044+0000 7f7b01a00640 1 Processor -- start 2024-04-17T19:49:59.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.048+0000 7f7b01a00640 1 -- start start 2024-04-17T19:49:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.048+0000 7f7b01a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7afc157250 0x7f7afc0cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:49:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.048+0000 7f7b01a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7afc0c9ea0 0x7f7afc0ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:49:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.048+0000 7f7b01a00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7afc0c4c10 con 0x7f7afc159ff0 2024-04-17T19:49:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.048+0000 7f7b01a00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f7afc06ef50 con 0x7f7afc0c9ea0 2024-04-17T19:49:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.048+0000 7f7b01a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7afc0c5a70 con 0x7f7afc157250 2024-04-17T19:49:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.048+0000 7f7afb400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7afc0c9ea0 0x7f7afc0ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:49:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.048+0000 7f7afb400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7afc0c9ea0 0x7f7afc0ca2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33410/0 (socket says 172.21.15.29:33410) 2024-04-17T19:49:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.048+0000 7f7afb400640 1 -- 172.21.15.29:0/1411470173 learned_addr learned my addr 172.21.15.29:0/1411470173 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:49:59.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.048+0000 7f7afbe00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7afc157250 0x7f7afc0cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:49:59.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.052+0000 7f7afb400640 1 -- 172.21.15.29:0/1411470173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7afc157250 msgr2=0x7f7afc0cb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:49:59.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.052+0000 7f7afb400640 1 --2- 172.21.15.29:0/1411470173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7afc157250 0x7f7afc0cb2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:49:59.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.052+0000 7f7afb400640 1 -- 172.21.15.29:0/1411470173 >> v1:172.21.15.29:6789/0 conn(0x7f7afc159ff0 legacy=0x7f7afc15a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:49:59.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.052+0000 7f7afb400640 1 -- 172.21.15.29:0/1411470173 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7afc0c4ef0 con 0x7f7afc0c9ea0 2024-04-17T19:49:59.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.056+0000 7f7afb400640 1 --2- 172.21.15.29:0/1411470173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7afc0c9ea0 0x7f7afc0ca2b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8009870 tx=0x7f7ae801ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86c0abc90b9bd8de server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:49:59.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.056+0000 7f7afaa00640 1 -- 172.21.15.29:0/1411470173 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae80b5070 con 0x7f7afc0c9ea0 2024-04-17T19:49:59.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.056+0000 7f7afaa00640 1 -- 172.21.15.29:0/1411470173 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ae80a8a50 con 0x7f7afc0c9ea0 2024-04-17T19:49:59.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.056+0000 7f7afaa00640 1 -- 172.21.15.29:0/1411470173 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ae80a8e20 con 0x7f7afc0c9ea0 2024-04-17T19:49:59.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.056+0000 7f7b01a00640 1 -- 172.21.15.29:0/1411470173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7afc0c9ea0 msgr2=0x7f7afc0ca2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:49:59.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.056+0000 7f7b01a00640 1 --2- 172.21.15.29:0/1411470173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7afc0c9ea0 0x7f7afc0ca2b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8009870 tx=0x7f7ae801ed50 comp rx=0 tx=0).stop 2024-04-17T19:49:59.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.056+0000 7f7b01a00640 1 -- 172.21.15.29:0/1411470173 shutdown_connections 2024-04-17T19:49:59.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.056+0000 7f7b01a00640 1 --2- 172.21.15.29:0/1411470173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7afc0c9ea0 0x7f7afc0ca2b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:49:59.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.056+0000 7f7b01a00640 1 --2- 172.21.15.29:0/1411470173 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7afc157250 0x7f7afc0cb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:49:59.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.056+0000 7f7b01a00640 1 -- 172.21.15.29:0/1411470173 >> 172.21.15.29:0/1411470173 conn(0x7f7afc0680e0 msgr2=0x7f7afc06dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:49:59.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.056+0000 7f7b01a00640 1 -- 172.21.15.29:0/1411470173 shutdown_connections 2024-04-17T19:49:59.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.056+0000 7f7b01a00640 1 -- 172.21.15.29:0/1411470173 wait complete. 2024-04-17T19:49:59.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.060+0000 7f7b01a00640 1 Processor -- start 2024-04-17T19:49:59.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.060+0000 7f7b01a00640 1 -- start start 2024-04-17T19:49:59.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7b01a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7afc0c9ea0 0x7f7afc1f0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:49:59.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7b01a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7afc157250 0x7f7afc0cd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:49:59.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7b01a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7afc159ff0 0x7f7afc0b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:49:59.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7b01a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7afc0c54b0 con 0x7f7afc159ff0 2024-04-17T19:49:59.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7b01a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7afc157660 con 0x7f7afc157250 2024-04-17T19:49:59.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7b01a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7afc0c5790 con 0x7f7afc0c9ea0 2024-04-17T19:49:59.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7b01000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7afc159ff0 0x7f7afc0b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:49:59.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7b01000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7afc159ff0 0x7f7afc0b3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33416/0 (socket says 172.21.15.29:33416) 2024-04-17T19:49:59.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7b01000640 1 -- 172.21.15.29:0/2180716186 learned_addr learned my addr 172.21.15.29:0/2180716186 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:49:59.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7afb400640 1 --2- 172.21.15.29:0/2180716186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7afc157250 0x7f7afc0cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:49:59.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7afbe00640 1 --2- 172.21.15.29:0/2180716186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7afc0c9ea0 0x7f7afc1f0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:49:59.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7b01000640 1 -- 172.21.15.29:0/2180716186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7afc0c9ea0 msgr2=0x7f7afc1f0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:49:59.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7b01000640 1 --2- 172.21.15.29:0/2180716186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7afc0c9ea0 0x7f7afc1f0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:49:59.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7b01000640 1 -- 172.21.15.29:0/2180716186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7afc157250 msgr2=0x7f7afc0cd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:49:59.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7b01000640 1 --2- 172.21.15.29:0/2180716186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7afc157250 0x7f7afc0cd790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:49:59.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7b01000640 1 -- 172.21.15.29:0/2180716186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7afc06ea90 con 0x7f7afc159ff0 2024-04-17T19:49:59.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7b01000640 1 --2- 172.21.15.29:0/2180716186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7afc159ff0 0x7f7afc0b3f60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7af400c760 tx=0x7f7af400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:49:59.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7af3e00640 1 -- 172.21.15.29:0/2180716186 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af4016020 con 0x7f7afc159ff0 2024-04-17T19:49:59.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7af3e00640 1 -- 172.21.15.29:0/2180716186 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7af4006d60 con 0x7f7afc159ff0 2024-04-17T19:49:59.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7af3e00640 1 -- 172.21.15.29:0/2180716186 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af4003bc0 con 0x7f7afc159ff0 2024-04-17T19:49:59.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7b01a00640 1 -- 172.21.15.29:0/2180716186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7afc0cdcd0 con 0x7f7afc159ff0 2024-04-17T19:49:59.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.064+0000 7f7b01a00640 1 -- 172.21.15.29:0/2180716186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7afc0ce990 con 0x7f7afc159ff0 2024-04-17T19:49:59.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.068+0000 7f7b01a00640 1 -- 172.21.15.29:0/2180716186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7afc0ce410 con 0x7f7afc159ff0 2024-04-17T19:49:59.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.068+0000 7f7af3e00640 1 -- 172.21.15.29:0/2180716186 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7af40050d0 con 0x7f7afc159ff0 2024-04-17T19:49:59.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.072+0000 7f7af3e00640 1 --2- 172.21.15.29:0/2180716186 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7adc068400 0x7f7adc06a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:49:59.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.072+0000 7f7af3e00640 1 -- 172.21.15.29:0/2180716186 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f7af4086910 con 0x7f7afc159ff0 2024-04-17T19:49:59.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.072+0000 7f7afbe00640 1 --2- 172.21.15.29:0/2180716186 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7adc068400 0x7f7adc06a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:49:59.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.072+0000 7f7afbe00640 1 --2- 172.21.15.29:0/2180716186 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7adc068400 0x7f7adc06a910 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7aec009ac0 tx=0x7f7aec009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:49:59.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.080+0000 7f7af3e00640 1 -- 172.21.15.29:0/2180716186 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7afc0ce410 con 0x7f7afc159ff0 2024-04-17T19:49:59.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.488+0000 7f7b01a00640 1 -- 172.21.15.29:0/2180716186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7afc04ff10 con 0x7f7afc159ff0 2024-04-17T19:49:59.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.492+0000 7f7af3e00640 1 -- 172.21.15.29:0/2180716186 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6902 (secure 0 0 0) 0x7f7af4020020 con 0x7f7afc159ff0 2024-04-17T19:49:59.481 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:49:59.481 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:48:34.874038+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4226":{"gid":4226,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.37:6838/332907250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":332907250},{"type":"v1","addr":"172.21.15.37:6838","nonce":332907250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4226,4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:49:59.481 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 19 2024-04-17T19:49:59.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.492+0000 7f7b01a00640 1 -- 172.21.15.29:0/2180716186 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7adc068400 msgr2=0x7f7adc06a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:49:59.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.492+0000 7f7b01a00640 1 --2- 172.21.15.29:0/2180716186 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7adc068400 0x7f7adc06a910 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7aec009ac0 tx=0x7f7aec009190 comp rx=0 tx=0).stop 2024-04-17T19:49:59.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.492+0000 7f7b01a00640 1 -- 172.21.15.29:0/2180716186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7afc159ff0 msgr2=0x7f7afc0b3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:49:59.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.492+0000 7f7b01a00640 1 --2- 172.21.15.29:0/2180716186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7afc159ff0 0x7f7afc0b3f60 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7af400c760 tx=0x7f7af400cc20 comp rx=0 tx=0).stop 2024-04-17T19:49:59.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.492+0000 7f7b01a00640 1 -- 172.21.15.29:0/2180716186 shutdown_connections 2024-04-17T19:49:59.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.492+0000 7f7b01a00640 1 --2- 172.21.15.29:0/2180716186 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7adc068400 0x7f7adc06a910 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:49:59.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.492+0000 7f7b01a00640 1 --2- 172.21.15.29:0/2180716186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7afc159ff0 0x7f7afc0b3f60 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:49:59.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.492+0000 7f7b01a00640 1 --2- 172.21.15.29:0/2180716186 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7afc157250 0x7f7afc0cd790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:49:59.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.492+0000 7f7b01a00640 1 --2- 172.21.15.29:0/2180716186 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7afc0c9ea0 0x7f7afc1f0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:49:59.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.492+0000 7f7b01a00640 1 -- 172.21.15.29:0/2180716186 >> 172.21.15.29:0/2180716186 conn(0x7f7afc0680e0 msgr2=0x7f7afc152be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:49:59.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.492+0000 7f7b01a00640 1 -- 172.21.15.29:0/2180716186 shutdown_connections 2024-04-17T19:49:59.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.492+0000 7f7b01a00640 1 -- 172.21.15.29:0/2180716186 wait complete. 2024-04-17T19:49:59.515 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=4) 2024-04-17T19:49:59.515 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T19:49:59.515 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T19:49:59.531 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T19:49:59.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:49:59.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.788+0000 7f40ce800640 1 Processor -- start 2024-04-17T19:49:59.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.788+0000 7f40ce800640 1 -- start start 2024-04-17T19:49:59.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.792+0000 7f40ce800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f40c8160030 0x7f40c80ce880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:49:59.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.792+0000 7f40ce800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40c806c980 con 0x7f40c80ce470 2024-04-17T19:49:59.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.792+0000 7f40ce800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40c806cc60 con 0x7f40c8155f10 2024-04-17T19:49:59.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.792+0000 7f40ce800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f40c806cf40 con 0x7f40c8160030 2024-04-17T19:49:59.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.792+0000 7f40cca00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f40c80ce470 0x7f40c815f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48962/0 (socket says 172.21.15.29:48962) 2024-04-17T19:49:59.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.792+0000 7f40cca00640 1 -- 172.21.15.29:0/1669668670 learned_addr learned my addr 172.21.15.29:0/1669668670 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:49:59.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.792+0000 7f40cde00640 1 --2- 172.21.15.29:0/1669668670 >> v2:172.21.15.37:3300/0 conn(0x7f40c8160030 0x7f40c80ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:49:59.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.792+0000 7f40cde00640 1 -- 172.21.15.29:0/1669668670 >> v1:172.21.15.37:6789/0 conn(0x7f40c8155f10 legacy=0x7f40c815c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:49:59.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.792+0000 7f40cde00640 1 -- 172.21.15.29:0/1669668670 >> v1:172.21.15.29:6789/0 conn(0x7f40c80ce470 legacy=0x7f40c815f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:49:59.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.792+0000 7f40cde00640 1 -- 172.21.15.29:0/1669668670 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c806d220 con 0x7f40c8160030 2024-04-17T19:49:59.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.796+0000 7f40cde00640 1 --2- 172.21.15.29:0/1669668670 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40c8160030 0x7f40c80ce880 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009d90 tx=0x7f40c001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=614b20b81177493b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:49:59.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.796+0000 7f40c7e00640 1 -- 172.21.15.29:0/1669668670 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c00b2070 con 0x7f40c8160030 2024-04-17T19:49:59.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.796+0000 7f40c7e00640 1 -- 172.21.15.29:0/1669668670 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c00a5830 con 0x7f40c8160030 2024-04-17T19:49:59.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.796+0000 7f40c7e00640 1 -- 172.21.15.29:0/1669668670 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c00a5c00 con 0x7f40c8160030 2024-04-17T19:49:59.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.800+0000 7f40ce800640 1 -- 172.21.15.29:0/1669668670 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40c8160030 msgr2=0x7f40c80ce880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:49:59.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.800+0000 7f40ce800640 1 --2- 172.21.15.29:0/1669668670 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40c8160030 0x7f40c80ce880 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f40c0009d90 tx=0x7f40c001c970 comp rx=0 tx=0).stop 2024-04-17T19:49:59.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.800+0000 7f40ce800640 1 -- 172.21.15.29:0/1669668670 shutdown_connections 2024-04-17T19:49:59.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.800+0000 7f40ce800640 1 --2- 172.21.15.29:0/1669668670 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40c8160030 0x7f40c80ce880 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:49:59.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.800+0000 7f40ce800640 1 -- 172.21.15.29:0/1669668670 >> 172.21.15.29:0/1669668670 conn(0x7f40c8072db0 msgr2=0x7f40c80731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:49:59.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.800+0000 7f40ce800640 1 -- 172.21.15.29:0/1669668670 shutdown_connections 2024-04-17T19:49:59.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.800+0000 7f40ce800640 1 -- 172.21.15.29:0/1669668670 wait complete. 2024-04-17T19:49:59.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.804+0000 7f40ce800640 1 Processor -- start 2024-04-17T19:49:59.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.804+0000 7f40ce800640 1 -- start start 2024-04-17T19:49:59.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.804+0000 7f40ce800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c80ce470 0x7f40c813e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:49:59.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.804+0000 7f40ce800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f40c8155f10 0x7f40c81f9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:49:59.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.804+0000 7f40ce800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40c8160030 0x7f40c80c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:49:59.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.804+0000 7f40ce800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f40c816a2f0 con 0x7f40c80ce470 2024-04-17T19:49:59.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.804+0000 7f40ce800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f40c816a170 con 0x7f40c8160030 2024-04-17T19:49:59.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.804+0000 7f40ce800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f40c816a470 con 0x7f40c8155f10 2024-04-17T19:49:59.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.804+0000 7f40cd400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c80ce470 0x7f40c813e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:49:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.804+0000 7f40cd400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c80ce470 0x7f40c813e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33418/0 (socket says 172.21.15.29:33418) 2024-04-17T19:49:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.804+0000 7f40cd400640 1 -- 172.21.15.29:0/1762376870 learned_addr learned my addr 172.21.15.29:0/1762376870 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:49:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.804+0000 7f40cca00640 1 --2- 172.21.15.29:0/1762376870 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f40c8155f10 0x7f40c81f9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:49:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.808+0000 7f40cde00640 1 --2- 172.21.15.29:0/1762376870 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40c8160030 0x7f40c80c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:49:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.808+0000 7f40cd400640 1 -- 172.21.15.29:0/1762376870 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f40c8155f10 msgr2=0x7f40c81f9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:49:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.808+0000 7f40cd400640 1 --2- 172.21.15.29:0/1762376870 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f40c8155f10 0x7f40c81f9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:49:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.808+0000 7f40cd400640 1 -- 172.21.15.29:0/1762376870 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40c8160030 msgr2=0x7f40c80c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:49:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.808+0000 7f40cd400640 1 --2- 172.21.15.29:0/1762376870 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40c8160030 0x7f40c80c7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:49:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.808+0000 7f40cd400640 1 -- 172.21.15.29:0/1762376870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c81f98b0 con 0x7f40c80ce470 2024-04-17T19:49:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.808+0000 7f40cd400640 1 --2- 172.21.15.29:0/1762376870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c80ce470 0x7f40c813e490 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00c3f0 tx=0x7f40bc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:49:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.808+0000 7f40c5600640 1 -- 172.21.15.29:0/1762376870 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc017070 con 0x7f40c80ce470 2024-04-17T19:49:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.808+0000 7f40c5600640 1 -- 172.21.15.29:0/1762376870 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40bc0052d0 con 0x7f40c80ce470 2024-04-17T19:49:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.808+0000 7f40c5600640 1 -- 172.21.15.29:0/1762376870 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40bc003840 con 0x7f40c80ce470 2024-04-17T19:49:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.808+0000 7f40ce800640 1 -- 172.21.15.29:0/1762376870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40c80cd850 con 0x7f40c80ce470 2024-04-17T19:49:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.808+0000 7f40ce800640 1 -- 172.21.15.29:0/1762376870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40c81f8b70 con 0x7f40c80ce470 2024-04-17T19:49:59.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.808+0000 7f40ce800640 1 -- 172.21.15.29:0/1762376870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40c804ff10 con 0x7f40c80ce470 2024-04-17T19:49:59.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.812+0000 7f40c5600640 1 -- 172.21.15.29:0/1762376870 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f40bc01a070 con 0x7f40c80ce470 2024-04-17T19:49:59.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.812+0000 7f40c5600640 1 --2- 172.21.15.29:0/1762376870 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f40a8068400 0x7f40a806a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:49:59.803 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.812+0000 7f40c5600640 1 -- 172.21.15.29:0/1762376870 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f40bc0860b0 con 0x7f40c80ce470 2024-04-17T19:49:59.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.812+0000 7f40cca00640 1 --2- 172.21.15.29:0/1762376870 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f40a8068400 0x7f40a806a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:49:59.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.816+0000 7f40cca00640 1 --2- 172.21.15.29:0/1762376870 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f40a8068400 0x7f40a806a910 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f40c815cdb0 tx=0x7f40b4007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:49:59.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:49:59.820+0000 7f40c5600640 1 -- 172.21.15.29:0/1762376870 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f40bc005480 con 0x7f40c80ce470 2024-04-17T19:49:59.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-04-17T19:50:00.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:00.232+0000 7f40ce800640 1 -- 172.21.15.29:0/1762376870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40c813f0b0 con 0x7f40c80ce470 2024-04-17T19:50:00.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:00.232+0000 7f40c5600640 1 -- 172.21.15.29:0/1762376870 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6902 (secure 0 0 0) 0x7f40bc055c00 con 0x7f40c80ce470 2024-04-17T19:50:00.223 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:50:00.223 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:48:34.874038+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4226":{"gid":4226,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.37:6838/332907250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":332907250},{"type":"v1","addr":"172.21.15.37:6838","nonce":332907250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4226,4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:50:00.223 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 19 2024-04-17T19:50:00.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:00.236+0000 7f40ce800640 1 -- 172.21.15.29:0/1762376870 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f40a8068400 msgr2=0x7f40a806a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:00.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:00.236+0000 7f40ce800640 1 --2- 172.21.15.29:0/1762376870 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f40a8068400 0x7f40a806a910 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f40c815cdb0 tx=0x7f40b4007e90 comp rx=0 tx=0).stop 2024-04-17T19:50:00.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:00.236+0000 7f40ce800640 1 -- 172.21.15.29:0/1762376870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c80ce470 msgr2=0x7f40c813e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:00.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:00.236+0000 7f40ce800640 1 --2- 172.21.15.29:0/1762376870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c80ce470 0x7f40c813e490 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f40bc00c3f0 tx=0x7f40bc00c8b0 comp rx=0 tx=0).stop 2024-04-17T19:50:00.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:00.236+0000 7f40ce800640 1 -- 172.21.15.29:0/1762376870 shutdown_connections 2024-04-17T19:50:00.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:00.236+0000 7f40ce800640 1 --2- 172.21.15.29:0/1762376870 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f40a8068400 0x7f40a806a910 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:00.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:00.236+0000 7f40ce800640 1 --2- 172.21.15.29:0/1762376870 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40c8160030 0x7f40c80c7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:00.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:00.236+0000 7f40ce800640 1 --2- 172.21.15.29:0/1762376870 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f40c8155f10 0x7f40c81f9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:00.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:00.236+0000 7f40ce800640 1 --2- 172.21.15.29:0/1762376870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c80ce470 0x7f40c813e490 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:00.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:00.236+0000 7f40ce800640 1 -- 172.21.15.29:0/1762376870 >> 172.21.15.29:0/1762376870 conn(0x7f40c8072db0 msgr2=0x7f40c80ca770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:00.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:00.236+0000 7f40ce800640 1 -- 172.21.15.29:0/1762376870 shutdown_connections 2024-04-17T19:50:00.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:00.236+0000 7f40ce800640 1 -- 172.21.15.29:0/1762376870 wait complete. 2024-04-17T19:50:00.258 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-17T19:50:02.258 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:50:02.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.516+0000 7f94e2c00640 1 Processor -- start 2024-04-17T19:50:02.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.516+0000 7f94e2c00640 1 -- start start 2024-04-17T19:50:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.520+0000 7f94e2c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94dc155f10 0x7f94dc15c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.520+0000 7f94e2c00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f94dc0ce470 0x7f94dc0ce880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.520+0000 7f94e2c00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94dc15cd70 con 0x7f94dc15f920 2024-04-17T19:50:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.520+0000 7f94e2c00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f94dc06c980 con 0x7f94dc0ce470 2024-04-17T19:50:02.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.520+0000 7f94e2c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f94dc06cc60 con 0x7f94dc155f10 2024-04-17T19:50:02.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.520+0000 7f94e1800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94dc155f10 0x7f94dc15c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:02.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.520+0000 7f94e2200640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f94dc15f920 0x7f94dc15fd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48968/0 (socket says 172.21.15.29:48968) 2024-04-17T19:50:02.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.520+0000 7f94e2200640 1 -- 172.21.15.29:0/2037911877 learned_addr learned my addr 172.21.15.29:0/2037911877 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:50:02.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.520+0000 7f94e0e00640 1 --2- 172.21.15.29:0/2037911877 >> v2:172.21.15.37:3300/0 conn(0x7f94dc0ce470 0x7f94dc0ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:02.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.520+0000 7f94dbe00640 1 -- 172.21.15.29:0/2037911877 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4114475024 0 0) 0x7f94d40014a0 con 0x7f94dc15f920 2024-04-17T19:50:02.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.520+0000 7f94dbe00640 1 -- 172.21.15.29:0/2037911877 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94c4003530 con 0x7f94dc15f920 2024-04-17T19:50:02.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.520+0000 7f94e1800640 1 -- 172.21.15.29:0/2037911877 >> v2:172.21.15.37:3300/0 conn(0x7f94dc0ce470 msgr2=0x7f94dc0ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:02.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.520+0000 7f94e1800640 1 --2- 172.21.15.29:0/2037911877 >> v2:172.21.15.37:3300/0 conn(0x7f94dc0ce470 0x7f94dc0ce880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:02.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.520+0000 7f94e1800640 1 -- 172.21.15.29:0/2037911877 >> v1:172.21.15.29:6789/0 conn(0x7f94dc15f920 legacy=0x7f94dc15fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:02.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.520+0000 7f94e1800640 1 -- 172.21.15.29:0/2037911877 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94dc06cf40 con 0x7f94dc155f10 2024-04-17T19:50:02.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.524+0000 7f94e1800640 1 --2- 172.21.15.29:0/2037911877 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94dc155f10 0x7f94dc15c830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f94d00097c0 tx=0x7f94d001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4891e03ba023411 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:02.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.524+0000 7f94dbe00640 1 -- 172.21.15.29:0/2037911877 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d00b0070 con 0x7f94dc155f10 2024-04-17T19:50:02.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.524+0000 7f94dbe00640 1 -- 172.21.15.29:0/2037911877 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94d00ab600 con 0x7f94dc155f10 2024-04-17T19:50:02.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.524+0000 7f94dbe00640 1 -- 172.21.15.29:0/2037911877 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d00ab9b0 con 0x7f94dc155f10 2024-04-17T19:50:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.528+0000 7f94e2c00640 1 -- 172.21.15.29:0/2037911877 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94dc155f10 msgr2=0x7f94dc15c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.528+0000 7f94e2c00640 1 --2- 172.21.15.29:0/2037911877 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94dc155f10 0x7f94dc15c830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f94d00097c0 tx=0x7f94d001c4a0 comp rx=0 tx=0).stop 2024-04-17T19:50:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.528+0000 7f94e2c00640 1 -- 172.21.15.29:0/2037911877 shutdown_connections 2024-04-17T19:50:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.528+0000 7f94e2c00640 1 --2- 172.21.15.29:0/2037911877 >> v2:172.21.15.37:3300/0 conn(0x7f94dc0ce470 0x7f94dc0ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.528+0000 7f94e2c00640 1 --2- 172.21.15.29:0/2037911877 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94dc155f10 0x7f94dc15c830 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.528+0000 7f94e2c00640 1 -- 172.21.15.29:0/2037911877 >> 172.21.15.29:0/2037911877 conn(0x7f94dc072db0 msgr2=0x7f94dc0731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:02.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.528+0000 7f94e2c00640 1 -- 172.21.15.29:0/2037911877 shutdown_connections 2024-04-17T19:50:02.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.528+0000 7f94e2c00640 1 -- 172.21.15.29:0/2037911877 wait complete. 2024-04-17T19:50:02.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.532+0000 7f94e2c00640 1 Processor -- start 2024-04-17T19:50:02.521 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.532+0000 7f94e2c00640 1 -- start start 2024-04-17T19:50:02.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.532+0000 7f94e2c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94dc0ce470 0x7f94dc0cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:02.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.532+0000 7f94e2c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94dc155f10 0x7f94dc13c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:02.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.532+0000 7f94e2c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94dc15f920 0x7f94dc1fd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:02.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.532+0000 7f94e2c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f94dc1690d0 con 0x7f94dc15f920 2024-04-17T19:50:02.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.532+0000 7f94e2c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f94dc168f50 con 0x7f94dc155f10 2024-04-17T19:50:02.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.532+0000 7f94e2c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f94dc169250 con 0x7f94dc0ce470 2024-04-17T19:50:02.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.532+0000 7f94e0e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94dc155f10 0x7f94dc13c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:02.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.532+0000 7f94e2200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94dc15f920 0x7f94dc1fd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:02.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.532+0000 7f94e0e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94dc155f10 0x7f94dc13c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:40918/0 (socket says 172.21.15.29:40918) 2024-04-17T19:50:02.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.532+0000 7f94e0e00640 1 -- 172.21.15.29:0/292986938 learned_addr learned my addr 172.21.15.29:0/292986938 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:02.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.532+0000 7f94e1800640 1 --2- 172.21.15.29:0/292986938 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94dc0ce470 0x7f94dc0cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:02.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.536+0000 7f94e2200640 1 -- 172.21.15.29:0/292986938 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94dc0ce470 msgr2=0x7f94dc0cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:02.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.536+0000 7f94e2200640 1 --2- 172.21.15.29:0/292986938 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94dc0ce470 0x7f94dc0cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:02.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.536+0000 7f94e2200640 1 -- 172.21.15.29:0/292986938 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94dc155f10 msgr2=0x7f94dc13c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:02.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.536+0000 7f94e2200640 1 --2- 172.21.15.29:0/292986938 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94dc155f10 0x7f94dc13c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:02.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.536+0000 7f94e2200640 1 -- 172.21.15.29:0/292986938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94dc0cd8e0 con 0x7f94dc15f920 2024-04-17T19:50:02.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.536+0000 7f94e2200640 1 --2- 172.21.15.29:0/292986938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94dc15f920 0x7f94dc1fd1d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f94d400ca70 tx=0x7f94d400cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:02.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.536+0000 7f94d9600640 1 -- 172.21.15.29:0/292986938 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d4017070 con 0x7f94dc15f920 2024-04-17T19:50:02.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.536+0000 7f94d9600640 1 -- 172.21.15.29:0/292986938 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94d400ab10 con 0x7f94dc15f920 2024-04-17T19:50:02.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.536+0000 7f94e2c00640 1 -- 172.21.15.29:0/292986938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94dc15d210 con 0x7f94dc15f920 2024-04-17T19:50:02.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.536+0000 7f94d9600640 1 -- 172.21.15.29:0/292986938 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d4012400 con 0x7f94dc15f920 2024-04-17T19:50:02.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.536+0000 7f94e2c00640 1 -- 172.21.15.29:0/292986938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94dc0cbdd0 con 0x7f94dc15f920 2024-04-17T19:50:02.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.536+0000 7f94e2c00640 1 -- 172.21.15.29:0/292986938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94dc15cf70 con 0x7f94dc15f920 2024-04-17T19:50:02.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.540+0000 7f94d9600640 1 -- 172.21.15.29:0/292986938 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f94d401a070 con 0x7f94dc15f920 2024-04-17T19:50:02.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.540+0000 7f94d9600640 1 --2- 172.21.15.29:0/292986938 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f94bc068400 0x7f94bc06a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:02.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.540+0000 7f94d9600640 1 -- 172.21.15.29:0/292986938 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f94d4086970 con 0x7f94dc15f920 2024-04-17T19:50:02.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.540+0000 7f94e1800640 1 --2- 172.21.15.29:0/292986938 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f94bc068400 0x7f94bc06a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:02.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.540+0000 7f94e1800640 1 --2- 172.21.15.29:0/292986938 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f94bc068400 0x7f94bc06a910 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f94d00ad000 tx=0x7f94d00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:02.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.548+0000 7f94d9600640 1 -- 172.21.15.29:0/292986938 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f94dc15cf70 con 0x7f94dc15f920 2024-04-17T19:50:02.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.956+0000 7f94e2c00640 1 -- 172.21.15.29:0/292986938 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94dc155000 con 0x7f94dc15f920 2024-04-17T19:50:02.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.956+0000 7f94d9600640 1 -- 172.21.15.29:0/292986938 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6902 (secure 0 0 0) 0x7f94dc15cd70 con 0x7f94dc15f920 2024-04-17T19:50:02.946 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:50:02.946 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:48:34.874038+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4226":{"gid":4226,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.37:6838/332907250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":332907250},{"type":"v1","addr":"172.21.15.37:6838","nonce":332907250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4226,4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:50:02.946 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 19 2024-04-17T19:50:02.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.960+0000 7f94e2c00640 1 -- 172.21.15.29:0/292986938 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f94bc068400 msgr2=0x7f94bc06a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:02.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.960+0000 7f94e2c00640 1 --2- 172.21.15.29:0/292986938 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f94bc068400 0x7f94bc06a910 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f94d00ad000 tx=0x7f94d00a6000 comp rx=0 tx=0).stop 2024-04-17T19:50:02.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.960+0000 7f94e2c00640 1 -- 172.21.15.29:0/292986938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94dc15f920 msgr2=0x7f94dc1fd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:02.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.960+0000 7f94e2c00640 1 --2- 172.21.15.29:0/292986938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94dc15f920 0x7f94dc1fd1d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f94d400ca70 tx=0x7f94d400cf30 comp rx=0 tx=0).stop 2024-04-17T19:50:02.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.960+0000 7f94e2c00640 1 -- 172.21.15.29:0/292986938 shutdown_connections 2024-04-17T19:50:02.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.960+0000 7f94e2c00640 1 --2- 172.21.15.29:0/292986938 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f94bc068400 0x7f94bc06a910 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:02.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.960+0000 7f94e2c00640 1 --2- 172.21.15.29:0/292986938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94dc15f920 0x7f94dc1fd1d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:02.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.960+0000 7f94e2c00640 1 --2- 172.21.15.29:0/292986938 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94dc155f10 0x7f94dc13c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:02.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.960+0000 7f94e2c00640 1 --2- 172.21.15.29:0/292986938 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f94dc0ce470 0x7f94dc0cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:02.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.960+0000 7f94e2c00640 1 -- 172.21.15.29:0/292986938 >> 172.21.15.29:0/292986938 conn(0x7f94dc072db0 msgr2=0x7f94dc0ca750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:02.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.960+0000 7f94e2c00640 1 -- 172.21.15.29:0/292986938 shutdown_connections 2024-04-17T19:50:02.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:02.960+0000 7f94e2c00640 1 -- 172.21.15.29:0/292986938 wait complete. 2024-04-17T19:50:02.981 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-17T19:50:04.983 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:50:05.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.240+0000 7fc20d400640 1 Processor -- start 2024-04-17T19:50:05.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.240+0000 7fc20d400640 1 -- start start 2024-04-17T19:50:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.244+0000 7fc20d400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc20815c870 0x7fc2080c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.244+0000 7fc20d400640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2080ca210 con 0x7fc208155ff0 2024-04-17T19:50:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.244+0000 7fc20d400640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc20806cc20 con 0x7fc2080c9e00 2024-04-17T19:50:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.244+0000 7fc20d400640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fc2080c5770 con 0x7fc20815c870 2024-04-17T19:50:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.244+0000 7fc207e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc208155ff0 0x7fc2080ce490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48982/0 (socket says 172.21.15.29:48982) 2024-04-17T19:50:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.244+0000 7fc207e00640 1 -- 172.21.15.29:0/2845338051 learned_addr learned my addr 172.21.15.29:0/2845338051 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:50:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.244+0000 7fc207400640 1 --2- 172.21.15.29:0/2845338051 >> v2:172.21.15.29:3300/0 conn(0x7fc20815c870 0x7fc2080c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.244+0000 7fc207400640 1 -- 172.21.15.29:0/2845338051 >> v1:172.21.15.37:6789/0 conn(0x7fc2080c9e00 legacy=0x7fc2081667c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.244+0000 7fc207400640 1 -- 172.21.15.29:0/2845338051 >> v1:172.21.15.29:6789/0 conn(0x7fc208155ff0 legacy=0x7fc2080ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.244+0000 7fc206a00640 1 -- 172.21.15.29:0/2845338051 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 377840659 0 0) 0x7fc1f8001120 con 0x7fc208155ff0 2024-04-17T19:50:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.244+0000 7fc207400640 1 -- 172.21.15.29:0/2845338051 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc20806cf00 con 0x7fc20815c870 2024-04-17T19:50:05.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.248+0000 7fc207400640 1 --2- 172.21.15.29:0/2845338051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc20815c870 0x7fc2080c98c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc1f40097c0 tx=0x7fc1f401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3592aa0b8969c413 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:05.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.248+0000 7fc206a00640 1 -- 172.21.15.29:0/2845338051 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f40b0070 con 0x7fc20815c870 2024-04-17T19:50:05.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.248+0000 7fc206a00640 1 -- 172.21.15.29:0/2845338051 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1f40ab600 con 0x7fc20815c870 2024-04-17T19:50:05.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.248+0000 7fc206a00640 1 -- 172.21.15.29:0/2845338051 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f40ab9d0 con 0x7fc20815c870 2024-04-17T19:50:05.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.248+0000 7fc20d400640 1 -- 172.21.15.29:0/2845338051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc20815c870 msgr2=0x7fc2080c98c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:05.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.248+0000 7fc20d400640 1 --2- 172.21.15.29:0/2845338051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc20815c870 0x7fc2080c98c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc1f40097c0 tx=0x7fc1f401c410 comp rx=0 tx=0).stop 2024-04-17T19:50:05.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.248+0000 7fc20d400640 1 -- 172.21.15.29:0/2845338051 shutdown_connections 2024-04-17T19:50:05.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.248+0000 7fc20d400640 1 --2- 172.21.15.29:0/2845338051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc20815c870 0x7fc2080c98c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:05.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.248+0000 7fc20d400640 1 -- 172.21.15.29:0/2845338051 >> 172.21.15.29:0/2845338051 conn(0x7fc208072db0 msgr2=0x7fc2080731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:05.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.252+0000 7fc20d400640 1 -- 172.21.15.29:0/2845338051 shutdown_connections 2024-04-17T19:50:05.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.252+0000 7fc20d400640 1 -- 172.21.15.29:0/2845338051 wait complete. 2024-04-17T19:50:05.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.252+0000 7fc20d400640 1 Processor -- start 2024-04-17T19:50:05.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.252+0000 7fc20d400640 1 -- start start 2024-04-17T19:50:05.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.256+0000 7fc20d400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2080c9e00 0x7fc208146a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:05.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.256+0000 7fc20d400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc208155ff0 0x7fc2080c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:05.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.256+0000 7fc20d400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc20815c870 0x7fc208147b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:05.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.256+0000 7fc20d400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc20806c320 con 0x7fc208155ff0 2024-04-17T19:50:05.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.256+0000 7fc20d400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc20815cc80 con 0x7fc2080c9e00 2024-04-17T19:50:05.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.256+0000 7fc20d400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc20806c4a0 con 0x7fc20815c870 2024-04-17T19:50:05.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.256+0000 7fc207400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc208155ff0 0x7fc2080c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:05.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.256+0000 7fc207400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc208155ff0 0x7fc2080c68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33440/0 (socket says 172.21.15.29:33440) 2024-04-17T19:50:05.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.256+0000 7fc207400640 1 -- 172.21.15.29:0/723045174 learned_addr learned my addr 172.21.15.29:0/723045174 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:05.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.256+0000 7fc207e00640 1 --2- 172.21.15.29:0/723045174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2080c9e00 0x7fc208146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:05.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.256+0000 7fc20ca00640 1 --2- 172.21.15.29:0/723045174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc20815c870 0x7fc208147b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:05.248 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.256+0000 7fc207e00640 1 -- 172.21.15.29:0/723045174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc20815c870 msgr2=0x7fc208147b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.256+0000 7fc207e00640 1 --2- 172.21.15.29:0/723045174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc20815c870 0x7fc208147b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.256+0000 7fc207e00640 1 -- 172.21.15.29:0/723045174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc208155ff0 msgr2=0x7fc2080c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.256+0000 7fc207e00640 1 --2- 172.21.15.29:0/723045174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc208155ff0 0x7fc2080c68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.256+0000 7fc207e00640 1 -- 172.21.15.29:0/723045174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc208146f70 con 0x7fc2080c9e00 2024-04-17T19:50:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.256+0000 7fc207400640 1 --2- 172.21.15.29:0/723045174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc208155ff0 0x7fc2080c68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:50:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.260+0000 7fc207e00640 1 --2- 172.21.15.29:0/723045174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2080c9e00 0x7fc208146a30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc1f800c410 tx=0x7fc1f800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:05.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.260+0000 7fc1ffe00640 1 -- 172.21.15.29:0/723045174 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f80074e0 con 0x7fc2080c9e00 2024-04-17T19:50:05.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.260+0000 7fc1ffe00640 1 -- 172.21.15.29:0/723045174 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1f80047f0 con 0x7fc2080c9e00 2024-04-17T19:50:05.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.260+0000 7fc20d400640 1 -- 172.21.15.29:0/723045174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2080c6e30 con 0x7fc2080c9e00 2024-04-17T19:50:05.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.260+0000 7fc1ffe00640 1 -- 172.21.15.29:0/723045174 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f8007680 con 0x7fc2080c9e00 2024-04-17T19:50:05.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.260+0000 7fc20d400640 1 -- 172.21.15.29:0/723045174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc208148c10 con 0x7fc2080c9e00 2024-04-17T19:50:05.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.260+0000 7fc20d400640 1 -- 172.21.15.29:0/723045174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc20804ff10 con 0x7fc2080c9e00 2024-04-17T19:50:05.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.264+0000 7fc1ffe00640 1 -- 172.21.15.29:0/723045174 <== mon.1 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc1f8011070 con 0x7fc2080c9e00 2024-04-17T19:50:05.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.264+0000 7fc1ffe00640 1 --2- 172.21.15.29:0/723045174 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc1e8068380 0x7fc1e806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:05.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.264+0000 7fc1ffe00640 1 -- 172.21.15.29:0/723045174 <== mon.1 v2:172.21.15.37:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fc1f8086e30 con 0x7fc2080c9e00 2024-04-17T19:50:05.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.264+0000 7fc207400640 1 --2- 172.21.15.29:0/723045174 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc1e8068380 0x7fc1e806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:05.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.264+0000 7fc207400640 1 --2- 172.21.15.29:0/723045174 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc1e8068380 0x7fc1e806a890 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc1f401a040 tx=0x7fc1f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:05.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.276+0000 7fc1ffe00640 1 -- 172.21.15.29:0/723045174 <== mon.1 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc2080ca210 con 0x7fc2080c9e00 2024-04-17T19:50:05.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.684+0000 7fc20d400640 1 -- 172.21.15.29:0/723045174 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc20806c6a0 con 0x7fc2080c9e00 2024-04-17T19:50:05.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.688+0000 7fc1ffe00640 1 -- 172.21.15.29:0/723045174 <== mon.1 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6902 (secure 0 0 0) 0x7fc1f8056a10 con 0x7fc2080c9e00 2024-04-17T19:50:05.678 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:50:05.678 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:48:34.874038+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4226":{"gid":4226,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.37:6838/332907250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":332907250},{"type":"v1","addr":"172.21.15.37:6838","nonce":332907250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4226,4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:50:05.678 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 19 2024-04-17T19:50:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.688+0000 7fc20d400640 1 -- 172.21.15.29:0/723045174 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc1e8068380 msgr2=0x7fc1e806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.688+0000 7fc20d400640 1 --2- 172.21.15.29:0/723045174 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc1e8068380 0x7fc1e806a890 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc1f401a040 tx=0x7fc1f4008040 comp rx=0 tx=0).stop 2024-04-17T19:50:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.688+0000 7fc20d400640 1 -- 172.21.15.29:0/723045174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2080c9e00 msgr2=0x7fc208146a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.688+0000 7fc20d400640 1 --2- 172.21.15.29:0/723045174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2080c9e00 0x7fc208146a30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc1f800c410 tx=0x7fc1f800c8d0 comp rx=0 tx=0).stop 2024-04-17T19:50:05.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.692+0000 7fc20d400640 1 -- 172.21.15.29:0/723045174 shutdown_connections 2024-04-17T19:50:05.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.692+0000 7fc20d400640 1 --2- 172.21.15.29:0/723045174 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc1e8068380 0x7fc1e806a890 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:05.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.692+0000 7fc20d400640 1 --2- 172.21.15.29:0/723045174 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc20815c870 0x7fc208147b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:05.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.692+0000 7fc20d400640 1 --2- 172.21.15.29:0/723045174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc208155ff0 0x7fc2080c68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:05.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.692+0000 7fc20d400640 1 --2- 172.21.15.29:0/723045174 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc2080c9e00 0x7fc208146a30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:05.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.692+0000 7fc20d400640 1 -- 172.21.15.29:0/723045174 >> 172.21.15.29:0/723045174 conn(0x7fc208072db0 msgr2=0x7fc208070750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:05.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.696+0000 7fc20d400640 1 -- 172.21.15.29:0/723045174 shutdown_connections 2024-04-17T19:50:05.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:05.696+0000 7fc20d400640 1 -- 172.21.15.29:0/723045174 wait complete. 2024-04-17T19:50:05.714 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-17T19:50:06.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-04-17T19:50:07.715 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:50:07.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.972+0000 7f3747200640 1 Processor -- start 2024-04-17T19:50:07.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.972+0000 7f3747200640 1 -- start start 2024-04-17T19:50:07.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.976+0000 7f3747200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37480cb370 0x7f37480c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:07.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.976+0000 7f3747200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3748159c70 con 0x7f37480c89d0 2024-04-17T19:50:07.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.976+0000 7f3747200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3748159f50 con 0x7f3748153d40 2024-04-17T19:50:07.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.976+0000 7f3747200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f374815a230 con 0x7f37480cb370 2024-04-17T19:50:07.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.976+0000 7f3745400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f37480c89d0 0x7f37480cac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48996/0 (socket says 172.21.15.29:48996) 2024-04-17T19:50:07.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.976+0000 7f3745400640 1 -- 172.21.15.29:0/1206093859 learned_addr learned my addr 172.21.15.29:0/1206093859 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:50:07.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.976+0000 7f3746800640 1 --2- 172.21.15.29:0/1206093859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37480cb370 0x7f37480c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:07.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.976+0000 7f3744a00640 1 -- 172.21.15.29:0/1206093859 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2768709854 0 0) 0x7f3748159c70 con 0x7f37480c89d0 2024-04-17T19:50:07.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.976+0000 7f3744a00640 1 -- 172.21.15.29:0/1206093859 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f372c003530 con 0x7f37480c89d0 2024-04-17T19:50:07.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.976+0000 7f3744a00640 1 -- 172.21.15.29:0/1206093859 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 341262685 0 0) 0x7f3748159f50 con 0x7f3748153d40 2024-04-17T19:50:07.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.976+0000 7f3744a00640 1 -- 172.21.15.29:0/1206093859 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3748159c70 con 0x7f3748153d40 2024-04-17T19:50:07.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.976+0000 7f3746800640 1 -- 172.21.15.29:0/1206093859 >> v1:172.21.15.37:6789/0 conn(0x7f3748153d40 legacy=0x7f3748067ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:07.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.980+0000 7f3744a00640 1 -- 172.21.15.29:0/1206093859 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1525373091 0 0) 0x7f372c003530 con 0x7f37480c89d0 2024-04-17T19:50:07.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.980+0000 7f3746800640 1 -- 172.21.15.29:0/1206093859 >> v1:172.21.15.29:6789/0 conn(0x7f37480c89d0 legacy=0x7f37480cac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:07.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.980+0000 7f3746800640 1 -- 172.21.15.29:0/1206093859 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374815a510 con 0x7f37480cb370 2024-04-17T19:50:07.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.984+0000 7f3746800640 1 --2- 172.21.15.29:0/1206093859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37480cb370 0x7f37480c8de0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3740009dc0 tx=0x7f374001f0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=371594c4775e4a1a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:07.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.984+0000 7f3744a00640 1 -- 172.21.15.29:0/1206093859 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37400b5070 con 0x7f37480cb370 2024-04-17T19:50:07.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.984+0000 7f3744a00640 1 -- 172.21.15.29:0/1206093859 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37400a8c30 con 0x7f37480cb370 2024-04-17T19:50:07.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.984+0000 7f3744a00640 1 -- 172.21.15.29:0/1206093859 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3734001680 con 0x7f37480cb370 2024-04-17T19:50:07.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.984+0000 7f3747200640 1 -- 172.21.15.29:0/1206093859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37480cb370 msgr2=0x7f37480c8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:07.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.984+0000 7f3747200640 1 --2- 172.21.15.29:0/1206093859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37480cb370 0x7f37480c8de0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3740009dc0 tx=0x7f374001f0a0 comp rx=0 tx=0).stop 2024-04-17T19:50:07.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.984+0000 7f3747200640 1 -- 172.21.15.29:0/1206093859 shutdown_connections 2024-04-17T19:50:07.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.984+0000 7f3747200640 1 --2- 172.21.15.29:0/1206093859 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37480cb370 0x7f37480c8de0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:07.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.984+0000 7f3747200640 1 -- 172.21.15.29:0/1206093859 >> 172.21.15.29:0/1206093859 conn(0x7f3748072db0 msgr2=0x7f37480731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:07.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.984+0000 7f3747200640 1 -- 172.21.15.29:0/1206093859 shutdown_connections 2024-04-17T19:50:07.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.984+0000 7f3747200640 1 -- 172.21.15.29:0/1206093859 wait complete. 2024-04-17T19:50:07.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.988+0000 7f3747200640 1 Processor -- start 2024-04-17T19:50:07.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.988+0000 7f3747200640 1 -- start start 2024-04-17T19:50:07.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3747200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37480c89d0 0x7f374814ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:07.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3747200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37480cb370 0x7f3748133660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:07.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3747200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3748153d40 0x7f3748133ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:07.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3747200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f37480683f0 con 0x7f3748153d40 2024-04-17T19:50:07.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3747200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3748154150 con 0x7f37480c89d0 2024-04-17T19:50:07.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3747200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3748068570 con 0x7f37480cb370 2024-04-17T19:50:07.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3745400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37480cb370 0x7f3748133660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:07.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3746800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3748153d40 0x7f3748133ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:07.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3745400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37480cb370 0x7f3748133660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:53208/0 (socket says 172.21.15.29:53208) 2024-04-17T19:50:07.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3745400640 1 -- 172.21.15.29:0/748766275 learned_addr learned my addr 172.21.15.29:0/748766275 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:07.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3745e00640 1 --2- 172.21.15.29:0/748766275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37480c89d0 0x7f374814ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:07.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3746800640 1 -- 172.21.15.29:0/748766275 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37480cb370 msgr2=0x7f3748133660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:07.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3746800640 1 --2- 172.21.15.29:0/748766275 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37480cb370 0x7f3748133660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:07.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3746800640 1 -- 172.21.15.29:0/748766275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37480c89d0 msgr2=0x7f374814ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:07.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3746800640 1 --2- 172.21.15.29:0/748766275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37480c89d0 0x7f374814ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:07.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3746800640 1 -- 172.21.15.29:0/748766275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37480c7300 con 0x7f3748153d40 2024-04-17T19:50:07.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3745400640 1 --2- 172.21.15.29:0/748766275 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37480cb370 0x7f3748133660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:50:07.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3746800640 1 --2- 172.21.15.29:0/748766275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3748153d40 0x7f3748133ba0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f37400af000 tx=0x7f374001be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:07.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f373e000640 1 -- 172.21.15.29:0/748766275 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37400b5050 con 0x7f3748153d40 2024-04-17T19:50:07.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3747200640 1 -- 172.21.15.29:0/748766275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3748147f50 con 0x7f3748153d40 2024-04-17T19:50:07.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f3747200640 1 -- 172.21.15.29:0/748766275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37480cd4f0 con 0x7f3748153d40 2024-04-17T19:50:07.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f373e000640 1 -- 172.21.15.29:0/748766275 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37400b6040 con 0x7f3748153d40 2024-04-17T19:50:07.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.992+0000 7f373e000640 1 -- 172.21.15.29:0/748766275 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3740009450 con 0x7f3748153d40 2024-04-17T19:50:07.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.996+0000 7f3747200640 1 -- 172.21.15.29:0/748766275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f374804ff10 con 0x7f3748153d40 2024-04-17T19:50:07.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:07.996+0000 7f373e000640 1 -- 172.21.15.29:0/748766275 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f37400c3020 con 0x7f3748153d40 2024-04-17T19:50:07.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.000+0000 7f373e000640 1 --2- 172.21.15.29:0/748766275 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3728068380 0x7f372806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:07.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.000+0000 7f373e000640 1 -- 172.21.15.29:0/748766275 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f3740125670 con 0x7f3748153d40 2024-04-17T19:50:07.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.000+0000 7f3745e00640 1 --2- 172.21.15.29:0/748766275 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3728068380 0x7f372806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:07.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.000+0000 7f3745e00640 1 --2- 172.21.15.29:0/748766275 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3728068380 0x7f372806a890 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3738007fd0 tx=0x7f3738009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:07.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.008+0000 7f373e000640 1 -- 172.21.15.29:0/748766275 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3740018c00 con 0x7f3748153d40 2024-04-17T19:50:08.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.412+0000 7f3747200640 1 -- 172.21.15.29:0/748766275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3748167260 con 0x7f3748153d40 2024-04-17T19:50:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.416+0000 7f373e000640 1 -- 172.21.15.29:0/748766275 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6902 (secure 0 0 0) 0x7f37400f51e0 con 0x7f3748153d40 2024-04-17T19:50:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:50:08.405 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:48:34.874038+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4226":{"gid":4226,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.37:6838/332907250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":332907250},{"type":"v1","addr":"172.21.15.37:6838","nonce":332907250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4226,4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:50:08.405 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 19 2024-04-17T19:50:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.416+0000 7f3747200640 1 -- 172.21.15.29:0/748766275 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3728068380 msgr2=0x7f372806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.416+0000 7f3747200640 1 --2- 172.21.15.29:0/748766275 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3728068380 0x7f372806a890 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3738007fd0 tx=0x7f3738009040 comp rx=0 tx=0).stop 2024-04-17T19:50:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.416+0000 7f3747200640 1 -- 172.21.15.29:0/748766275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3748153d40 msgr2=0x7f3748133ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.416+0000 7f3747200640 1 --2- 172.21.15.29:0/748766275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3748153d40 0x7f3748133ba0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f37400af000 tx=0x7f374001be40 comp rx=0 tx=0).stop 2024-04-17T19:50:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.416+0000 7f3747200640 1 -- 172.21.15.29:0/748766275 shutdown_connections 2024-04-17T19:50:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.416+0000 7f3747200640 1 --2- 172.21.15.29:0/748766275 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3728068380 0x7f372806a890 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:08.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.416+0000 7f3747200640 1 --2- 172.21.15.29:0/748766275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3748153d40 0x7f3748133ba0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:08.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.416+0000 7f3747200640 1 --2- 172.21.15.29:0/748766275 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37480cb370 0x7f3748133660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:08.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.416+0000 7f3747200640 1 --2- 172.21.15.29:0/748766275 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37480c89d0 0x7f374814ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:08.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.416+0000 7f3747200640 1 -- 172.21.15.29:0/748766275 >> 172.21.15.29:0/748766275 conn(0x7f3748072db0 msgr2=0x7f374806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:08.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.416+0000 7f3747200640 1 -- 172.21.15.29:0/748766275 shutdown_connections 2024-04-17T19:50:08.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:08.416+0000 7f3747200640 1 -- 172.21.15.29:0/748766275 wait complete. 2024-04-17T19:50:08.441 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-17T19:50:10.473 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:50:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.640+0000 7f2a8be00640 1 Processor -- start 2024-04-17T19:50:10.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.640+0000 7f2a8be00640 1 -- start start 2024-04-17T19:50:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.640+0000 7f2a8be00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2a84160090 0x7f2a84167900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.640+0000 7f2a8be00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a8406c900 con 0x7f2a840caf80 2024-04-17T19:50:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.640+0000 7f2a8be00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a8406c690 con 0x7f2a8415fc80 2024-04-17T19:50:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.640+0000 7f2a8be00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f2a8415c050 con 0x7f2a84160090 2024-04-17T19:50:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.640+0000 7f2a89400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2a840caf80 0x7f2a8415c9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40324/0 (socket says 172.21.15.29:40324) 2024-04-17T19:50:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.640+0000 7f2a89400640 1 -- 172.21.15.29:0/440905770 learned_addr learned my addr 172.21.15.29:0/440905770 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:50:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.640+0000 7f2a89e00640 1 --2- 172.21.15.29:0/440905770 >> v2:172.21.15.29:3300/0 conn(0x7f2a84160090 0x7f2a84167900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:10.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.640+0000 7f2a83e00640 1 -- 172.21.15.29:0/440905770 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1978450926 0 0) 0x7f2a78001120 con 0x7f2a840caf80 2024-04-17T19:50:10.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.640+0000 7f2a83e00640 1 -- 172.21.15.29:0/440905770 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a6c003530 con 0x7f2a840caf80 2024-04-17T19:50:10.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.640+0000 7f2a89e00640 1 -- 172.21.15.29:0/440905770 >> v1:172.21.15.37:6789/0 conn(0x7f2a8415fc80 legacy=0x7f2a840cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:10.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.640+0000 7f2a89e00640 1 -- 172.21.15.29:0/440905770 >> v1:172.21.15.29:6789/0 conn(0x7f2a840caf80 legacy=0x7f2a8415c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:10.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.640+0000 7f2a89e00640 1 -- 172.21.15.29:0/440905770 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a841604a0 con 0x7f2a84160090 2024-04-17T19:50:10.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.644+0000 7f2a89e00640 1 --2- 172.21.15.29:0/440905770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a84160090 0x7f2a84167900 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009d90 tx=0x7f2a7c01f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21fafd58c2322b3c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:10.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.644+0000 7f2a83e00640 1 -- 172.21.15.29:0/440905770 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c0b4070 con 0x7f2a84160090 2024-04-17T19:50:10.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.644+0000 7f2a83e00640 1 -- 172.21.15.29:0/440905770 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a7c0af600 con 0x7f2a84160090 2024-04-17T19:50:10.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.644+0000 7f2a83e00640 1 -- 172.21.15.29:0/440905770 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c0af9b0 con 0x7f2a84160090 2024-04-17T19:50:10.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.644+0000 7f2a8be00640 1 -- 172.21.15.29:0/440905770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a84160090 msgr2=0x7f2a84167900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:10.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.644+0000 7f2a8be00640 1 --2- 172.21.15.29:0/440905770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a84160090 0x7f2a84167900 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009d90 tx=0x7f2a7c01f150 comp rx=0 tx=0).stop 2024-04-17T19:50:10.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.644+0000 7f2a8be00640 1 -- 172.21.15.29:0/440905770 shutdown_connections 2024-04-17T19:50:10.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.644+0000 7f2a8be00640 1 --2- 172.21.15.29:0/440905770 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a84160090 0x7f2a84167900 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:10.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.644+0000 7f2a8be00640 1 -- 172.21.15.29:0/440905770 >> 172.21.15.29:0/440905770 conn(0x7f2a8406e3d0 msgr2=0x7f2a8406e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:10.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.644+0000 7f2a8be00640 1 -- 172.21.15.29:0/440905770 shutdown_connections 2024-04-17T19:50:10.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.648+0000 7f2a8be00640 1 -- 172.21.15.29:0/440905770 wait complete. 2024-04-17T19:50:10.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.648+0000 7f2a8be00640 1 Processor -- start 2024-04-17T19:50:10.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.648+0000 7f2a8be00640 1 -- start start 2024-04-17T19:50:10.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.648+0000 7f2a8be00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a840caf80 0x7f2a840af7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:10.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.648+0000 7f2a8be00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a8415fc80 0x7f2a841f5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:10.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.648+0000 7f2a8be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a84160090 0x7f2a84158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:10.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.648+0000 7f2a8be00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8406d140 con 0x7f2a84160090 2024-04-17T19:50:10.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.648+0000 7f2a8be00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2a8406ce60 con 0x7f2a840caf80 2024-04-17T19:50:10.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.648+0000 7f2a8be00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2a8406d420 con 0x7f2a8415fc80 2024-04-17T19:50:10.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.652+0000 7f2a89e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a84160090 0x7f2a84158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:10.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.652+0000 7f2a89e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a84160090 0x7f2a84158940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46680/0 (socket says 172.21.15.29:46680) 2024-04-17T19:50:10.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.652+0000 7f2a89e00640 1 -- 172.21.15.29:0/1815443719 learned_addr learned my addr 172.21.15.29:0/1815443719 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:10.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.652+0000 7f2a89400640 1 --2- 172.21.15.29:0/1815443719 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a840caf80 0x7f2a840af7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:10.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.652+0000 7f2a88a00640 1 --2- 172.21.15.29:0/1815443719 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a8415fc80 0x7f2a841f5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:10.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.652+0000 7f2a89e00640 1 -- 172.21.15.29:0/1815443719 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a8415fc80 msgr2=0x7f2a841f5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:10.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.652+0000 7f2a89e00640 1 --2- 172.21.15.29:0/1815443719 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a8415fc80 0x7f2a841f5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:10.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.652+0000 7f2a89e00640 1 -- 172.21.15.29:0/1815443719 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a840caf80 msgr2=0x7f2a840af7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:10.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.652+0000 7f2a89e00640 1 --2- 172.21.15.29:0/1815443719 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a840caf80 0x7f2a840af7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:10.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.652+0000 7f2a89e00640 1 -- 172.21.15.29:0/1815443719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a840b0160 con 0x7f2a84160090 2024-04-17T19:50:10.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.652+0000 7f2a89e00640 1 --2- 172.21.15.29:0/1815443719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a84160090 0x7f2a84158940 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009d90 tx=0x7f2a7c01bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:10.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.652+0000 7f2a81600640 1 -- 172.21.15.29:0/1815443719 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c0b4040 con 0x7f2a84160090 2024-04-17T19:50:10.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.652+0000 7f2a81600640 1 -- 172.21.15.29:0/1815443719 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a7c0039a0 con 0x7f2a84160090 2024-04-17T19:50:10.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.652+0000 7f2a81600640 1 -- 172.21.15.29:0/1815443719 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c009450 con 0x7f2a84160090 2024-04-17T19:50:10.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.652+0000 7f2a8be00640 1 -- 172.21.15.29:0/1815443719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a841e7510 con 0x7f2a84160090 2024-04-17T19:50:10.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.652+0000 7f2a8be00640 1 -- 172.21.15.29:0/1815443719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a841e7ad0 con 0x7f2a84160090 2024-04-17T19:50:10.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.652+0000 7f2a8be00640 1 -- 172.21.15.29:0/1815443719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a84069b90 con 0x7f2a84160090 2024-04-17T19:50:10.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.652+0000 7f2a81600640 1 -- 172.21.15.29:0/1815443719 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2a7c0c4060 con 0x7f2a84160090 2024-04-17T19:50:10.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.656+0000 7f2a81600640 1 --2- 172.21.15.29:0/1815443719 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2a64068380 0x7f2a6406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:10.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.656+0000 7f2a81600640 1 -- 172.21.15.29:0/1815443719 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f2a7c124640 con 0x7f2a84160090 2024-04-17T19:50:10.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.656+0000 7f2a89400640 1 --2- 172.21.15.29:0/1815443719 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2a64068380 0x7f2a6406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:10.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.656+0000 7f2a89400640 1 --2- 172.21.15.29:0/1815443719 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2a64068380 0x7f2a6406a890 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2a780012c0 tx=0x7f2a78007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:10.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.660+0000 7f2a81600640 1 -- 172.21.15.29:0/1815443719 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2a7c018cb0 con 0x7f2a84160090 2024-04-17T19:50:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.860+0000 7f2a8be00640 1 -- 172.21.15.29:0/1815443719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a840cad60 con 0x7f2a84160090 2024-04-17T19:50:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.860+0000 7f2a81600640 1 -- 172.21.15.29:0/1815443719 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6902 (secure 0 0 0) 0x7f2a7c0f4080 con 0x7f2a84160090 2024-04-17T19:50:10.852 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:50:10.852 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:48:34.874038+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4226":{"gid":4226,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.37:6838/332907250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":332907250},{"type":"v1","addr":"172.21.15.37:6838","nonce":332907250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4226,4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:50:10.852 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 19 2024-04-17T19:50:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.864+0000 7f2a8be00640 1 -- 172.21.15.29:0/1815443719 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2a64068380 msgr2=0x7f2a6406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.864+0000 7f2a8be00640 1 --2- 172.21.15.29:0/1815443719 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2a64068380 0x7f2a6406a890 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2a780012c0 tx=0x7f2a78007b80 comp rx=0 tx=0).stop 2024-04-17T19:50:10.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.864+0000 7f2a8be00640 1 -- 172.21.15.29:0/1815443719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a84160090 msgr2=0x7f2a84158940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.864+0000 7f2a8be00640 1 --2- 172.21.15.29:0/1815443719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a84160090 0x7f2a84158940 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009d90 tx=0x7f2a7c01bef0 comp rx=0 tx=0).stop 2024-04-17T19:50:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.864+0000 7f2a8be00640 1 -- 172.21.15.29:0/1815443719 shutdown_connections 2024-04-17T19:50:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.864+0000 7f2a8be00640 1 --2- 172.21.15.29:0/1815443719 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2a64068380 0x7f2a6406a890 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.864+0000 7f2a8be00640 1 --2- 172.21.15.29:0/1815443719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a84160090 0x7f2a84158940 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.864+0000 7f2a8be00640 1 --2- 172.21.15.29:0/1815443719 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2a8415fc80 0x7f2a841f5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.864+0000 7f2a8be00640 1 --2- 172.21.15.29:0/1815443719 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2a840caf80 0x7f2a840af7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.864+0000 7f2a8be00640 1 -- 172.21.15.29:0/1815443719 >> 172.21.15.29:0/1815443719 conn(0x7f2a8406e3d0 msgr2=0x7f2a840cc350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.864+0000 7f2a8be00640 1 -- 172.21.15.29:0/1815443719 shutdown_connections 2024-04-17T19:50:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:10.864+0000 7f2a8be00640 1 -- 172.21.15.29:0/1815443719 wait complete. 2024-04-17T19:50:10.876 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-17T19:50:12.877 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:50:13.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.056+0000 7f6b3be00640 1 Processor -- start 2024-04-17T19:50:13.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.056+0000 7f6b3be00640 1 -- start start 2024-04-17T19:50:13.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.060+0000 7f6b3be00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b3c158c70 0x7f6b3c141c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:13.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.060+0000 7f6b3be00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6b3c0c8460 0x7f6b3c1635f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:13.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.060+0000 7f6b3be00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b3c159c10 con 0x7f6b3c0c8050 2024-04-17T19:50:13.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.060+0000 7f6b3be00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f6b3c0c7bc0 con 0x7f6b3c0c8460 2024-04-17T19:50:13.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.060+0000 7f6b3be00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6b3c159370 con 0x7f6b3c158c70 2024-04-17T19:50:13.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.060+0000 7f6b3aa00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b3c158c70 0x7f6b3c141c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:13.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.060+0000 7f6b3aa00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b3c158c70 0x7f6b3c141c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:47534/0 (socket says 172.21.15.29:47534) 2024-04-17T19:50:13.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.060+0000 7f6b3aa00640 1 -- 172.21.15.29:0/601751646 learned_addr learned my addr 172.21.15.29:0/601751646 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:13.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.060+0000 7f6b3b400640 1 --2- 172.21.15.29:0/601751646 >> v2:172.21.15.37:3300/0 conn(0x7f6b3c0c8460 0x7f6b3c1635f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:13.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.060+0000 7f6b3aa00640 1 -- 172.21.15.29:0/601751646 >> v2:172.21.15.37:3300/0 conn(0x7f6b3c0c8460 msgr2=0x7f6b3c1635f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:13.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.060+0000 7f6b3aa00640 1 --2- 172.21.15.29:0/601751646 >> v2:172.21.15.37:3300/0 conn(0x7f6b3c0c8460 0x7f6b3c1635f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:13.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.060+0000 7f6b3aa00640 1 -- 172.21.15.29:0/601751646 >> v1:172.21.15.37:6789/0 conn(0x7f6b3c0c8050 legacy=0x7f6b3c0c5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:13.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.060+0000 7f6b3aa00640 1 -- 172.21.15.29:0/601751646 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b3c159650 con 0x7f6b3c158c70 2024-04-17T19:50:13.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.064+0000 7f6b3aa00640 1 --2- 172.21.15.29:0/601751646 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b3c158c70 0x7f6b3c141c20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009870 tx=0x7f6b2c01c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85f2dab1fe1ea265 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:13.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.064+0000 7f6b39600640 1 -- 172.21.15.29:0/601751646 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c0ab650 con 0x7f6b3c158c70 2024-04-17T19:50:13.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.064+0000 7f6b39600640 1 -- 172.21.15.29:0/601751646 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b2c0b0070 con 0x7f6b3c158c70 2024-04-17T19:50:13.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.064+0000 7f6b39600640 1 -- 172.21.15.29:0/601751646 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b2c0aba10 con 0x7f6b3c158c70 2024-04-17T19:50:13.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.064+0000 7f6b3be00640 1 -- 172.21.15.29:0/601751646 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b3c158c70 msgr2=0x7f6b3c141c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:13.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.064+0000 7f6b3be00640 1 --2- 172.21.15.29:0/601751646 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b3c158c70 0x7f6b3c141c20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c009870 tx=0x7f6b2c01c560 comp rx=0 tx=0).stop 2024-04-17T19:50:13.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.068+0000 7f6b3be00640 1 -- 172.21.15.29:0/601751646 shutdown_connections 2024-04-17T19:50:13.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.068+0000 7f6b3be00640 1 --2- 172.21.15.29:0/601751646 >> v2:172.21.15.37:3300/0 conn(0x7f6b3c0c8460 0x7f6b3c1635f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:13.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.068+0000 7f6b3be00640 1 --2- 172.21.15.29:0/601751646 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b3c158c70 0x7f6b3c141c20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:13.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.068+0000 7f6b3be00640 1 -- 172.21.15.29:0/601751646 >> 172.21.15.29:0/601751646 conn(0x7f6b3c0cee40 msgr2=0x7f6b3c0cad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:13.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.068+0000 7f6b3be00640 1 -- 172.21.15.29:0/601751646 shutdown_connections 2024-04-17T19:50:13.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.068+0000 7f6b3be00640 1 -- 172.21.15.29:0/601751646 wait complete. 2024-04-17T19:50:13.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.068+0000 7f6b3be00640 1 Processor -- start 2024-04-17T19:50:13.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.068+0000 7f6b3be00640 1 -- start start 2024-04-17T19:50:13.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3be00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b3c0c8050 0x7f6b3c135110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:13.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3be00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b3c0c8460 0x7f6b3c157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:13.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b3c158c70 0x7f6b3c1f5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:13.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3be00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c0c8f40 con 0x7f6b3c158c70 2024-04-17T19:50:13.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3be00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6b3c0c9a80 con 0x7f6b3c0c8050 2024-04-17T19:50:13.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3be00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6b3c142160 con 0x7f6b3c0c8460 2024-04-17T19:50:13.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b3c158c70 0x7f6b3c1f5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:13.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b3c158c70 0x7f6b3c1f5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46694/0 (socket says 172.21.15.29:46694) 2024-04-17T19:50:13.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3a000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b3c0c8460 0x7f6b3c157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:13.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3aa00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b3c0c8050 0x7f6b3c135110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:13.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3b400640 1 -- 172.21.15.29:0/1816794646 learned_addr learned my addr 172.21.15.29:0/1816794646 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:13.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3b400640 1 -- 172.21.15.29:0/1816794646 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b3c0c8460 msgr2=0x7f6b3c157bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:13.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3b400640 1 --2- 172.21.15.29:0/1816794646 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b3c0c8460 0x7f6b3c157bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:13.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3b400640 1 -- 172.21.15.29:0/1816794646 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b3c0c8050 msgr2=0x7f6b3c135110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:13.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3b400640 1 --2- 172.21.15.29:0/1816794646 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b3c0c8050 0x7f6b3c135110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:13.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3b400640 1 -- 172.21.15.29:0/1816794646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b3c135650 con 0x7f6b3c158c70 2024-04-17T19:50:13.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3b400640 1 --2- 172.21.15.29:0/1816794646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b3c158c70 0x7f6b3c1f5aa0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6b34009cb0 tx=0x7f6b3400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:13.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b32a00640 1 -- 172.21.15.29:0/1816794646 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34018020 con 0x7f6b3c158c70 2024-04-17T19:50:13.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b32a00640 1 -- 172.21.15.29:0/1816794646 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b34014480 con 0x7f6b3c158c70 2024-04-17T19:50:13.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3be00640 1 -- 172.21.15.29:0/1816794646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b3c1f68f0 con 0x7f6b3c158c70 2024-04-17T19:50:13.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b32a00640 1 -- 172.21.15.29:0/1816794646 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34014870 con 0x7f6b3c158c70 2024-04-17T19:50:13.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.072+0000 7f6b3be00640 1 -- 172.21.15.29:0/1816794646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b3c134630 con 0x7f6b3c158c70 2024-04-17T19:50:13.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.076+0000 7f6b3be00640 1 -- 172.21.15.29:0/1816794646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b3c1f6660 con 0x7f6b3c158c70 2024-04-17T19:50:13.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.076+0000 7f6b32a00640 1 -- 172.21.15.29:0/1816794646 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6b3401b020 con 0x7f6b3c158c70 2024-04-17T19:50:13.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.076+0000 7f6b32a00640 1 --2- 172.21.15.29:0/1816794646 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6b1c068380 0x7f6b1c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:13.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.076+0000 7f6b32a00640 1 -- 172.21.15.29:0/1816794646 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f6b34087a40 con 0x7f6b3c158c70 2024-04-17T19:50:13.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.076+0000 7f6b3aa00640 1 --2- 172.21.15.29:0/1816794646 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6b1c068380 0x7f6b1c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:13.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.080+0000 7f6b3aa00640 1 --2- 172.21.15.29:0/1816794646 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6b1c068380 0x7f6b1c06a890 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c0af000 tx=0x7f6b2c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:13.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.080+0000 7f6b32a00640 1 -- 172.21.15.29:0/1816794646 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6b34022030 con 0x7f6b3c158c70 2024-04-17T19:50:13.272 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.280+0000 7f6b3be00640 1 -- 172.21.15.29:0/1816794646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b3c04ff10 con 0x7f6b3c158c70 2024-04-17T19:50:13.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.284+0000 7f6b32a00640 1 -- 172.21.15.29:0/1816794646 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6902 (secure 0 0 0) 0x7f6b34010040 con 0x7f6b3c158c70 2024-04-17T19:50:13.273 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:50:13.274 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:48:34.874038+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4226},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4226":{"gid":4226,"name":"b","rank":4,"incarnation":18,"state":"up:active","state_seq":14,"addr":"172.21.15.37:6838/332907250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":332907250},{"type":"v1","addr":"172.21.15.37:6838","nonce":332907250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4226,4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:50:13.274 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 19 2024-04-17T19:50:13.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.284+0000 7f6b3be00640 1 -- 172.21.15.29:0/1816794646 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6b1c068380 msgr2=0x7f6b1c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:13.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.284+0000 7f6b3be00640 1 --2- 172.21.15.29:0/1816794646 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6b1c068380 0x7f6b1c06a890 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c0af000 tx=0x7f6b2c0a7000 comp rx=0 tx=0).stop 2024-04-17T19:50:13.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.284+0000 7f6b3be00640 1 -- 172.21.15.29:0/1816794646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b3c158c70 msgr2=0x7f6b3c1f5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:13.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.284+0000 7f6b3be00640 1 --2- 172.21.15.29:0/1816794646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b3c158c70 0x7f6b3c1f5aa0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f6b34009cb0 tx=0x7f6b3400c490 comp rx=0 tx=0).stop 2024-04-17T19:50:13.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.284+0000 7f6b3be00640 1 -- 172.21.15.29:0/1816794646 shutdown_connections 2024-04-17T19:50:13.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.284+0000 7f6b3be00640 1 --2- 172.21.15.29:0/1816794646 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6b1c068380 0x7f6b1c06a890 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:13.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.284+0000 7f6b3be00640 1 --2- 172.21.15.29:0/1816794646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b3c158c70 0x7f6b3c1f5aa0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:13.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.284+0000 7f6b3be00640 1 --2- 172.21.15.29:0/1816794646 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6b3c0c8460 0x7f6b3c157bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:13.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.284+0000 7f6b3be00640 1 --2- 172.21.15.29:0/1816794646 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6b3c0c8050 0x7f6b3c135110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:13.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.284+0000 7f6b3be00640 1 -- 172.21.15.29:0/1816794646 >> 172.21.15.29:0/1816794646 conn(0x7f6b3c0cee40 msgr2=0x7f6b3c0c5550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:13.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.284+0000 7f6b3be00640 1 -- 172.21.15.29:0/1816794646 shutdown_connections 2024-04-17T19:50:13.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:13.284+0000 7f6b3be00640 1 -- 172.21.15.29:0/1816794646 wait complete. 2024-04-17T19:50:13.296 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-17T19:50:15.073 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 63% (7791/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-17T19:50:15.297 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:50:15.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.460+0000 7ffa44000640 1 Processor -- start 2024-04-17T19:50:15.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.460+0000 7ffa44000640 1 -- start start 2024-04-17T19:50:15.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.460+0000 7ffa44000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ffa3c141f50 0x7ffa3c0cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:15.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.460+0000 7ffa44000640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa3c15c2e0 con 0x7ffa3c15caa0 2024-04-17T19:50:15.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.460+0000 7ffa44000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa3c142360 con 0x7ffa3c15ceb0 2024-04-17T19:50:15.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.460+0000 7ffa44000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ffa3c15c610 con 0x7ffa3c141f50 2024-04-17T19:50:15.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.460+0000 7ffa40c00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ffa3c15caa0 0x7ffa3c0ca240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40336/0 (socket says 172.21.15.29:40336) 2024-04-17T19:50:15.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.460+0000 7ffa40c00640 1 -- 172.21.15.29:0/3532220139 learned_addr learned my addr 172.21.15.29:0/3532220139 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:50:15.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.460+0000 7ffa41600640 1 --2- 172.21.15.29:0/3532220139 >> v2:172.21.15.37:3300/0 conn(0x7ffa3c141f50 0x7ffa3c0cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:15.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.464+0000 7ffa3be00640 1 -- 172.21.15.29:0/3532220139 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1476332832 0 0) 0x7ffa3c15c2e0 con 0x7ffa3c15caa0 2024-04-17T19:50:15.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.464+0000 7ffa3be00640 1 -- 172.21.15.29:0/3532220139 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa24005740 con 0x7ffa3c15caa0 2024-04-17T19:50:15.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.464+0000 7ffa3be00640 1 -- 172.21.15.29:0/3532220139 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 762193963 0 0) 0x7ffa3c142360 con 0x7ffa3c15ceb0 2024-04-17T19:50:15.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.464+0000 7ffa3be00640 1 -- 172.21.15.29:0/3532220139 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa3c15c2e0 con 0x7ffa3c15ceb0 2024-04-17T19:50:15.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.464+0000 7ffa41600640 1 -- 172.21.15.29:0/3532220139 >> v1:172.21.15.37:6789/0 conn(0x7ffa3c15ceb0 legacy=0x7ffa3c1636c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:15.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.464+0000 7ffa3be00640 1 -- 172.21.15.29:0/3532220139 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1161726760 0 0) 0x7ffa24005740 con 0x7ffa3c15caa0 2024-04-17T19:50:15.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.464+0000 7ffa41600640 1 -- 172.21.15.29:0/3532220139 >> v1:172.21.15.29:6789/0 conn(0x7ffa3c15caa0 legacy=0x7ffa3c0ca240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:15.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.464+0000 7ffa41600640 1 -- 172.21.15.29:0/3532220139 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa3c0cb140 con 0x7ffa3c141f50 2024-04-17T19:50:15.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.464+0000 7ffa41600640 1 --2- 172.21.15.29:0/3532220139 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa3c141f50 0x7ffa3c0cc8c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c15c790 tx=0x7ffa3001c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1aaaa34e86e4bd32 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:15.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.464+0000 7ffa3be00640 1 -- 172.21.15.29:0/3532220139 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa300b0070 con 0x7ffa3c141f50 2024-04-17T19:50:15.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.468+0000 7ffa3be00640 1 -- 172.21.15.29:0/3532220139 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa300ab400 con 0x7ffa3c141f50 2024-04-17T19:50:15.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.468+0000 7ffa3be00640 1 -- 172.21.15.29:0/3532220139 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa280016a0 con 0x7ffa3c141f50 2024-04-17T19:50:15.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.468+0000 7ffa44000640 1 -- 172.21.15.29:0/3532220139 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa3c141f50 msgr2=0x7ffa3c0cc8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:15.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.468+0000 7ffa44000640 1 --2- 172.21.15.29:0/3532220139 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa3c141f50 0x7ffa3c0cc8c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ffa3c15c790 tx=0x7ffa3001c3c0 comp rx=0 tx=0).stop 2024-04-17T19:50:15.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.468+0000 7ffa44000640 1 -- 172.21.15.29:0/3532220139 shutdown_connections 2024-04-17T19:50:15.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.468+0000 7ffa44000640 1 --2- 172.21.15.29:0/3532220139 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa3c141f50 0x7ffa3c0cc8c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:15.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.468+0000 7ffa44000640 1 -- 172.21.15.29:0/3532220139 >> 172.21.15.29:0/3532220139 conn(0x7ffa3c06fde0 msgr2=0x7ffa3c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:15.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.468+0000 7ffa44000640 1 -- 172.21.15.29:0/3532220139 shutdown_connections 2024-04-17T19:50:15.458 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.468+0000 7ffa44000640 1 -- 172.21.15.29:0/3532220139 wait complete. 2024-04-17T19:50:15.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.468+0000 7ffa44000640 1 Processor -- start 2024-04-17T19:50:15.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.468+0000 7ffa44000640 1 -- start start 2024-04-17T19:50:15.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa44000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3c141f50 0x7ffa3c1ebd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:15.462 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa44000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa3c15caa0 0x7ffa3c072e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa44000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa3c15ceb0 0x7ffa3c13b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa44000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c0c8110 con 0x7ffa3c141f50 2024-04-17T19:50:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa44000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ffa3c159b60 con 0x7ffa3c15caa0 2024-04-17T19:50:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa44000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ffa3c0ca950 con 0x7ffa3c15ceb0 2024-04-17T19:50:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa41600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3c141f50 0x7ffa3c1ebd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa41600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3c141f50 0x7ffa3c1ebd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46706/0 (socket says 172.21.15.29:46706) 2024-04-17T19:50:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa41600640 1 -- 172.21.15.29:0/2369093005 learned_addr learned my addr 172.21.15.29:0/2369093005 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa40c00640 1 --2- 172.21.15.29:0/2369093005 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa3c15caa0 0x7ffa3c072e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa42000640 1 --2- 172.21.15.29:0/2369093005 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa3c15ceb0 0x7ffa3c13b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa41600640 1 -- 172.21.15.29:0/2369093005 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa3c15ceb0 msgr2=0x7ffa3c13b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa41600640 1 --2- 172.21.15.29:0/2369093005 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa3c15ceb0 0x7ffa3c13b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa41600640 1 -- 172.21.15.29:0/2369093005 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa3c15caa0 msgr2=0x7ffa3c072e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa41600640 1 --2- 172.21.15.29:0/2369093005 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa3c15caa0 0x7ffa3c072e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa41600640 1 -- 172.21.15.29:0/2369093005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa3c1ec280 con 0x7ffa3c141f50 2024-04-17T19:50:15.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa41600640 1 --2- 172.21.15.29:0/2369093005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3c141f50 0x7ffa3c1ebd40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ffa30018fd0 tx=0x7ffa3001ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:15.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa39600640 1 -- 172.21.15.29:0/2369093005 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa300b0050 con 0x7ffa3c141f50 2024-04-17T19:50:15.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa39600640 1 -- 172.21.15.29:0/2369093005 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa300bc070 con 0x7ffa3c141f50 2024-04-17T19:50:15.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa44000640 1 -- 172.21.15.29:0/2369093005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa3c15a030 con 0x7ffa3c141f50 2024-04-17T19:50:15.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa39600640 1 -- 172.21.15.29:0/2369093005 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa30006e20 con 0x7ffa3c141f50 2024-04-17T19:50:15.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa44000640 1 -- 172.21.15.29:0/2369093005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa3c15a5b0 con 0x7ffa3c141f50 2024-04-17T19:50:15.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.472+0000 7ffa44000640 1 -- 172.21.15.29:0/2369093005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa3c069b90 con 0x7ffa3c141f50 2024-04-17T19:50:15.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.476+0000 7ffa39600640 1 -- 172.21.15.29:0/2369093005 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ffa300b7850 con 0x7ffa3c141f50 2024-04-17T19:50:15.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.476+0000 7ffa39600640 1 --2- 172.21.15.29:0/2369093005 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ffa1c068380 0x7ffa1c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:15.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.476+0000 7ffa39600640 1 -- 172.21.15.29:0/2369093005 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7ffa301213b0 con 0x7ffa3c141f50 2024-04-17T19:50:15.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.476+0000 7ffa40c00640 1 --2- 172.21.15.29:0/2369093005 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ffa1c068380 0x7ffa1c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:15.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.476+0000 7ffa40c00640 1 --2- 172.21.15.29:0/2369093005 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ffa1c068380 0x7ffa1c06a890 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ffa28005890 tx=0x7ffa280016c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:15.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.484+0000 7ffa39600640 1 -- 172.21.15.29:0/2369093005 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ffa300c6020 con 0x7ffa3c141f50 2024-04-17T19:50:15.618 INFO:tasks.workunit.client.0.smithi029.stderr:Note: switching to 'f3a98d3f110319c567ec86bf70955c141495ccf6'. 2024-04-17T19:50:15.618 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-17T19:50:15.618 INFO:tasks.workunit.client.0.smithi029.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-17T19:50:15.618 INFO:tasks.workunit.client.0.smithi029.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-17T19:50:15.618 INFO:tasks.workunit.client.0.smithi029.stderr:state without impacting any branches by switching back to a branch. 2024-04-17T19:50:15.618 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-17T19:50:15.618 INFO:tasks.workunit.client.0.smithi029.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-17T19:50:15.618 INFO:tasks.workunit.client.0.smithi029.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-17T19:50:15.619 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-17T19:50:15.619 INFO:tasks.workunit.client.0.smithi029.stderr: git switch -c 2024-04-17T19:50:15.619 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-17T19:50:15.619 INFO:tasks.workunit.client.0.smithi029.stderr:Or undo this operation with: 2024-04-17T19:50:15.619 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-17T19:50:15.619 INFO:tasks.workunit.client.0.smithi029.stderr: git switch - 2024-04-17T19:50:15.619 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-17T19:50:15.619 INFO:tasks.workunit.client.0.smithi029.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-17T19:50:15.619 INFO:tasks.workunit.client.0.smithi029.stderr: 2024-04-17T19:50:15.619 INFO:tasks.workunit.client.0.smithi029.stderr:HEAD is now at f3a98d3f110 Merge PR #54067 into wip-pdonnell-testing-20240417.021458-debug 2024-04-17T19:50:15.631 DEBUG:teuthology.orchestra.run.smithi029:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-17T19:50:15.640 INFO:tasks.workunit.client.0.smithi029.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-17T19:50:15.643 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-17T19:50:15.643 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-17T19:50:15.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.684+0000 7ffa44000640 1 -- 172.21.15.29:0/2369093005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa3c0c7e80 con 0x7ffa3c141f50 2024-04-17T19:50:15.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.684+0000 7ffa39600640 1 -- 172.21.15.29:0/2369093005 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6077 (secure 0 0 0) 0x7ffa30015fb0 con 0x7ffa3c141f50 2024-04-17T19:50:15.676 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:50:15.676 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:13.820171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:50:15.676 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 20 2024-04-17T19:50:15.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.688+0000 7ffa44000640 1 -- 172.21.15.29:0/2369093005 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ffa1c068380 msgr2=0x7ffa1c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:15.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.688+0000 7ffa44000640 1 --2- 172.21.15.29:0/2369093005 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ffa1c068380 0x7ffa1c06a890 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ffa28005890 tx=0x7ffa280016c0 comp rx=0 tx=0).stop 2024-04-17T19:50:15.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.688+0000 7ffa44000640 1 -- 172.21.15.29:0/2369093005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3c141f50 msgr2=0x7ffa3c1ebd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:15.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.688+0000 7ffa44000640 1 --2- 172.21.15.29:0/2369093005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3c141f50 0x7ffa3c1ebd40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ffa30018fd0 tx=0x7ffa3001ac00 comp rx=0 tx=0).stop 2024-04-17T19:50:15.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.688+0000 7ffa44000640 1 -- 172.21.15.29:0/2369093005 shutdown_connections 2024-04-17T19:50:15.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.688+0000 7ffa44000640 1 --2- 172.21.15.29:0/2369093005 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ffa1c068380 0x7ffa1c06a890 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:15.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.688+0000 7ffa44000640 1 --2- 172.21.15.29:0/2369093005 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ffa3c15ceb0 0x7ffa3c13b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:15.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.688+0000 7ffa44000640 1 --2- 172.21.15.29:0/2369093005 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ffa3c15caa0 0x7ffa3c072e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:15.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.688+0000 7ffa44000640 1 --2- 172.21.15.29:0/2369093005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffa3c141f50 0x7ffa3c1ebd40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:15.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.688+0000 7ffa44000640 1 -- 172.21.15.29:0/2369093005 >> 172.21.15.29:0/2369093005 conn(0x7ffa3c06fde0 msgr2=0x7ffa3c0c7270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:15.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.688+0000 7ffa44000640 1 -- 172.21.15.29:0/2369093005 shutdown_connections 2024-04-17T19:50:15.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.688+0000 7ffa44000640 1 -- 172.21.15.29:0/2369093005 wait complete. 2024-04-17T19:50:15.698 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2024-04-17T19:50:15.698 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-17T19:50:15.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:50:15.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.904+0000 7f4d4ca00640 1 Processor -- start 2024-04-17T19:50:15.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.904+0000 7f4d4ca00640 1 -- start start 2024-04-17T19:50:15.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.908+0000 7f4d4ca00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d48159540 0x7f4d480cb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.908+0000 7f4d4ca00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4d480cc300 0x7f4d480cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.908+0000 7f4d4ca00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d4806cab0 con 0x7f4d4815c6f0 2024-04-17T19:50:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.908+0000 7f4d4ca00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f4d4806c860 con 0x7f4d480cc300 2024-04-17T19:50:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.908+0000 7f4d4ca00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4d4806cd90 con 0x7f4d48159540 2024-04-17T19:50:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.908+0000 7f4d45400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4d480cc300 0x7f4d480cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.908+0000 7f4d45400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4d480cc300 0x7f4d480cc710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46720/0 (socket says 172.21.15.29:46720) 2024-04-17T19:50:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.908+0000 7f4d45400640 1 -- 172.21.15.29:0/1385189141 learned_addr learned my addr 172.21.15.29:0/1385189141 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:15.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.908+0000 7f4d45e00640 1 --2- 172.21.15.29:0/1385189141 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d48159540 0x7f4d480cb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.908+0000 7f4d45400640 1 -- 172.21.15.29:0/1385189141 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d48159540 msgr2=0x7f4d480cb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.908+0000 7f4d45400640 1 --2- 172.21.15.29:0/1385189141 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d48159540 0x7f4d480cb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.908+0000 7f4d45400640 1 -- 172.21.15.29:0/1385189141 >> v1:172.21.15.37:6789/0 conn(0x7f4d4815c6f0 legacy=0x7f4d4815f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.908+0000 7f4d45400640 1 -- 172.21.15.29:0/1385189141 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d4806d070 con 0x7f4d480cc300 2024-04-17T19:50:15.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.908+0000 7f4d45e00640 1 --2- 172.21.15.29:0/1385189141 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d48159540 0x7f4d480cb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:50:15.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.912+0000 7f4d45400640 1 --2- 172.21.15.29:0/1385189141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d480cc300 0x7f4d480cc710 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4d34009870 tx=0x7f4d3401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af104941f04b44f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:15.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.912+0000 7f4d44a00640 1 -- 172.21.15.29:0/1385189141 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d340b5070 con 0x7f4d480cc300 2024-04-17T19:50:15.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.912+0000 7f4d44a00640 1 -- 172.21.15.29:0/1385189141 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d340a8a50 con 0x7f4d480cc300 2024-04-17T19:50:15.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.912+0000 7f4d44a00640 1 -- 172.21.15.29:0/1385189141 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d340a8e20 con 0x7f4d480cc300 2024-04-17T19:50:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.912+0000 7f4d4ca00640 1 -- 172.21.15.29:0/1385189141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d480cc300 msgr2=0x7f4d480cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.912+0000 7f4d4ca00640 1 --2- 172.21.15.29:0/1385189141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d480cc300 0x7f4d480cc710 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4d34009870 tx=0x7f4d3401ed50 comp rx=0 tx=0).stop 2024-04-17T19:50:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.912+0000 7f4d4ca00640 1 -- 172.21.15.29:0/1385189141 shutdown_connections 2024-04-17T19:50:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.912+0000 7f4d4ca00640 1 --2- 172.21.15.29:0/1385189141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d480cc300 0x7f4d480cc710 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.912+0000 7f4d4ca00640 1 --2- 172.21.15.29:0/1385189141 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d48159540 0x7f4d480cb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.912+0000 7f4d4ca00640 1 -- 172.21.15.29:0/1385189141 >> 172.21.15.29:0/1385189141 conn(0x7f4d4806e1a0 msgr2=0x7f4d4806e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.912+0000 7f4d4ca00640 1 -- 172.21.15.29:0/1385189141 shutdown_connections 2024-04-17T19:50:15.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.912+0000 7f4d4ca00640 1 -- 172.21.15.29:0/1385189141 wait complete. 2024-04-17T19:50:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.912+0000 7f4d4ca00640 1 Processor -- start 2024-04-17T19:50:15.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.912+0000 7f4d4ca00640 1 -- start start 2024-04-17T19:50:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d4ca00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d480cc300 0x7f4d480b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d4ca00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d48159540 0x7f4d480b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d4ca00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d4815c6f0 0x7f4d480c50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d4ca00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4806d630 con 0x7f4d4815c6f0 2024-04-17T19:50:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d4ca00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4815c1d0 con 0x7f4d480cc300 2024-04-17T19:50:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d4ca00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4d48159950 con 0x7f4d48159540 2024-04-17T19:50:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d46800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d4815c6f0 0x7f4d480c50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d46800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d4815c6f0 0x7f4d480c50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46734/0 (socket says 172.21.15.29:46734) 2024-04-17T19:50:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d46800640 1 -- 172.21.15.29:0/3549220060 learned_addr learned my addr 172.21.15.29:0/3549220060 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:15.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d45400640 1 --2- 172.21.15.29:0/3549220060 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d48159540 0x7f4d480b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d45e00640 1 --2- 172.21.15.29:0/3549220060 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d480cc300 0x7f4d480b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d46800640 1 -- 172.21.15.29:0/3549220060 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d48159540 msgr2=0x7f4d480b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d46800640 1 --2- 172.21.15.29:0/3549220060 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d48159540 0x7f4d480b4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d46800640 1 -- 172.21.15.29:0/3549220060 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d480cc300 msgr2=0x7f4d480b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d46800640 1 --2- 172.21.15.29:0/3549220060 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d480cc300 0x7f4d480b0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d46800640 1 -- 172.21.15.29:0/3549220060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d480b1070 con 0x7f4d4815c6f0 2024-04-17T19:50:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d45e00640 1 --2- 172.21.15.29:0/3549220060 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d480cc300 0x7f4d480b0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:50:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d46800640 1 --2- 172.21.15.29:0/3549220060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d4815c6f0 0x7f4d480c50f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000c760 tx=0x7f4d4000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d3e000640 1 -- 172.21.15.29:0/3549220060 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d40003b80 con 0x7f4d4815c6f0 2024-04-17T19:50:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d3e000640 1 -- 172.21.15.29:0/3549220060 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d40003d20 con 0x7f4d4815c6f0 2024-04-17T19:50:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d3e000640 1 -- 172.21.15.29:0/3549220060 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4000aa90 con 0x7f4d4815c6f0 2024-04-17T19:50:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d4ca00640 1 -- 172.21.15.29:0/3549220060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d481f2740 con 0x7f4d4815c6f0 2024-04-17T19:50:15.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d4ca00640 1 -- 172.21.15.29:0/3549220060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d481f2d00 con 0x7f4d4815c6f0 2024-04-17T19:50:15.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.916+0000 7f4d4ca00640 1 -- 172.21.15.29:0/3549220060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d48069b90 con 0x7f4d4815c6f0 2024-04-17T19:50:15.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.920+0000 7f4d3e000640 1 -- 172.21.15.29:0/3549220060 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4d4000ac30 con 0x7f4d4815c6f0 2024-04-17T19:50:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.920+0000 7f4d3e000640 1 --2- 172.21.15.29:0/3549220060 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4d280683b0 0x7f4d2806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.920+0000 7f4d3e000640 1 -- 172.21.15.29:0/3549220060 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f4d40086c60 con 0x7f4d4815c6f0 2024-04-17T19:50:15.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.920+0000 7f4d45e00640 1 --2- 172.21.15.29:0/3549220060 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4d280683b0 0x7f4d2806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:15.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.920+0000 7f4d45e00640 1 --2- 172.21.15.29:0/3549220060 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4d280683b0 0x7f4d2806a8c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4d380079d0 tx=0x7f4d38008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:15.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:15.924+0000 7f4d3e000640 1 -- 172.21.15.29:0/3549220060 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4d40056d20 con 0x7f4d4815c6f0 2024-04-17T19:50:16.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:16.124+0000 7f4d4ca00640 1 -- 172.21.15.29:0/3549220060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d480cad20 con 0x7f4d4815c6f0 2024-04-17T19:50:16.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:16.124+0000 7f4d3e000640 1 -- 172.21.15.29:0/3549220060 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6077 (secure 0 0 0) 0x7f4d40056620 con 0x7f4d4815c6f0 2024-04-17T19:50:16.114 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:50:16.114 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:13.820171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:50:16.114 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 20 2024-04-17T19:50:16.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:16.124+0000 7f4d4ca00640 1 -- 172.21.15.29:0/3549220060 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4d280683b0 msgr2=0x7f4d2806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:16.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:16.124+0000 7f4d4ca00640 1 --2- 172.21.15.29:0/3549220060 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4d280683b0 0x7f4d2806a8c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4d380079d0 tx=0x7f4d38008040 comp rx=0 tx=0).stop 2024-04-17T19:50:16.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:16.124+0000 7f4d4ca00640 1 -- 172.21.15.29:0/3549220060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d4815c6f0 msgr2=0x7f4d480c50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:16.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:16.124+0000 7f4d4ca00640 1 --2- 172.21.15.29:0/3549220060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d4815c6f0 0x7f4d480c50f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000c760 tx=0x7f4d4000cc20 comp rx=0 tx=0).stop 2024-04-17T19:50:16.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:16.124+0000 7f4d4ca00640 1 -- 172.21.15.29:0/3549220060 shutdown_connections 2024-04-17T19:50:16.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:16.124+0000 7f4d4ca00640 1 --2- 172.21.15.29:0/3549220060 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4d280683b0 0x7f4d2806a8c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:16.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:16.124+0000 7f4d4ca00640 1 --2- 172.21.15.29:0/3549220060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4d4815c6f0 0x7f4d480c50f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:16.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:16.124+0000 7f4d4ca00640 1 --2- 172.21.15.29:0/3549220060 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4d48159540 0x7f4d480b4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:16.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:16.124+0000 7f4d4ca00640 1 --2- 172.21.15.29:0/3549220060 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4d480cc300 0x7f4d480b0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:16.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:16.124+0000 7f4d4ca00640 1 -- 172.21.15.29:0/3549220060 >> 172.21.15.29:0/3549220060 conn(0x7f4d4806e1a0 msgr2=0x7f4d48156bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:16.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:16.124+0000 7f4d4ca00640 1 -- 172.21.15.29:0/3549220060 shutdown_connections 2024-04-17T19:50:16.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:16.124+0000 7f4d4ca00640 1 -- 172.21.15.29:0/3549220060 wait complete. 2024-04-17T19:50:16.136 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 20, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-17T19:47:49.315515+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 20, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4223": { "addr": "172.21.15.37:6837/274798483", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6834", "nonce": 274798483, "type": "v2" }, { "addr": "172.21.15.37:6837", "nonce": 274798483, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4223, "incarnation": 20, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:replay", "state_seq": 1 }, "gid_4229": { "addr": "172.21.15.37:6839/1100598562", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6836", "nonce": 1100598562, "type": "v2" }, { "addr": "172.21.15.37:6839", "nonce": 1100598562, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4229, "incarnation": 16, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 13 }, "gid_4259": { "addr": "172.21.15.29:6836/4184160618", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6832", "nonce": 4184160618, "type": "v2" }, { "addr": "172.21.15.29:6836", "nonce": 4184160618, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4259, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 11 }, "gid_4302": { "addr": "172.21.15.29:6835/270402476", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6834", "nonce": 270402476, "type": "v2" }, { "addr": "172.21.15.29:6835", "nonce": 270402476, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 10, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 9 }, "gid_4305": { "addr": "172.21.15.29:6837/2691590158", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6833", "nonce": 2691590158, "type": "v2" }, { "addr": "172.21.15.29:6837", "nonce": 2691590158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4305, "incarnation": 5, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-17T19:50:13.820171+0000", "qdb_cluster": [ 4229, 4259, 4302, 4305 ], "qdb_leader": 4305, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4305, "mds_1": 4302, "mds_2": 4259, "mds_3": 4229, "mds_4": 4223 } } } ], "standbys": [] } 2024-04-17T19:50:16.734 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-17T19:50:16.872 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-17T19:50:16.929 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-17T19:50:16.931 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-17T19:50:16.931 INFO:tasks.workunit.client.0.smithi029.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-17T19:50:16.986 INFO:tasks.workunit.client.0.smithi029.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-17T19:50:16.991 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-04-17T19:50:16.992 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-17T19:50:17.000 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2024-04-17T19:50:17.001 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2024-04-17T19:50:17.001 DEBUG:teuthology.orchestra.run.smithi029:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f3a98d3f110319c567ec86bf70955c141495ccf6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2024-04-17T19:50:18.137 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:50:18.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.312+0000 7fd01b800640 1 Processor -- start 2024-04-17T19:50:18.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.312+0000 7fd01b800640 1 -- start start 2024-04-17T19:50:18.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.312+0000 7fd01b800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd0141596e0 0x7fd01415ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:18.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.312+0000 7fd01b800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd01415c050 con 0x7fd0140b1e60 2024-04-17T19:50:18.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.312+0000 7fd01b800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0140cafb0 con 0x7fd01415c510 2024-04-17T19:50:18.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.312+0000 7fd01b800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd0140cb550 con 0x7fd0141596e0 2024-04-17T19:50:18.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.312+0000 7fd018e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd0140b1e60 0x7fd0140cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40338/0 (socket says 172.21.15.29:40338) 2024-04-17T19:50:18.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.312+0000 7fd018e00640 1 -- 172.21.15.29:0/201751667 learned_addr learned my addr 172.21.15.29:0/201751667 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:50:18.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.312+0000 7fd013e00640 1 --2- 172.21.15.29:0/201751667 >> v2:172.21.15.37:3300/0 conn(0x7fd0141596e0 0x7fd01415ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:18.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.312+0000 7fd013400640 1 -- 172.21.15.29:0/201751667 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 333806822 0 0) 0x7fd008001120 con 0x7fd0140b1e60 2024-04-17T19:50:18.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.312+0000 7fd013400640 1 -- 172.21.15.29:0/201751667 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcffc003530 con 0x7fd0140b1e60 2024-04-17T19:50:18.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd013400640 1 -- 172.21.15.29:0/201751667 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2843895875 0 0) 0x7fcffc003530 con 0x7fd0140b1e60 2024-04-17T19:50:18.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd013400640 1 -- 172.21.15.29:0/201751667 >> v2:172.21.15.37:3300/0 conn(0x7fd0141596e0 msgr2=0x7fd01415ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:18.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd013400640 1 --2- 172.21.15.29:0/201751667 >> v2:172.21.15.37:3300/0 conn(0x7fd0141596e0 0x7fd01415ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:18.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd013400640 1 -- 172.21.15.29:0/201751667 >> v1:172.21.15.37:6789/0 conn(0x7fd01415c510 legacy=0x7fd01415c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:18.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd013400640 1 -- 172.21.15.29:0/201751667 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0140cb270 con 0x7fd0140b1e60 2024-04-17T19:50:18.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd013400640 1 -- 172.21.15.29:0/201751667 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fd008001850 con 0x7fd0140b1e60 2024-04-17T19:50:18.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd013400640 1 -- 172.21.15.29:0/201751667 >> v1:172.21.15.29:6789/0 conn(0x7fd0140b1e60 legacy=0x7fd0140cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:18.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd013400640 1 --2- 172.21.15.29:0/201751667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcffc003a30 0x7fcffc003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd013400640 1 --2- 172.21.15.29:0/201751667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcffc004810 0x7fcffc004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd013400640 1 --2- 172.21.15.29:0/201751667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcffc005280 0x7fcffc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd013400640 1 -- 172.21.15.29:0/201751667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd0140cb550 con 0x7fcffc003a30 2024-04-17T19:50:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd013400640 1 -- 172.21.15.29:0/201751667 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcffc00c1a0 con 0x7fcffc005280 2024-04-17T19:50:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd013400640 1 -- 172.21.15.29:0/201751667 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcffc00c3a0 con 0x7fcffc004810 2024-04-17T19:50:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd018e00640 1 --2- 172.21.15.29:0/201751667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcffc003a30 0x7fcffc003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd013e00640 1 --2- 172.21.15.29:0/201751667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcffc004810 0x7fcffc004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd019800640 1 --2- 172.21.15.29:0/201751667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcffc005280 0x7fcffc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd018e00640 1 -- 172.21.15.29:0/201751667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcffc004810 msgr2=0x7fcffc004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd018e00640 1 --2- 172.21.15.29:0/201751667 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcffc004810 0x7fcffc004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd018e00640 1 -- 172.21.15.29:0/201751667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcffc005280 msgr2=0x7fcffc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd018e00640 1 --2- 172.21.15.29:0/201751667 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcffc005280 0x7fcffc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd018e00640 1 -- 172.21.15.29:0/201751667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcffc003530 con 0x7fcffc003a30 2024-04-17T19:50:18.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd019800640 1 -- 172.21.15.29:0/201751667 reap_dead start 2024-04-17T19:50:18.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.316+0000 7fd018e00640 1 --2- 172.21.15.29:0/201751667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcffc003a30 0x7fcffc003e60 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd00800c320 tx=0x7fd00801cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=561981a9f0bad80e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:18.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.320+0000 7fd013400640 1 -- 172.21.15.29:0/201751667 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd008007e40 con 0x7fcffc003a30 2024-04-17T19:50:18.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.320+0000 7fd013400640 1 -- 172.21.15.29:0/201751667 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0080a9070 con 0x7fcffc003a30 2024-04-17T19:50:18.309 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.320+0000 7fd013400640 1 -- 172.21.15.29:0/201751667 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd008005d60 con 0x7fcffc003a30 2024-04-17T19:50:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.320+0000 7fd01b800640 1 -- 172.21.15.29:0/201751667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcffc003a30 msgr2=0x7fcffc003e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.320+0000 7fd01b800640 1 --2- 172.21.15.29:0/201751667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcffc003a30 0x7fcffc003e60 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd00800c320 tx=0x7fd00801cbc0 comp rx=0 tx=0).stop 2024-04-17T19:50:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.320+0000 7fd01b800640 1 -- 172.21.15.29:0/201751667 shutdown_connections 2024-04-17T19:50:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.320+0000 7fd01b800640 1 --2- 172.21.15.29:0/201751667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcffc003a30 0x7fcffc003e60 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.320+0000 7fd01b800640 1 -- 172.21.15.29:0/201751667 >> 172.21.15.29:0/201751667 conn(0x7fd01406e3d0 msgr2=0x7fd01406e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.320+0000 7fd01b800640 1 -- 172.21.15.29:0/201751667 shutdown_connections 2024-04-17T19:50:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.320+0000 7fd01b800640 1 -- 172.21.15.29:0/201751667 wait complete. 2024-04-17T19:50:18.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.320+0000 7fd01b800640 1 Processor -- start 2024-04-17T19:50:18.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.320+0000 7fd01b800640 1 -- start start 2024-04-17T19:50:18.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd01b800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcffc003a30 0x7fd0140b43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:18.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd01b800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0141f9970 0x7fd0140b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:18.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd01b800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0140b0a10 0x7fd014151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:18.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd01b800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd01415cd30 con 0x7fd0141f9970 2024-04-17T19:50:18.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd01b800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd0140cccc0 con 0x7fcffc003a30 2024-04-17T19:50:18.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd01b800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd01415ceb0 con 0x7fd0140b0a10 2024-04-17T19:50:18.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd013e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0141f9970 0x7fd0140b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:18.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd013e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0141f9970 0x7fd0140b4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46756/0 (socket says 172.21.15.29:46756) 2024-04-17T19:50:18.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd013e00640 1 -- 172.21.15.29:0/3894006592 learned_addr learned my addr 172.21.15.29:0/3894006592 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:18.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd018e00640 1 --2- 172.21.15.29:0/3894006592 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcffc003a30 0x7fd0140b43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:18.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd019800640 1 --2- 172.21.15.29:0/3894006592 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0140b0a10 0x7fd014151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:18.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd013e00640 1 -- 172.21.15.29:0/3894006592 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0140b0a10 msgr2=0x7fd014151e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:18.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd013e00640 1 --2- 172.21.15.29:0/3894006592 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0140b0a10 0x7fd014151e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:18.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd013e00640 1 -- 172.21.15.29:0/3894006592 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcffc003a30 msgr2=0x7fd0140b43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:18.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd013e00640 1 --2- 172.21.15.29:0/3894006592 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcffc003a30 0x7fd0140b43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:18.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd013e00640 1 -- 172.21.15.29:0/3894006592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0140cf4d0 con 0x7fd0141f9970 2024-04-17T19:50:18.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd013e00640 1 --2- 172.21.15.29:0/3894006592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0141f9970 0x7fd0140b4f50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd000011650 tx=0x7fd000011b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:18.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd010c00640 1 -- 172.21.15.29:0/3894006592 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00000f9c0 con 0x7fd0141f9970 2024-04-17T19:50:18.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd010c00640 1 -- 172.21.15.29:0/3894006592 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd00000fb60 con 0x7fd0141f9970 2024-04-17T19:50:18.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd01b800640 1 -- 172.21.15.29:0/3894006592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0140cf770 con 0x7fd0141f9970 2024-04-17T19:50:18.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd010c00640 1 -- 172.21.15.29:0/3894006592 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd000015460 con 0x7fd0141f9970 2024-04-17T19:50:18.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd01b800640 1 -- 172.21.15.29:0/3894006592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0140cfd30 con 0x7fd0141f9970 2024-04-17T19:50:18.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.324+0000 7fd01b800640 1 -- 172.21.15.29:0/3894006592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd014069b90 con 0x7fd0141f9970 2024-04-17T19:50:18.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.328+0000 7fd010c00640 1 -- 172.21.15.29:0/3894006592 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd000015600 con 0x7fd0141f9970 2024-04-17T19:50:18.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.328+0000 7fd010c00640 1 --2- 172.21.15.29:0/3894006592 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcff4068380 0x7fcff406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:18.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.328+0000 7fd010c00640 1 -- 172.21.15.29:0/3894006592 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7fd0000878a0 con 0x7fd0141f9970 2024-04-17T19:50:18.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.328+0000 7fd018e00640 1 --2- 172.21.15.29:0/3894006592 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcff4068380 0x7fcff406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:18.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.328+0000 7fd018e00640 1 --2- 172.21.15.29:0/3894006592 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcff4068380 0x7fcff406a890 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd0080ab000 tx=0x7fd0080b0040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:18.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.332+0000 7fd010c00640 1 -- 172.21.15.29:0/3894006592 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd000019030 con 0x7fd0141f9970 2024-04-17T19:50:18.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.532+0000 7fd01b800640 1 -- 172.21.15.29:0/3894006592 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd014154870 con 0x7fd0141f9970 2024-04-17T19:50:18.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.532+0000 7fd010c00640 1 -- 172.21.15.29:0/3894006592 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+6077 (secure 0 0 0) 0x7fd000057960 con 0x7fd0141f9970 2024-04-17T19:50:18.523 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:50:18.523 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:13.820171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:50:18.523 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 20 2024-04-17T19:50:18.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.532+0000 7fd01b800640 1 -- 172.21.15.29:0/3894006592 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcff4068380 msgr2=0x7fcff406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:18.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.532+0000 7fd01b800640 1 --2- 172.21.15.29:0/3894006592 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcff4068380 0x7fcff406a890 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd0080ab000 tx=0x7fd0080b0040 comp rx=0 tx=0).stop 2024-04-17T19:50:18.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.532+0000 7fd01b800640 1 -- 172.21.15.29:0/3894006592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0141f9970 msgr2=0x7fd0140b4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:18.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.532+0000 7fd01b800640 1 --2- 172.21.15.29:0/3894006592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0141f9970 0x7fd0140b4f50 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd000011650 tx=0x7fd000011b10 comp rx=0 tx=0).stop 2024-04-17T19:50:18.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.532+0000 7fd01b800640 1 -- 172.21.15.29:0/3894006592 shutdown_connections 2024-04-17T19:50:18.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.532+0000 7fd01b800640 1 --2- 172.21.15.29:0/3894006592 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcff4068380 0x7fcff406a890 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:18.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.532+0000 7fd01b800640 1 --2- 172.21.15.29:0/3894006592 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0140b0a10 0x7fd014151e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:18.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.532+0000 7fd01b800640 1 --2- 172.21.15.29:0/3894006592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0141f9970 0x7fd0140b4f50 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:18.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.532+0000 7fd01b800640 1 --2- 172.21.15.29:0/3894006592 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcffc003a30 0x7fd0140b43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:18.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.532+0000 7fd01b800640 1 -- 172.21.15.29:0/3894006592 >> 172.21.15.29:0/3894006592 conn(0x7fd01406e3d0 msgr2=0x7fd01415a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:18.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.532+0000 7fd01b800640 1 -- 172.21.15.29:0/3894006592 shutdown_connections 2024-04-17T19:50:18.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:18.532+0000 7fd01b800640 1 -- 172.21.15.29:0/3894006592 wait complete. 2024-04-17T19:50:18.528 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:50:18.532 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:50:18.546 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:50:20.547 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:50:20.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.724+0000 7f35d6400640 1 Processor -- start 2024-04-17T19:50:20.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.724+0000 7f35d6400640 1 -- start start 2024-04-17T19:50:20.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.724+0000 7f35d6400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35d00cc8c0 0x7f35d00cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:20.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.724+0000 7f35d6400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f35d01597a0 0x7f35d01635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:20.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.724+0000 7f35d6400640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35d00c88b0 con 0x7f35d0159390 2024-04-17T19:50:20.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.724+0000 7f35d6400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f35d00c6ce0 con 0x7f35d01597a0 2024-04-17T19:50:20.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.724+0000 7f35d6400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f35d00c6fc0 con 0x7f35d00cc8c0 2024-04-17T19:50:20.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.724+0000 7f35c7e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f35d0159390 0x7f35d00ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56884/0 (socket says 172.21.15.29:56884) 2024-04-17T19:50:20.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.724+0000 7f35c7e00640 1 -- 172.21.15.29:0/3624714221 learned_addr learned my addr 172.21.15.29:0/3624714221 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:50:20.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.724+0000 7f35cf400640 1 --2- 172.21.15.29:0/3624714221 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35d00cc8c0 0x7f35d00cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:20.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.724+0000 7f35cfe00640 1 --2- 172.21.15.29:0/3624714221 >> v2:172.21.15.37:3300/0 conn(0x7f35d01597a0 0x7f35d01635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:20.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.724+0000 7f35cea00640 1 -- 172.21.15.29:0/3624714221 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3693710635 0 0) 0x7f35d00c88b0 con 0x7f35d0159390 2024-04-17T19:50:20.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.724+0000 7f35cea00640 1 -- 172.21.15.29:0/3624714221 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35b4003530 con 0x7f35d0159390 2024-04-17T19:50:20.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.724+0000 7f35cf400640 1 -- 172.21.15.29:0/3624714221 >> v2:172.21.15.37:3300/0 conn(0x7f35d01597a0 msgr2=0x7f35d01635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:20.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.724+0000 7f35cf400640 1 --2- 172.21.15.29:0/3624714221 >> v2:172.21.15.37:3300/0 conn(0x7f35d01597a0 0x7f35d01635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:20.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.724+0000 7f35cf400640 1 -- 172.21.15.29:0/3624714221 >> v1:172.21.15.29:6789/0 conn(0x7f35d0159390 legacy=0x7f35d00ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:20.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.724+0000 7f35cea00640 1 -- 172.21.15.29:0/3624714221 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2929591847 0 0) 0x7f35b4003530 con 0x7f35d0159390 2024-04-17T19:50:20.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.724+0000 7f35cf400640 1 -- 172.21.15.29:0/3624714221 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35d00c72a0 con 0x7f35d00cc8c0 2024-04-17T19:50:20.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.728+0000 7f35cf400640 1 --2- 172.21.15.29:0/3624714221 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35d00cc8c0 0x7f35d00cccd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f35c000ba60 tx=0x7f35c001ea90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23edfad02a239847 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:20.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.728+0000 7f35cea00640 1 -- 172.21.15.29:0/3624714221 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c00b4070 con 0x7f35d00cc8c0 2024-04-17T19:50:20.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.728+0000 7f35cea00640 1 -- 172.21.15.29:0/3624714221 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35c00a75a0 con 0x7f35d00cc8c0 2024-04-17T19:50:20.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.728+0000 7f35cea00640 1 -- 172.21.15.29:0/3624714221 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b8001680 con 0x7f35d00cc8c0 2024-04-17T19:50:20.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.728+0000 7f35d6400640 1 -- 172.21.15.29:0/3624714221 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35d00cc8c0 msgr2=0x7f35d00cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:20.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.728+0000 7f35d6400640 1 --2- 172.21.15.29:0/3624714221 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35d00cc8c0 0x7f35d00cccd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f35c000ba60 tx=0x7f35c001ea90 comp rx=0 tx=0).stop 2024-04-17T19:50:20.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.728+0000 7f35d6400640 1 -- 172.21.15.29:0/3624714221 shutdown_connections 2024-04-17T19:50:20.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.732+0000 7f35d6400640 1 --2- 172.21.15.29:0/3624714221 >> v2:172.21.15.37:3300/0 conn(0x7f35d01597a0 0x7f35d01635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:20.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.732+0000 7f35d6400640 1 --2- 172.21.15.29:0/3624714221 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35d00cc8c0 0x7f35d00cccd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:20.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.732+0000 7f35d6400640 1 -- 172.21.15.29:0/3624714221 >> 172.21.15.29:0/3624714221 conn(0x7f35d006fde0 msgr2=0x7f35d00701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:20.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.732+0000 7f35d6400640 1 -- 172.21.15.29:0/3624714221 shutdown_connections 2024-04-17T19:50:20.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.732+0000 7f35d6400640 1 -- 172.21.15.29:0/3624714221 wait complete. 2024-04-17T19:50:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.732+0000 7f35d6400640 1 Processor -- start 2024-04-17T19:50:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.732+0000 7f35d6400640 1 -- start start 2024-04-17T19:50:20.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.732+0000 7f35d6400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35d00cc8c0 0x7f35d015cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:20.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.732+0000 7f35d6400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35d0159390 0x7f35d015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:20.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.732+0000 7f35d6400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35d01597a0 0x7f35d00cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:20.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.732+0000 7f35d6400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f35d00cabc0 con 0x7f35d01597a0 2024-04-17T19:50:20.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.732+0000 7f35d6400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f35d00caa40 con 0x7f35d00cc8c0 2024-04-17T19:50:20.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.732+0000 7f35d6400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f35d00c6880 con 0x7f35d0159390 2024-04-17T19:50:20.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.736+0000 7f35cfe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35d01597a0 0x7f35d00cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:20.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.736+0000 7f35cfe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35d01597a0 0x7f35d00cd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38408/0 (socket says 172.21.15.29:38408) 2024-04-17T19:50:20.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.736+0000 7f35cfe00640 1 -- 172.21.15.29:0/3723605059 learned_addr learned my addr 172.21.15.29:0/3723605059 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:20.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.736+0000 7f35c7e00640 1 --2- 172.21.15.29:0/3723605059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35d0159390 0x7f35d015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:20.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.736+0000 7f35cf400640 1 --2- 172.21.15.29:0/3723605059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35d00cc8c0 0x7f35d015cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:20.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.736+0000 7f35cfe00640 1 -- 172.21.15.29:0/3723605059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35d0159390 msgr2=0x7f35d015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:20.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.736+0000 7f35cfe00640 1 --2- 172.21.15.29:0/3723605059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35d0159390 0x7f35d015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:20.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.736+0000 7f35cfe00640 1 -- 172.21.15.29:0/3723605059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35d00cc8c0 msgr2=0x7f35d015cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:20.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.736+0000 7f35cfe00640 1 --2- 172.21.15.29:0/3723605059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35d00cc8c0 0x7f35d015cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:20.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.736+0000 7f35cfe00640 1 -- 172.21.15.29:0/3723605059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35d013ec40 con 0x7f35d01597a0 2024-04-17T19:50:20.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.736+0000 7f35cfe00640 1 --2- 172.21.15.29:0/3723605059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35d01597a0 0x7f35d00cd130 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f35c800efc0 tx=0x7f35c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:20.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.736+0000 7f35c7400640 1 -- 172.21.15.29:0/3723605059 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c8018020 con 0x7f35d01597a0 2024-04-17T19:50:20.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.736+0000 7f35c7400640 1 -- 172.21.15.29:0/3723605059 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35c8014480 con 0x7f35d01597a0 2024-04-17T19:50:20.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.736+0000 7f35d6400640 1 -- 172.21.15.29:0/3723605059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35d015b2e0 con 0x7f35d01597a0 2024-04-17T19:50:20.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.736+0000 7f35c7400640 1 -- 172.21.15.29:0/3723605059 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c8014870 con 0x7f35d01597a0 2024-04-17T19:50:20.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.736+0000 7f35d6400640 1 -- 172.21.15.29:0/3723605059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35d01f58a0 con 0x7f35d01597a0 2024-04-17T19:50:20.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.736+0000 7f35d6400640 1 -- 172.21.15.29:0/3723605059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35d01f53b0 con 0x7f35d01597a0 2024-04-17T19:50:20.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.736+0000 7f35c7400640 1 -- 172.21.15.29:0/3723605059 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f35c801b020 con 0x7f35d01597a0 2024-04-17T19:50:20.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.740+0000 7f35c7400640 1 --2- 172.21.15.29:0/3723605059 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f35ac068380 0x7f35ac06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:20.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.740+0000 7f35c7400640 1 -- 172.21.15.29:0/3723605059 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f35c8087aa0 con 0x7f35d01597a0 2024-04-17T19:50:20.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.740+0000 7f35cf400640 1 --2- 172.21.15.29:0/3723605059 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f35ac068380 0x7f35ac06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:20.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.740+0000 7f35cf400640 1 --2- 172.21.15.29:0/3723605059 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f35ac068380 0x7f35ac06a890 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f35c00b1000 tx=0x7f35c00a9070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:20.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.744+0000 7f35c7400640 1 -- 172.21.15.29:0/3723605059 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f35c8022030 con 0x7f35d01597a0 2024-04-17T19:50:20.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.948+0000 7f35d6400640 1 -- 172.21.15.29:0/3723605059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35d015b070 con 0x7f35d01597a0 2024-04-17T19:50:20.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.948+0000 7f35c7400640 1 -- 172.21.15.29:0/3723605059 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6079 (secure 0 0 0) 0x7f35c8010040 con 0x7f35d01597a0 2024-04-17T19:50:20.939 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:50:20.939 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:18.836074+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:resolve","state_seq":41,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229]},"id":1}]} 2024-04-17T19:50:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 21 2024-04-17T19:50:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.948+0000 7f35d6400640 1 -- 172.21.15.29:0/3723605059 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f35ac068380 msgr2=0x7f35ac06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.948+0000 7f35d6400640 1 --2- 172.21.15.29:0/3723605059 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f35ac068380 0x7f35ac06a890 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f35c00b1000 tx=0x7f35c00a9070 comp rx=0 tx=0).stop 2024-04-17T19:50:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.948+0000 7f35d6400640 1 -- 172.21.15.29:0/3723605059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35d01597a0 msgr2=0x7f35d00cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.948+0000 7f35d6400640 1 --2- 172.21.15.29:0/3723605059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35d01597a0 0x7f35d00cd130 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f35c800efc0 tx=0x7f35c800c490 comp rx=0 tx=0).stop 2024-04-17T19:50:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.948+0000 7f35d6400640 1 -- 172.21.15.29:0/3723605059 shutdown_connections 2024-04-17T19:50:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.948+0000 7f35d6400640 1 --2- 172.21.15.29:0/3723605059 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f35ac068380 0x7f35ac06a890 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.948+0000 7f35d6400640 1 --2- 172.21.15.29:0/3723605059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35d01597a0 0x7f35d00cd130 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.948+0000 7f35d6400640 1 --2- 172.21.15.29:0/3723605059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f35d0159390 0x7f35d015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.948+0000 7f35d6400640 1 --2- 172.21.15.29:0/3723605059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f35d00cc8c0 0x7f35d015cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.948+0000 7f35d6400640 1 -- 172.21.15.29:0/3723605059 >> 172.21.15.29:0/3723605059 conn(0x7f35d006fde0 msgr2=0x7f35d01577e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.948+0000 7f35d6400640 1 -- 172.21.15.29:0/3723605059 shutdown_connections 2024-04-17T19:50:20.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:20.948+0000 7f35d6400640 1 -- 172.21.15.29:0/3723605059 wait complete. 2024-04-17T19:50:20.962 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:50:22.963 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:50:23.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.136+0000 7f6436000640 1 Processor -- start 2024-04-17T19:50:23.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.136+0000 7f6436000640 1 -- start start 2024-04-17T19:50:23.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.136+0000 7f6436000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f64300c7f50 0x7f64300ca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:23.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.136+0000 7f6436000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6430155da0 0x7f64301561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:23.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.136+0000 7f6436000640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64301566f0 con 0x7f64300cbf90 2024-04-17T19:50:23.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.136+0000 7f6436000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f643006bdc0 con 0x7f64300c7f50 2024-04-17T19:50:23.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.136+0000 7f6436000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f64300cbba0 con 0x7f6430155da0 2024-04-17T19:50:23.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.136+0000 7f642f400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f64300cbf90 0x7f643015ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56900/0 (socket says 172.21.15.29:56900) 2024-04-17T19:50:23.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.136+0000 7f642f400640 1 -- 172.21.15.29:0/2429591251 learned_addr learned my addr 172.21.15.29:0/2429591251 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:50:23.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642fe00640 1 --2- 172.21.15.29:0/2429591251 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6430155da0 0x7f64301561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:23.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f6427e00640 1 --2- 172.21.15.29:0/2429591251 >> v2:172.21.15.37:3300/0 conn(0x7f64300c7f50 0x7f64300ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:23.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642ea00640 1 -- 172.21.15.29:0/2429591251 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 899473149 0 0) 0x7f64301566f0 con 0x7f64300cbf90 2024-04-17T19:50:23.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642ea00640 1 -- 172.21.15.29:0/2429591251 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6414003530 con 0x7f64300cbf90 2024-04-17T19:50:23.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642ea00640 1 -- 172.21.15.29:0/2429591251 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3458443886 0 0) 0x7f6414003530 con 0x7f64300cbf90 2024-04-17T19:50:23.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642ea00640 1 -- 172.21.15.29:0/2429591251 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6430155da0 msgr2=0x7f64301561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:23.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642ea00640 1 --2- 172.21.15.29:0/2429591251 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6430155da0 0x7f64301561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:23.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642ea00640 1 -- 172.21.15.29:0/2429591251 >> v2:172.21.15.37:3300/0 conn(0x7f64300c7f50 msgr2=0x7f64300ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:23.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642ea00640 1 --2- 172.21.15.29:0/2429591251 >> v2:172.21.15.37:3300/0 conn(0x7f64300c7f50 0x7f64300ca410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:23.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642ea00640 1 -- 172.21.15.29:0/2429591251 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f643006c010 con 0x7f64300cbf90 2024-04-17T19:50:23.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642ea00640 1 -- 172.21.15.29:0/2429591251 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f6420001680 con 0x7f64300cbf90 2024-04-17T19:50:23.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642ea00640 1 -- 172.21.15.29:0/2429591251 >> v1:172.21.15.29:6789/0 conn(0x7f64300cbf90 legacy=0x7f643015ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:23.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642ea00640 1 --2- 172.21.15.29:0/2429591251 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6414003a30 0x7f6414003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:23.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642ea00640 1 --2- 172.21.15.29:0/2429591251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6414004810 0x7f6414004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:23.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642ea00640 1 --2- 172.21.15.29:0/2429591251 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6430155da0 0x7f6414005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:23.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642ea00640 1 -- 172.21.15.29:0/2429591251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f643006bdc0 con 0x7f6414004810 2024-04-17T19:50:23.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642ea00640 1 -- 172.21.15.29:0/2429591251 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f64300cbba0 con 0x7f6414003a30 2024-04-17T19:50:23.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642ea00640 1 -- 172.21.15.29:0/2429591251 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f641400bab0 con 0x7f6430155da0 2024-04-17T19:50:23.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f6427e00640 1 --2- 172.21.15.29:0/2429591251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6414004810 0x7f6414004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:23.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642f400640 1 --2- 172.21.15.29:0/2429591251 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6414003a30 0x7f6414003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:23.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642fe00640 1 --2- 172.21.15.29:0/2429591251 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6430155da0 0x7f6414005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:23.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f6427e00640 1 -- 172.21.15.29:0/2429591251 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6430155da0 msgr2=0x7f6414005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:23.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f6427e00640 1 --2- 172.21.15.29:0/2429591251 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6430155da0 0x7f6414005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:23.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f6427e00640 1 -- 172.21.15.29:0/2429591251 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6414003a30 msgr2=0x7f6414003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:23.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f6427e00640 1 --2- 172.21.15.29:0/2429591251 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6414003a30 0x7f6414003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:23.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f6427e00640 1 -- 172.21.15.29:0/2429591251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6414003530 con 0x7f6414004810 2024-04-17T19:50:23.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.140+0000 7f642fe00640 1 --2- 172.21.15.29:0/2429591251 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6430155da0 0x7f6414005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:50:23.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.144+0000 7f6427e00640 1 --2- 172.21.15.29:0/2429591251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6414004810 0x7f6414004c20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6418009ae0 tx=0x7f641801cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a48923e2bd03fcef server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:23.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.144+0000 7f642ea00640 1 -- 172.21.15.29:0/2429591251 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64180b2070 con 0x7f6414004810 2024-04-17T19:50:23.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.144+0000 7f642ea00640 1 -- 172.21.15.29:0/2429591251 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64180a56a0 con 0x7f6414004810 2024-04-17T19:50:23.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.144+0000 7f642ea00640 1 -- 172.21.15.29:0/2429591251 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64180096b0 con 0x7f6414004810 2024-04-17T19:50:23.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.144+0000 7f6436000640 1 -- 172.21.15.29:0/2429591251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6414004810 msgr2=0x7f6414004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:23.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.144+0000 7f6436000640 1 --2- 172.21.15.29:0/2429591251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6414004810 0x7f6414004c20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6418009ae0 tx=0x7f641801cab0 comp rx=0 tx=0).stop 2024-04-17T19:50:23.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.144+0000 7f642fe00640 1 -- 172.21.15.29:0/2429591251 reap_dead start 2024-04-17T19:50:23.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.144+0000 7f6436000640 1 -- 172.21.15.29:0/2429591251 shutdown_connections 2024-04-17T19:50:23.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.144+0000 7f6436000640 1 -- 172.21.15.29:0/2429591251 >> 172.21.15.29:0/2429591251 conn(0x7f643006fde0 msgr2=0x7f64300701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:23.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.144+0000 7f6436000640 1 -- 172.21.15.29:0/2429591251 shutdown_connections 2024-04-17T19:50:23.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.144+0000 7f6436000640 1 -- 172.21.15.29:0/2429591251 wait complete. 2024-04-17T19:50:23.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.144+0000 7f6436000640 1 Processor -- start 2024-04-17T19:50:23.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.144+0000 7f6436000640 1 -- start start 2024-04-17T19:50:23.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6436000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6414004810 0x7f6430158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:23.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6436000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f643015d1a0 0x7f643015d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:23.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6436000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6430156b90 0x7f6430156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:23.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6436000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f64300c8360 con 0x7f643015d1a0 2024-04-17T19:50:23.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6436000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f643006c5d0 con 0x7f6414004810 2024-04-17T19:50:23.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6436000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f64300c84e0 con 0x7f6430156b90 2024-04-17T19:50:23.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6427e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f643015d1a0 0x7f643015d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:23.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6427e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f643015d1a0 0x7f643015d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38422/0 (socket says 172.21.15.29:38422) 2024-04-17T19:50:23.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6427e00640 1 -- 172.21.15.29:0/1031773367 learned_addr learned my addr 172.21.15.29:0/1031773367 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:23.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f642f400640 1 --2- 172.21.15.29:0/1031773367 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6414004810 0x7f6430158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:23.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f642fe00640 1 --2- 172.21.15.29:0/1031773367 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6430156b90 0x7f6430156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:23.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6427e00640 1 -- 172.21.15.29:0/1031773367 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6430156b90 msgr2=0x7f6430156fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:23.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6427e00640 1 --2- 172.21.15.29:0/1031773367 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6430156b90 0x7f6430156fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:23.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6427e00640 1 -- 172.21.15.29:0/1031773367 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6414004810 msgr2=0x7f6430158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:23.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6427e00640 1 --2- 172.21.15.29:0/1031773367 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6414004810 0x7f6430158a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:23.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6427e00640 1 -- 172.21.15.29:0/1031773367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64301553e0 con 0x7f643015d1a0 2024-04-17T19:50:23.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6427e00640 1 --2- 172.21.15.29:0/1031773367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f643015d1a0 0x7f643015d5b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6418009ae0 tx=0x7f6414005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:23.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6427400640 1 -- 172.21.15.29:0/1031773367 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64180b2040 con 0x7f643015d1a0 2024-04-17T19:50:23.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6427400640 1 -- 172.21.15.29:0/1031773367 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64180a5df0 con 0x7f643015d1a0 2024-04-17T19:50:23.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6436000640 1 -- 172.21.15.29:0/1031773367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6430158f50 con 0x7f643015d1a0 2024-04-17T19:50:23.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6427400640 1 -- 172.21.15.29:0/1031773367 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64180096b0 con 0x7f643015d1a0 2024-04-17T19:50:23.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6436000640 1 -- 172.21.15.29:0/1031773367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f643015f990 con 0x7f643015d1a0 2024-04-17T19:50:23.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.148+0000 7f6436000640 1 -- 172.21.15.29:0/1031773367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6430069b90 con 0x7f643015d1a0 2024-04-17T19:50:23.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.152+0000 7f6427400640 1 -- 172.21.15.29:0/1031773367 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f64180c0030 con 0x7f643015d1a0 2024-04-17T19:50:23.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.152+0000 7f6427400640 1 --2- 172.21.15.29:0/1031773367 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f640c0683b0 0x7f640c06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:23.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.152+0000 7f6427400640 1 -- 172.21.15.29:0/1031773367 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f6418120890 con 0x7f643015d1a0 2024-04-17T19:50:23.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.152+0000 7f642f400640 1 --2- 172.21.15.29:0/1031773367 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f640c0683b0 0x7f640c06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:23.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.152+0000 7f642f400640 1 --2- 172.21.15.29:0/1031773367 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f640c0683b0 0x7f640c06a8c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6420000c10 tx=0x7f6420001e10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:23.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.156+0000 7f6427400640 1 -- 172.21.15.29:0/1031773367 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f64180167c0 con 0x7f643015d1a0 2024-04-17T19:50:23.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.352+0000 7f6436000640 1 -- 172.21.15.29:0/1031773367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f643015fcf0 con 0x7f643015d1a0 2024-04-17T19:50:23.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.356+0000 7f6427400640 1 -- 172.21.15.29:0/1031773367 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+6079 (secure 0 0 0) 0x7f64180f09e0 con 0x7f643015d1a0 2024-04-17T19:50:23.345 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:50:23.346 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:18.836074+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:resolve","state_seq":41,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229]},"id":1}]} 2024-04-17T19:50:23.346 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 21 2024-04-17T19:50:23.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.356+0000 7f6436000640 1 -- 172.21.15.29:0/1031773367 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f640c0683b0 msgr2=0x7f640c06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:23.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.356+0000 7f6436000640 1 --2- 172.21.15.29:0/1031773367 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f640c0683b0 0x7f640c06a8c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6420000c10 tx=0x7f6420001e10 comp rx=0 tx=0).stop 2024-04-17T19:50:23.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.356+0000 7f6436000640 1 -- 172.21.15.29:0/1031773367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f643015d1a0 msgr2=0x7f643015d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:23.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.356+0000 7f6436000640 1 --2- 172.21.15.29:0/1031773367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f643015d1a0 0x7f643015d5b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6418009ae0 tx=0x7f6414005840 comp rx=0 tx=0).stop 2024-04-17T19:50:23.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.356+0000 7f6436000640 1 -- 172.21.15.29:0/1031773367 shutdown_connections 2024-04-17T19:50:23.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.356+0000 7f6436000640 1 --2- 172.21.15.29:0/1031773367 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f640c0683b0 0x7f640c06a8c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:23.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.356+0000 7f6436000640 1 --2- 172.21.15.29:0/1031773367 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6430156b90 0x7f6430156fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:23.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.356+0000 7f6436000640 1 --2- 172.21.15.29:0/1031773367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f643015d1a0 0x7f643015d5b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:23.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.356+0000 7f6436000640 1 --2- 172.21.15.29:0/1031773367 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6414004810 0x7f6430158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:23.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.356+0000 7f6436000640 1 -- 172.21.15.29:0/1031773367 >> 172.21.15.29:0/1031773367 conn(0x7f643006fde0 msgr2=0x7f64300755b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:23.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.356+0000 7f6436000640 1 -- 172.21.15.29:0/1031773367 shutdown_connections 2024-04-17T19:50:23.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:23.356+0000 7f6436000640 1 -- 172.21.15.29:0/1031773367 wait complete. 2024-04-17T19:50:23.367 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:50:25.368 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:50:25.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.536+0000 7f788c800640 1 Processor -- start 2024-04-17T19:50:25.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.536+0000 7f788c800640 1 -- start start 2024-04-17T19:50:25.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.536+0000 7f788c800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f78840cccd0 0x7f78840ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:25.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.536+0000 7f788c800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f78840ca870 0x7f78841635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:25.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.536+0000 7f788c800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78840cd0e0 con 0x7f78840cc8c0 2024-04-17T19:50:25.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.536+0000 7f788c800640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f78840c88b0 con 0x7f78840cccd0 2024-04-17T19:50:25.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.536+0000 7f788c800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f78840c6ce0 con 0x7f78840ca870 2024-04-17T19:50:25.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.540+0000 7f7889400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f78840cccd0 0x7f78840ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:25.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.540+0000 7f7889400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f78840cccd0 0x7f78840ca330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38430/0 (socket says 172.21.15.29:38430) 2024-04-17T19:50:25.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.540+0000 7f7889400640 1 -- 172.21.15.29:0/4038234069 learned_addr learned my addr 172.21.15.29:0/4038234069 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:25.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.540+0000 7f788a800640 1 --2- 172.21.15.29:0/4038234069 >> v2:172.21.15.37:3300/0 conn(0x7f78840ca870 0x7f78841635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:25.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.540+0000 7f7889400640 1 -- 172.21.15.29:0/4038234069 >> v2:172.21.15.37:3300/0 conn(0x7f78840ca870 msgr2=0x7f78841635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:25.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.540+0000 7f7889400640 1 --2- 172.21.15.29:0/4038234069 >> v2:172.21.15.37:3300/0 conn(0x7f78840ca870 0x7f78841635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:25.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.540+0000 7f7889400640 1 -- 172.21.15.29:0/4038234069 >> v1:172.21.15.37:6789/0 conn(0x7f78840cc8c0 legacy=0x7f7884159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:25.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.540+0000 7f7889400640 1 -- 172.21.15.29:0/4038234069 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78840c6fc0 con 0x7f78840cccd0 2024-04-17T19:50:25.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.540+0000 7f7889400640 1 --2- 172.21.15.29:0/4038234069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78840cccd0 0x7f78840ca330 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f78840722d0 tx=0x7f787401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c0e2097b6c446e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:25.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.540+0000 7f7888a00640 1 -- 172.21.15.29:0/4038234069 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78740b0070 con 0x7f78840cccd0 2024-04-17T19:50:25.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.540+0000 7f7888a00640 1 -- 172.21.15.29:0/4038234069 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78740ab600 con 0x7f78840cccd0 2024-04-17T19:50:25.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.540+0000 7f7888a00640 1 -- 172.21.15.29:0/4038234069 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78740ab9d0 con 0x7f78840cccd0 2024-04-17T19:50:25.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.544+0000 7f788c800640 1 -- 172.21.15.29:0/4038234069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78840cccd0 msgr2=0x7f78840ca330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:25.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.544+0000 7f788c800640 1 --2- 172.21.15.29:0/4038234069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78840cccd0 0x7f78840ca330 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f78840722d0 tx=0x7f787401c4e0 comp rx=0 tx=0).stop 2024-04-17T19:50:25.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.544+0000 7f788c800640 1 -- 172.21.15.29:0/4038234069 shutdown_connections 2024-04-17T19:50:25.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.544+0000 7f788c800640 1 --2- 172.21.15.29:0/4038234069 >> v2:172.21.15.37:3300/0 conn(0x7f78840ca870 0x7f78841635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:25.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.544+0000 7f788c800640 1 --2- 172.21.15.29:0/4038234069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78840cccd0 0x7f78840ca330 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:25.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.544+0000 7f788c800640 1 -- 172.21.15.29:0/4038234069 >> 172.21.15.29:0/4038234069 conn(0x7f788406fde0 msgr2=0x7f78840701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:25.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.544+0000 7f788c800640 1 -- 172.21.15.29:0/4038234069 shutdown_connections 2024-04-17T19:50:25.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.544+0000 7f788c800640 1 -- 172.21.15.29:0/4038234069 wait complete. 2024-04-17T19:50:25.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.544+0000 7f788c800640 1 Processor -- start 2024-04-17T19:50:25.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.544+0000 7f788c800640 1 -- start start 2024-04-17T19:50:25.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.544+0000 7f788c800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78840ca870 0x7f7884165eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:25.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.544+0000 7f788c800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78840cc8c0 0x7f7884073360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:25.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.544+0000 7f788c800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78840cccd0 0x7f78840738a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:25.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.544+0000 7f788c800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7884159c20 con 0x7f78840cccd0 2024-04-17T19:50:25.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.544+0000 7f788c800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7884159aa0 con 0x7f78840ca870 2024-04-17T19:50:25.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.544+0000 7f788c800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f78840c6880 con 0x7f78840cc8c0 2024-04-17T19:50:25.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.548+0000 7f788a800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78840cccd0 0x7f78840738a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:25.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.548+0000 7f788a800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78840cccd0 0x7f78840738a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38440/0 (socket says 172.21.15.29:38440) 2024-04-17T19:50:25.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.548+0000 7f788a800640 1 -- 172.21.15.29:0/3160291084 learned_addr learned my addr 172.21.15.29:0/3160291084 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:25.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.548+0000 7f7889400640 1 --2- 172.21.15.29:0/3160291084 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78840cc8c0 0x7f7884073360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:25.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.548+0000 7f7889e00640 1 --2- 172.21.15.29:0/3160291084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78840ca870 0x7f7884165eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:25.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.548+0000 7f788a800640 1 -- 172.21.15.29:0/3160291084 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78840cc8c0 msgr2=0x7f7884073360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:25.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.548+0000 7f788a800640 1 --2- 172.21.15.29:0/3160291084 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78840cc8c0 0x7f7884073360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:25.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.548+0000 7f788a800640 1 -- 172.21.15.29:0/3160291084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78840ca870 msgr2=0x7f7884165eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:25.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.548+0000 7f788a800640 1 --2- 172.21.15.29:0/3160291084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78840ca870 0x7f7884165eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:25.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.548+0000 7f788a800640 1 -- 172.21.15.29:0/3160291084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78841656f0 con 0x7f78840cccd0 2024-04-17T19:50:25.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.548+0000 7f788a800640 1 --2- 172.21.15.29:0/3160291084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78840cccd0 0x7f78840738a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7880009fd0 tx=0x7f788000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:25.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.548+0000 7f787e000640 1 -- 172.21.15.29:0/3160291084 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7880014420 con 0x7f78840cccd0 2024-04-17T19:50:25.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.548+0000 7f787e000640 1 -- 172.21.15.29:0/3160291084 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7880019070 con 0x7f78840cccd0 2024-04-17T19:50:25.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.548+0000 7f788c800640 1 -- 172.21.15.29:0/3160291084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7884074040 con 0x7f78840cccd0 2024-04-17T19:50:25.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.548+0000 7f787e000640 1 -- 172.21.15.29:0/3160291084 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7880014800 con 0x7f78840cccd0 2024-04-17T19:50:25.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.548+0000 7f788c800640 1 -- 172.21.15.29:0/3160291084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78841f8a10 con 0x7f78840cccd0 2024-04-17T19:50:25.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.548+0000 7f788c800640 1 -- 172.21.15.29:0/3160291084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7884069b90 con 0x7f78840cccd0 2024-04-17T19:50:25.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.548+0000 7f787e000640 1 -- 172.21.15.29:0/3160291084 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f78800149a0 con 0x7f78840cccd0 2024-04-17T19:50:25.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.552+0000 7f787e000640 1 --2- 172.21.15.29:0/3160291084 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7868068380 0x7f786806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:25.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.552+0000 7f787e000640 1 -- 172.21.15.29:0/3160291084 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f78800875d0 con 0x7f78840cccd0 2024-04-17T19:50:25.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.552+0000 7f7889e00640 1 --2- 172.21.15.29:0/3160291084 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7868068380 0x7f786806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:25.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.552+0000 7f7889e00640 1 --2- 172.21.15.29:0/3160291084 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7868068380 0x7f786806a890 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f78780098e0 tx=0x7f7878009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:25.546 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.556+0000 7f787e000640 1 -- 172.21.15.29:0/3160291084 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7880057040 con 0x7f78840cccd0 2024-04-17T19:50:25.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.756+0000 7f788c800640 1 -- 172.21.15.29:0/3160291084 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78840cb8d0 con 0x7f78840cccd0 2024-04-17T19:50:25.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.756+0000 7f787e000640 1 -- 172.21.15.29:0/3160291084 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6081 (secure 0 0 0) 0x7f788005b340 con 0x7f78840cccd0 2024-04-17T19:50:25.748 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:50:25.748 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:23.849622+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:reconnect","state_seq":43,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:50:25.748 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 22 2024-04-17T19:50:25.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.760+0000 7f788c800640 1 -- 172.21.15.29:0/3160291084 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7868068380 msgr2=0x7f786806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:25.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.760+0000 7f788c800640 1 --2- 172.21.15.29:0/3160291084 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7868068380 0x7f786806a890 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f78780098e0 tx=0x7f7878009060 comp rx=0 tx=0).stop 2024-04-17T19:50:25.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.760+0000 7f788c800640 1 -- 172.21.15.29:0/3160291084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78840cccd0 msgr2=0x7f78840738a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:25.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.760+0000 7f788c800640 1 --2- 172.21.15.29:0/3160291084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78840cccd0 0x7f78840738a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7880009fd0 tx=0x7f788000e4b0 comp rx=0 tx=0).stop 2024-04-17T19:50:25.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.760+0000 7f788c800640 1 -- 172.21.15.29:0/3160291084 shutdown_connections 2024-04-17T19:50:25.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.760+0000 7f788c800640 1 --2- 172.21.15.29:0/3160291084 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7868068380 0x7f786806a890 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:25.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.760+0000 7f788c800640 1 --2- 172.21.15.29:0/3160291084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78840cccd0 0x7f78840738a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:25.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.760+0000 7f788c800640 1 --2- 172.21.15.29:0/3160291084 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f78840cc8c0 0x7f7884073360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:25.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.760+0000 7f788c800640 1 --2- 172.21.15.29:0/3160291084 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f78840ca870 0x7f7884165eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:25.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.760+0000 7f788c800640 1 -- 172.21.15.29:0/3160291084 >> 172.21.15.29:0/3160291084 conn(0x7f788406fde0 msgr2=0x7f78840c5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:25.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.760+0000 7f788c800640 1 -- 172.21.15.29:0/3160291084 shutdown_connections 2024-04-17T19:50:25.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:25.760+0000 7f788c800640 1 -- 172.21.15.29:0/3160291084 wait complete. 2024-04-17T19:50:25.770 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:50:27.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:50:27.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.944+0000 7f36ba000640 1 Processor -- start 2024-04-17T19:50:27.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.944+0000 7f36ba000640 1 -- start start 2024-04-17T19:50:27.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.948+0000 7f36ba000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f36b4156210 0x7f36b4070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:27.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.948+0000 7f36ba000640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36b415c960 con 0x7f36b4160160 2024-04-17T19:50:27.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.948+0000 7f36ba000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36b415cc40 con 0x7f36b4155e00 2024-04-17T19:50:27.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.948+0000 7f36ba000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f36b40c9210 con 0x7f36b4156210 2024-04-17T19:50:27.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.948+0000 7f36b3e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f36b4156210 0x7f36b4070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:27.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.948+0000 7f36b3400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f36b4160160 0x7f36b4073c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56902/0 (socket says 172.21.15.29:56902) 2024-04-17T19:50:27.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.948+0000 7f36b3400640 1 -- 172.21.15.29:0/325263170 learned_addr learned my addr 172.21.15.29:0/325263170 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:50:27.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.948+0000 7f36b2a00640 1 -- 172.21.15.29:0/325263170 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2424474333 0 0) 0x7f36b415c960 con 0x7f36b4160160 2024-04-17T19:50:27.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.948+0000 7f36b2a00640 1 -- 172.21.15.29:0/325263170 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3698003530 con 0x7f36b4160160 2024-04-17T19:50:27.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.948+0000 7f36b3e00640 1 -- 172.21.15.29:0/325263170 >> v1:172.21.15.37:6789/0 conn(0x7f36b4155e00 legacy=0x7f36b415bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:27.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.948+0000 7f36b3e00640 1 -- 172.21.15.29:0/325263170 >> v1:172.21.15.29:6789/0 conn(0x7f36b4160160 legacy=0x7f36b4073c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:27.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.948+0000 7f36b3e00640 1 -- 172.21.15.29:0/325263170 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b415cf20 con 0x7f36b4156210 2024-04-17T19:50:27.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.952+0000 7f36b3e00640 1 --2- 172.21.15.29:0/325263170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b4156210 0x7f36b4070bc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f36ac009d90 tx=0x7f36ac01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17b6229f0e5db8ef server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:27.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.952+0000 7f36b2a00640 1 -- 172.21.15.29:0/325263170 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac0b2070 con 0x7f36b4156210 2024-04-17T19:50:27.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.952+0000 7f36b2a00640 1 -- 172.21.15.29:0/325263170 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36ac0a58b0 con 0x7f36b4156210 2024-04-17T19:50:27.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.952+0000 7f36b2a00640 1 -- 172.21.15.29:0/325263170 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac0a5c80 con 0x7f36b4156210 2024-04-17T19:50:27.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.952+0000 7f36ba000640 1 -- 172.21.15.29:0/325263170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b4156210 msgr2=0x7f36b4070bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:27.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.952+0000 7f36ba000640 1 --2- 172.21.15.29:0/325263170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b4156210 0x7f36b4070bc0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f36ac009d90 tx=0x7f36ac01c8e0 comp rx=0 tx=0).stop 2024-04-17T19:50:27.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.952+0000 7f36ba000640 1 -- 172.21.15.29:0/325263170 shutdown_connections 2024-04-17T19:50:27.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.952+0000 7f36ba000640 1 --2- 172.21.15.29:0/325263170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b4156210 0x7f36b4070bc0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:27.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.952+0000 7f36ba000640 1 -- 172.21.15.29:0/325263170 >> 172.21.15.29:0/325263170 conn(0x7f36b40ce5e0 msgr2=0x7f36b40ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:27.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.952+0000 7f36ba000640 1 -- 172.21.15.29:0/325263170 shutdown_connections 2024-04-17T19:50:27.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.952+0000 7f36ba000640 1 -- 172.21.15.29:0/325263170 wait complete. 2024-04-17T19:50:27.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36ba000640 1 Processor -- start 2024-04-17T19:50:27.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36ba000640 1 -- start start 2024-04-17T19:50:27.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36ba000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36b4155e00 0x7f36b4157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:27.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36ba000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b4156210 0x7f36b40cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:27.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36ba000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36b4160160 0x7f36b4158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:27.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36ba000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f36b415c700 con 0x7f36b4156210 2024-04-17T19:50:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36ba000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f36b4160570 con 0x7f36b4155e00 2024-04-17T19:50:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36ba000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f36b416a7c0 con 0x7f36b4160160 2024-04-17T19:50:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36abe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b4156210 0x7f36b40cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36abe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b4156210 0x7f36b40cd2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38456/0 (socket says 172.21.15.29:38456) 2024-04-17T19:50:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36abe00640 1 -- 172.21.15.29:0/3313908040 learned_addr learned my addr 172.21.15.29:0/3313908040 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36b3e00640 1 --2- 172.21.15.29:0/3313908040 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36b4160160 0x7f36b4158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36b3400640 1 --2- 172.21.15.29:0/3313908040 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36b4155e00 0x7f36b4157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36abe00640 1 -- 172.21.15.29:0/3313908040 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36b4160160 msgr2=0x7f36b4158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36abe00640 1 --2- 172.21.15.29:0/3313908040 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36b4160160 0x7f36b4158f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36abe00640 1 -- 172.21.15.29:0/3313908040 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36b4155e00 msgr2=0x7f36b4157d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36abe00640 1 --2- 172.21.15.29:0/3313908040 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36b4155e00 0x7f36b4157d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36abe00640 1 -- 172.21.15.29:0/3313908040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b41582b0 con 0x7f36b4156210 2024-04-17T19:50:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36b3e00640 1 --2- 172.21.15.29:0/3313908040 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36b4160160 0x7f36b4158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:50:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36abe00640 1 --2- 172.21.15.29:0/3313908040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b4156210 0x7f36b40cd2c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f369c00c370 tx=0x7f369c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36ab400640 1 -- 172.21.15.29:0/3313908040 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369c0052d0 con 0x7f36b4156210 2024-04-17T19:50:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36ab400640 1 -- 172.21.15.29:0/3313908040 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f369c005470 con 0x7f36b4156210 2024-04-17T19:50:27.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36ab400640 1 -- 172.21.15.29:0/3313908040 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369c007260 con 0x7f36b4156210 2024-04-17T19:50:27.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36ba000640 1 -- 172.21.15.29:0/3313908040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36b41f5930 con 0x7f36b4156210 2024-04-17T19:50:27.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.956+0000 7f36ba000640 1 -- 172.21.15.29:0/3313908040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b415bdb0 con 0x7f36b4156210 2024-04-17T19:50:27.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.960+0000 7f36ba000640 1 -- 172.21.15.29:0/3313908040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36b4069b90 con 0x7f36b4156210 2024-04-17T19:50:27.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.960+0000 7f36ab400640 1 -- 172.21.15.29:0/3313908040 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f369c007400 con 0x7f36b4156210 2024-04-17T19:50:27.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.960+0000 7f36ab400640 1 --2- 172.21.15.29:0/3313908040 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f36900683b0 0x7f369006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:27.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.960+0000 7f36ab400640 1 -- 172.21.15.29:0/3313908040 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f369c086320 con 0x7f36b4156210 2024-04-17T19:50:27.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.960+0000 7f36b3400640 1 --2- 172.21.15.29:0/3313908040 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f36900683b0 0x7f369006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:27.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.960+0000 7f36b3400640 1 --2- 172.21.15.29:0/3313908040 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f36900683b0 0x7f369006a8c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f36a40011f0 tx=0x7f36a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:27.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:27.964+0000 7f36ab400640 1 -- 172.21.15.29:0/3313908040 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f369c0563e0 con 0x7f36b4156210 2024-04-17T19:50:28.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:28.164+0000 7f36ba000640 1 -- 172.21.15.29:0/3313908040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36b4069560 con 0x7f36b4156210 2024-04-17T19:50:28.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:28.164+0000 7f36ab400640 1 -- 172.21.15.29:0/3313908040 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+6081 (secure 0 0 0) 0x7f369c055ce0 con 0x7f36b4156210 2024-04-17T19:50:28.156 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:50:28.156 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:23.849622+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:reconnect","state_seq":43,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:50:28.157 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 22 2024-04-17T19:50:28.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:28.168+0000 7f36ba000640 1 -- 172.21.15.29:0/3313908040 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f36900683b0 msgr2=0x7f369006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:28.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:28.168+0000 7f36ba000640 1 --2- 172.21.15.29:0/3313908040 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f36900683b0 0x7f369006a8c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f36a40011f0 tx=0x7f36a4009040 comp rx=0 tx=0).stop 2024-04-17T19:50:28.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:28.168+0000 7f36ba000640 1 -- 172.21.15.29:0/3313908040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b4156210 msgr2=0x7f36b40cd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:28.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:28.168+0000 7f36ba000640 1 --2- 172.21.15.29:0/3313908040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b4156210 0x7f36b40cd2c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f369c00c370 tx=0x7f369c00c830 comp rx=0 tx=0).stop 2024-04-17T19:50:28.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:28.168+0000 7f36ba000640 1 -- 172.21.15.29:0/3313908040 shutdown_connections 2024-04-17T19:50:28.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:28.168+0000 7f36ba000640 1 --2- 172.21.15.29:0/3313908040 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f36900683b0 0x7f369006a8c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:28.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:28.168+0000 7f36ba000640 1 --2- 172.21.15.29:0/3313908040 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36b4160160 0x7f36b4158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:28.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:28.168+0000 7f36ba000640 1 --2- 172.21.15.29:0/3313908040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36b4156210 0x7f36b40cd2c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:28.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:28.168+0000 7f36ba000640 1 --2- 172.21.15.29:0/3313908040 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36b4155e00 0x7f36b4157d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:28.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:28.168+0000 7f36ba000640 1 -- 172.21.15.29:0/3313908040 >> 172.21.15.29:0/3313908040 conn(0x7f36b40ce5e0 msgr2=0x7f36b4155020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:28.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:28.168+0000 7f36ba000640 1 -- 172.21.15.29:0/3313908040 shutdown_connections 2024-04-17T19:50:28.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:28.168+0000 7f36ba000640 1 -- 172.21.15.29:0/3313908040 wait complete. 2024-04-17T19:50:28.179 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 22, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-17T19:47:49.315515+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 22, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4223": { "addr": "172.21.15.37:6837/274798483", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6834", "nonce": 274798483, "type": "v2" }, { "addr": "172.21.15.37:6837", "nonce": 274798483, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4223, "incarnation": 20, "join_fscid": -1, "name": "d", "rank": 4, "state": "up:reconnect", "state_seq": 43 }, "gid_4229": { "addr": "172.21.15.37:6839/1100598562", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6836", "nonce": 1100598562, "type": "v2" }, { "addr": "172.21.15.37:6839", "nonce": 1100598562, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4229, "incarnation": 16, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 13 }, "gid_4259": { "addr": "172.21.15.29:6836/4184160618", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6832", "nonce": 4184160618, "type": "v2" }, { "addr": "172.21.15.29:6836", "nonce": 4184160618, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4259, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 11 }, "gid_4302": { "addr": "172.21.15.29:6835/270402476", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6834", "nonce": 270402476, "type": "v2" }, { "addr": "172.21.15.29:6835", "nonce": 270402476, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 10, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 9 }, "gid_4305": { "addr": "172.21.15.29:6837/2691590158", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6833", "nonce": 2691590158, "type": "v2" }, { "addr": "172.21.15.29:6837", "nonce": 2691590158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4305, "incarnation": 5, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-17T19:50:23.849622+0000", "qdb_cluster": [ 4229, 4259, 4302, 4305 ], "qdb_leader": 4305, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4305, "mds_1": 4302, "mds_2": 4259, "mds_3": 4229, "mds_4": 4223 } } } ], "standbys": [] } 2024-04-17T19:50:30.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:50:30.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.344+0000 7fd787a00640 1 Processor -- start 2024-04-17T19:50:30.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.344+0000 7fd787a00640 1 -- start start 2024-04-17T19:50:30.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.344+0000 7fd787a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7801531b0 0x7fd7800c8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:30.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.344+0000 7fd787a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7800c8f10 0x7fd7800c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:30.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.344+0000 7fd787a00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd7800c9f90 0x7fd7800c5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:30.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.344+0000 7fd787a00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fd780159cc0 con 0x7fd7800c8f10 2024-04-17T19:50:30.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.344+0000 7fd787a00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd7800718b0 con 0x7fd7800c9f90 2024-04-17T19:50:30.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.344+0000 7fd787a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd7800715d0 con 0x7fd7801531b0 2024-04-17T19:50:30.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.344+0000 7fd77fe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7800c8f10 0x7fd7800c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:30.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.344+0000 7fd77fe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7800c8f10 0x7fd7800c9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34892/0 (socket says 172.21.15.29:34892) 2024-04-17T19:50:30.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.344+0000 7fd77fe00640 1 -- 172.21.15.29:0/2635071991 learned_addr learned my addr 172.21.15.29:0/2635071991 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:30.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.344+0000 7fd785000640 1 --2- 172.21.15.29:0/2635071991 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7801531b0 0x7fd7800c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:30.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.344+0000 7fd785a00640 1 --2- 172.21.15.29:0/2635071991 >> v2:172.21.15.37:3300/0 conn(0x7fd7800c9f90 0x7fd7800c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:30.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.344+0000 7fd77fe00640 1 -- 172.21.15.29:0/2635071991 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7801531b0 msgr2=0x7fd7800c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:30.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.344+0000 7fd77fe00640 1 --2- 172.21.15.29:0/2635071991 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7801531b0 0x7fd7800c8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:30.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.344+0000 7fd77fe00640 1 -- 172.21.15.29:0/2635071991 >> v2:172.21.15.37:3300/0 conn(0x7fd7800c9f90 msgr2=0x7fd7800c5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:30.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.344+0000 7fd77fe00640 1 --2- 172.21.15.29:0/2635071991 >> v2:172.21.15.37:3300/0 conn(0x7fd7800c9f90 0x7fd7800c5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:30.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.344+0000 7fd77fe00640 1 -- 172.21.15.29:0/2635071991 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd780158f00 con 0x7fd7800c8f10 2024-04-17T19:50:30.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.348+0000 7fd77fe00640 1 --2- 172.21.15.29:0/2635071991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7800c8f10 0x7fd7800c9a50 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd78006fc40 tx=0x7fd76c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd480a229405a677 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:30.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.348+0000 7fd77f400640 1 -- 172.21.15.29:0/2635071991 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c0b5070 con 0x7fd7800c8f10 2024-04-17T19:50:30.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.348+0000 7fd77f400640 1 -- 172.21.15.29:0/2635071991 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd76c0a89d0 con 0x7fd7800c8f10 2024-04-17T19:50:30.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.348+0000 7fd77f400640 1 -- 172.21.15.29:0/2635071991 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76c0a8da0 con 0x7fd7800c8f10 2024-04-17T19:50:30.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.348+0000 7fd787a00640 1 -- 172.21.15.29:0/2635071991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7800c8f10 msgr2=0x7fd7800c9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:30.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.348+0000 7fd787a00640 1 --2- 172.21.15.29:0/2635071991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7800c8f10 0x7fd7800c9a50 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd78006fc40 tx=0x7fd76c01ed10 comp rx=0 tx=0).stop 2024-04-17T19:50:30.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.348+0000 7fd787a00640 1 -- 172.21.15.29:0/2635071991 shutdown_connections 2024-04-17T19:50:30.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.348+0000 7fd787a00640 1 --2- 172.21.15.29:0/2635071991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7800c8f10 0x7fd7800c9a50 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:30.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.348+0000 7fd787a00640 1 --2- 172.21.15.29:0/2635071991 >> v2:172.21.15.37:3300/0 conn(0x7fd7800c9f90 0x7fd7800c5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:30.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.348+0000 7fd787a00640 1 --2- 172.21.15.29:0/2635071991 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7801531b0 0x7fd7800c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:30.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.348+0000 7fd787a00640 1 -- 172.21.15.29:0/2635071991 >> 172.21.15.29:0/2635071991 conn(0x7fd780072e50 msgr2=0x7fd780073260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:30.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.348+0000 7fd787a00640 1 -- 172.21.15.29:0/2635071991 shutdown_connections 2024-04-17T19:50:30.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.348+0000 7fd787a00640 1 -- 172.21.15.29:0/2635071991 wait complete. 2024-04-17T19:50:30.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd787a00640 1 Processor -- start 2024-04-17T19:50:30.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd787a00640 1 -- start start 2024-04-17T19:50:30.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd787a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7800c8f10 0x7fd7801deef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:30.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd787a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7800c9f90 0x7fd7801ee6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:30.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd787a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7801531b0 0x7fd780070080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:30.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd787a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd7800c5b80 con 0x7fd7801531b0 2024-04-17T19:50:30.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd787a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd7801594c0 con 0x7fd7800c8f10 2024-04-17T19:50:30.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd787a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd7800c5d00 con 0x7fd7800c9f90 2024-04-17T19:50:30.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd785a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7801531b0 0x7fd780070080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:30.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd785a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7801531b0 0x7fd780070080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34898/0 (socket says 172.21.15.29:34898) 2024-04-17T19:50:30.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd785a00640 1 -- 172.21.15.29:0/3049529024 learned_addr learned my addr 172.21.15.29:0/3049529024 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:30.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd77fe00640 1 --2- 172.21.15.29:0/3049529024 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7800c9f90 0x7fd7801ee6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:30.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd785000640 1 --2- 172.21.15.29:0/3049529024 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7800c8f10 0x7fd7801deef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:30.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd785a00640 1 -- 172.21.15.29:0/3049529024 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7800c9f90 msgr2=0x7fd7801ee6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:30.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd785a00640 1 --2- 172.21.15.29:0/3049529024 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7800c9f90 0x7fd7801ee6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:30.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd785a00640 1 -- 172.21.15.29:0/3049529024 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7800c8f10 msgr2=0x7fd7801deef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:30.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd785a00640 1 --2- 172.21.15.29:0/3049529024 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7800c8f10 0x7fd7801deef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:30.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd785a00640 1 -- 172.21.15.29:0/3049529024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7801eebe0 con 0x7fd7801531b0 2024-04-17T19:50:30.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd785a00640 1 --2- 172.21.15.29:0/3049529024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7801531b0 0x7fd780070080 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd778007d50 tx=0x7fd77800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:30.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd77cc00640 1 -- 172.21.15.29:0/3049529024 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd778014420 con 0x7fd7801531b0 2024-04-17T19:50:30.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd77cc00640 1 -- 172.21.15.29:0/3049529024 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd778019070 con 0x7fd7801531b0 2024-04-17T19:50:30.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.352+0000 7fd77cc00640 1 -- 172.21.15.29:0/3049529024 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd778014820 con 0x7fd7801531b0 2024-04-17T19:50:30.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.356+0000 7fd787a00640 1 -- 172.21.15.29:0/3049529024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7801f1490 con 0x7fd7801531b0 2024-04-17T19:50:30.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.356+0000 7fd787a00640 1 -- 172.21.15.29:0/3049529024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7801edec0 con 0x7fd7801531b0 2024-04-17T19:50:30.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.356+0000 7fd787a00640 1 -- 172.21.15.29:0/3049529024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd780154db0 con 0x7fd7801531b0 2024-04-17T19:50:30.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.356+0000 7fd77cc00640 1 -- 172.21.15.29:0/3049529024 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd7780149c0 con 0x7fd7801531b0 2024-04-17T19:50:30.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.356+0000 7fd77cc00640 1 --2- 172.21.15.29:0/3049529024 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd760068380 0x7fd76006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:30.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.356+0000 7fd77cc00640 1 -- 172.21.15.29:0/3049529024 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7fd778087760 con 0x7fd7801531b0 2024-04-17T19:50:30.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.356+0000 7fd785000640 1 --2- 172.21.15.29:0/3049529024 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd760068380 0x7fd76006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:30.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.360+0000 7fd785000640 1 --2- 172.21.15.29:0/3049529024 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd760068380 0x7fd76006a890 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd77400b930 tx=0x7fd774008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:30.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.364+0000 7fd77cc00640 1 -- 172.21.15.29:0/3049529024 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd778057790 con 0x7fd7801531b0 2024-04-17T19:50:30.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.560+0000 7fd787a00640 1 -- 172.21.15.29:0/3049529024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd780069b90 con 0x7fd7801531b0 2024-04-17T19:50:30.552 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.560+0000 7fd77cc00640 1 -- 172.21.15.29:0/3049529024 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6078 (secure 0 0 0) 0x7fd778057090 con 0x7fd7801531b0 2024-04-17T19:50:30.552 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:50:30.553 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:28.863399+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:rejoin","state_seq":45,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229]},"id":1}]} 2024-04-17T19:50:30.553 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 23 2024-04-17T19:50:30.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.564+0000 7fd787a00640 1 -- 172.21.15.29:0/3049529024 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd760068380 msgr2=0x7fd76006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:30.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.564+0000 7fd787a00640 1 --2- 172.21.15.29:0/3049529024 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd760068380 0x7fd76006a890 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd77400b930 tx=0x7fd774008040 comp rx=0 tx=0).stop 2024-04-17T19:50:30.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.564+0000 7fd787a00640 1 -- 172.21.15.29:0/3049529024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7801531b0 msgr2=0x7fd780070080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:30.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.564+0000 7fd787a00640 1 --2- 172.21.15.29:0/3049529024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7801531b0 0x7fd780070080 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd778007d50 tx=0x7fd77800e4b0 comp rx=0 tx=0).stop 2024-04-17T19:50:30.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.564+0000 7fd787a00640 1 -- 172.21.15.29:0/3049529024 shutdown_connections 2024-04-17T19:50:30.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.564+0000 7fd787a00640 1 --2- 172.21.15.29:0/3049529024 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd760068380 0x7fd76006a890 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:30.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.564+0000 7fd787a00640 1 --2- 172.21.15.29:0/3049529024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7801531b0 0x7fd780070080 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:30.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.564+0000 7fd787a00640 1 --2- 172.21.15.29:0/3049529024 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7800c9f90 0x7fd7801ee6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:30.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.564+0000 7fd787a00640 1 --2- 172.21.15.29:0/3049529024 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7800c8f10 0x7fd7801deef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:30.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.564+0000 7fd787a00640 1 -- 172.21.15.29:0/3049529024 >> 172.21.15.29:0/3049529024 conn(0x7fd780072e50 msgr2=0x7fd7800ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:30.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.564+0000 7fd787a00640 1 -- 172.21.15.29:0/3049529024 shutdown_connections 2024-04-17T19:50:30.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:30.564+0000 7fd787a00640 1 -- 172.21.15.29:0/3049529024 wait complete. 2024-04-17T19:50:30.576 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:50:32.577 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:50:32.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.752+0000 7f078e400640 1 Processor -- start 2024-04-17T19:50:32.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.752+0000 7f078e400640 1 -- start start 2024-04-17T19:50:32.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.752+0000 7f078e400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f07880cbf90 0x7f078815ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:32.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.752+0000 7f078e400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0788155da0 0x7f07881561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:32.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.752+0000 7f078e400640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f078815cf70 con 0x7f07880c7f50 2024-04-17T19:50:32.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.752+0000 7f078e400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f078806bdc0 con 0x7f07880cbf90 2024-04-17T19:50:32.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.752+0000 7f078e400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f07880cab20 con 0x7f0788155da0 2024-04-17T19:50:32.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.756+0000 7f077fe00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f07880c7f50 0x7f07880ca410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52406/0 (socket says 172.21.15.29:52406) 2024-04-17T19:50:32.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.756+0000 7f077fe00640 1 -- 172.21.15.29:0/568299382 learned_addr learned my addr 172.21.15.29:0/568299382 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:50:32.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.756+0000 7f0787e00640 1 --2- 172.21.15.29:0/568299382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0788155da0 0x7f07881561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:32.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.756+0000 7f0787400640 1 --2- 172.21.15.29:0/568299382 >> v2:172.21.15.37:3300/0 conn(0x7f07880cbf90 0x7f078815ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:32.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.756+0000 7f0786a00640 1 -- 172.21.15.29:0/568299382 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1826410678 0 0) 0x7f078815cf70 con 0x7f07880c7f50 2024-04-17T19:50:32.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.756+0000 7f0786a00640 1 -- 172.21.15.29:0/568299382 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0768003530 con 0x7f07880c7f50 2024-04-17T19:50:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.756+0000 7f0786a00640 1 -- 172.21.15.29:0/568299382 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 161152143 0 0) 0x7f0768003530 con 0x7f07880c7f50 2024-04-17T19:50:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.756+0000 7f0787e00640 1 -- 172.21.15.29:0/568299382 >> v2:172.21.15.37:3300/0 conn(0x7f07880cbf90 msgr2=0x7f078815ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.756+0000 7f0787e00640 1 --2- 172.21.15.29:0/568299382 >> v2:172.21.15.37:3300/0 conn(0x7f07880cbf90 0x7f078815ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.756+0000 7f0787e00640 1 -- 172.21.15.29:0/568299382 >> v1:172.21.15.29:6789/0 conn(0x7f07880c7f50 legacy=0x7f07880ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.756+0000 7f0787e00640 1 -- 172.21.15.29:0/568299382 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07881566f0 con 0x7f0788155da0 2024-04-17T19:50:32.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.760+0000 7f0787e00640 1 --2- 172.21.15.29:0/568299382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0788155da0 0x7f07881561b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0780009d90 tx=0x7f078001c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1ceace6c83b456a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:32.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.760+0000 7f0786a00640 1 -- 172.21.15.29:0/568299382 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07800b2070 con 0x7f0788155da0 2024-04-17T19:50:32.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.760+0000 7f0786a00640 1 -- 172.21.15.29:0/568299382 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07800a5580 con 0x7f0788155da0 2024-04-17T19:50:32.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.760+0000 7f0786a00640 1 -- 172.21.15.29:0/568299382 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778001680 con 0x7f0788155da0 2024-04-17T19:50:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.760+0000 7f078e400640 1 -- 172.21.15.29:0/568299382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0788155da0 msgr2=0x7f07881561b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.760+0000 7f078e400640 1 --2- 172.21.15.29:0/568299382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0788155da0 0x7f07881561b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0780009d90 tx=0x7f078001c890 comp rx=0 tx=0).stop 2024-04-17T19:50:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.760+0000 7f078e400640 1 -- 172.21.15.29:0/568299382 shutdown_connections 2024-04-17T19:50:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.760+0000 7f078e400640 1 --2- 172.21.15.29:0/568299382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0788155da0 0x7f07881561b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.760+0000 7f078e400640 1 --2- 172.21.15.29:0/568299382 >> v2:172.21.15.37:3300/0 conn(0x7f07880cbf90 0x7f078815ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.760+0000 7f078e400640 1 -- 172.21.15.29:0/568299382 >> 172.21.15.29:0/568299382 conn(0x7f078806fde0 msgr2=0x7f07880701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.760+0000 7f078e400640 1 -- 172.21.15.29:0/568299382 shutdown_connections 2024-04-17T19:50:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.760+0000 7f078e400640 1 -- 172.21.15.29:0/568299382 wait complete. 2024-04-17T19:50:32.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.760+0000 7f078e400640 1 Processor -- start 2024-04-17T19:50:32.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.760+0000 7f078e400640 1 -- start start 2024-04-17T19:50:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f078e400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07880c7f50 0x7f07881430d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f078e400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07880cbf90 0x7f0788140eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f078e400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0788155da0 0x7f078815e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f078e400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f07880c8360 con 0x7f07880cbf90 2024-04-17T19:50:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f078e400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f078806c5d0 con 0x7f0788155da0 2024-04-17T19:50:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f078e400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f07880c84e0 con 0x7f07880c7f50 2024-04-17T19:50:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f077fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07880cbf90 0x7f0788140eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f0787400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07880c7f50 0x7f07881430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f077fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07880cbf90 0x7f0788140eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34904/0 (socket says 172.21.15.29:34904) 2024-04-17T19:50:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f077fe00640 1 -- 172.21.15.29:0/4045307175 learned_addr learned my addr 172.21.15.29:0/4045307175 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f0787e00640 1 --2- 172.21.15.29:0/4045307175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0788155da0 0x7f078815e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f077fe00640 1 -- 172.21.15.29:0/4045307175 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07880c7f50 msgr2=0x7f07881430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f077fe00640 1 --2- 172.21.15.29:0/4045307175 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07880c7f50 0x7f07881430d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f077fe00640 1 -- 172.21.15.29:0/4045307175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0788155da0 msgr2=0x7f078815e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f077fe00640 1 --2- 172.21.15.29:0/4045307175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0788155da0 0x7f078815e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f077fe00640 1 -- 172.21.15.29:0/4045307175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07880c70d0 con 0x7f07880cbf90 2024-04-17T19:50:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f0787e00640 1 --2- 172.21.15.29:0/4045307175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0788155da0 0x7f078815e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:50:32.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f077fe00640 1 --2- 172.21.15.29:0/4045307175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07880cbf90 0x7f0788140eb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f07780017f0 tx=0x7f077800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:32.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f077f400640 1 -- 172.21.15.29:0/4045307175 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778005ae0 con 0x7f07880cbf90 2024-04-17T19:50:32.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f077f400640 1 -- 172.21.15.29:0/4045307175 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0778005c80 con 0x7f07880cbf90 2024-04-17T19:50:32.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f078e400640 1 -- 172.21.15.29:0/4045307175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07880c73b0 con 0x7f07880cbf90 2024-04-17T19:50:32.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f077f400640 1 -- 172.21.15.29:0/4045307175 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778007290 con 0x7f07880cbf90 2024-04-17T19:50:32.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f078e400640 1 -- 172.21.15.29:0/4045307175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07881413f0 con 0x7f07880cbf90 2024-04-17T19:50:32.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.764+0000 7f078e400640 1 -- 172.21.15.29:0/4045307175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0788069b90 con 0x7f07880cbf90 2024-04-17T19:50:32.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.768+0000 7f077f400640 1 -- 172.21.15.29:0/4045307175 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0778007430 con 0x7f07880cbf90 2024-04-17T19:50:32.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.768+0000 7f077f400640 1 --2- 172.21.15.29:0/4045307175 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f07640683b0 0x7f076406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:32.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.768+0000 7f077f400640 1 -- 172.21.15.29:0/4045307175 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f0778086a90 con 0x7f07880cbf90 2024-04-17T19:50:32.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.768+0000 7f0787400640 1 --2- 172.21.15.29:0/4045307175 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f07640683b0 0x7f076406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:32.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.768+0000 7f0787400640 1 --2- 172.21.15.29:0/4045307175 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f07640683b0 0x7f076406a8c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0770009ac0 tx=0x7f0770009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:32.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.772+0000 7f077f400640 1 -- 172.21.15.29:0/4045307175 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0778007720 con 0x7f07880cbf90 2024-04-17T19:50:32.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.976+0000 7f078e400640 1 -- 172.21.15.29:0/4045307175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0788149740 con 0x7f07880cbf90 2024-04-17T19:50:32.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.976+0000 7f077f400640 1 -- 172.21.15.29:0/4045307175 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+6078 (secure 0 0 0) 0x7f0778056be0 con 0x7f07880cbf90 2024-04-17T19:50:32.967 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:50:32.967 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:28.863399+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:rejoin","state_seq":45,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229]},"id":1}]} 2024-04-17T19:50:32.968 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 23 2024-04-17T19:50:32.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.980+0000 7f078e400640 1 -- 172.21.15.29:0/4045307175 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f07640683b0 msgr2=0x7f076406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:32.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.980+0000 7f078e400640 1 --2- 172.21.15.29:0/4045307175 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f07640683b0 0x7f076406a8c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0770009ac0 tx=0x7f0770009190 comp rx=0 tx=0).stop 2024-04-17T19:50:32.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.980+0000 7f078e400640 1 -- 172.21.15.29:0/4045307175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07880cbf90 msgr2=0x7f0788140eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:32.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.980+0000 7f078e400640 1 --2- 172.21.15.29:0/4045307175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07880cbf90 0x7f0788140eb0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f07780017f0 tx=0x7f077800cbe0 comp rx=0 tx=0).stop 2024-04-17T19:50:32.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.980+0000 7f078e400640 1 -- 172.21.15.29:0/4045307175 shutdown_connections 2024-04-17T19:50:32.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.980+0000 7f078e400640 1 --2- 172.21.15.29:0/4045307175 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f07640683b0 0x7f076406a8c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:32.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.980+0000 7f078e400640 1 --2- 172.21.15.29:0/4045307175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0788155da0 0x7f078815e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:32.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.980+0000 7f078e400640 1 --2- 172.21.15.29:0/4045307175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07880cbf90 0x7f0788140eb0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:32.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.980+0000 7f078e400640 1 --2- 172.21.15.29:0/4045307175 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f07880c7f50 0x7f07881430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:32.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.980+0000 7f078e400640 1 -- 172.21.15.29:0/4045307175 >> 172.21.15.29:0/4045307175 conn(0x7f078806fde0 msgr2=0x7f078815b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:32.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.980+0000 7f078e400640 1 -- 172.21.15.29:0/4045307175 shutdown_connections 2024-04-17T19:50:32.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:32.980+0000 7f078e400640 1 -- 172.21.15.29:0/4045307175 wait complete. 2024-04-17T19:50:32.990 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:50:33.883 INFO:tasks.workunit.client.0.smithi029.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2024-04-17T19:50:34.812 INFO:tasks.ceph.mds.c.smithi029.stderr:2024-04-17T19:50:34.820+0000 7ff3b0200640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-17T19:50:34.991 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:50:35.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.151+0000 7f549dc00640 1 Processor -- start 2024-04-17T19:50:35.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.151+0000 7f549dc00640 1 -- start start 2024-04-17T19:50:35.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.151+0000 7f549dc00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f549815c900 0x7f54980cb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:35.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.151+0000 7f549dc00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54980ca790 con 0x7f54980ce470 2024-04-17T19:50:35.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.151+0000 7f549dc00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f549815cd10 con 0x7f54980c8be0 2024-04-17T19:50:35.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.151+0000 7f549dc00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f54980c8ff0 con 0x7f549815c900 2024-04-17T19:50:35.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.151+0000 7f549d200640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f54980ce470 0x7f5498166800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52420/0 (socket says 172.21.15.29:52420) 2024-04-17T19:50:35.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.151+0000 7f549d200640 1 -- 172.21.15.29:0/2867896106 learned_addr learned my addr 172.21.15.29:0/2867896106 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:50:35.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.151+0000 7f548fe00640 1 --2- 172.21.15.29:0/2867896106 >> v2:172.21.15.37:3300/0 conn(0x7f549815c900 0x7f54980cb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:35.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497400640 1 -- 172.21.15.29:0/2867896106 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 841388817 0 0) 0x7f54900014a0 con 0x7f54980ce470 2024-04-17T19:50:35.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497400640 1 -- 172.21.15.29:0/2867896106 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f547c003530 con 0x7f54980ce470 2024-04-17T19:50:35.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497400640 1 -- 172.21.15.29:0/2867896106 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 150573252 0 0) 0x7f547c003530 con 0x7f54980ce470 2024-04-17T19:50:35.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497400640 1 -- 172.21.15.29:0/2867896106 >> v2:172.21.15.37:3300/0 conn(0x7f549815c900 msgr2=0x7f54980cb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:35.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497400640 1 --2- 172.21.15.29:0/2867896106 >> v2:172.21.15.37:3300/0 conn(0x7f549815c900 0x7f54980cb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:35.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497400640 1 -- 172.21.15.29:0/2867896106 >> v1:172.21.15.37:6789/0 conn(0x7f54980c8be0 legacy=0x7f5498152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:35.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497400640 1 -- 172.21.15.29:0/2867896106 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5498156520 con 0x7f54980ce470 2024-04-17T19:50:35.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497400640 1 -- 172.21.15.29:0/2867896106 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f5490001bd0 con 0x7f54980ce470 2024-04-17T19:50:35.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f548fe00640 1 --2- 172.21.15.29:0/2867896106 >> v2:172.21.15.37:3300/0 conn(0x7f549815c900 0x7f54980cb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:50:35.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497400640 1 -- 172.21.15.29:0/2867896106 >> v1:172.21.15.29:6789/0 conn(0x7f54980ce470 legacy=0x7f5498166800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:35.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497400640 1 --2- 172.21.15.29:0/2867896106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f547c003a30 0x7f547c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:35.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497400640 1 --2- 172.21.15.29:0/2867896106 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f547c004810 0x7f547c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:35.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497400640 1 --2- 172.21.15.29:0/2867896106 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f547c005230 0x7f547c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:35.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497400640 1 -- 172.21.15.29:0/2867896106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f54980c8ff0 con 0x7f547c003a30 2024-04-17T19:50:35.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497400640 1 -- 172.21.15.29:0/2867896106 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f547c00c1c0 con 0x7f547c005230 2024-04-17T19:50:35.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497400640 1 -- 172.21.15.29:0/2867896106 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f547c00c3c0 con 0x7f547c004810 2024-04-17T19:50:35.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497e00640 1 --2- 172.21.15.29:0/2867896106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f547c003a30 0x7f547c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:35.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f548fe00640 1 --2- 172.21.15.29:0/2867896106 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f547c004810 0x7f547c004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:35.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f549d200640 1 --2- 172.21.15.29:0/2867896106 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f547c005230 0x7f547c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:35.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497e00640 1 -- 172.21.15.29:0/2867896106 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f547c004810 msgr2=0x7f547c004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:35.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497e00640 1 --2- 172.21.15.29:0/2867896106 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f547c004810 0x7f547c004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:35.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497e00640 1 -- 172.21.15.29:0/2867896106 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f547c005230 msgr2=0x7f547c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:35.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497e00640 1 --2- 172.21.15.29:0/2867896106 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f547c005230 0x7f547c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:35.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f549d200640 1 -- 172.21.15.29:0/2867896106 reap_dead start 2024-04-17T19:50:35.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.155+0000 7f5497e00640 1 -- 172.21.15.29:0/2867896106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f547c003530 con 0x7f547c003a30 2024-04-17T19:50:35.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.159+0000 7f5497e00640 1 --2- 172.21.15.29:0/2867896106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f547c003a30 0x7f547c003ea0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5488001780 tx=0x7f548801c220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ecd36344a0fe01b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:35.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.159+0000 7f5497400640 1 -- 172.21.15.29:0/2867896106 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54880013a0 con 0x7f547c003a30 2024-04-17T19:50:35.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.159+0000 7f5497400640 1 -- 172.21.15.29:0/2867896106 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54880050d0 con 0x7f547c003a30 2024-04-17T19:50:35.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.159+0000 7f5497400640 1 -- 172.21.15.29:0/2867896106 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54880054c0 con 0x7f547c003a30 2024-04-17T19:50:35.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.159+0000 7f549dc00640 1 -- 172.21.15.29:0/2867896106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f547c003a30 msgr2=0x7f547c003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:35.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.159+0000 7f549dc00640 1 --2- 172.21.15.29:0/2867896106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f547c003a30 0x7f547c003ea0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5488001780 tx=0x7f548801c220 comp rx=0 tx=0).stop 2024-04-17T19:50:35.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.159+0000 7f549dc00640 1 -- 172.21.15.29:0/2867896106 shutdown_connections 2024-04-17T19:50:35.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.159+0000 7f549dc00640 1 --2- 172.21.15.29:0/2867896106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f547c003a30 0x7f547c003ea0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:35.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.159+0000 7f549dc00640 1 -- 172.21.15.29:0/2867896106 >> 172.21.15.29:0/2867896106 conn(0x7f5498072de0 msgr2=0x7f54980731f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:35.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.159+0000 7f549dc00640 1 -- 172.21.15.29:0/2867896106 shutdown_connections 2024-04-17T19:50:35.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.159+0000 7f549dc00640 1 -- 172.21.15.29:0/2867896106 wait complete. 2024-04-17T19:50:35.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f549dc00640 1 Processor -- start 2024-04-17T19:50:35.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f549dc00640 1 -- start start 2024-04-17T19:50:35.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f549dc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f547c003a30 0x7f54981f9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:35.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f549dc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f549814a6c0 0x7f5498153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:35.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f549dc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f549814e200 0x7f5498148820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:35.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f549dc00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f54980cec90 con 0x7f549814e200 2024-04-17T19:50:35.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f549dc00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f54980cafd0 con 0x7f547c003a30 2024-04-17T19:50:35.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f549dc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f549806c450 con 0x7f549814a6c0 2024-04-17T19:50:35.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f549d200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f549814e200 0x7f5498148820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:35.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f549d200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f549814e200 0x7f5498148820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34926/0 (socket says 172.21.15.29:34926) 2024-04-17T19:50:35.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f549d200640 1 -- 172.21.15.29:0/4046616446 learned_addr learned my addr 172.21.15.29:0/4046616446 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:35.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f548fe00640 1 --2- 172.21.15.29:0/4046616446 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f549814a6c0 0x7f5498153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:35.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f5497e00640 1 --2- 172.21.15.29:0/4046616446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f547c003a30 0x7f54981f9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:35.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f549d200640 1 -- 172.21.15.29:0/4046616446 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f549814a6c0 msgr2=0x7f5498153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:35.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f549d200640 1 --2- 172.21.15.29:0/4046616446 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f549814a6c0 0x7f5498153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:35.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f549d200640 1 -- 172.21.15.29:0/4046616446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f547c003a30 msgr2=0x7f54981f9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:35.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f549d200640 1 --2- 172.21.15.29:0/4046616446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f547c003a30 0x7f54981f9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:35.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f549d200640 1 -- 172.21.15.29:0/4046616446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54980c7720 con 0x7f549814e200 2024-04-17T19:50:35.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f549d200640 1 --2- 172.21.15.29:0/4046616446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f549814e200 0x7f5498148820 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f54980c5380 tx=0x7f549000af40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:35.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.163+0000 7f5494c00640 1 -- 172.21.15.29:0/4046616446 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5490017020 con 0x7f549814e200 2024-04-17T19:50:35.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.167+0000 7f5494c00640 1 -- 172.21.15.29:0/4046616446 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54900077c0 con 0x7f549814e200 2024-04-17T19:50:35.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.167+0000 7f5494c00640 1 -- 172.21.15.29:0/4046616446 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54900133f0 con 0x7f549814e200 2024-04-17T19:50:35.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.167+0000 7f549dc00640 1 -- 172.21.15.29:0/4046616446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54980c79e0 con 0x7f549814e200 2024-04-17T19:50:35.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.167+0000 7f549dc00640 1 -- 172.21.15.29:0/4046616446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54981f95e0 con 0x7f549814e200 2024-04-17T19:50:35.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.167+0000 7f549dc00640 1 -- 172.21.15.29:0/4046616446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f549804ff10 con 0x7f549814e200 2024-04-17T19:50:35.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.167+0000 7f5494c00640 1 -- 172.21.15.29:0/4046616446 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f549001a020 con 0x7f549814e200 2024-04-17T19:50:35.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.167+0000 7f5494c00640 1 --2- 172.21.15.29:0/4046616446 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54740683b0 0x7f547406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:35.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.167+0000 7f5494c00640 1 -- 172.21.15.29:0/4046616446 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f54900873d0 con 0x7f549814e200 2024-04-17T19:50:35.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.167+0000 7f5497e00640 1 --2- 172.21.15.29:0/4046616446 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54740683b0 0x7f547406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:35.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.171+0000 7f5497e00640 1 --2- 172.21.15.29:0/4046616446 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54740683b0 0x7f547406a8c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5488001750 tx=0x7f5488009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:35.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.171+0000 7f5494c00640 1 -- 172.21.15.29:0/4046616446 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5490021020 con 0x7f549814e200 2024-04-17T19:50:35.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.371+0000 7f549dc00640 1 -- 172.21.15.29:0/4046616446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f549806c760 con 0x7f549814e200 2024-04-17T19:50:35.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.371+0000 7f5494c00640 1 -- 172.21.15.29:0/4046616446 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+6083 (secure 0 0 0) 0x7f54980ca790 con 0x7f549814e200 2024-04-17T19:50:35.365 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:50:35.365 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:34.757701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:active","state_seq":47,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4223,4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:50:35.365 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 24 2024-04-17T19:50:35.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.375+0000 7f549dc00640 1 -- 172.21.15.29:0/4046616446 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54740683b0 msgr2=0x7f547406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:35.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.375+0000 7f549dc00640 1 --2- 172.21.15.29:0/4046616446 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54740683b0 0x7f547406a8c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5488001750 tx=0x7f5488009040 comp rx=0 tx=0).stop 2024-04-17T19:50:35.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.375+0000 7f549dc00640 1 -- 172.21.15.29:0/4046616446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f549814e200 msgr2=0x7f5498148820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:35.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.375+0000 7f549dc00640 1 --2- 172.21.15.29:0/4046616446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f549814e200 0x7f5498148820 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f54980c5380 tx=0x7f549000af40 comp rx=0 tx=0).stop 2024-04-17T19:50:35.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.375+0000 7f549dc00640 1 -- 172.21.15.29:0/4046616446 shutdown_connections 2024-04-17T19:50:35.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.375+0000 7f549dc00640 1 --2- 172.21.15.29:0/4046616446 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54740683b0 0x7f547406a8c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:35.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.375+0000 7f549dc00640 1 --2- 172.21.15.29:0/4046616446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f549814e200 0x7f5498148820 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:35.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.375+0000 7f549dc00640 1 --2- 172.21.15.29:0/4046616446 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f549814a6c0 0x7f5498153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:35.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.375+0000 7f549dc00640 1 --2- 172.21.15.29:0/4046616446 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f547c003a30 0x7f54981f9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:35.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.375+0000 7f549dc00640 1 -- 172.21.15.29:0/4046616446 >> 172.21.15.29:0/4046616446 conn(0x7f5498072de0 msgr2=0x7f54980c9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:35.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.375+0000 7f549dc00640 1 -- 172.21.15.29:0/4046616446 shutdown_connections 2024-04-17T19:50:35.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:35.375+0000 7f549dc00640 1 -- 172.21.15.29:0/4046616446 wait complete. 2024-04-17T19:50:35.387 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=4, replacing gid=4226 2024-04-17T19:50:35.387 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.b 2024-04-17T19:50:37.400 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir foo 2024-04-17T19:50:37.408 INFO:tasks.workunit.client.0.smithi029.stderr:+ cp linux-2.6.33.tar.bz2 foo 2024-04-17T19:50:38.318 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir foo/.snap/barsnap 2024-04-17T19:50:39.763 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm foo/linux-2.6.33.tar.bz2 2024-04-17T19:50:39.774 INFO:tasks.workunit.client.0.smithi029.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2024-04-17T19:50:40.047 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo passed: files are identical 2024-04-17T19:50:40.048 INFO:tasks.workunit.client.0.smithi029.stdout:passed: files are identical 2024-04-17T19:50:40.048 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir foo/.snap/barsnap 2024-04-17T19:50:40.073 INFO:tasks.workunit.client.0.smithi029.stdout:OK 2024-04-17T19:50:40.074 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo OK 2024-04-17T19:50:40.075 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-17T19:50:40.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-17T19:50:40.124 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2024-04-17T19:50:40.124 DEBUG:teuthology.orchestra.run.smithi029:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f3a98d3f110319c567ec86bf70955c141495ccf6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2024-04-17T19:50:40.141 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 1 2024-04-17T19:50:40.142 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 2 2024-04-17T19:50:40.143 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 3 2024-04-17T19:50:40.144 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' -e file4 ']' 2024-04-17T19:50:40.144 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/snap1 2024-04-17T19:50:40.160 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 4 2024-04-17T19:50:40.162 INFO:tasks.workunit.client.0.smithi029.stderr:++ ls 2024-04-17T19:50:40.168 INFO:tasks.workunit.client.0.smithi029.stderr:+ now='file1 2024-04-17T19:50:40.168 INFO:tasks.workunit.client.0.smithi029.stderr:file2 2024-04-17T19:50:40.168 INFO:tasks.workunit.client.0.smithi029.stderr:file3 2024-04-17T19:50:40.168 INFO:tasks.workunit.client.0.smithi029.stderr:file4' 2024-04-17T19:50:40.168 INFO:tasks.workunit.client.0.smithi029.stderr:++ ls .snap/snap1 2024-04-17T19:50:40.455 INFO:tasks.workunit.client.0.smithi029.stderr:+ then='file1 2024-04-17T19:50:40.455 INFO:tasks.workunit.client.0.smithi029.stderr:file2 2024-04-17T19:50:40.456 INFO:tasks.workunit.client.0.smithi029.stderr:file3' 2024-04-17T19:50:40.456 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir .snap/snap1 2024-04-17T19:50:40.466 INFO:tasks.workunit.client.0.smithi029.stderr:+ '[' 'file1 2024-04-17T19:50:40.466 INFO:tasks.workunit.client.0.smithi029.stderr:file2 2024-04-17T19:50:40.466 INFO:tasks.workunit.client.0.smithi029.stderr:file3 2024-04-17T19:50:40.466 INFO:tasks.workunit.client.0.smithi029.stderr:file4' = 'file1 2024-04-17T19:50:40.466 INFO:tasks.workunit.client.0.smithi029.stderr:file2 2024-04-17T19:50:40.466 INFO:tasks.workunit.client.0.smithi029.stderr:file3' ']' 2024-04-17T19:50:40.466 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 1 2024-04-17T19:50:40.553 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 2 2024-04-17T19:50:40.566 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 3 2024-04-17T19:50:40.580 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/snap1 2024-04-17T19:50:40.838 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo 4 2024-04-17T19:50:40.847 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir .snap/snap1 2024-04-17T19:50:40.881 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm file1 file2 file3 file4 2024-04-17T19:50:40.889 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo OK 2024-04-17T19:50:40.890 INFO:tasks.workunit.client.0.smithi029.stdout:OK 2024-04-17T19:50:40.890 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-17T19:50:40.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-17T19:50:40.959 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2024-04-17T19:50:40.959 DEBUG:teuthology.orchestra.run.smithi029:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f3a98d3f110319c567ec86bf70955c141495ccf6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2024-04-17T19:50:43.388 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2024-04-17T19:50:43.388 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T19:50:43.388 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T19:50:43.392 INFO:tasks.ceph.mds.b:Started 2024-04-17T19:50:43.392 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:50:43.502 INFO:tasks.ceph.mds.b.smithi037.stdout:starting mds.b at 2024-04-17T19:50:43.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.567+0000 7f99b7400640 1 Processor -- start 2024-04-17T19:50:43.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.567+0000 7f99b7400640 1 -- start start 2024-04-17T19:50:43.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.567+0000 7f99b7400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f99b01531b0 0x7f99b00c8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:43.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.567+0000 7f99b7400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f99b00c5640 0x7f99b00c5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:43.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.567+0000 7f99b7400640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99b00715d0 con 0x7f99b00c8f10 2024-04-17T19:50:43.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.567+0000 7f99b7400640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f99b0159cc0 con 0x7f99b00c5640 2024-04-17T19:50:43.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.567+0000 7f99b7400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f99b00718b0 con 0x7f99b01531b0 2024-04-17T19:50:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.567+0000 7f99b5400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f99b00c5640 0x7f99b00c5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.567+0000 7f99afe00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f99b00c8f10 0x7f99b00c9a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35648/0 (socket says 172.21.15.29:35648) 2024-04-17T19:50:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.567+0000 7f99afe00640 1 -- 172.21.15.29:0/3697749952 learned_addr learned my addr 172.21.15.29:0/3697749952 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:50:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.567+0000 7f99b4a00640 1 --2- 172.21.15.29:0/3697749952 >> v2:172.21.15.37:3300/0 conn(0x7f99b01531b0 0x7f99b00c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.567+0000 7f99b5400640 1 -- 172.21.15.29:0/3697749952 >> v2:172.21.15.37:3300/0 conn(0x7f99b01531b0 msgr2=0x7f99b00c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:50:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.567+0000 7f99b5400640 1 --2- 172.21.15.29:0/3697749952 >> v2:172.21.15.37:3300/0 conn(0x7f99b01531b0 0x7f99b00c8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.567+0000 7f99b5400640 1 -- 172.21.15.29:0/3697749952 >> v1:172.21.15.29:6789/0 conn(0x7f99b00c8f10 legacy=0x7f99b00c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.567+0000 7f99b5400640 1 -- 172.21.15.29:0/3697749952 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b0158f00 con 0x7f99b00c5640 2024-04-17T19:50:43.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.571+0000 7f99b5400640 1 --2- 172.21.15.29:0/3697749952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b00c5640 0x7f99b00c5a50 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f99b0072860 tx=0x7f99a801f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15069171f053b82c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:43.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.571+0000 7f99af400640 1 -- 172.21.15.29:0/3697749952 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a80b4070 con 0x7f99b00c5640 2024-04-17T19:50:43.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.571+0000 7f99af400640 1 -- 172.21.15.29:0/3697749952 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99a80af600 con 0x7f99b00c5640 2024-04-17T19:50:43.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.571+0000 7f99af400640 1 -- 172.21.15.29:0/3697749952 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a80af9d0 con 0x7f99b00c5640 2024-04-17T19:50:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.571+0000 7f99b7400640 1 -- 172.21.15.29:0/3697749952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b00c5640 msgr2=0x7f99b00c5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.571+0000 7f99b7400640 1 --2- 172.21.15.29:0/3697749952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b00c5640 0x7f99b00c5a50 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f99b0072860 tx=0x7f99a801f110 comp rx=0 tx=0).stop 2024-04-17T19:50:43.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.571+0000 7f99b7400640 1 -- 172.21.15.29:0/3697749952 shutdown_connections 2024-04-17T19:50:43.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.571+0000 7f99b7400640 1 --2- 172.21.15.29:0/3697749952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b00c5640 0x7f99b00c5a50 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:43.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.571+0000 7f99b7400640 1 --2- 172.21.15.29:0/3697749952 >> v2:172.21.15.37:3300/0 conn(0x7f99b01531b0 0x7f99b00c8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:43.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.571+0000 7f99b7400640 1 -- 172.21.15.29:0/3697749952 >> 172.21.15.29:0/3697749952 conn(0x7f99b0072e50 msgr2=0x7f99b0073260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:43.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.571+0000 7f99b7400640 1 -- 172.21.15.29:0/3697749952 shutdown_connections 2024-04-17T19:50:43.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.571+0000 7f99b7400640 1 -- 172.21.15.29:0/3697749952 wait complete. 2024-04-17T19:50:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.575+0000 7f99b7400640 1 Processor -- start 2024-04-17T19:50:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.575+0000 7f99b7400640 1 -- start start 2024-04-17T19:50:43.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.575+0000 7f99b7400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b00c5640 0x7f99b0141830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:43.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.575+0000 7f99b7400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f99b00c8f10 0x7f99b01358f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:43.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.575+0000 7f99b7400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99b01531b0 0x7f99b0135e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:43.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.575+0000 7f99b7400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f99b00ca160 con 0x7f99b00c5640 2024-04-17T19:50:43.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.575+0000 7f99b7400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f99b01594c0 con 0x7f99b01531b0 2024-04-17T19:50:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.575+0000 7f99b7400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f99b014a2c0 con 0x7f99b00c8f10 2024-04-17T19:50:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.575+0000 7f99b4a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b00c5640 0x7f99b0141830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.575+0000 7f99b4a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b00c5640 0x7f99b0141830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54616/0 (socket says 172.21.15.29:54616) 2024-04-17T19:50:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.575+0000 7f99b4a00640 1 -- 172.21.15.29:0/2838061636 learned_addr learned my addr 172.21.15.29:0/2838061636 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:50:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.575+0000 7f99b5400640 1 --2- 172.21.15.29:0/2838061636 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99b01531b0 0x7f99b0135e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.575+0000 7f99afe00640 1 --2- 172.21.15.29:0/2838061636 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f99b00c8f10 0x7f99b01358f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.579+0000 7f99b4a00640 1 -- 172.21.15.29:0/2838061636 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f99b00c8f10 msgr2=0x7f99b01358f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.579+0000 7f99b4a00640 1 --2- 172.21.15.29:0/2838061636 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f99b00c8f10 0x7f99b01358f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.579+0000 7f99b4a00640 1 -- 172.21.15.29:0/2838061636 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99b01531b0 msgr2=0x7f99b0135e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.579+0000 7f99b4a00640 1 --2- 172.21.15.29:0/2838061636 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99b01531b0 0x7f99b0135e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.579+0000 7f99b4a00640 1 -- 172.21.15.29:0/2838061636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b014af00 con 0x7f99b00c5640 2024-04-17T19:50:43.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.579+0000 7f99afe00640 1 --2- 172.21.15.29:0/2838061636 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f99b00c8f10 0x7f99b01358f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:50:43.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.579+0000 7f99b4a00640 1 --2- 172.21.15.29:0/2838061636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b00c5640 0x7f99b0141830 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f99a400d5f0 tx=0x7f99a400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:43.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.579+0000 7f99acc00640 1 -- 172.21.15.29:0/2838061636 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a40125f0 con 0x7f99b00c5640 2024-04-17T19:50:43.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.579+0000 7f99acc00640 1 -- 172.21.15.29:0/2838061636 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99a4019070 con 0x7f99b00c5640 2024-04-17T19:50:43.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.579+0000 7f99b7400640 1 -- 172.21.15.29:0/2838061636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99b014b430 con 0x7f99b00c5640 2024-04-17T19:50:43.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.579+0000 7f99acc00640 1 -- 172.21.15.29:0/2838061636 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a40129b0 con 0x7f99b00c5640 2024-04-17T19:50:43.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.579+0000 7f99b7400640 1 -- 172.21.15.29:0/2838061636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99b014b9d0 con 0x7f99b00c5640 2024-04-17T19:50:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.579+0000 7f99b7400640 1 -- 172.21.15.29:0/2838061636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99b0069b90 con 0x7f99b00c5640 2024-04-17T19:50:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.579+0000 7f99acc00640 1 -- 172.21.15.29:0/2838061636 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f99a4012b50 con 0x7f99b00c5640 2024-04-17T19:50:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.579+0000 7f99acc00640 1 --2- 172.21.15.29:0/2838061636 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f99900683b0 0x7f999006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:50:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.579+0000 7f99acc00640 1 -- 172.21.15.29:0/2838061636 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f99a40871d0 con 0x7f99b00c5640 2024-04-17T19:50:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.579+0000 7f99afe00640 1 --2- 172.21.15.29:0/2838061636 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f99900683b0 0x7f999006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:50:43.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.583+0000 7f99afe00640 1 --2- 172.21.15.29:0/2838061636 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f99900683b0 0x7f999006a8c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f99b0136820 tx=0x7f999c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:50:43.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.587+0000 7f99acc00640 1 -- 172.21.15.29:0/2838061636 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f99a4057200 con 0x7f99b00c5640 2024-04-17T19:50:43.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.787+0000 7f99b7400640 1 -- 172.21.15.29:0/2838061636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99b00c8cf0 con 0x7f99b00c5640 2024-04-17T19:50:43.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.787+0000 7f99acc00640 1 -- 172.21.15.29:0/2838061636 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6904 (secure 0 0 0) 0x7f99a4056b00 con 0x7f99b00c5640 2024-04-17T19:50:43.779 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:50:43.779 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5212,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:34.757701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:active","state_seq":47,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229,4223]},"id":1}]} 2024-04-17T19:50:43.779 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2024-04-17T19:50:43.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.787+0000 7f99b7400640 1 -- 172.21.15.29:0/2838061636 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f99900683b0 msgr2=0x7f999006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:43.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.787+0000 7f99b7400640 1 --2- 172.21.15.29:0/2838061636 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f99900683b0 0x7f999006a8c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f99b0136820 tx=0x7f999c009040 comp rx=0 tx=0).stop 2024-04-17T19:50:43.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.787+0000 7f99b7400640 1 -- 172.21.15.29:0/2838061636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b00c5640 msgr2=0x7f99b0141830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:50:43.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.787+0000 7f99b7400640 1 --2- 172.21.15.29:0/2838061636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b00c5640 0x7f99b0141830 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f99a400d5f0 tx=0x7f99a400dab0 comp rx=0 tx=0).stop 2024-04-17T19:50:43.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.787+0000 7f99b7400640 1 -- 172.21.15.29:0/2838061636 shutdown_connections 2024-04-17T19:50:43.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.787+0000 7f99b7400640 1 --2- 172.21.15.29:0/2838061636 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f99900683b0 0x7f999006a8c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:43.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.787+0000 7f99b7400640 1 --2- 172.21.15.29:0/2838061636 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99b01531b0 0x7f99b0135e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:43.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.787+0000 7f99b7400640 1 --2- 172.21.15.29:0/2838061636 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f99b00c8f10 0x7f99b01358f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:43.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.787+0000 7f99b7400640 1 --2- 172.21.15.29:0/2838061636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b00c5640 0x7f99b0141830 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:50:43.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.787+0000 7f99b7400640 1 -- 172.21.15.29:0/2838061636 >> 172.21.15.29:0/2838061636 conn(0x7f99b0072e50 msgr2=0x7f99b00cc260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:50:43.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.787+0000 7f99b7400640 1 -- 172.21.15.29:0/2838061636 shutdown_connections 2024-04-17T19:50:43.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:50:43.787+0000 7f99b7400640 1 -- 172.21.15.29:0/2838061636 wait complete. 2024-04-17T19:50:43.801 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2024-04-17T19:50:43.801 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 74 secs before thrashing 2024-04-17T19:50:44.972 INFO:tasks.workunit.client.0.smithi029.stdout:Create dir 100 to 199 ... 2024-04-17T19:50:44.974 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 100 2024-04-17T19:50:44.988 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 101 2024-04-17T19:50:45.001 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 102 2024-04-17T19:50:45.015 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 103 2024-04-17T19:50:45.029 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 104 2024-04-17T19:50:45.042 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 105 2024-04-17T19:50:45.056 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 106 2024-04-17T19:50:45.075 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 107 2024-04-17T19:50:45.096 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 108 2024-04-17T19:50:45.111 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 109 2024-04-17T19:50:45.124 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 110 2024-04-17T19:50:45.138 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 111 2024-04-17T19:50:45.813 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 112 2024-04-17T19:50:45.833 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 113 2024-04-17T19:50:45.852 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 114 2024-04-17T19:50:46.788 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 115 2024-04-17T19:50:46.803 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 116 2024-04-17T19:50:46.818 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 117 2024-04-17T19:50:46.832 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 118 2024-04-17T19:50:46.846 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 119 2024-04-17T19:50:46.860 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 120 2024-04-17T19:50:46.874 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 121 2024-04-17T19:50:46.887 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 122 2024-04-17T19:50:46.901 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 123 2024-04-17T19:50:46.994 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 124 2024-04-17T19:50:47.008 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 125 2024-04-17T19:50:47.021 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 126 2024-04-17T19:50:47.035 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 127 2024-04-17T19:50:47.049 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 128 2024-04-17T19:50:47.062 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 129 2024-04-17T19:50:47.076 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 130 2024-04-17T19:50:47.090 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 131 2024-04-17T19:50:47.104 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 132 2024-04-17T19:50:47.118 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 133 2024-04-17T19:50:47.133 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 134 2024-04-17T19:50:47.146 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 135 2024-04-17T19:50:47.160 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 136 2024-04-17T19:50:47.174 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 137 2024-04-17T19:50:47.188 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 138 2024-04-17T19:50:47.202 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 139 2024-04-17T19:50:47.216 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 140 2024-04-17T19:50:47.229 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 141 2024-04-17T19:50:47.243 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 142 2024-04-17T19:50:47.257 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 143 2024-04-17T19:50:47.271 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 144 2024-04-17T19:50:47.284 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 145 2024-04-17T19:50:47.298 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 146 2024-04-17T19:50:47.894 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 147 2024-04-17T19:50:47.909 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 148 2024-04-17T19:50:47.924 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 149 2024-04-17T19:50:47.938 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 150 2024-04-17T19:50:47.955 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 151 2024-04-17T19:50:47.974 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 152 2024-04-17T19:50:47.993 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 153 2024-04-17T19:50:48.012 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 154 2024-04-17T19:50:48.032 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 155 2024-04-17T19:50:48.051 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 156 2024-04-17T19:50:48.070 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 157 2024-04-17T19:50:48.089 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 158 2024-04-17T19:50:48.108 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 159 2024-04-17T19:50:48.336 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 160 2024-04-17T19:50:48.356 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 161 2024-04-17T19:50:48.376 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 162 2024-04-17T19:50:48.395 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 163 2024-04-17T19:50:48.409 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 164 2024-04-17T19:50:48.585 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:50:48.589 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:50:49.306 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 165 2024-04-17T19:50:49.321 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 166 2024-04-17T19:50:49.335 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 167 2024-04-17T19:50:49.350 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 168 2024-04-17T19:50:49.364 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 169 2024-04-17T19:50:49.386 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 170 2024-04-17T19:50:49.401 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 171 2024-04-17T19:50:49.415 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 172 2024-04-17T19:50:49.429 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 173 2024-04-17T19:50:49.442 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 174 2024-04-17T19:50:49.710 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 175 2024-04-17T19:50:49.724 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 176 2024-04-17T19:50:49.738 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 177 2024-04-17T19:50:49.752 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 178 2024-04-17T19:50:49.765 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 179 2024-04-17T19:50:50.960 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 180 2024-04-17T19:50:50.982 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 181 2024-04-17T19:50:51.002 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 182 2024-04-17T19:50:51.023 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 183 2024-04-17T19:50:51.043 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 184 2024-04-17T19:50:51.063 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 185 2024-04-17T19:50:51.083 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 186 2024-04-17T19:50:51.103 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 187 2024-04-17T19:50:51.122 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 188 2024-04-17T19:50:51.142 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 189 2024-04-17T19:50:51.155 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 190 2024-04-17T19:50:51.169 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 191 2024-04-17T19:50:51.185 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 192 2024-04-17T19:50:51.205 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 193 2024-04-17T19:50:51.224 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 194 2024-04-17T19:50:51.243 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 195 2024-04-17T19:50:51.263 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 196 2024-04-17T19:50:51.282 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 197 2024-04-17T19:50:51.297 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 198 2024-04-17T19:50:51.310 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 199 2024-04-17T19:50:51.324 INFO:tasks.workunit.client.0.smithi029.stdout:Take first snapshot .snap/test1 2024-04-17T19:50:51.702 INFO:tasks.workunit.client.0.smithi029.stdout:Create dir 200 to 299 ... 2024-04-17T19:50:51.704 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 200 2024-04-17T19:50:51.718 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 201 2024-04-17T19:50:51.731 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 202 2024-04-17T19:50:51.745 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 203 2024-04-17T19:50:51.757 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 204 2024-04-17T19:50:51.969 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 205 2024-04-17T19:50:52.927 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 206 2024-04-17T19:50:53.195 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 207 2024-04-17T19:50:53.211 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 208 2024-04-17T19:50:53.477 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 209 2024-04-17T19:50:53.493 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 210 2024-04-17T19:50:53.508 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 211 2024-04-17T19:50:53.522 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 212 2024-04-17T19:50:53.536 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 213 2024-04-17T19:50:53.550 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 214 2024-04-17T19:50:53.565 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 215 2024-04-17T19:50:53.578 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 216 2024-04-17T19:50:53.591 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 217 2024-04-17T19:50:53.604 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 218 2024-04-17T19:50:53.617 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 219 2024-04-17T19:50:54.103 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 220 2024-04-17T19:50:54.122 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 221 2024-04-17T19:50:54.141 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 222 2024-04-17T19:50:54.159 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 223 2024-04-17T19:50:54.178 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 224 2024-04-17T19:50:54.196 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 225 2024-04-17T19:50:54.215 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 226 2024-04-17T19:50:54.233 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 227 2024-04-17T19:50:54.252 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 228 2024-04-17T19:50:54.268 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 229 2024-04-17T19:50:54.282 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 230 2024-04-17T19:50:54.295 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 231 2024-04-17T19:50:54.311 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 232 2024-04-17T19:50:54.330 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 233 2024-04-17T19:50:54.544 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 234 2024-04-17T19:50:54.558 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 235 2024-04-17T19:50:54.571 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 236 2024-04-17T19:50:54.585 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 237 2024-04-17T19:50:54.598 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 238 2024-04-17T19:50:54.612 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 239 2024-04-17T19:50:54.630 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 240 2024-04-17T19:50:54.648 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 241 2024-04-17T19:50:54.666 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 242 2024-04-17T19:50:54.681 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 243 2024-04-17T19:50:54.694 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 244 2024-04-17T19:50:54.707 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 245 2024-04-17T19:50:54.720 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 246 2024-04-17T19:50:54.737 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 247 2024-04-17T19:50:54.755 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 248 2024-04-17T19:50:54.767 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 249 2024-04-17T19:50:54.780 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 250 2024-04-17T19:50:54.797 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 251 2024-04-17T19:50:55.948 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 252 2024-04-17T19:50:55.964 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 253 2024-04-17T19:50:55.978 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 254 2024-04-17T19:50:55.993 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 255 2024-04-17T19:50:56.007 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 256 2024-04-17T19:50:56.352 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 257 2024-04-17T19:50:56.372 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 258 2024-04-17T19:50:56.391 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 259 2024-04-17T19:50:56.410 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 260 2024-04-17T19:50:56.429 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 261 2024-04-17T19:50:56.447 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 262 2024-04-17T19:50:56.466 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 263 2024-04-17T19:50:56.481 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 264 2024-04-17T19:50:56.494 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 265 2024-04-17T19:50:56.507 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 266 2024-04-17T19:50:56.520 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 267 2024-04-17T19:50:56.534 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 268 2024-04-17T19:50:56.549 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 269 2024-04-17T19:50:56.564 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 270 2024-04-17T19:50:56.580 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 271 2024-04-17T19:50:56.596 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 272 2024-04-17T19:50:56.611 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 273 2024-04-17T19:50:56.627 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 274 2024-04-17T19:50:56.643 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 275 2024-04-17T19:50:56.658 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 276 2024-04-17T19:50:56.671 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 277 2024-04-17T19:50:56.684 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 278 2024-04-17T19:50:56.696 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 279 2024-04-17T19:50:56.709 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 280 2024-04-17T19:50:56.722 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 281 2024-04-17T19:50:56.734 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 282 2024-04-17T19:50:56.748 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 283 2024-04-17T19:50:56.761 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 284 2024-04-17T19:50:56.774 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 285 2024-04-17T19:50:56.787 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 286 2024-04-17T19:50:56.805 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 287 2024-04-17T19:50:56.823 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 288 2024-04-17T19:50:56.842 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 289 2024-04-17T19:50:56.856 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 290 2024-04-17T19:50:56.869 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 291 2024-04-17T19:50:56.882 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 292 2024-04-17T19:50:56.895 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 293 2024-04-17T19:50:56.909 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 294 2024-04-17T19:50:56.921 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 295 2024-04-17T19:50:56.934 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 296 2024-04-17T19:50:56.947 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 297 2024-04-17T19:50:56.963 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 298 2024-04-17T19:50:56.976 INFO:tasks.workunit.client.0.smithi029.stdout: create dir 299 2024-04-17T19:50:56.989 INFO:tasks.workunit.client.0.smithi029.stdout:Create a snapshot in every first level dir ... 2024-04-17T19:50:57.634 INFO:tasks.workunit.client.0.smithi029.stdout: create 100/.snap/snap-subdir-test 2024-04-17T19:50:57.651 INFO:tasks.workunit.client.0.smithi029.stdout: create 100/30 file after the snapshot 2024-04-17T19:50:57.653 INFO:tasks.workunit.client.0.smithi029.stdout: create 100/31 file after the snapshot 2024-04-17T19:50:57.654 INFO:tasks.workunit.client.0.smithi029.stdout: create 100/32 file after the snapshot 2024-04-17T19:50:57.656 INFO:tasks.workunit.client.0.smithi029.stdout: create 100/33 file after the snapshot 2024-04-17T19:50:57.662 INFO:tasks.workunit.client.0.smithi029.stdout: create 100/34 file after the snapshot 2024-04-17T19:50:57.671 INFO:tasks.workunit.client.0.smithi029.stdout: create 100/35 file after the snapshot 2024-04-17T19:50:57.672 INFO:tasks.workunit.client.0.smithi029.stdout: create 100/36 file after the snapshot 2024-04-17T19:50:57.673 INFO:tasks.workunit.client.0.smithi029.stdout: create 100/37 file after the snapshot 2024-04-17T19:50:57.674 INFO:tasks.workunit.client.0.smithi029.stdout: create 100/38 file after the snapshot 2024-04-17T19:50:57.676 INFO:tasks.workunit.client.0.smithi029.stdout: create 100/39 file after the snapshot 2024-04-17T19:50:57.677 INFO:tasks.workunit.client.0.smithi029.stdout: create 101/.snap/snap-subdir-test 2024-04-17T19:50:57.710 INFO:tasks.workunit.client.0.smithi029.stdout: create 101/30 file after the snapshot 2024-04-17T19:50:57.711 INFO:tasks.workunit.client.0.smithi029.stdout: create 101/31 file after the snapshot 2024-04-17T19:50:57.713 INFO:tasks.workunit.client.0.smithi029.stdout: create 101/32 file after the snapshot 2024-04-17T19:50:57.716 INFO:tasks.workunit.client.0.smithi029.stdout: create 101/33 file after the snapshot 2024-04-17T19:50:57.722 INFO:tasks.workunit.client.0.smithi029.stdout: create 101/34 file after the snapshot 2024-04-17T19:50:57.728 INFO:tasks.workunit.client.0.smithi029.stdout: create 101/35 file after the snapshot 2024-04-17T19:50:57.730 INFO:tasks.workunit.client.0.smithi029.stdout: create 101/36 file after the snapshot 2024-04-17T19:50:57.730 INFO:tasks.workunit.client.0.smithi029.stdout: create 101/37 file after the snapshot 2024-04-17T19:50:57.731 INFO:tasks.workunit.client.0.smithi029.stdout: create 101/38 file after the snapshot 2024-04-17T19:50:57.732 INFO:tasks.workunit.client.0.smithi029.stdout: create 101/39 file after the snapshot 2024-04-17T19:50:57.733 INFO:tasks.workunit.client.0.smithi029.stdout: create 102/.snap/snap-subdir-test 2024-04-17T19:50:57.757 INFO:tasks.workunit.client.0.smithi029.stdout: create 102/30 file after the snapshot 2024-04-17T19:50:57.759 INFO:tasks.workunit.client.0.smithi029.stdout: create 102/31 file after the snapshot 2024-04-17T19:50:57.762 INFO:tasks.workunit.client.0.smithi029.stdout: create 102/32 file after the snapshot 2024-04-17T19:50:57.767 INFO:tasks.workunit.client.0.smithi029.stdout: create 102/33 file after the snapshot 2024-04-17T19:50:57.775 INFO:tasks.workunit.client.0.smithi029.stdout: create 102/34 file after the snapshot 2024-04-17T19:50:57.776 INFO:tasks.workunit.client.0.smithi029.stdout: create 102/35 file after the snapshot 2024-04-17T19:50:57.777 INFO:tasks.workunit.client.0.smithi029.stdout: create 102/36 file after the snapshot 2024-04-17T19:50:57.777 INFO:tasks.workunit.client.0.smithi029.stdout: create 102/37 file after the snapshot 2024-04-17T19:50:57.779 INFO:tasks.workunit.client.0.smithi029.stdout: create 102/38 file after the snapshot 2024-04-17T19:50:57.779 INFO:tasks.workunit.client.0.smithi029.stdout: create 102/39 file after the snapshot 2024-04-17T19:50:57.780 INFO:tasks.workunit.client.0.smithi029.stdout: create 103/.snap/snap-subdir-test 2024-04-17T19:50:57.812 INFO:tasks.workunit.client.0.smithi029.stdout: create 103/30 file after the snapshot 2024-04-17T19:50:57.815 INFO:tasks.workunit.client.0.smithi029.stdout: create 103/31 file after the snapshot 2024-04-17T19:50:57.817 INFO:tasks.workunit.client.0.smithi029.stdout: create 103/32 file after the snapshot 2024-04-17T19:50:57.822 INFO:tasks.workunit.client.0.smithi029.stdout: create 103/33 file after the snapshot 2024-04-17T19:50:57.828 INFO:tasks.workunit.client.0.smithi029.stdout: create 103/34 file after the snapshot 2024-04-17T19:50:57.830 INFO:tasks.workunit.client.0.smithi029.stdout: create 103/35 file after the snapshot 2024-04-17T19:50:57.831 INFO:tasks.workunit.client.0.smithi029.stdout: create 103/36 file after the snapshot 2024-04-17T19:50:57.832 INFO:tasks.workunit.client.0.smithi029.stdout: create 103/37 file after the snapshot 2024-04-17T19:50:57.833 INFO:tasks.workunit.client.0.smithi029.stdout: create 103/38 file after the snapshot 2024-04-17T19:50:57.834 INFO:tasks.workunit.client.0.smithi029.stdout: create 103/39 file after the snapshot 2024-04-17T19:50:57.835 INFO:tasks.workunit.client.0.smithi029.stdout: create 104/.snap/snap-subdir-test 2024-04-17T19:50:57.857 INFO:tasks.workunit.client.0.smithi029.stdout: create 104/30 file after the snapshot 2024-04-17T19:50:57.859 INFO:tasks.workunit.client.0.smithi029.stdout: create 104/31 file after the snapshot 2024-04-17T19:50:58.112 INFO:tasks.workunit.client.0.smithi029.stdout: create 104/32 file after the snapshot 2024-04-17T19:50:58.120 INFO:tasks.workunit.client.0.smithi029.stdout: create 104/33 file after the snapshot 2024-04-17T19:50:58.121 INFO:tasks.workunit.client.0.smithi029.stdout: create 104/34 file after the snapshot 2024-04-17T19:50:58.122 INFO:tasks.workunit.client.0.smithi029.stdout: create 104/35 file after the snapshot 2024-04-17T19:50:58.123 INFO:tasks.workunit.client.0.smithi029.stdout: create 104/36 file after the snapshot 2024-04-17T19:50:58.124 INFO:tasks.workunit.client.0.smithi029.stdout: create 104/37 file after the snapshot 2024-04-17T19:50:58.125 INFO:tasks.workunit.client.0.smithi029.stdout: create 104/38 file after the snapshot 2024-04-17T19:50:58.126 INFO:tasks.workunit.client.0.smithi029.stdout: create 104/39 file after the snapshot 2024-04-17T19:50:58.127 INFO:tasks.workunit.client.0.smithi029.stdout: create 105/.snap/snap-subdir-test 2024-04-17T19:50:58.151 INFO:tasks.workunit.client.0.smithi029.stdout: create 105/30 file after the snapshot 2024-04-17T19:50:58.152 INFO:tasks.workunit.client.0.smithi029.stdout: create 105/31 file after the snapshot 2024-04-17T19:50:58.156 INFO:tasks.workunit.client.0.smithi029.stdout: create 105/32 file after the snapshot 2024-04-17T19:50:58.159 INFO:tasks.workunit.client.0.smithi029.stdout: create 105/33 file after the snapshot 2024-04-17T19:50:58.164 INFO:tasks.workunit.client.0.smithi029.stdout: create 105/34 file after the snapshot 2024-04-17T19:50:58.169 INFO:tasks.workunit.client.0.smithi029.stdout: create 105/35 file after the snapshot 2024-04-17T19:50:58.169 INFO:tasks.workunit.client.0.smithi029.stdout: create 105/36 file after the snapshot 2024-04-17T19:50:58.170 INFO:tasks.workunit.client.0.smithi029.stdout: create 105/37 file after the snapshot 2024-04-17T19:50:58.171 INFO:tasks.workunit.client.0.smithi029.stdout: create 105/38 file after the snapshot 2024-04-17T19:50:58.172 INFO:tasks.workunit.client.0.smithi029.stdout: create 105/39 file after the snapshot 2024-04-17T19:50:58.173 INFO:tasks.workunit.client.0.smithi029.stdout: create 106/.snap/snap-subdir-test 2024-04-17T19:50:58.204 INFO:tasks.workunit.client.0.smithi029.stdout: create 106/30 file after the snapshot 2024-04-17T19:50:58.207 INFO:tasks.workunit.client.0.smithi029.stdout: create 106/31 file after the snapshot 2024-04-17T19:50:58.209 INFO:tasks.workunit.client.0.smithi029.stdout: create 106/32 file after the snapshot 2024-04-17T19:50:58.213 INFO:tasks.workunit.client.0.smithi029.stdout: create 106/33 file after the snapshot 2024-04-17T19:50:58.221 INFO:tasks.workunit.client.0.smithi029.stdout: create 106/34 file after the snapshot 2024-04-17T19:50:58.222 INFO:tasks.workunit.client.0.smithi029.stdout: create 106/35 file after the snapshot 2024-04-17T19:50:58.223 INFO:tasks.workunit.client.0.smithi029.stdout: create 106/36 file after the snapshot 2024-04-17T19:50:58.224 INFO:tasks.workunit.client.0.smithi029.stdout: create 106/37 file after the snapshot 2024-04-17T19:50:58.225 INFO:tasks.workunit.client.0.smithi029.stdout: create 106/38 file after the snapshot 2024-04-17T19:50:58.226 INFO:tasks.workunit.client.0.smithi029.stdout: create 106/39 file after the snapshot 2024-04-17T19:50:58.227 INFO:tasks.workunit.client.0.smithi029.stdout: create 107/.snap/snap-subdir-test 2024-04-17T19:50:58.251 INFO:tasks.workunit.client.0.smithi029.stdout: create 107/30 file after the snapshot 2024-04-17T19:50:58.252 INFO:tasks.workunit.client.0.smithi029.stdout: create 107/31 file after the snapshot 2024-04-17T19:50:58.254 INFO:tasks.workunit.client.0.smithi029.stdout: create 107/32 file after the snapshot 2024-04-17T19:50:58.260 INFO:tasks.workunit.client.0.smithi029.stdout: create 107/33 file after the snapshot 2024-04-17T19:50:58.267 INFO:tasks.workunit.client.0.smithi029.stdout: create 107/34 file after the snapshot 2024-04-17T19:50:58.268 INFO:tasks.workunit.client.0.smithi029.stdout: create 107/35 file after the snapshot 2024-04-17T19:50:58.269 INFO:tasks.workunit.client.0.smithi029.stdout: create 107/36 file after the snapshot 2024-04-17T19:50:58.270 INFO:tasks.workunit.client.0.smithi029.stdout: create 107/37 file after the snapshot 2024-04-17T19:50:58.271 INFO:tasks.workunit.client.0.smithi029.stdout: create 107/38 file after the snapshot 2024-04-17T19:50:58.272 INFO:tasks.workunit.client.0.smithi029.stdout: create 107/39 file after the snapshot 2024-04-17T19:50:58.273 INFO:tasks.workunit.client.0.smithi029.stdout: create 108/.snap/snap-subdir-test 2024-04-17T19:50:58.296 INFO:tasks.workunit.client.0.smithi029.stdout: create 108/30 file after the snapshot 2024-04-17T19:50:58.297 INFO:tasks.workunit.client.0.smithi029.stdout: create 108/31 file after the snapshot 2024-04-17T19:50:58.299 INFO:tasks.workunit.client.0.smithi029.stdout: create 108/32 file after the snapshot 2024-04-17T19:50:58.301 INFO:tasks.workunit.client.0.smithi029.stdout: create 108/33 file after the snapshot 2024-04-17T19:50:58.303 INFO:tasks.workunit.client.0.smithi029.stdout: create 108/34 file after the snapshot 2024-04-17T19:50:58.309 INFO:tasks.workunit.client.0.smithi029.stdout: create 108/35 file after the snapshot 2024-04-17T19:50:58.320 INFO:tasks.workunit.client.0.smithi029.stdout: create 108/36 file after the snapshot 2024-04-17T19:50:58.321 INFO:tasks.workunit.client.0.smithi029.stdout: create 108/37 file after the snapshot 2024-04-17T19:50:58.323 INFO:tasks.workunit.client.0.smithi029.stdout: create 108/38 file after the snapshot 2024-04-17T19:50:58.324 INFO:tasks.workunit.client.0.smithi029.stdout: create 108/39 file after the snapshot 2024-04-17T19:50:58.326 INFO:tasks.workunit.client.0.smithi029.stdout: create 109/.snap/snap-subdir-test 2024-04-17T19:50:58.359 INFO:tasks.workunit.client.0.smithi029.stdout: create 109/30 file after the snapshot 2024-04-17T19:50:58.360 INFO:tasks.workunit.client.0.smithi029.stdout: create 109/31 file after the snapshot 2024-04-17T19:50:58.362 INFO:tasks.workunit.client.0.smithi029.stdout: create 109/32 file after the snapshot 2024-04-17T19:50:58.366 INFO:tasks.workunit.client.0.smithi029.stdout: create 109/33 file after the snapshot 2024-04-17T19:50:58.374 INFO:tasks.workunit.client.0.smithi029.stdout: create 109/34 file after the snapshot 2024-04-17T19:50:58.377 INFO:tasks.workunit.client.0.smithi029.stdout: create 109/35 file after the snapshot 2024-04-17T19:50:58.378 INFO:tasks.workunit.client.0.smithi029.stdout: create 109/36 file after the snapshot 2024-04-17T19:50:58.379 INFO:tasks.workunit.client.0.smithi029.stdout: create 109/37 file after the snapshot 2024-04-17T19:50:58.380 INFO:tasks.workunit.client.0.smithi029.stdout: create 109/38 file after the snapshot 2024-04-17T19:50:58.381 INFO:tasks.workunit.client.0.smithi029.stdout: create 109/39 file after the snapshot 2024-04-17T19:50:58.381 INFO:tasks.workunit.client.0.smithi029.stdout: create 110/.snap/snap-subdir-test 2024-04-17T19:50:58.410 INFO:tasks.workunit.client.0.smithi029.stdout: create 110/30 file after the snapshot 2024-04-17T19:50:58.411 INFO:tasks.workunit.client.0.smithi029.stdout: create 110/31 file after the snapshot 2024-04-17T19:50:58.412 INFO:tasks.workunit.client.0.smithi029.stdout: create 110/32 file after the snapshot 2024-04-17T19:50:58.413 INFO:tasks.workunit.client.0.smithi029.stdout: create 110/33 file after the snapshot 2024-04-17T19:50:58.416 INFO:tasks.workunit.client.0.smithi029.stdout: create 110/34 file after the snapshot 2024-04-17T19:50:58.426 INFO:tasks.workunit.client.0.smithi029.stdout: create 110/35 file after the snapshot 2024-04-17T19:50:58.436 INFO:tasks.workunit.client.0.smithi029.stdout: create 110/36 file after the snapshot 2024-04-17T19:50:58.437 INFO:tasks.workunit.client.0.smithi029.stdout: create 110/37 file after the snapshot 2024-04-17T19:50:58.438 INFO:tasks.workunit.client.0.smithi029.stdout: create 110/38 file after the snapshot 2024-04-17T19:50:58.440 INFO:tasks.workunit.client.0.smithi029.stdout: create 110/39 file after the snapshot 2024-04-17T19:50:58.441 INFO:tasks.workunit.client.0.smithi029.stdout: create 111/.snap/snap-subdir-test 2024-04-17T19:50:58.465 INFO:tasks.workunit.client.0.smithi029.stdout: create 111/30 file after the snapshot 2024-04-17T19:50:58.466 INFO:tasks.workunit.client.0.smithi029.stdout: create 111/31 file after the snapshot 2024-04-17T19:50:58.467 INFO:tasks.workunit.client.0.smithi029.stdout: create 111/32 file after the snapshot 2024-04-17T19:50:58.469 INFO:tasks.workunit.client.0.smithi029.stdout: create 111/33 file after the snapshot 2024-04-17T19:50:58.474 INFO:tasks.workunit.client.0.smithi029.stdout: create 111/34 file after the snapshot 2024-04-17T19:50:58.489 INFO:tasks.workunit.client.0.smithi029.stdout: create 111/35 file after the snapshot 2024-04-17T19:50:58.491 INFO:tasks.workunit.client.0.smithi029.stdout: create 111/36 file after the snapshot 2024-04-17T19:50:58.492 INFO:tasks.workunit.client.0.smithi029.stdout: create 111/37 file after the snapshot 2024-04-17T19:50:58.493 INFO:tasks.workunit.client.0.smithi029.stdout: create 111/38 file after the snapshot 2024-04-17T19:50:58.495 INFO:tasks.workunit.client.0.smithi029.stdout: create 111/39 file after the snapshot 2024-04-17T19:50:58.496 INFO:tasks.workunit.client.0.smithi029.stdout: create 112/.snap/snap-subdir-test 2024-04-17T19:50:58.527 INFO:tasks.workunit.client.0.smithi029.stdout: create 112/30 file after the snapshot 2024-04-17T19:50:58.779 INFO:tasks.workunit.client.0.smithi029.stdout: create 112/31 file after the snapshot 2024-04-17T19:50:58.789 INFO:tasks.workunit.client.0.smithi029.stdout: create 112/32 file after the snapshot 2024-04-17T19:50:58.790 INFO:tasks.workunit.client.0.smithi029.stdout: create 112/33 file after the snapshot 2024-04-17T19:50:58.791 INFO:tasks.workunit.client.0.smithi029.stdout: create 112/34 file after the snapshot 2024-04-17T19:50:58.792 INFO:tasks.workunit.client.0.smithi029.stdout: create 112/35 file after the snapshot 2024-04-17T19:50:58.793 INFO:tasks.workunit.client.0.smithi029.stdout: create 112/36 file after the snapshot 2024-04-17T19:50:58.794 INFO:tasks.workunit.client.0.smithi029.stdout: create 112/37 file after the snapshot 2024-04-17T19:50:58.795 INFO:tasks.workunit.client.0.smithi029.stdout: create 112/38 file after the snapshot 2024-04-17T19:50:58.796 INFO:tasks.workunit.client.0.smithi029.stdout: create 112/39 file after the snapshot 2024-04-17T19:50:58.797 INFO:tasks.workunit.client.0.smithi029.stdout: create 113/.snap/snap-subdir-test 2024-04-17T19:50:58.821 INFO:tasks.workunit.client.0.smithi029.stdout: create 113/30 file after the snapshot 2024-04-17T19:50:58.824 INFO:tasks.workunit.client.0.smithi029.stdout: create 113/31 file after the snapshot 2024-04-17T19:50:58.829 INFO:tasks.workunit.client.0.smithi029.stdout: create 113/32 file after the snapshot 2024-04-17T19:50:58.836 INFO:tasks.workunit.client.0.smithi029.stdout: create 113/33 file after the snapshot 2024-04-17T19:50:58.844 INFO:tasks.workunit.client.0.smithi029.stdout: create 113/34 file after the snapshot 2024-04-17T19:50:58.845 INFO:tasks.workunit.client.0.smithi029.stdout: create 113/35 file after the snapshot 2024-04-17T19:50:58.846 INFO:tasks.workunit.client.0.smithi029.stdout: create 113/36 file after the snapshot 2024-04-17T19:50:58.848 INFO:tasks.workunit.client.0.smithi029.stdout: create 113/37 file after the snapshot 2024-04-17T19:50:58.849 INFO:tasks.workunit.client.0.smithi029.stdout: create 113/38 file after the snapshot 2024-04-17T19:50:58.851 INFO:tasks.workunit.client.0.smithi029.stdout: create 113/39 file after the snapshot 2024-04-17T19:50:58.852 INFO:tasks.workunit.client.0.smithi029.stdout: create 114/.snap/snap-subdir-test 2024-04-17T19:50:58.877 INFO:tasks.workunit.client.0.smithi029.stdout: create 114/30 file after the snapshot 2024-04-17T19:50:58.878 INFO:tasks.workunit.client.0.smithi029.stdout: create 114/31 file after the snapshot 2024-04-17T19:50:58.881 INFO:tasks.workunit.client.0.smithi029.stdout: create 114/32 file after the snapshot 2024-04-17T19:50:58.883 INFO:tasks.workunit.client.0.smithi029.stdout: create 114/33 file after the snapshot 2024-04-17T19:50:58.888 INFO:tasks.workunit.client.0.smithi029.stdout: create 114/34 file after the snapshot 2024-04-17T19:50:58.894 INFO:tasks.workunit.client.0.smithi029.stdout: create 114/35 file after the snapshot 2024-04-17T19:50:58.896 INFO:tasks.workunit.client.0.smithi029.stdout: create 114/36 file after the snapshot 2024-04-17T19:50:58.897 INFO:tasks.workunit.client.0.smithi029.stdout: create 114/37 file after the snapshot 2024-04-17T19:50:58.898 INFO:tasks.workunit.client.0.smithi029.stdout: create 114/38 file after the snapshot 2024-04-17T19:50:58.899 INFO:tasks.workunit.client.0.smithi029.stdout: create 114/39 file after the snapshot 2024-04-17T19:50:58.899 INFO:tasks.workunit.client.0.smithi029.stdout: create 115/.snap/snap-subdir-test 2024-04-17T19:50:58.928 INFO:tasks.workunit.client.0.smithi029.stdout: create 115/30 file after the snapshot 2024-04-17T19:50:58.930 INFO:tasks.workunit.client.0.smithi029.stdout: create 115/31 file after the snapshot 2024-04-17T19:50:58.934 INFO:tasks.workunit.client.0.smithi029.stdout: create 115/32 file after the snapshot 2024-04-17T19:50:58.939 INFO:tasks.workunit.client.0.smithi029.stdout: create 115/33 file after the snapshot 2024-04-17T19:50:58.945 INFO:tasks.workunit.client.0.smithi029.stdout: create 115/34 file after the snapshot 2024-04-17T19:50:58.946 INFO:tasks.workunit.client.0.smithi029.stdout: create 115/35 file after the snapshot 2024-04-17T19:50:58.947 INFO:tasks.workunit.client.0.smithi029.stdout: create 115/36 file after the snapshot 2024-04-17T19:50:58.948 INFO:tasks.workunit.client.0.smithi029.stdout: create 115/37 file after the snapshot 2024-04-17T19:50:58.949 INFO:tasks.workunit.client.0.smithi029.stdout: create 115/38 file after the snapshot 2024-04-17T19:50:58.949 INFO:tasks.workunit.client.0.smithi029.stdout: create 115/39 file after the snapshot 2024-04-17T19:50:58.950 INFO:tasks.workunit.client.0.smithi029.stdout: create 116/.snap/snap-subdir-test 2024-04-17T19:50:58.974 INFO:tasks.workunit.client.0.smithi029.stdout: create 116/30 file after the snapshot 2024-04-17T19:50:58.976 INFO:tasks.workunit.client.0.smithi029.stdout: create 116/31 file after the snapshot 2024-04-17T19:50:58.978 INFO:tasks.workunit.client.0.smithi029.stdout: create 116/32 file after the snapshot 2024-04-17T19:50:58.981 INFO:tasks.workunit.client.0.smithi029.stdout: create 116/33 file after the snapshot 2024-04-17T19:50:58.986 INFO:tasks.workunit.client.0.smithi029.stdout: create 116/34 file after the snapshot 2024-04-17T19:50:58.991 INFO:tasks.workunit.client.0.smithi029.stdout: create 116/35 file after the snapshot 2024-04-17T19:50:59.275 INFO:tasks.workunit.client.0.smithi029.stdout: create 116/36 file after the snapshot 2024-04-17T19:50:59.276 INFO:tasks.workunit.client.0.smithi029.stdout: create 116/37 file after the snapshot 2024-04-17T19:50:59.277 INFO:tasks.workunit.client.0.smithi029.stdout: create 116/38 file after the snapshot 2024-04-17T19:50:59.278 INFO:tasks.workunit.client.0.smithi029.stdout: create 116/39 file after the snapshot 2024-04-17T19:50:59.279 INFO:tasks.workunit.client.0.smithi029.stdout: create 117/.snap/snap-subdir-test 2024-04-17T19:50:59.305 INFO:tasks.workunit.client.0.smithi029.stdout: create 117/30 file after the snapshot 2024-04-17T19:50:59.306 INFO:tasks.workunit.client.0.smithi029.stdout: create 117/31 file after the snapshot 2024-04-17T19:50:59.308 INFO:tasks.workunit.client.0.smithi029.stdout: create 117/32 file after the snapshot 2024-04-17T19:50:59.312 INFO:tasks.workunit.client.0.smithi029.stdout: create 117/33 file after the snapshot 2024-04-17T19:50:59.322 INFO:tasks.workunit.client.0.smithi029.stdout: create 117/34 file after the snapshot 2024-04-17T19:50:59.330 INFO:tasks.workunit.client.0.smithi029.stdout: create 117/35 file after the snapshot 2024-04-17T19:50:59.333 INFO:tasks.workunit.client.0.smithi029.stdout: create 117/36 file after the snapshot 2024-04-17T19:50:59.334 INFO:tasks.workunit.client.0.smithi029.stdout: create 117/37 file after the snapshot 2024-04-17T19:50:59.335 INFO:tasks.workunit.client.0.smithi029.stdout: create 117/38 file after the snapshot 2024-04-17T19:50:59.337 INFO:tasks.workunit.client.0.smithi029.stdout: create 117/39 file after the snapshot 2024-04-17T19:50:59.338 INFO:tasks.workunit.client.0.smithi029.stdout: create 118/.snap/snap-subdir-test 2024-04-17T19:50:59.365 INFO:tasks.workunit.client.0.smithi029.stdout: create 118/30 file after the snapshot 2024-04-17T19:50:59.367 INFO:tasks.workunit.client.0.smithi029.stdout: create 118/31 file after the snapshot 2024-04-17T19:50:59.368 INFO:tasks.workunit.client.0.smithi029.stdout: create 118/32 file after the snapshot 2024-04-17T19:50:59.373 INFO:tasks.workunit.client.0.smithi029.stdout: create 118/33 file after the snapshot 2024-04-17T19:50:59.382 INFO:tasks.workunit.client.0.smithi029.stdout: create 118/34 file after the snapshot 2024-04-17T19:50:59.383 INFO:tasks.workunit.client.0.smithi029.stdout: create 118/35 file after the snapshot 2024-04-17T19:50:59.384 INFO:tasks.workunit.client.0.smithi029.stdout: create 118/36 file after the snapshot 2024-04-17T19:50:59.385 INFO:tasks.workunit.client.0.smithi029.stdout: create 118/37 file after the snapshot 2024-04-17T19:50:59.386 INFO:tasks.workunit.client.0.smithi029.stdout: create 118/38 file after the snapshot 2024-04-17T19:50:59.387 INFO:tasks.workunit.client.0.smithi029.stdout: create 118/39 file after the snapshot 2024-04-17T19:50:59.388 INFO:tasks.workunit.client.0.smithi029.stdout: create 119/.snap/snap-subdir-test 2024-04-17T19:50:59.411 INFO:tasks.workunit.client.0.smithi029.stdout: create 119/30 file after the snapshot 2024-04-17T19:50:59.413 INFO:tasks.workunit.client.0.smithi029.stdout: create 119/31 file after the snapshot 2024-04-17T19:50:59.414 INFO:tasks.workunit.client.0.smithi029.stdout: create 119/32 file after the snapshot 2024-04-17T19:50:59.416 INFO:tasks.workunit.client.0.smithi029.stdout: create 119/33 file after the snapshot 2024-04-17T19:50:59.421 INFO:tasks.workunit.client.0.smithi029.stdout: create 119/34 file after the snapshot 2024-04-17T19:50:59.428 INFO:tasks.workunit.client.0.smithi029.stdout: create 119/35 file after the snapshot 2024-04-17T19:50:59.430 INFO:tasks.workunit.client.0.smithi029.stdout: create 119/36 file after the snapshot 2024-04-17T19:50:59.430 INFO:tasks.workunit.client.0.smithi029.stdout: create 119/37 file after the snapshot 2024-04-17T19:50:59.431 INFO:tasks.workunit.client.0.smithi029.stdout: create 119/38 file after the snapshot 2024-04-17T19:50:59.432 INFO:tasks.workunit.client.0.smithi029.stdout: create 119/39 file after the snapshot 2024-04-17T19:50:59.433 INFO:tasks.workunit.client.0.smithi029.stdout: create 120/.snap/snap-subdir-test 2024-04-17T19:50:59.456 INFO:tasks.workunit.client.0.smithi029.stdout: create 120/30 file after the snapshot 2024-04-17T19:50:59.458 INFO:tasks.workunit.client.0.smithi029.stdout: create 120/31 file after the snapshot 2024-04-17T19:50:59.459 INFO:tasks.workunit.client.0.smithi029.stdout: create 120/32 file after the snapshot 2024-04-17T19:50:59.461 INFO:tasks.workunit.client.0.smithi029.stdout: create 120/33 file after the snapshot 2024-04-17T19:50:59.462 INFO:tasks.workunit.client.0.smithi029.stdout: create 120/34 file after the snapshot 2024-04-17T19:50:59.466 INFO:tasks.workunit.client.0.smithi029.stdout: create 120/35 file after the snapshot 2024-04-17T19:50:59.479 INFO:tasks.workunit.client.0.smithi029.stdout: create 120/36 file after the snapshot 2024-04-17T19:50:59.485 INFO:tasks.workunit.client.0.smithi029.stdout: create 120/37 file after the snapshot 2024-04-17T19:50:59.487 INFO:tasks.workunit.client.0.smithi029.stdout: create 120/38 file after the snapshot 2024-04-17T19:50:59.488 INFO:tasks.workunit.client.0.smithi029.stdout: create 120/39 file after the snapshot 2024-04-17T19:50:59.490 INFO:tasks.workunit.client.0.smithi029.stdout: create 121/.snap/snap-subdir-test 2024-04-17T19:50:59.514 INFO:tasks.workunit.client.0.smithi029.stdout: create 121/30 file after the snapshot 2024-04-17T19:50:59.516 INFO:tasks.workunit.client.0.smithi029.stdout: create 121/31 file after the snapshot 2024-04-17T19:50:59.517 INFO:tasks.workunit.client.0.smithi029.stdout: create 121/32 file after the snapshot 2024-04-17T19:50:59.518 INFO:tasks.workunit.client.0.smithi029.stdout: create 121/33 file after the snapshot 2024-04-17T19:50:59.523 INFO:tasks.workunit.client.0.smithi029.stdout: create 121/34 file after the snapshot 2024-04-17T19:50:59.530 INFO:tasks.workunit.client.0.smithi029.stdout: create 121/35 file after the snapshot 2024-04-17T19:50:59.635 INFO:tasks.workunit.client.0.smithi029.stdout: create 121/36 file after the snapshot 2024-04-17T19:51:00.075 INFO:tasks.workunit.client.0.smithi029.stdout: create 121/37 file after the snapshot 2024-04-17T19:51:00.076 INFO:tasks.workunit.client.0.smithi029.stdout: create 121/38 file after the snapshot 2024-04-17T19:51:00.078 INFO:tasks.workunit.client.0.smithi029.stdout: create 121/39 file after the snapshot 2024-04-17T19:51:00.079 INFO:tasks.workunit.client.0.smithi029.stdout: create 122/.snap/snap-subdir-test 2024-04-17T19:51:00.098 INFO:tasks.workunit.client.0.smithi029.stdout: create 122/30 file after the snapshot 2024-04-17T19:51:00.100 INFO:tasks.workunit.client.0.smithi029.stdout: create 122/31 file after the snapshot 2024-04-17T19:51:00.102 INFO:tasks.workunit.client.0.smithi029.stdout: create 122/32 file after the snapshot 2024-04-17T19:51:00.103 INFO:tasks.workunit.client.0.smithi029.stdout: create 122/33 file after the snapshot 2024-04-17T19:51:00.108 INFO:tasks.workunit.client.0.smithi029.stdout: create 122/34 file after the snapshot 2024-04-17T19:51:00.116 INFO:tasks.workunit.client.0.smithi029.stdout: create 122/35 file after the snapshot 2024-04-17T19:51:00.627 INFO:tasks.workunit.client.0.smithi029.stdout: create 122/36 file after the snapshot 2024-04-17T19:51:00.682 INFO:tasks.workunit.client.0.smithi029.stdout: create 122/37 file after the snapshot 2024-04-17T19:51:00.683 INFO:tasks.workunit.client.0.smithi029.stdout: create 122/38 file after the snapshot 2024-04-17T19:51:00.684 INFO:tasks.workunit.client.0.smithi029.stdout: create 122/39 file after the snapshot 2024-04-17T19:51:00.685 INFO:tasks.workunit.client.0.smithi029.stdout: create 123/.snap/snap-subdir-test 2024-04-17T19:51:00.708 INFO:tasks.workunit.client.0.smithi029.stdout: create 123/30 file after the snapshot 2024-04-17T19:51:00.710 INFO:tasks.workunit.client.0.smithi029.stdout: create 123/31 file after the snapshot 2024-04-17T19:51:00.711 INFO:tasks.workunit.client.0.smithi029.stdout: create 123/32 file after the snapshot 2024-04-17T19:51:00.716 INFO:tasks.workunit.client.0.smithi029.stdout: create 123/33 file after the snapshot 2024-04-17T19:51:00.721 INFO:tasks.workunit.client.0.smithi029.stdout: create 123/34 file after the snapshot 2024-04-17T19:51:00.727 INFO:tasks.workunit.client.0.smithi029.stdout: create 123/35 file after the snapshot 2024-04-17T19:51:00.728 INFO:tasks.workunit.client.0.smithi029.stdout: create 123/36 file after the snapshot 2024-04-17T19:51:00.729 INFO:tasks.workunit.client.0.smithi029.stdout: create 123/37 file after the snapshot 2024-04-17T19:51:00.730 INFO:tasks.workunit.client.0.smithi029.stdout: create 123/38 file after the snapshot 2024-04-17T19:51:00.731 INFO:tasks.workunit.client.0.smithi029.stdout: create 123/39 file after the snapshot 2024-04-17T19:51:00.732 INFO:tasks.workunit.client.0.smithi029.stdout: create 124/.snap/snap-subdir-test 2024-04-17T19:51:00.762 INFO:tasks.workunit.client.0.smithi029.stdout: create 124/30 file after the snapshot 2024-04-17T19:51:00.764 INFO:tasks.workunit.client.0.smithi029.stdout: create 124/31 file after the snapshot 2024-04-17T19:51:00.765 INFO:tasks.workunit.client.0.smithi029.stdout: create 124/32 file after the snapshot 2024-04-17T19:51:00.767 INFO:tasks.workunit.client.0.smithi029.stdout: create 124/33 file after the snapshot 2024-04-17T19:51:00.772 INFO:tasks.workunit.client.0.smithi029.stdout: create 124/34 file after the snapshot 2024-04-17T19:51:00.779 INFO:tasks.workunit.client.0.smithi029.stdout: create 124/35 file after the snapshot 2024-04-17T19:51:01.319 INFO:tasks.workunit.client.0.smithi029.stdout: create 124/36 file after the snapshot 2024-04-17T19:51:01.320 INFO:tasks.workunit.client.0.smithi029.stdout: create 124/37 file after the snapshot 2024-04-17T19:51:01.321 INFO:tasks.workunit.client.0.smithi029.stdout: create 124/38 file after the snapshot 2024-04-17T19:51:01.322 INFO:tasks.workunit.client.0.smithi029.stdout: create 124/39 file after the snapshot 2024-04-17T19:51:01.323 INFO:tasks.workunit.client.0.smithi029.stdout: create 125/.snap/snap-subdir-test 2024-04-17T19:51:01.350 INFO:tasks.workunit.client.0.smithi029.stdout: create 125/30 file after the snapshot 2024-04-17T19:51:01.351 INFO:tasks.workunit.client.0.smithi029.stdout: create 125/31 file after the snapshot 2024-04-17T19:51:01.352 INFO:tasks.workunit.client.0.smithi029.stdout: create 125/32 file after the snapshot 2024-04-17T19:51:01.353 INFO:tasks.workunit.client.0.smithi029.stdout: create 125/33 file after the snapshot 2024-04-17T19:51:01.354 INFO:tasks.workunit.client.0.smithi029.stdout: create 125/34 file after the snapshot 2024-04-17T19:51:01.358 INFO:tasks.workunit.client.0.smithi029.stdout: create 125/35 file after the snapshot 2024-04-17T19:51:01.363 INFO:tasks.workunit.client.0.smithi029.stdout: create 125/36 file after the snapshot 2024-04-17T19:51:01.370 INFO:tasks.workunit.client.0.smithi029.stdout: create 125/37 file after the snapshot 2024-04-17T19:51:01.371 INFO:tasks.workunit.client.0.smithi029.stdout: create 125/38 file after the snapshot 2024-04-17T19:51:01.372 INFO:tasks.workunit.client.0.smithi029.stdout: create 125/39 file after the snapshot 2024-04-17T19:51:01.373 INFO:tasks.workunit.client.0.smithi029.stdout: create 126/.snap/snap-subdir-test 2024-04-17T19:51:01.402 INFO:tasks.workunit.client.0.smithi029.stdout: create 126/30 file after the snapshot 2024-04-17T19:51:01.404 INFO:tasks.workunit.client.0.smithi029.stdout: create 126/31 file after the snapshot 2024-04-17T19:51:01.407 INFO:tasks.workunit.client.0.smithi029.stdout: create 126/32 file after the snapshot 2024-04-17T19:51:01.413 INFO:tasks.workunit.client.0.smithi029.stdout: create 126/33 file after the snapshot 2024-04-17T19:51:01.419 INFO:tasks.workunit.client.0.smithi029.stdout: create 126/34 file after the snapshot 2024-04-17T19:51:01.420 INFO:tasks.workunit.client.0.smithi029.stdout: create 126/35 file after the snapshot 2024-04-17T19:51:01.421 INFO:tasks.workunit.client.0.smithi029.stdout: create 126/36 file after the snapshot 2024-04-17T19:51:01.422 INFO:tasks.workunit.client.0.smithi029.stdout: create 126/37 file after the snapshot 2024-04-17T19:51:01.423 INFO:tasks.workunit.client.0.smithi029.stdout: create 126/38 file after the snapshot 2024-04-17T19:51:01.424 INFO:tasks.workunit.client.0.smithi029.stdout: create 126/39 file after the snapshot 2024-04-17T19:51:01.425 INFO:tasks.workunit.client.0.smithi029.stdout: create 127/.snap/snap-subdir-test 2024-04-17T19:51:01.451 INFO:tasks.workunit.client.0.smithi029.stdout: create 127/30 file after the snapshot 2024-04-17T19:51:01.452 INFO:tasks.workunit.client.0.smithi029.stdout: create 127/31 file after the snapshot 2024-04-17T19:51:01.454 INFO:tasks.workunit.client.0.smithi029.stdout: create 127/32 file after the snapshot 2024-04-17T19:51:01.455 INFO:tasks.workunit.client.0.smithi029.stdout: create 127/33 file after the snapshot 2024-04-17T19:51:01.457 INFO:tasks.workunit.client.0.smithi029.stdout: create 127/34 file after the snapshot 2024-04-17T19:51:01.464 INFO:tasks.workunit.client.0.smithi029.stdout: create 127/35 file after the snapshot 2024-04-17T19:51:01.470 INFO:tasks.workunit.client.0.smithi029.stdout: create 127/36 file after the snapshot 2024-04-17T19:51:01.471 INFO:tasks.workunit.client.0.smithi029.stdout: create 127/37 file after the snapshot 2024-04-17T19:51:01.472 INFO:tasks.workunit.client.0.smithi029.stdout: create 127/38 file after the snapshot 2024-04-17T19:51:01.473 INFO:tasks.workunit.client.0.smithi029.stdout: create 127/39 file after the snapshot 2024-04-17T19:51:01.474 INFO:tasks.workunit.client.0.smithi029.stdout: create 128/.snap/snap-subdir-test 2024-04-17T19:51:01.502 INFO:tasks.workunit.client.0.smithi029.stdout: create 128/30 file after the snapshot 2024-04-17T19:51:01.755 INFO:tasks.workunit.client.0.smithi029.stdout: create 128/31 file after the snapshot 2024-04-17T19:51:02.508 INFO:tasks.workunit.client.0.smithi029.stdout: create 128/32 file after the snapshot 2024-04-17T19:51:02.509 INFO:tasks.workunit.client.0.smithi029.stdout: create 128/33 file after the snapshot 2024-04-17T19:51:02.510 INFO:tasks.workunit.client.0.smithi029.stdout: create 128/34 file after the snapshot 2024-04-17T19:51:02.511 INFO:tasks.workunit.client.0.smithi029.stdout: create 128/35 file after the snapshot 2024-04-17T19:51:02.512 INFO:tasks.workunit.client.0.smithi029.stdout: create 128/36 file after the snapshot 2024-04-17T19:51:02.513 INFO:tasks.workunit.client.0.smithi029.stdout: create 128/37 file after the snapshot 2024-04-17T19:51:02.515 INFO:tasks.workunit.client.0.smithi029.stdout: create 128/38 file after the snapshot 2024-04-17T19:51:02.516 INFO:tasks.workunit.client.0.smithi029.stdout: create 128/39 file after the snapshot 2024-04-17T19:51:02.517 INFO:tasks.workunit.client.0.smithi029.stdout: create 129/.snap/snap-subdir-test 2024-04-17T19:51:02.551 INFO:tasks.workunit.client.0.smithi029.stdout: create 129/30 file after the snapshot 2024-04-17T19:51:02.553 INFO:tasks.workunit.client.0.smithi029.stdout: create 129/31 file after the snapshot 2024-04-17T19:51:02.554 INFO:tasks.workunit.client.0.smithi029.stdout: create 129/32 file after the snapshot 2024-04-17T19:51:02.556 INFO:tasks.workunit.client.0.smithi029.stdout: create 129/33 file after the snapshot 2024-04-17T19:51:02.561 INFO:tasks.workunit.client.0.smithi029.stdout: create 129/34 file after the snapshot 2024-04-17T19:51:02.568 INFO:tasks.workunit.client.0.smithi029.stdout: create 129/35 file after the snapshot 2024-04-17T19:51:02.642 INFO:tasks.workunit.client.0.smithi029.stdout: create 129/36 file after the snapshot 2024-04-17T19:51:03.273 INFO:tasks.workunit.client.0.smithi029.stdout: create 129/37 file after the snapshot 2024-04-17T19:51:03.274 INFO:tasks.workunit.client.0.smithi029.stdout: create 129/38 file after the snapshot 2024-04-17T19:51:03.275 INFO:tasks.workunit.client.0.smithi029.stdout: create 129/39 file after the snapshot 2024-04-17T19:51:03.276 INFO:tasks.workunit.client.0.smithi029.stdout: create 130/.snap/snap-subdir-test 2024-04-17T19:51:03.310 INFO:tasks.workunit.client.0.smithi029.stdout: create 130/30 file after the snapshot 2024-04-17T19:51:03.313 INFO:tasks.workunit.client.0.smithi029.stdout: create 130/31 file after the snapshot 2024-04-17T19:51:03.316 INFO:tasks.workunit.client.0.smithi029.stdout: create 130/32 file after the snapshot 2024-04-17T19:51:03.325 INFO:tasks.workunit.client.0.smithi029.stdout: create 130/33 file after the snapshot 2024-04-17T19:51:03.331 INFO:tasks.workunit.client.0.smithi029.stdout: create 130/34 file after the snapshot 2024-04-17T19:51:03.332 INFO:tasks.workunit.client.0.smithi029.stdout: create 130/35 file after the snapshot 2024-04-17T19:51:03.333 INFO:tasks.workunit.client.0.smithi029.stdout: create 130/36 file after the snapshot 2024-04-17T19:51:03.334 INFO:tasks.workunit.client.0.smithi029.stdout: create 130/37 file after the snapshot 2024-04-17T19:51:03.335 INFO:tasks.workunit.client.0.smithi029.stdout: create 130/38 file after the snapshot 2024-04-17T19:51:03.336 INFO:tasks.workunit.client.0.smithi029.stdout: create 130/39 file after the snapshot 2024-04-17T19:51:03.337 INFO:tasks.workunit.client.0.smithi029.stdout: create 131/.snap/snap-subdir-test 2024-04-17T19:51:03.371 INFO:tasks.workunit.client.0.smithi029.stdout: create 131/30 file after the snapshot 2024-04-17T19:51:03.373 INFO:tasks.workunit.client.0.smithi029.stdout: create 131/31 file after the snapshot 2024-04-17T19:51:03.375 INFO:tasks.workunit.client.0.smithi029.stdout: create 131/32 file after the snapshot 2024-04-17T19:51:03.378 INFO:tasks.workunit.client.0.smithi029.stdout: create 131/33 file after the snapshot 2024-04-17T19:51:03.388 INFO:tasks.workunit.client.0.smithi029.stdout: create 131/34 file after the snapshot 2024-04-17T19:51:03.390 INFO:tasks.workunit.client.0.smithi029.stdout: create 131/35 file after the snapshot 2024-04-17T19:51:03.391 INFO:tasks.workunit.client.0.smithi029.stdout: create 131/36 file after the snapshot 2024-04-17T19:51:03.392 INFO:tasks.workunit.client.0.smithi029.stdout: create 131/37 file after the snapshot 2024-04-17T19:51:03.393 INFO:tasks.workunit.client.0.smithi029.stdout: create 131/38 file after the snapshot 2024-04-17T19:51:03.394 INFO:tasks.workunit.client.0.smithi029.stdout: create 131/39 file after the snapshot 2024-04-17T19:51:03.395 INFO:tasks.workunit.client.0.smithi029.stdout: create 132/.snap/snap-subdir-test 2024-04-17T19:51:03.424 INFO:tasks.workunit.client.0.smithi029.stdout: create 132/30 file after the snapshot 2024-04-17T19:51:03.427 INFO:tasks.workunit.client.0.smithi029.stdout: create 132/31 file after the snapshot 2024-04-17T19:51:03.443 INFO:tasks.workunit.client.0.smithi029.stdout: create 132/32 file after the snapshot 2024-04-17T19:51:03.446 INFO:tasks.workunit.client.0.smithi029.stdout: create 132/33 file after the snapshot 2024-04-17T19:51:03.447 INFO:tasks.workunit.client.0.smithi029.stdout: create 132/34 file after the snapshot 2024-04-17T19:51:03.449 INFO:tasks.workunit.client.0.smithi029.stdout: create 132/35 file after the snapshot 2024-04-17T19:51:03.450 INFO:tasks.workunit.client.0.smithi029.stdout: create 132/36 file after the snapshot 2024-04-17T19:51:03.452 INFO:tasks.workunit.client.0.smithi029.stdout: create 132/37 file after the snapshot 2024-04-17T19:51:03.453 INFO:tasks.workunit.client.0.smithi029.stdout: create 132/38 file after the snapshot 2024-04-17T19:51:03.455 INFO:tasks.workunit.client.0.smithi029.stdout: create 132/39 file after the snapshot 2024-04-17T19:51:03.457 INFO:tasks.workunit.client.0.smithi029.stdout: create 133/.snap/snap-subdir-test 2024-04-17T19:51:03.483 INFO:tasks.workunit.client.0.smithi029.stdout: create 133/30 file after the snapshot 2024-04-17T19:51:03.485 INFO:tasks.workunit.client.0.smithi029.stdout: create 133/31 file after the snapshot 2024-04-17T19:51:03.486 INFO:tasks.workunit.client.0.smithi029.stdout: create 133/32 file after the snapshot 2024-04-17T19:51:03.740 INFO:tasks.workunit.client.0.smithi029.stdout: create 133/33 file after the snapshot 2024-04-17T19:51:04.150 INFO:tasks.workunit.client.0.smithi029.stdout: create 133/34 file after the snapshot 2024-04-17T19:51:04.151 INFO:tasks.workunit.client.0.smithi029.stdout: create 133/35 file after the snapshot 2024-04-17T19:51:04.152 INFO:tasks.workunit.client.0.smithi029.stdout: create 133/36 file after the snapshot 2024-04-17T19:51:04.153 INFO:tasks.workunit.client.0.smithi029.stdout: create 133/37 file after the snapshot 2024-04-17T19:51:04.154 INFO:tasks.workunit.client.0.smithi029.stdout: create 133/38 file after the snapshot 2024-04-17T19:51:04.155 INFO:tasks.workunit.client.0.smithi029.stdout: create 133/39 file after the snapshot 2024-04-17T19:51:04.156 INFO:tasks.workunit.client.0.smithi029.stdout: create 134/.snap/snap-subdir-test 2024-04-17T19:51:04.187 INFO:tasks.workunit.client.0.smithi029.stdout: create 134/30 file after the snapshot 2024-04-17T19:51:04.190 INFO:tasks.workunit.client.0.smithi029.stdout: create 134/31 file after the snapshot 2024-04-17T19:51:04.194 INFO:tasks.workunit.client.0.smithi029.stdout: create 134/32 file after the snapshot 2024-04-17T19:51:04.203 INFO:tasks.workunit.client.0.smithi029.stdout: create 134/33 file after the snapshot 2024-04-17T19:51:04.207 INFO:tasks.workunit.client.0.smithi029.stdout: create 134/34 file after the snapshot 2024-04-17T19:51:04.208 INFO:tasks.workunit.client.0.smithi029.stdout: create 134/35 file after the snapshot 2024-04-17T19:51:04.209 INFO:tasks.workunit.client.0.smithi029.stdout: create 134/36 file after the snapshot 2024-04-17T19:51:04.210 INFO:tasks.workunit.client.0.smithi029.stdout: create 134/37 file after the snapshot 2024-04-17T19:51:04.211 INFO:tasks.workunit.client.0.smithi029.stdout: create 134/38 file after the snapshot 2024-04-17T19:51:04.212 INFO:tasks.workunit.client.0.smithi029.stdout: create 134/39 file after the snapshot 2024-04-17T19:51:04.213 INFO:tasks.workunit.client.0.smithi029.stdout: create 135/.snap/snap-subdir-test 2024-04-17T19:51:04.239 INFO:tasks.workunit.client.0.smithi029.stdout: create 135/30 file after the snapshot 2024-04-17T19:51:04.241 INFO:tasks.workunit.client.0.smithi029.stdout: create 135/31 file after the snapshot 2024-04-17T19:51:04.242 INFO:tasks.workunit.client.0.smithi029.stdout: create 135/32 file after the snapshot 2024-04-17T19:51:04.243 INFO:tasks.workunit.client.0.smithi029.stdout: create 135/33 file after the snapshot 2024-04-17T19:51:04.248 INFO:tasks.workunit.client.0.smithi029.stdout: create 135/34 file after the snapshot 2024-04-17T19:51:04.254 INFO:tasks.workunit.client.0.smithi029.stdout: create 135/35 file after the snapshot 2024-04-17T19:51:04.259 INFO:tasks.workunit.client.0.smithi029.stdout: create 135/36 file after the snapshot 2024-04-17T19:51:04.260 INFO:tasks.workunit.client.0.smithi029.stdout: create 135/37 file after the snapshot 2024-04-17T19:51:04.261 INFO:tasks.workunit.client.0.smithi029.stdout: create 135/38 file after the snapshot 2024-04-17T19:51:04.262 INFO:tasks.workunit.client.0.smithi029.stdout: create 135/39 file after the snapshot 2024-04-17T19:51:04.263 INFO:tasks.workunit.client.0.smithi029.stdout: create 136/.snap/snap-subdir-test 2024-04-17T19:51:04.292 INFO:tasks.workunit.client.0.smithi029.stdout: create 136/30 file after the snapshot 2024-04-17T19:51:04.294 INFO:tasks.workunit.client.0.smithi029.stdout: create 136/31 file after the snapshot 2024-04-17T19:51:04.296 INFO:tasks.workunit.client.0.smithi029.stdout: create 136/32 file after the snapshot 2024-04-17T19:51:04.297 INFO:tasks.workunit.client.0.smithi029.stdout: create 136/33 file after the snapshot 2024-04-17T19:51:04.300 INFO:tasks.workunit.client.0.smithi029.stdout: create 136/34 file after the snapshot 2024-04-17T19:51:04.307 INFO:tasks.workunit.client.0.smithi029.stdout: create 136/35 file after the snapshot 2024-04-17T19:51:04.312 INFO:tasks.workunit.client.0.smithi029.stdout: create 136/36 file after the snapshot 2024-04-17T19:51:04.313 INFO:tasks.workunit.client.0.smithi029.stdout: create 136/37 file after the snapshot 2024-04-17T19:51:04.314 INFO:tasks.workunit.client.0.smithi029.stdout: create 136/38 file after the snapshot 2024-04-17T19:51:04.315 INFO:tasks.workunit.client.0.smithi029.stdout: create 136/39 file after the snapshot 2024-04-17T19:51:04.316 INFO:tasks.workunit.client.0.smithi029.stdout: create 137/.snap/snap-subdir-test 2024-04-17T19:51:04.343 INFO:tasks.workunit.client.0.smithi029.stdout: create 137/30 file after the snapshot 2024-04-17T19:51:04.345 INFO:tasks.workunit.client.0.smithi029.stdout: create 137/31 file after the snapshot 2024-04-17T19:51:04.347 INFO:tasks.workunit.client.0.smithi029.stdout: create 137/32 file after the snapshot 2024-04-17T19:51:04.351 INFO:tasks.workunit.client.0.smithi029.stdout: create 137/33 file after the snapshot 2024-04-17T19:51:04.360 INFO:tasks.workunit.client.0.smithi029.stdout: create 137/34 file after the snapshot 2024-04-17T19:51:04.362 INFO:tasks.workunit.client.0.smithi029.stdout: create 137/35 file after the snapshot 2024-04-17T19:51:04.363 INFO:tasks.workunit.client.0.smithi029.stdout: create 137/36 file after the snapshot 2024-04-17T19:51:04.364 INFO:tasks.workunit.client.0.smithi029.stdout: create 137/37 file after the snapshot 2024-04-17T19:51:04.365 INFO:tasks.workunit.client.0.smithi029.stdout: create 137/38 file after the snapshot 2024-04-17T19:51:04.366 INFO:tasks.workunit.client.0.smithi029.stdout: create 137/39 file after the snapshot 2024-04-17T19:51:04.367 INFO:tasks.workunit.client.0.smithi029.stdout: create 138/.snap/snap-subdir-test 2024-04-17T19:51:04.394 INFO:tasks.workunit.client.0.smithi029.stdout: create 138/30 file after the snapshot 2024-04-17T19:51:04.396 INFO:tasks.workunit.client.0.smithi029.stdout: create 138/31 file after the snapshot 2024-04-17T19:51:04.397 INFO:tasks.workunit.client.0.smithi029.stdout: create 138/32 file after the snapshot 2024-04-17T19:51:04.398 INFO:tasks.workunit.client.0.smithi029.stdout: create 138/33 file after the snapshot 2024-04-17T19:51:04.400 INFO:tasks.workunit.client.0.smithi029.stdout: create 138/34 file after the snapshot 2024-04-17T19:51:04.401 INFO:tasks.workunit.client.0.smithi029.stdout: create 138/35 file after the snapshot 2024-04-17T19:51:04.402 INFO:tasks.workunit.client.0.smithi029.stdout: create 138/36 file after the snapshot 2024-04-17T19:51:04.404 INFO:tasks.workunit.client.0.smithi029.stdout: create 138/37 file after the snapshot 2024-04-17T19:51:04.409 INFO:tasks.workunit.client.0.smithi029.stdout: create 138/38 file after the snapshot 2024-04-17T19:51:04.416 INFO:tasks.workunit.client.0.smithi029.stdout: create 138/39 file after the snapshot 2024-04-17T19:51:04.416 INFO:tasks.workunit.client.0.smithi029.stdout: create 139/.snap/snap-subdir-test 2024-04-17T19:51:04.451 INFO:tasks.workunit.client.0.smithi029.stdout: create 139/30 file after the snapshot 2024-04-17T19:51:04.453 INFO:tasks.workunit.client.0.smithi029.stdout: create 139/31 file after the snapshot 2024-04-17T19:51:04.454 INFO:tasks.workunit.client.0.smithi029.stdout: create 139/32 file after the snapshot 2024-04-17T19:51:04.456 INFO:tasks.workunit.client.0.smithi029.stdout: create 139/33 file after the snapshot 2024-04-17T19:51:04.462 INFO:tasks.workunit.client.0.smithi029.stdout: create 139/34 file after the snapshot 2024-04-17T19:51:04.468 INFO:tasks.workunit.client.0.smithi029.stdout: create 139/35 file after the snapshot 2024-04-17T19:51:04.469 INFO:tasks.workunit.client.0.smithi029.stdout: create 139/36 file after the snapshot 2024-04-17T19:51:04.470 INFO:tasks.workunit.client.0.smithi029.stdout: create 139/37 file after the snapshot 2024-04-17T19:51:04.471 INFO:tasks.workunit.client.0.smithi029.stdout: create 139/38 file after the snapshot 2024-04-17T19:51:04.472 INFO:tasks.workunit.client.0.smithi029.stdout: create 139/39 file after the snapshot 2024-04-17T19:51:04.473 INFO:tasks.workunit.client.0.smithi029.stdout: create 140/.snap/snap-subdir-test 2024-04-17T19:51:04.499 INFO:tasks.workunit.client.0.smithi029.stdout: create 140/30 file after the snapshot 2024-04-17T19:51:04.500 INFO:tasks.workunit.client.0.smithi029.stdout: create 140/31 file after the snapshot 2024-04-17T19:51:04.501 INFO:tasks.workunit.client.0.smithi029.stdout: create 140/32 file after the snapshot 2024-04-17T19:51:04.502 INFO:tasks.workunit.client.0.smithi029.stdout: create 140/33 file after the snapshot 2024-04-17T19:51:04.503 INFO:tasks.workunit.client.0.smithi029.stdout: create 140/34 file after the snapshot 2024-04-17T19:51:04.506 INFO:tasks.workunit.client.0.smithi029.stdout: create 140/35 file after the snapshot 2024-04-17T19:51:04.508 INFO:tasks.workunit.client.0.smithi029.stdout: create 140/36 file after the snapshot 2024-04-17T19:51:04.509 INFO:tasks.workunit.client.0.smithi029.stdout: create 140/37 file after the snapshot 2024-04-17T19:51:04.510 INFO:tasks.workunit.client.0.smithi029.stdout: create 140/38 file after the snapshot 2024-04-17T19:51:04.511 INFO:tasks.workunit.client.0.smithi029.stdout: create 140/39 file after the snapshot 2024-04-17T19:51:04.515 INFO:tasks.workunit.client.0.smithi029.stdout: create 141/.snap/snap-subdir-test 2024-04-17T19:51:04.938 INFO:tasks.workunit.client.0.smithi029.stdout: create 141/30 file after the snapshot 2024-04-17T19:51:04.940 INFO:tasks.workunit.client.0.smithi029.stdout: create 141/31 file after the snapshot 2024-04-17T19:51:04.942 INFO:tasks.workunit.client.0.smithi029.stdout: create 141/32 file after the snapshot 2024-04-17T19:51:04.951 INFO:tasks.workunit.client.0.smithi029.stdout: create 141/33 file after the snapshot 2024-04-17T19:51:04.957 INFO:tasks.workunit.client.0.smithi029.stdout: create 141/34 file after the snapshot 2024-04-17T19:51:04.958 INFO:tasks.workunit.client.0.smithi029.stdout: create 141/35 file after the snapshot 2024-04-17T19:51:04.959 INFO:tasks.workunit.client.0.smithi029.stdout: create 141/36 file after the snapshot 2024-04-17T19:51:04.960 INFO:tasks.workunit.client.0.smithi029.stdout: create 141/37 file after the snapshot 2024-04-17T19:51:04.961 INFO:tasks.workunit.client.0.smithi029.stdout: create 141/38 file after the snapshot 2024-04-17T19:51:04.962 INFO:tasks.workunit.client.0.smithi029.stdout: create 141/39 file after the snapshot 2024-04-17T19:51:04.963 INFO:tasks.workunit.client.0.smithi029.stdout: create 142/.snap/snap-subdir-test 2024-04-17T19:51:04.990 INFO:tasks.workunit.client.0.smithi029.stdout: create 142/30 file after the snapshot 2024-04-17T19:51:04.992 INFO:tasks.workunit.client.0.smithi029.stdout: create 142/31 file after the snapshot 2024-04-17T19:51:04.997 INFO:tasks.workunit.client.0.smithi029.stdout: create 142/32 file after the snapshot 2024-04-17T19:51:05.001 INFO:tasks.workunit.client.0.smithi029.stdout: create 142/33 file after the snapshot 2024-04-17T19:51:05.619 INFO:tasks.workunit.client.0.smithi029.stdout: create 142/34 file after the snapshot 2024-04-17T19:51:05.719 INFO:tasks.workunit.client.0.smithi029.stdout: create 142/35 file after the snapshot 2024-04-17T19:51:05.720 INFO:tasks.workunit.client.0.smithi029.stdout: create 142/36 file after the snapshot 2024-04-17T19:51:05.721 INFO:tasks.workunit.client.0.smithi029.stdout: create 142/37 file after the snapshot 2024-04-17T19:51:05.722 INFO:tasks.workunit.client.0.smithi029.stdout: create 142/38 file after the snapshot 2024-04-17T19:51:05.723 INFO:tasks.workunit.client.0.smithi029.stdout: create 142/39 file after the snapshot 2024-04-17T19:51:05.724 INFO:tasks.workunit.client.0.smithi029.stdout: create 143/.snap/snap-subdir-test 2024-04-17T19:51:05.754 INFO:tasks.workunit.client.0.smithi029.stdout: create 143/30 file after the snapshot 2024-04-17T19:51:05.757 INFO:tasks.workunit.client.0.smithi029.stdout: create 143/31 file after the snapshot 2024-04-17T19:51:05.758 INFO:tasks.workunit.client.0.smithi029.stdout: create 143/32 file after the snapshot 2024-04-17T19:51:05.760 INFO:tasks.workunit.client.0.smithi029.stdout: create 143/33 file after the snapshot 2024-04-17T19:51:05.768 INFO:tasks.workunit.client.0.smithi029.stdout: create 143/34 file after the snapshot 2024-04-17T19:51:06.611 INFO:tasks.workunit.client.0.smithi029.stdout: create 143/35 file after the snapshot 2024-04-17T19:51:06.727 INFO:tasks.workunit.client.0.smithi029.stdout: create 143/36 file after the snapshot 2024-04-17T19:51:06.728 INFO:tasks.workunit.client.0.smithi029.stdout: create 143/37 file after the snapshot 2024-04-17T19:51:06.730 INFO:tasks.workunit.client.0.smithi029.stdout: create 143/38 file after the snapshot 2024-04-17T19:51:06.731 INFO:tasks.workunit.client.0.smithi029.stdout: create 143/39 file after the snapshot 2024-04-17T19:51:06.732 INFO:tasks.workunit.client.0.smithi029.stdout: create 144/.snap/snap-subdir-test 2024-04-17T19:51:06.770 INFO:tasks.workunit.client.0.smithi029.stdout: create 144/30 file after the snapshot 2024-04-17T19:51:06.771 INFO:tasks.workunit.client.0.smithi029.stdout: create 144/31 file after the snapshot 2024-04-17T19:51:06.775 INFO:tasks.workunit.client.0.smithi029.stdout: create 144/32 file after the snapshot 2024-04-17T19:51:06.781 INFO:tasks.workunit.client.0.smithi029.stdout: create 144/33 file after the snapshot 2024-04-17T19:51:06.789 INFO:tasks.workunit.client.0.smithi029.stdout: create 144/34 file after the snapshot 2024-04-17T19:51:06.790 INFO:tasks.workunit.client.0.smithi029.stdout: create 144/35 file after the snapshot 2024-04-17T19:51:06.791 INFO:tasks.workunit.client.0.smithi029.stdout: create 144/36 file after the snapshot 2024-04-17T19:51:06.792 INFO:tasks.workunit.client.0.smithi029.stdout: create 144/37 file after the snapshot 2024-04-17T19:51:06.793 INFO:tasks.workunit.client.0.smithi029.stdout: create 144/38 file after the snapshot 2024-04-17T19:51:06.794 INFO:tasks.workunit.client.0.smithi029.stdout: create 144/39 file after the snapshot 2024-04-17T19:51:06.796 INFO:tasks.workunit.client.0.smithi029.stdout: create 145/.snap/snap-subdir-test 2024-04-17T19:51:06.823 INFO:tasks.workunit.client.0.smithi029.stdout: create 145/30 file after the snapshot 2024-04-17T19:51:06.826 INFO:tasks.workunit.client.0.smithi029.stdout: create 145/31 file after the snapshot 2024-04-17T19:51:06.831 INFO:tasks.workunit.client.0.smithi029.stdout: create 145/32 file after the snapshot 2024-04-17T19:51:06.838 INFO:tasks.workunit.client.0.smithi029.stdout: create 145/33 file after the snapshot 2024-04-17T19:51:06.841 INFO:tasks.workunit.client.0.smithi029.stdout: create 145/34 file after the snapshot 2024-04-17T19:51:06.842 INFO:tasks.workunit.client.0.smithi029.stdout: create 145/35 file after the snapshot 2024-04-17T19:51:06.843 INFO:tasks.workunit.client.0.smithi029.stdout: create 145/36 file after the snapshot 2024-04-17T19:51:06.844 INFO:tasks.workunit.client.0.smithi029.stdout: create 145/37 file after the snapshot 2024-04-17T19:51:06.845 INFO:tasks.workunit.client.0.smithi029.stdout: create 145/38 file after the snapshot 2024-04-17T19:51:06.846 INFO:tasks.workunit.client.0.smithi029.stdout: create 145/39 file after the snapshot 2024-04-17T19:51:06.847 INFO:tasks.workunit.client.0.smithi029.stdout: create 146/.snap/snap-subdir-test 2024-04-17T19:51:06.875 INFO:tasks.workunit.client.0.smithi029.stdout: create 146/30 file after the snapshot 2024-04-17T19:51:06.876 INFO:tasks.workunit.client.0.smithi029.stdout: create 146/31 file after the snapshot 2024-04-17T19:51:06.877 INFO:tasks.workunit.client.0.smithi029.stdout: create 146/32 file after the snapshot 2024-04-17T19:51:06.878 INFO:tasks.workunit.client.0.smithi029.stdout: create 146/33 file after the snapshot 2024-04-17T19:51:06.880 INFO:tasks.workunit.client.0.smithi029.stdout: create 146/34 file after the snapshot 2024-04-17T19:51:06.883 INFO:tasks.workunit.client.0.smithi029.stdout: create 146/35 file after the snapshot 2024-04-17T19:51:06.884 INFO:tasks.workunit.client.0.smithi029.stdout: create 146/36 file after the snapshot 2024-04-17T19:51:06.885 INFO:tasks.workunit.client.0.smithi029.stdout: create 146/37 file after the snapshot 2024-04-17T19:51:06.890 INFO:tasks.workunit.client.0.smithi029.stdout: create 146/38 file after the snapshot 2024-04-17T19:51:06.896 INFO:tasks.workunit.client.0.smithi029.stdout: create 146/39 file after the snapshot 2024-04-17T19:51:06.897 INFO:tasks.workunit.client.0.smithi029.stdout: create 147/.snap/snap-subdir-test 2024-04-17T19:51:06.927 INFO:tasks.workunit.client.0.smithi029.stdout: create 147/30 file after the snapshot 2024-04-17T19:51:06.928 INFO:tasks.workunit.client.0.smithi029.stdout: create 147/31 file after the snapshot 2024-04-17T19:51:06.929 INFO:tasks.workunit.client.0.smithi029.stdout: create 147/32 file after the snapshot 2024-04-17T19:51:06.931 INFO:tasks.workunit.client.0.smithi029.stdout: create 147/33 file after the snapshot 2024-04-17T19:51:07.183 INFO:tasks.workunit.client.0.smithi029.stdout: create 147/34 file after the snapshot 2024-04-17T19:51:07.192 INFO:tasks.workunit.client.0.smithi029.stdout: create 147/35 file after the snapshot 2024-04-17T19:51:07.194 INFO:tasks.workunit.client.0.smithi029.stdout: create 147/36 file after the snapshot 2024-04-17T19:51:07.194 INFO:tasks.workunit.client.0.smithi029.stdout: create 147/37 file after the snapshot 2024-04-17T19:51:07.195 INFO:tasks.workunit.client.0.smithi029.stdout: create 147/38 file after the snapshot 2024-04-17T19:51:07.196 INFO:tasks.workunit.client.0.smithi029.stdout: create 147/39 file after the snapshot 2024-04-17T19:51:07.197 INFO:tasks.workunit.client.0.smithi029.stdout: create 148/.snap/snap-subdir-test 2024-04-17T19:51:07.218 INFO:tasks.workunit.client.0.smithi029.stdout: create 148/30 file after the snapshot 2024-04-17T19:51:07.219 INFO:tasks.workunit.client.0.smithi029.stdout: create 148/31 file after the snapshot 2024-04-17T19:51:07.220 INFO:tasks.workunit.client.0.smithi029.stdout: create 148/32 file after the snapshot 2024-04-17T19:51:07.225 INFO:tasks.workunit.client.0.smithi029.stdout: create 148/33 file after the snapshot 2024-04-17T19:51:07.379 INFO:tasks.workunit.client.0.smithi029.stdout: create 148/34 file after the snapshot 2024-04-17T19:51:07.380 INFO:tasks.workunit.client.0.smithi029.stdout: create 148/35 file after the snapshot 2024-04-17T19:51:07.381 INFO:tasks.workunit.client.0.smithi029.stdout: create 148/36 file after the snapshot 2024-04-17T19:51:07.382 INFO:tasks.workunit.client.0.smithi029.stdout: create 148/37 file after the snapshot 2024-04-17T19:51:07.383 INFO:tasks.workunit.client.0.smithi029.stdout: create 148/38 file after the snapshot 2024-04-17T19:51:07.384 INFO:tasks.workunit.client.0.smithi029.stdout: create 148/39 file after the snapshot 2024-04-17T19:51:07.385 INFO:tasks.workunit.client.0.smithi029.stdout: create 149/.snap/snap-subdir-test 2024-04-17T19:51:07.589 INFO:tasks.workunit.client.0.smithi029.stdout: create 149/30 file after the snapshot 2024-04-17T19:51:07.590 INFO:tasks.workunit.client.0.smithi029.stdout: create 149/31 file after the snapshot 2024-04-17T19:51:07.591 INFO:tasks.workunit.client.0.smithi029.stdout: create 149/32 file after the snapshot 2024-04-17T19:51:07.592 INFO:tasks.workunit.client.0.smithi029.stdout: create 149/33 file after the snapshot 2024-04-17T19:51:07.595 INFO:tasks.workunit.client.0.smithi029.stdout: create 149/34 file after the snapshot 2024-04-17T19:51:07.598 INFO:tasks.workunit.client.0.smithi029.stdout: create 149/35 file after the snapshot 2024-04-17T19:51:07.603 INFO:tasks.workunit.client.0.smithi029.stdout: create 149/36 file after the snapshot 2024-04-17T19:51:07.667 INFO:tasks.workunit.client.0.smithi029.stdout: create 149/37 file after the snapshot 2024-04-17T19:51:08.368 INFO:tasks.workunit.client.0.smithi029.stdout: create 149/38 file after the snapshot 2024-04-17T19:51:08.369 INFO:tasks.workunit.client.0.smithi029.stdout: create 149/39 file after the snapshot 2024-04-17T19:51:08.370 INFO:tasks.workunit.client.0.smithi029.stdout: create 150/.snap/snap-subdir-test 2024-04-17T19:51:08.803 INFO:tasks.workunit.client.0.smithi029.stdout: create 150/30 file after the snapshot 2024-04-17T19:51:08.806 INFO:tasks.workunit.client.0.smithi029.stdout: create 150/31 file after the snapshot 2024-04-17T19:51:08.811 INFO:tasks.workunit.client.0.smithi029.stdout: create 150/32 file after the snapshot 2024-04-17T19:51:09.063 INFO:tasks.workunit.client.0.smithi029.stdout: create 150/33 file after the snapshot 2024-04-17T19:51:09.652 INFO:tasks.workunit.client.0.smithi029.stdout: create 150/34 file after the snapshot 2024-04-17T19:51:09.745 INFO:tasks.workunit.client.0.smithi029.stdout: create 150/35 file after the snapshot 2024-04-17T19:51:09.746 INFO:tasks.workunit.client.0.smithi029.stdout: create 150/36 file after the snapshot 2024-04-17T19:51:09.747 INFO:tasks.workunit.client.0.smithi029.stdout: create 150/37 file after the snapshot 2024-04-17T19:51:09.749 INFO:tasks.workunit.client.0.smithi029.stdout: create 150/38 file after the snapshot 2024-04-17T19:51:09.750 INFO:tasks.workunit.client.0.smithi029.stdout: create 150/39 file after the snapshot 2024-04-17T19:51:09.751 INFO:tasks.workunit.client.0.smithi029.stdout: create 151/.snap/snap-subdir-test 2024-04-17T19:51:09.779 INFO:tasks.workunit.client.0.smithi029.stdout: create 151/30 file after the snapshot 2024-04-17T19:51:09.781 INFO:tasks.workunit.client.0.smithi029.stdout: create 151/31 file after the snapshot 2024-04-17T19:51:09.783 INFO:tasks.workunit.client.0.smithi029.stdout: create 151/32 file after the snapshot 2024-04-17T19:51:09.784 INFO:tasks.workunit.client.0.smithi029.stdout: create 151/33 file after the snapshot 2024-04-17T19:51:09.789 INFO:tasks.workunit.client.0.smithi029.stdout: create 151/34 file after the snapshot 2024-04-17T19:51:09.796 INFO:tasks.workunit.client.0.smithi029.stdout: create 151/35 file after the snapshot 2024-04-17T19:51:09.800 INFO:tasks.workunit.client.0.smithi029.stdout: create 151/36 file after the snapshot 2024-04-17T19:51:09.801 INFO:tasks.workunit.client.0.smithi029.stdout: create 151/37 file after the snapshot 2024-04-17T19:51:09.802 INFO:tasks.workunit.client.0.smithi029.stdout: create 151/38 file after the snapshot 2024-04-17T19:51:09.804 INFO:tasks.workunit.client.0.smithi029.stdout: create 151/39 file after the snapshot 2024-04-17T19:51:09.805 INFO:tasks.workunit.client.0.smithi029.stdout: create 152/.snap/snap-subdir-test 2024-04-17T19:51:09.831 INFO:tasks.workunit.client.0.smithi029.stdout: create 152/30 file after the snapshot 2024-04-17T19:51:09.832 INFO:tasks.workunit.client.0.smithi029.stdout: create 152/31 file after the snapshot 2024-04-17T19:51:09.834 INFO:tasks.workunit.client.0.smithi029.stdout: create 152/32 file after the snapshot 2024-04-17T19:51:09.836 INFO:tasks.workunit.client.0.smithi029.stdout: create 152/33 file after the snapshot 2024-04-17T19:51:09.839 INFO:tasks.workunit.client.0.smithi029.stdout: create 152/34 file after the snapshot 2024-04-17T19:51:09.849 INFO:tasks.workunit.client.0.smithi029.stdout: create 152/35 file after the snapshot 2024-04-17T19:51:09.854 INFO:tasks.workunit.client.0.smithi029.stdout: create 152/36 file after the snapshot 2024-04-17T19:51:09.855 INFO:tasks.workunit.client.0.smithi029.stdout: create 152/37 file after the snapshot 2024-04-17T19:51:09.856 INFO:tasks.workunit.client.0.smithi029.stdout: create 152/38 file after the snapshot 2024-04-17T19:51:09.858 INFO:tasks.workunit.client.0.smithi029.stdout: create 152/39 file after the snapshot 2024-04-17T19:51:09.858 INFO:tasks.workunit.client.0.smithi029.stdout: create 153/.snap/snap-subdir-test 2024-04-17T19:51:09.885 INFO:tasks.workunit.client.0.smithi029.stdout: create 153/30 file after the snapshot 2024-04-17T19:51:09.891 INFO:tasks.workunit.client.0.smithi029.stdout: create 153/31 file after the snapshot 2024-04-17T19:51:09.896 INFO:tasks.workunit.client.0.smithi029.stdout: create 153/32 file after the snapshot 2024-04-17T19:51:09.906 INFO:tasks.workunit.client.0.smithi029.stdout: create 153/33 file after the snapshot 2024-04-17T19:51:10.232 INFO:tasks.workunit.client.0.smithi029.stdout: create 153/34 file after the snapshot 2024-04-17T19:51:10.234 INFO:tasks.workunit.client.0.smithi029.stdout: create 153/35 file after the snapshot 2024-04-17T19:51:10.236 INFO:tasks.workunit.client.0.smithi029.stdout: create 153/36 file after the snapshot 2024-04-17T19:51:10.242 INFO:tasks.workunit.client.0.smithi029.stdout: create 153/37 file after the snapshot 2024-04-17T19:51:10.243 INFO:tasks.workunit.client.0.smithi029.stdout: create 153/38 file after the snapshot 2024-04-17T19:51:10.244 INFO:tasks.workunit.client.0.smithi029.stdout: create 153/39 file after the snapshot 2024-04-17T19:51:10.245 INFO:tasks.workunit.client.0.smithi029.stdout: create 154/.snap/snap-subdir-test 2024-04-17T19:51:10.265 INFO:tasks.workunit.client.0.smithi029.stdout: create 154/30 file after the snapshot 2024-04-17T19:51:10.266 INFO:tasks.workunit.client.0.smithi029.stdout: create 154/31 file after the snapshot 2024-04-17T19:51:10.267 INFO:tasks.workunit.client.0.smithi029.stdout: create 154/32 file after the snapshot 2024-04-17T19:51:10.269 INFO:tasks.workunit.client.0.smithi029.stdout: create 154/33 file after the snapshot 2024-04-17T19:51:10.273 INFO:tasks.workunit.client.0.smithi029.stdout: create 154/34 file after the snapshot 2024-04-17T19:51:10.281 INFO:tasks.workunit.client.0.smithi029.stdout: create 154/35 file after the snapshot 2024-04-17T19:51:10.602 INFO:tasks.workunit.client.0.smithi029.stdout: create 154/36 file after the snapshot 2024-04-17T19:51:10.603 INFO:tasks.workunit.client.0.smithi029.stdout: create 154/37 file after the snapshot 2024-04-17T19:51:10.604 INFO:tasks.workunit.client.0.smithi029.stdout: create 154/38 file after the snapshot 2024-04-17T19:51:10.605 INFO:tasks.workunit.client.0.smithi029.stdout: create 154/39 file after the snapshot 2024-04-17T19:51:10.606 INFO:tasks.workunit.client.0.smithi029.stdout: create 155/.snap/snap-subdir-test 2024-04-17T19:51:10.635 INFO:tasks.workunit.client.0.smithi029.stdout: create 155/30 file after the snapshot 2024-04-17T19:51:10.637 INFO:tasks.workunit.client.0.smithi029.stdout: create 155/31 file after the snapshot 2024-04-17T19:51:10.638 INFO:tasks.workunit.client.0.smithi029.stdout: create 155/32 file after the snapshot 2024-04-17T19:51:10.641 INFO:tasks.workunit.client.0.smithi029.stdout: create 155/33 file after the snapshot 2024-04-17T19:51:10.646 INFO:tasks.workunit.client.0.smithi029.stdout: create 155/34 file after the snapshot 2024-04-17T19:51:10.678 INFO:tasks.workunit.client.0.smithi029.stdout: create 155/35 file after the snapshot 2024-04-17T19:51:11.447 INFO:tasks.workunit.client.0.smithi029.stdout: create 155/36 file after the snapshot 2024-04-17T19:51:11.448 INFO:tasks.workunit.client.0.smithi029.stdout: create 155/37 file after the snapshot 2024-04-17T19:51:11.449 INFO:tasks.workunit.client.0.smithi029.stdout: create 155/38 file after the snapshot 2024-04-17T19:51:11.450 INFO:tasks.workunit.client.0.smithi029.stdout: create 155/39 file after the snapshot 2024-04-17T19:51:11.451 INFO:tasks.workunit.client.0.smithi029.stdout: create 156/.snap/snap-subdir-test 2024-04-17T19:51:11.480 INFO:tasks.workunit.client.0.smithi029.stdout: create 156/30 file after the snapshot 2024-04-17T19:51:11.483 INFO:tasks.workunit.client.0.smithi029.stdout: create 156/31 file after the snapshot 2024-04-17T19:51:11.490 INFO:tasks.workunit.client.0.smithi029.stdout: create 156/32 file after the snapshot 2024-04-17T19:51:11.496 INFO:tasks.workunit.client.0.smithi029.stdout: create 156/33 file after the snapshot 2024-04-17T19:51:11.496 INFO:tasks.workunit.client.0.smithi029.stdout: create 156/34 file after the snapshot 2024-04-17T19:51:11.497 INFO:tasks.workunit.client.0.smithi029.stdout: create 156/35 file after the snapshot 2024-04-17T19:51:11.498 INFO:tasks.workunit.client.0.smithi029.stdout: create 156/36 file after the snapshot 2024-04-17T19:51:11.499 INFO:tasks.workunit.client.0.smithi029.stdout: create 156/37 file after the snapshot 2024-04-17T19:51:11.500 INFO:tasks.workunit.client.0.smithi029.stdout: create 156/38 file after the snapshot 2024-04-17T19:51:11.501 INFO:tasks.workunit.client.0.smithi029.stdout: create 156/39 file after the snapshot 2024-04-17T19:51:11.502 INFO:tasks.workunit.client.0.smithi029.stdout: create 157/.snap/snap-subdir-test 2024-04-17T19:51:11.539 INFO:tasks.workunit.client.0.smithi029.stdout: create 157/30 file after the snapshot 2024-04-17T19:51:11.541 INFO:tasks.workunit.client.0.smithi029.stdout: create 157/31 file after the snapshot 2024-04-17T19:51:11.542 INFO:tasks.workunit.client.0.smithi029.stdout: create 157/32 file after the snapshot 2024-04-17T19:51:11.547 INFO:tasks.workunit.client.0.smithi029.stdout: create 157/33 file after the snapshot 2024-04-17T19:51:11.556 INFO:tasks.workunit.client.0.smithi029.stdout: create 157/34 file after the snapshot 2024-04-17T19:51:11.564 INFO:tasks.workunit.client.0.smithi029.stdout: create 157/35 file after the snapshot 2024-04-17T19:51:11.565 INFO:tasks.workunit.client.0.smithi029.stdout: create 157/36 file after the snapshot 2024-04-17T19:51:11.566 INFO:tasks.workunit.client.0.smithi029.stdout: create 157/37 file after the snapshot 2024-04-17T19:51:11.567 INFO:tasks.workunit.client.0.smithi029.stdout: create 157/38 file after the snapshot 2024-04-17T19:51:11.568 INFO:tasks.workunit.client.0.smithi029.stdout: create 157/39 file after the snapshot 2024-04-17T19:51:11.569 INFO:tasks.workunit.client.0.smithi029.stdout: create 158/.snap/snap-subdir-test 2024-04-17T19:51:11.593 INFO:tasks.workunit.client.0.smithi029.stdout: create 158/30 file after the snapshot 2024-04-17T19:51:11.595 INFO:tasks.workunit.client.0.smithi029.stdout: create 158/31 file after the snapshot 2024-04-17T19:51:11.598 INFO:tasks.workunit.client.0.smithi029.stdout: create 158/32 file after the snapshot 2024-04-17T19:51:11.603 INFO:tasks.workunit.client.0.smithi029.stdout: create 158/33 file after the snapshot 2024-04-17T19:51:11.611 INFO:tasks.workunit.client.0.smithi029.stdout: create 158/34 file after the snapshot 2024-04-17T19:51:11.612 INFO:tasks.workunit.client.0.smithi029.stdout: create 158/35 file after the snapshot 2024-04-17T19:51:11.613 INFO:tasks.workunit.client.0.smithi029.stdout: create 158/36 file after the snapshot 2024-04-17T19:51:11.614 INFO:tasks.workunit.client.0.smithi029.stdout: create 158/37 file after the snapshot 2024-04-17T19:51:11.614 INFO:tasks.workunit.client.0.smithi029.stdout: create 158/38 file after the snapshot 2024-04-17T19:51:11.615 INFO:tasks.workunit.client.0.smithi029.stdout: create 158/39 file after the snapshot 2024-04-17T19:51:11.616 INFO:tasks.workunit.client.0.smithi029.stdout: create 159/.snap/snap-subdir-test 2024-04-17T19:51:11.642 INFO:tasks.workunit.client.0.smithi029.stdout: create 159/30 file after the snapshot 2024-04-17T19:51:11.644 INFO:tasks.workunit.client.0.smithi029.stdout: create 159/31 file after the snapshot 2024-04-17T19:51:11.645 INFO:tasks.workunit.client.0.smithi029.stdout: create 159/32 file after the snapshot 2024-04-17T19:51:11.646 INFO:tasks.workunit.client.0.smithi029.stdout: create 159/33 file after the snapshot 2024-04-17T19:51:11.652 INFO:tasks.workunit.client.0.smithi029.stdout: create 159/34 file after the snapshot 2024-04-17T19:51:11.666 INFO:tasks.workunit.client.0.smithi029.stdout: create 159/35 file after the snapshot 2024-04-17T19:51:11.669 INFO:tasks.workunit.client.0.smithi029.stdout: create 159/36 file after the snapshot 2024-04-17T19:51:11.670 INFO:tasks.workunit.client.0.smithi029.stdout: create 159/37 file after the snapshot 2024-04-17T19:51:11.672 INFO:tasks.workunit.client.0.smithi029.stdout: create 159/38 file after the snapshot 2024-04-17T19:51:11.673 INFO:tasks.workunit.client.0.smithi029.stdout: create 159/39 file after the snapshot 2024-04-17T19:51:11.675 INFO:tasks.workunit.client.0.smithi029.stdout: create 160/.snap/snap-subdir-test 2024-04-17T19:51:11.699 INFO:tasks.workunit.client.0.smithi029.stdout: create 160/30 file after the snapshot 2024-04-17T19:51:11.701 INFO:tasks.workunit.client.0.smithi029.stdout: create 160/31 file after the snapshot 2024-04-17T19:51:11.703 INFO:tasks.workunit.client.0.smithi029.stdout: create 160/32 file after the snapshot 2024-04-17T19:51:11.710 INFO:tasks.workunit.client.0.smithi029.stdout: create 160/33 file after the snapshot 2024-04-17T19:51:11.967 INFO:tasks.workunit.client.0.smithi029.stdout: create 160/34 file after the snapshot 2024-04-17T19:51:11.969 INFO:tasks.workunit.client.0.smithi029.stdout: create 160/35 file after the snapshot 2024-04-17T19:51:11.970 INFO:tasks.workunit.client.0.smithi029.stdout: create 160/36 file after the snapshot 2024-04-17T19:51:11.972 INFO:tasks.workunit.client.0.smithi029.stdout: create 160/37 file after the snapshot 2024-04-17T19:51:11.973 INFO:tasks.workunit.client.0.smithi029.stdout: create 160/38 file after the snapshot 2024-04-17T19:51:11.974 INFO:tasks.workunit.client.0.smithi029.stdout: create 160/39 file after the snapshot 2024-04-17T19:51:11.976 INFO:tasks.workunit.client.0.smithi029.stdout: create 161/.snap/snap-subdir-test 2024-04-17T19:51:12.001 INFO:tasks.workunit.client.0.smithi029.stdout: create 161/30 file after the snapshot 2024-04-17T19:51:12.002 INFO:tasks.workunit.client.0.smithi029.stdout: create 161/31 file after the snapshot 2024-04-17T19:51:12.004 INFO:tasks.workunit.client.0.smithi029.stdout: create 161/32 file after the snapshot 2024-04-17T19:51:12.008 INFO:tasks.workunit.client.0.smithi029.stdout: create 161/33 file after the snapshot 2024-04-17T19:51:12.013 INFO:tasks.workunit.client.0.smithi029.stdout: create 161/34 file after the snapshot 2024-04-17T19:51:12.659 INFO:tasks.workunit.client.0.smithi029.stdout: create 161/35 file after the snapshot 2024-04-17T19:51:12.787 INFO:tasks.workunit.client.0.smithi029.stdout: create 161/36 file after the snapshot 2024-04-17T19:51:12.789 INFO:tasks.workunit.client.0.smithi029.stdout: create 161/37 file after the snapshot 2024-04-17T19:51:13.620 INFO:tasks.workunit.client.0.smithi029.stdout: create 161/38 file after the snapshot 2024-04-17T19:51:13.703 INFO:tasks.workunit.client.0.smithi029.stdout: create 161/39 file after the snapshot 2024-04-17T19:51:13.704 INFO:tasks.workunit.client.0.smithi029.stdout: create 162/.snap/snap-subdir-test 2024-04-17T19:51:13.737 INFO:tasks.workunit.client.0.smithi029.stdout: create 162/30 file after the snapshot 2024-04-17T19:51:13.744 INFO:tasks.workunit.client.0.smithi029.stdout: create 162/31 file after the snapshot 2024-04-17T19:51:13.751 INFO:tasks.workunit.client.0.smithi029.stdout: create 162/32 file after the snapshot 2024-04-17T19:51:14.460 INFO:tasks.workunit.client.0.smithi029.stdout: create 162/33 file after the snapshot 2024-04-17T19:51:14.461 INFO:tasks.workunit.client.0.smithi029.stdout: create 162/34 file after the snapshot 2024-04-17T19:51:14.462 INFO:tasks.workunit.client.0.smithi029.stdout: create 162/35 file after the snapshot 2024-04-17T19:51:14.463 INFO:tasks.workunit.client.0.smithi029.stdout: create 162/36 file after the snapshot 2024-04-17T19:51:14.464 INFO:tasks.workunit.client.0.smithi029.stdout: create 162/37 file after the snapshot 2024-04-17T19:51:14.465 INFO:tasks.workunit.client.0.smithi029.stdout: create 162/38 file after the snapshot 2024-04-17T19:51:14.466 INFO:tasks.workunit.client.0.smithi029.stdout: create 162/39 file after the snapshot 2024-04-17T19:51:14.468 INFO:tasks.workunit.client.0.smithi029.stdout: create 163/.snap/snap-subdir-test 2024-04-17T19:51:14.504 INFO:tasks.workunit.client.0.smithi029.stdout: create 163/30 file after the snapshot 2024-04-17T19:51:14.508 INFO:tasks.workunit.client.0.smithi029.stdout: create 163/31 file after the snapshot 2024-04-17T19:51:14.510 INFO:tasks.workunit.client.0.smithi029.stdout: create 163/32 file after the snapshot 2024-04-17T19:51:14.764 INFO:tasks.workunit.client.0.smithi029.stdout: create 163/33 file after the snapshot 2024-04-17T19:51:14.772 INFO:tasks.workunit.client.0.smithi029.stdout: create 163/34 file after the snapshot 2024-04-17T19:51:14.773 INFO:tasks.workunit.client.0.smithi029.stdout: create 163/35 file after the snapshot 2024-04-17T19:51:14.774 INFO:tasks.workunit.client.0.smithi029.stdout: create 163/36 file after the snapshot 2024-04-17T19:51:14.775 INFO:tasks.workunit.client.0.smithi029.stdout: create 163/37 file after the snapshot 2024-04-17T19:51:14.776 INFO:tasks.workunit.client.0.smithi029.stdout: create 163/38 file after the snapshot 2024-04-17T19:51:14.777 INFO:tasks.workunit.client.0.smithi029.stdout: create 163/39 file after the snapshot 2024-04-17T19:51:14.778 INFO:tasks.workunit.client.0.smithi029.stdout: create 164/.snap/snap-subdir-test 2024-04-17T19:51:14.815 INFO:tasks.workunit.client.0.smithi029.stdout: create 164/30 file after the snapshot 2024-04-17T19:51:14.819 INFO:tasks.workunit.client.0.smithi029.stdout: create 164/31 file after the snapshot 2024-04-17T19:51:14.826 INFO:tasks.workunit.client.0.smithi029.stdout: create 164/32 file after the snapshot 2024-04-17T19:51:14.835 INFO:tasks.workunit.client.0.smithi029.stdout: create 164/33 file after the snapshot 2024-04-17T19:51:14.836 INFO:tasks.workunit.client.0.smithi029.stdout: create 164/34 file after the snapshot 2024-04-17T19:51:14.837 INFO:tasks.workunit.client.0.smithi029.stdout: create 164/35 file after the snapshot 2024-04-17T19:51:14.838 INFO:tasks.workunit.client.0.smithi029.stdout: create 164/36 file after the snapshot 2024-04-17T19:51:14.839 INFO:tasks.workunit.client.0.smithi029.stdout: create 164/37 file after the snapshot 2024-04-17T19:51:14.840 INFO:tasks.workunit.client.0.smithi029.stdout: create 164/38 file after the snapshot 2024-04-17T19:51:14.841 INFO:tasks.workunit.client.0.smithi029.stdout: create 164/39 file after the snapshot 2024-04-17T19:51:14.842 INFO:tasks.workunit.client.0.smithi029.stdout: create 165/.snap/snap-subdir-test 2024-04-17T19:51:14.878 INFO:tasks.workunit.client.0.smithi029.stdout: create 165/30 file after the snapshot 2024-04-17T19:51:14.884 INFO:tasks.workunit.client.0.smithi029.stdout: create 165/31 file after the snapshot 2024-04-17T19:51:15.562 INFO:tasks.workunit.client.0.smithi029.stdout: create 165/32 file after the snapshot 2024-04-17T19:51:15.563 INFO:tasks.workunit.client.0.smithi029.stdout: create 165/33 file after the snapshot 2024-04-17T19:51:15.564 INFO:tasks.workunit.client.0.smithi029.stdout: create 165/34 file after the snapshot 2024-04-17T19:51:15.565 INFO:tasks.workunit.client.0.smithi029.stdout: create 165/35 file after the snapshot 2024-04-17T19:51:15.566 INFO:tasks.workunit.client.0.smithi029.stdout: create 165/36 file after the snapshot 2024-04-17T19:51:15.567 INFO:tasks.workunit.client.0.smithi029.stdout: create 165/37 file after the snapshot 2024-04-17T19:51:15.568 INFO:tasks.workunit.client.0.smithi029.stdout: create 165/38 file after the snapshot 2024-04-17T19:51:15.569 INFO:tasks.workunit.client.0.smithi029.stdout: create 165/39 file after the snapshot 2024-04-17T19:51:15.570 INFO:tasks.workunit.client.0.smithi029.stdout: create 166/.snap/snap-subdir-test 2024-04-17T19:51:15.593 INFO:tasks.workunit.client.0.smithi029.stdout: create 166/30 file after the snapshot 2024-04-17T19:51:15.595 INFO:tasks.workunit.client.0.smithi029.stdout: create 166/31 file after the snapshot 2024-04-17T19:51:15.598 INFO:tasks.workunit.client.0.smithi029.stdout: create 166/32 file after the snapshot 2024-04-17T19:51:15.602 INFO:tasks.workunit.client.0.smithi029.stdout: create 166/33 file after the snapshot 2024-04-17T19:51:15.610 INFO:tasks.workunit.client.0.smithi029.stdout: create 166/34 file after the snapshot 2024-04-17T19:51:15.612 INFO:tasks.workunit.client.0.smithi029.stdout: create 166/35 file after the snapshot 2024-04-17T19:51:15.613 INFO:tasks.workunit.client.0.smithi029.stdout: create 166/36 file after the snapshot 2024-04-17T19:51:15.614 INFO:tasks.workunit.client.0.smithi029.stdout: create 166/37 file after the snapshot 2024-04-17T19:51:15.615 INFO:tasks.workunit.client.0.smithi029.stdout: create 166/38 file after the snapshot 2024-04-17T19:51:15.616 INFO:tasks.workunit.client.0.smithi029.stdout: create 166/39 file after the snapshot 2024-04-17T19:51:15.617 INFO:tasks.workunit.client.0.smithi029.stdout: create 167/.snap/snap-subdir-test 2024-04-17T19:51:15.643 INFO:tasks.workunit.client.0.smithi029.stdout: create 167/30 file after the snapshot 2024-04-17T19:51:15.645 INFO:tasks.workunit.client.0.smithi029.stdout: create 167/31 file after the snapshot 2024-04-17T19:51:15.646 INFO:tasks.workunit.client.0.smithi029.stdout: create 167/32 file after the snapshot 2024-04-17T19:51:15.649 INFO:tasks.workunit.client.0.smithi029.stdout: create 167/33 file after the snapshot 2024-04-17T19:51:15.656 INFO:tasks.workunit.client.0.smithi029.stdout: create 167/34 file after the snapshot 2024-04-17T19:51:15.670 INFO:tasks.workunit.client.0.smithi029.stdout: create 167/35 file after the snapshot 2024-04-17T19:51:15.671 INFO:tasks.workunit.client.0.smithi029.stdout: create 167/36 file after the snapshot 2024-04-17T19:51:15.673 INFO:tasks.workunit.client.0.smithi029.stdout: create 167/37 file after the snapshot 2024-04-17T19:51:15.674 INFO:tasks.workunit.client.0.smithi029.stdout: create 167/38 file after the snapshot 2024-04-17T19:51:15.676 INFO:tasks.workunit.client.0.smithi029.stdout: create 167/39 file after the snapshot 2024-04-17T19:51:15.677 INFO:tasks.workunit.client.0.smithi029.stdout: create 168/.snap/snap-subdir-test 2024-04-17T19:51:15.710 INFO:tasks.workunit.client.0.smithi029.stdout: create 168/30 file after the snapshot 2024-04-17T19:51:15.712 INFO:tasks.workunit.client.0.smithi029.stdout: create 168/31 file after the snapshot 2024-04-17T19:51:15.713 INFO:tasks.workunit.client.0.smithi029.stdout: create 168/32 file after the snapshot 2024-04-17T19:51:15.714 INFO:tasks.workunit.client.0.smithi029.stdout: create 168/33 file after the snapshot 2024-04-17T19:51:15.716 INFO:tasks.workunit.client.0.smithi029.stdout: create 168/34 file after the snapshot 2024-04-17T19:51:15.718 INFO:tasks.workunit.client.0.smithi029.stdout: create 168/35 file after the snapshot 2024-04-17T19:51:15.724 INFO:tasks.workunit.client.0.smithi029.stdout: create 168/36 file after the snapshot 2024-04-17T19:51:15.731 INFO:tasks.workunit.client.0.smithi029.stdout: create 168/37 file after the snapshot 2024-04-17T19:51:15.732 INFO:tasks.workunit.client.0.smithi029.stdout: create 168/38 file after the snapshot 2024-04-17T19:51:15.733 INFO:tasks.workunit.client.0.smithi029.stdout: create 168/39 file after the snapshot 2024-04-17T19:51:15.734 INFO:tasks.workunit.client.0.smithi029.stdout: create 169/.snap/snap-subdir-test 2024-04-17T19:51:15.761 INFO:tasks.workunit.client.0.smithi029.stdout: create 169/30 file after the snapshot 2024-04-17T19:51:15.763 INFO:tasks.workunit.client.0.smithi029.stdout: create 169/31 file after the snapshot 2024-04-17T19:51:15.765 INFO:tasks.workunit.client.0.smithi029.stdout: create 169/32 file after the snapshot 2024-04-17T19:51:15.769 INFO:tasks.workunit.client.0.smithi029.stdout: create 169/33 file after the snapshot 2024-04-17T19:51:15.778 INFO:tasks.workunit.client.0.smithi029.stdout: create 169/34 file after the snapshot 2024-04-17T19:51:15.780 INFO:tasks.workunit.client.0.smithi029.stdout: create 169/35 file after the snapshot 2024-04-17T19:51:15.780 INFO:tasks.workunit.client.0.smithi029.stdout: create 169/36 file after the snapshot 2024-04-17T19:51:15.781 INFO:tasks.workunit.client.0.smithi029.stdout: create 169/37 file after the snapshot 2024-04-17T19:51:15.782 INFO:tasks.workunit.client.0.smithi029.stdout: create 169/38 file after the snapshot 2024-04-17T19:51:15.783 INFO:tasks.workunit.client.0.smithi029.stdout: create 169/39 file after the snapshot 2024-04-17T19:51:15.784 INFO:tasks.workunit.client.0.smithi029.stdout: create 170/.snap/snap-subdir-test 2024-04-17T19:51:15.808 INFO:tasks.workunit.client.0.smithi029.stdout: create 170/30 file after the snapshot 2024-04-17T19:51:15.810 INFO:tasks.workunit.client.0.smithi029.stdout: create 170/31 file after the snapshot 2024-04-17T19:51:15.811 INFO:tasks.workunit.client.0.smithi029.stdout: create 170/32 file after the snapshot 2024-04-17T19:51:15.816 INFO:tasks.workunit.client.0.smithi029.stdout: create 170/33 file after the snapshot 2024-04-17T19:51:15.822 INFO:tasks.workunit.client.0.smithi029.stdout: create 170/34 file after the snapshot 2024-04-17T19:51:16.628 INFO:tasks.workunit.client.0.smithi029.stdout: create 170/35 file after the snapshot 2024-04-17T19:51:16.662 INFO:tasks.workunit.client.0.smithi029.stdout: create 170/36 file after the snapshot 2024-04-17T19:51:16.663 INFO:tasks.workunit.client.0.smithi029.stdout: create 170/37 file after the snapshot 2024-04-17T19:51:16.664 INFO:tasks.workunit.client.0.smithi029.stdout: create 170/38 file after the snapshot 2024-04-17T19:51:16.665 INFO:tasks.workunit.client.0.smithi029.stdout: create 170/39 file after the snapshot 2024-04-17T19:51:16.666 INFO:tasks.workunit.client.0.smithi029.stdout: create 171/.snap/snap-subdir-test 2024-04-17T19:51:16.695 INFO:tasks.workunit.client.0.smithi029.stdout: create 171/30 file after the snapshot 2024-04-17T19:51:16.696 INFO:tasks.workunit.client.0.smithi029.stdout: create 171/31 file after the snapshot 2024-04-17T19:51:16.698 INFO:tasks.workunit.client.0.smithi029.stdout: create 171/32 file after the snapshot 2024-04-17T19:51:16.703 INFO:tasks.workunit.client.0.smithi029.stdout: create 171/33 file after the snapshot 2024-04-17T19:51:16.709 INFO:tasks.workunit.client.0.smithi029.stdout: create 171/34 file after the snapshot 2024-04-17T19:51:16.711 INFO:tasks.workunit.client.0.smithi029.stdout: create 171/35 file after the snapshot 2024-04-17T19:51:16.713 INFO:tasks.workunit.client.0.smithi029.stdout: create 171/36 file after the snapshot 2024-04-17T19:51:16.714 INFO:tasks.workunit.client.0.smithi029.stdout: create 171/37 file after the snapshot 2024-04-17T19:51:16.714 INFO:tasks.workunit.client.0.smithi029.stdout: create 171/38 file after the snapshot 2024-04-17T19:51:16.716 INFO:tasks.workunit.client.0.smithi029.stdout: create 171/39 file after the snapshot 2024-04-17T19:51:16.716 INFO:tasks.workunit.client.0.smithi029.stdout: create 172/.snap/snap-subdir-test 2024-04-17T19:51:16.749 INFO:tasks.workunit.client.0.smithi029.stdout: create 172/30 file after the snapshot 2024-04-17T19:51:16.751 INFO:tasks.workunit.client.0.smithi029.stdout: create 172/31 file after the snapshot 2024-04-17T19:51:16.755 INFO:tasks.workunit.client.0.smithi029.stdout: create 172/32 file after the snapshot 2024-04-17T19:51:16.766 INFO:tasks.workunit.client.0.smithi029.stdout: create 172/33 file after the snapshot 2024-04-17T19:51:16.773 INFO:tasks.workunit.client.0.smithi029.stdout: create 172/34 file after the snapshot 2024-04-17T19:51:16.775 INFO:tasks.workunit.client.0.smithi029.stdout: create 172/35 file after the snapshot 2024-04-17T19:51:16.776 INFO:tasks.workunit.client.0.smithi029.stdout: create 172/36 file after the snapshot 2024-04-17T19:51:16.778 INFO:tasks.workunit.client.0.smithi029.stdout: create 172/37 file after the snapshot 2024-04-17T19:51:16.780 INFO:tasks.workunit.client.0.smithi029.stdout: create 172/38 file after the snapshot 2024-04-17T19:51:16.781 INFO:tasks.workunit.client.0.smithi029.stdout: create 172/39 file after the snapshot 2024-04-17T19:51:16.782 INFO:tasks.workunit.client.0.smithi029.stdout: create 173/.snap/snap-subdir-test 2024-04-17T19:51:16.812 INFO:tasks.workunit.client.0.smithi029.stdout: create 173/30 file after the snapshot 2024-04-17T19:51:16.813 INFO:tasks.workunit.client.0.smithi029.stdout: create 173/31 file after the snapshot 2024-04-17T19:51:16.818 INFO:tasks.workunit.client.0.smithi029.stdout: create 173/32 file after the snapshot 2024-04-17T19:51:16.823 INFO:tasks.workunit.client.0.smithi029.stdout: create 173/33 file after the snapshot 2024-04-17T19:51:16.827 INFO:tasks.workunit.client.0.smithi029.stdout: create 173/34 file after the snapshot 2024-04-17T19:51:16.831 INFO:tasks.workunit.client.0.smithi029.stdout: create 173/35 file after the snapshot 2024-04-17T19:51:16.832 INFO:tasks.workunit.client.0.smithi029.stdout: create 173/36 file after the snapshot 2024-04-17T19:51:16.832 INFO:tasks.workunit.client.0.smithi029.stdout: create 173/37 file after the snapshot 2024-04-17T19:51:16.834 INFO:tasks.workunit.client.0.smithi029.stdout: create 173/38 file after the snapshot 2024-04-17T19:51:16.834 INFO:tasks.workunit.client.0.smithi029.stdout: create 173/39 file after the snapshot 2024-04-17T19:51:16.835 INFO:tasks.workunit.client.0.smithi029.stdout: create 174/.snap/snap-subdir-test 2024-04-17T19:51:16.867 INFO:tasks.workunit.client.0.smithi029.stdout: create 174/30 file after the snapshot 2024-04-17T19:51:17.120 INFO:tasks.workunit.client.0.smithi029.stdout: create 174/31 file after the snapshot 2024-04-17T19:51:17.130 INFO:tasks.workunit.client.0.smithi029.stdout: create 174/32 file after the snapshot 2024-04-17T19:51:17.131 INFO:tasks.workunit.client.0.smithi029.stdout: create 174/33 file after the snapshot 2024-04-17T19:51:17.132 INFO:tasks.workunit.client.0.smithi029.stdout: create 174/34 file after the snapshot 2024-04-17T19:51:17.133 INFO:tasks.workunit.client.0.smithi029.stdout: create 174/35 file after the snapshot 2024-04-17T19:51:17.134 INFO:tasks.workunit.client.0.smithi029.stdout: create 174/36 file after the snapshot 2024-04-17T19:51:17.135 INFO:tasks.workunit.client.0.smithi029.stdout: create 174/37 file after the snapshot 2024-04-17T19:51:17.136 INFO:tasks.workunit.client.0.smithi029.stdout: create 174/38 file after the snapshot 2024-04-17T19:51:17.137 INFO:tasks.workunit.client.0.smithi029.stdout: create 174/39 file after the snapshot 2024-04-17T19:51:17.138 INFO:tasks.workunit.client.0.smithi029.stdout: create 175/.snap/snap-subdir-test 2024-04-17T19:51:17.173 INFO:tasks.workunit.client.0.smithi029.stdout: create 175/30 file after the snapshot 2024-04-17T19:51:17.175 INFO:tasks.workunit.client.0.smithi029.stdout: create 175/31 file after the snapshot 2024-04-17T19:51:17.176 INFO:tasks.workunit.client.0.smithi029.stdout: create 175/32 file after the snapshot 2024-04-17T19:51:17.178 INFO:tasks.workunit.client.0.smithi029.stdout: create 175/33 file after the snapshot 2024-04-17T19:51:17.180 INFO:tasks.workunit.client.0.smithi029.stdout: create 175/34 file after the snapshot 2024-04-17T19:51:17.184 INFO:tasks.workunit.client.0.smithi029.stdout: create 175/35 file after the snapshot 2024-04-17T19:51:17.199 INFO:tasks.workunit.client.0.smithi029.stdout: create 175/36 file after the snapshot 2024-04-17T19:51:17.200 INFO:tasks.workunit.client.0.smithi029.stdout: create 175/37 file after the snapshot 2024-04-17T19:51:17.202 INFO:tasks.workunit.client.0.smithi029.stdout: create 175/38 file after the snapshot 2024-04-17T19:51:17.203 INFO:tasks.workunit.client.0.smithi029.stdout: create 175/39 file after the snapshot 2024-04-17T19:51:17.205 INFO:tasks.workunit.client.0.smithi029.stdout: create 176/.snap/snap-subdir-test 2024-04-17T19:51:17.239 INFO:tasks.workunit.client.0.smithi029.stdout: create 176/30 file after the snapshot 2024-04-17T19:51:17.241 INFO:tasks.workunit.client.0.smithi029.stdout: create 176/31 file after the snapshot 2024-04-17T19:51:17.242 INFO:tasks.workunit.client.0.smithi029.stdout: create 176/32 file after the snapshot 2024-04-17T19:51:17.246 INFO:tasks.workunit.client.0.smithi029.stdout: create 176/33 file after the snapshot 2024-04-17T19:51:17.254 INFO:tasks.workunit.client.0.smithi029.stdout: create 176/34 file after the snapshot 2024-04-17T19:51:17.258 INFO:tasks.workunit.client.0.smithi029.stdout: create 176/35 file after the snapshot 2024-04-17T19:51:17.259 INFO:tasks.workunit.client.0.smithi029.stdout: create 176/36 file after the snapshot 2024-04-17T19:51:17.260 INFO:tasks.workunit.client.0.smithi029.stdout: create 176/37 file after the snapshot 2024-04-17T19:51:17.261 INFO:tasks.workunit.client.0.smithi029.stdout: create 176/38 file after the snapshot 2024-04-17T19:51:17.262 INFO:tasks.workunit.client.0.smithi029.stdout: create 176/39 file after the snapshot 2024-04-17T19:51:17.262 INFO:tasks.workunit.client.0.smithi029.stdout: create 177/.snap/snap-subdir-test 2024-04-17T19:51:17.289 INFO:tasks.workunit.client.0.smithi029.stdout: create 177/30 file after the snapshot 2024-04-17T19:51:17.292 INFO:tasks.workunit.client.0.smithi029.stdout: create 177/31 file after the snapshot 2024-04-17T19:51:17.298 INFO:tasks.workunit.client.0.smithi029.stdout: create 177/32 file after the snapshot 2024-04-17T19:51:17.307 INFO:tasks.workunit.client.0.smithi029.stdout: create 177/33 file after the snapshot 2024-04-17T19:51:17.308 INFO:tasks.workunit.client.0.smithi029.stdout: create 177/34 file after the snapshot 2024-04-17T19:51:17.309 INFO:tasks.workunit.client.0.smithi029.stdout: create 177/35 file after the snapshot 2024-04-17T19:51:17.310 INFO:tasks.workunit.client.0.smithi029.stdout: create 177/36 file after the snapshot 2024-04-17T19:51:17.311 INFO:tasks.workunit.client.0.smithi029.stdout: create 177/37 file after the snapshot 2024-04-17T19:51:17.312 INFO:tasks.workunit.client.0.smithi029.stdout: create 177/38 file after the snapshot 2024-04-17T19:51:17.313 INFO:tasks.workunit.client.0.smithi029.stdout: create 177/39 file after the snapshot 2024-04-17T19:51:17.313 INFO:tasks.workunit.client.0.smithi029.stdout: create 178/.snap/snap-subdir-test 2024-04-17T19:51:17.340 INFO:tasks.workunit.client.0.smithi029.stdout: create 178/30 file after the snapshot 2024-04-17T19:51:17.342 INFO:tasks.workunit.client.0.smithi029.stdout: create 178/31 file after the snapshot 2024-04-17T19:51:17.343 INFO:tasks.workunit.client.0.smithi029.stdout: create 178/32 file after the snapshot 2024-04-17T19:51:17.345 INFO:tasks.workunit.client.0.smithi029.stdout: create 178/33 file after the snapshot 2024-04-17T19:51:17.349 INFO:tasks.workunit.client.0.smithi029.stdout: create 178/34 file after the snapshot 2024-04-17T19:51:17.355 INFO:tasks.workunit.client.0.smithi029.stdout: create 178/35 file after the snapshot 2024-04-17T19:51:17.361 INFO:tasks.workunit.client.0.smithi029.stdout: create 178/36 file after the snapshot 2024-04-17T19:51:17.361 INFO:tasks.workunit.client.0.smithi029.stdout: create 178/37 file after the snapshot 2024-04-17T19:51:17.362 INFO:tasks.workunit.client.0.smithi029.stdout: create 178/38 file after the snapshot 2024-04-17T19:51:17.363 INFO:tasks.workunit.client.0.smithi029.stdout: create 178/39 file after the snapshot 2024-04-17T19:51:17.364 INFO:tasks.workunit.client.0.smithi029.stdout: create 179/.snap/snap-subdir-test 2024-04-17T19:51:17.399 INFO:tasks.workunit.client.0.smithi029.stdout: create 179/30 file after the snapshot 2024-04-17T19:51:17.401 INFO:tasks.workunit.client.0.smithi029.stdout: create 179/31 file after the snapshot 2024-04-17T19:51:17.405 INFO:tasks.workunit.client.0.smithi029.stdout: create 179/32 file after the snapshot 2024-04-17T19:51:17.413 INFO:tasks.workunit.client.0.smithi029.stdout: create 179/33 file after the snapshot 2024-04-17T19:51:17.416 INFO:tasks.workunit.client.0.smithi029.stdout: create 179/34 file after the snapshot 2024-04-17T19:51:17.416 INFO:tasks.workunit.client.0.smithi029.stdout: create 179/35 file after the snapshot 2024-04-17T19:51:17.417 INFO:tasks.workunit.client.0.smithi029.stdout: create 179/36 file after the snapshot 2024-04-17T19:51:17.418 INFO:tasks.workunit.client.0.smithi029.stdout: create 179/37 file after the snapshot 2024-04-17T19:51:17.419 INFO:tasks.workunit.client.0.smithi029.stdout: create 179/38 file after the snapshot 2024-04-17T19:51:17.420 INFO:tasks.workunit.client.0.smithi029.stdout: create 179/39 file after the snapshot 2024-04-17T19:51:17.421 INFO:tasks.workunit.client.0.smithi029.stdout: create 180/.snap/snap-subdir-test 2024-04-17T19:51:17.453 INFO:tasks.workunit.client.0.smithi029.stdout: create 180/30 file after the snapshot 2024-04-17T19:51:17.456 INFO:tasks.workunit.client.0.smithi029.stdout: create 180/31 file after the snapshot 2024-04-17T19:51:17.458 INFO:tasks.workunit.client.0.smithi029.stdout: create 180/32 file after the snapshot 2024-04-17T19:51:17.462 INFO:tasks.workunit.client.0.smithi029.stdout: create 180/33 file after the snapshot 2024-04-17T19:51:17.465 INFO:tasks.workunit.client.0.smithi029.stdout: create 180/34 file after the snapshot 2024-04-17T19:51:17.470 INFO:tasks.workunit.client.0.smithi029.stdout: create 180/35 file after the snapshot 2024-04-17T19:51:17.472 INFO:tasks.workunit.client.0.smithi029.stdout: create 180/36 file after the snapshot 2024-04-17T19:51:17.473 INFO:tasks.workunit.client.0.smithi029.stdout: create 180/37 file after the snapshot 2024-04-17T19:51:17.474 INFO:tasks.workunit.client.0.smithi029.stdout: create 180/38 file after the snapshot 2024-04-17T19:51:17.475 INFO:tasks.workunit.client.0.smithi029.stdout: create 180/39 file after the snapshot 2024-04-17T19:51:17.476 INFO:tasks.workunit.client.0.smithi029.stdout: create 181/.snap/snap-subdir-test 2024-04-17T19:51:17.502 INFO:tasks.workunit.client.0.smithi029.stdout: create 181/30 file after the snapshot 2024-04-17T19:51:17.504 INFO:tasks.workunit.client.0.smithi029.stdout: create 181/31 file after the snapshot 2024-04-17T19:51:17.510 INFO:tasks.workunit.client.0.smithi029.stdout: create 181/32 file after the snapshot 2024-04-17T19:51:17.518 INFO:tasks.workunit.client.0.smithi029.stdout: create 181/33 file after the snapshot 2024-04-17T19:51:17.519 INFO:tasks.workunit.client.0.smithi029.stdout: create 181/34 file after the snapshot 2024-04-17T19:51:17.520 INFO:tasks.workunit.client.0.smithi029.stdout: create 181/35 file after the snapshot 2024-04-17T19:51:17.521 INFO:tasks.workunit.client.0.smithi029.stdout: create 181/36 file after the snapshot 2024-04-17T19:51:17.521 INFO:tasks.workunit.client.0.smithi029.stdout: create 181/37 file after the snapshot 2024-04-17T19:51:17.522 INFO:tasks.workunit.client.0.smithi029.stdout: create 181/38 file after the snapshot 2024-04-17T19:51:17.523 INFO:tasks.workunit.client.0.smithi029.stdout: create 181/39 file after the snapshot 2024-04-17T19:51:17.524 INFO:tasks.workunit.client.0.smithi029.stdout: create 182/.snap/snap-subdir-test 2024-04-17T19:51:17.552 INFO:tasks.workunit.client.0.smithi029.stdout: create 182/30 file after the snapshot 2024-04-17T19:51:17.554 INFO:tasks.workunit.client.0.smithi029.stdout: create 182/31 file after the snapshot 2024-04-17T19:51:17.559 INFO:tasks.workunit.client.0.smithi029.stdout: create 182/32 file after the snapshot 2024-04-17T19:51:17.567 INFO:tasks.workunit.client.0.smithi029.stdout: create 182/33 file after the snapshot 2024-04-17T19:51:17.569 INFO:tasks.workunit.client.0.smithi029.stdout: create 182/34 file after the snapshot 2024-04-17T19:51:17.570 INFO:tasks.workunit.client.0.smithi029.stdout: create 182/35 file after the snapshot 2024-04-17T19:51:17.571 INFO:tasks.workunit.client.0.smithi029.stdout: create 182/36 file after the snapshot 2024-04-17T19:51:17.572 INFO:tasks.workunit.client.0.smithi029.stdout: create 182/37 file after the snapshot 2024-04-17T19:51:17.573 INFO:tasks.workunit.client.0.smithi029.stdout: create 182/38 file after the snapshot 2024-04-17T19:51:17.574 INFO:tasks.workunit.client.0.smithi029.stdout: create 182/39 file after the snapshot 2024-04-17T19:51:17.575 INFO:tasks.workunit.client.0.smithi029.stdout: create 183/.snap/snap-subdir-test 2024-04-17T19:51:17.601 INFO:tasks.workunit.client.0.smithi029.stdout: create 183/30 file after the snapshot 2024-04-17T19:51:17.604 INFO:tasks.workunit.client.0.smithi029.stdout: create 183/31 file after the snapshot 2024-04-17T19:51:17.605 INFO:tasks.workunit.client.0.smithi029.stdout: create 183/32 file after the snapshot 2024-04-17T19:51:17.611 INFO:tasks.workunit.client.0.smithi029.stdout: create 183/33 file after the snapshot 2024-04-17T19:51:17.619 INFO:tasks.workunit.client.0.smithi029.stdout: create 183/34 file after the snapshot 2024-04-17T19:51:17.620 INFO:tasks.workunit.client.0.smithi029.stdout: create 183/35 file after the snapshot 2024-04-17T19:51:17.621 INFO:tasks.workunit.client.0.smithi029.stdout: create 183/36 file after the snapshot 2024-04-17T19:51:17.621 INFO:tasks.workunit.client.0.smithi029.stdout: create 183/37 file after the snapshot 2024-04-17T19:51:17.622 INFO:tasks.workunit.client.0.smithi029.stdout: create 183/38 file after the snapshot 2024-04-17T19:51:17.623 INFO:tasks.workunit.client.0.smithi029.stdout: create 183/39 file after the snapshot 2024-04-17T19:51:17.624 INFO:tasks.workunit.client.0.smithi029.stdout: create 184/.snap/snap-subdir-test 2024-04-17T19:51:17.655 INFO:tasks.workunit.client.0.smithi029.stdout: create 184/30 file after the snapshot 2024-04-17T19:51:17.656 INFO:tasks.workunit.client.0.smithi029.stdout: create 184/31 file after the snapshot 2024-04-17T19:51:17.657 INFO:tasks.workunit.client.0.smithi029.stdout: create 184/32 file after the snapshot 2024-04-17T19:51:17.659 INFO:tasks.workunit.client.0.smithi029.stdout: create 184/33 file after the snapshot 2024-04-17T19:51:17.660 INFO:tasks.workunit.client.0.smithi029.stdout: create 184/34 file after the snapshot 2024-04-17T19:51:17.662 INFO:tasks.workunit.client.0.smithi029.stdout: create 184/35 file after the snapshot 2024-04-17T19:51:17.664 INFO:tasks.workunit.client.0.smithi029.stdout: create 184/36 file after the snapshot 2024-04-17T19:51:17.915 INFO:tasks.workunit.client.0.smithi029.stdout: create 184/37 file after the snapshot 2024-04-17T19:51:17.923 INFO:tasks.workunit.client.0.smithi029.stdout: create 184/38 file after the snapshot 2024-04-17T19:51:17.925 INFO:tasks.workunit.client.0.smithi029.stdout: create 184/39 file after the snapshot 2024-04-17T19:51:17.926 INFO:tasks.workunit.client.0.smithi029.stdout: create 185/.snap/snap-subdir-test 2024-04-17T19:51:17.962 INFO:tasks.workunit.client.0.smithi029.stdout: create 185/30 file after the snapshot 2024-04-17T19:51:17.964 INFO:tasks.workunit.client.0.smithi029.stdout: create 185/31 file after the snapshot 2024-04-17T19:51:17.965 INFO:tasks.workunit.client.0.smithi029.stdout: create 185/32 file after the snapshot 2024-04-17T19:51:17.968 INFO:tasks.workunit.client.0.smithi029.stdout: create 185/33 file after the snapshot 2024-04-17T19:51:17.970 INFO:tasks.workunit.client.0.smithi029.stdout: create 185/34 file after the snapshot 2024-04-17T19:51:17.975 INFO:tasks.workunit.client.0.smithi029.stdout: create 185/35 file after the snapshot 2024-04-17T19:51:17.981 INFO:tasks.workunit.client.0.smithi029.stdout: create 185/36 file after the snapshot 2024-04-17T19:51:17.982 INFO:tasks.workunit.client.0.smithi029.stdout: create 185/37 file after the snapshot 2024-04-17T19:51:17.983 INFO:tasks.workunit.client.0.smithi029.stdout: create 185/38 file after the snapshot 2024-04-17T19:51:17.984 INFO:tasks.workunit.client.0.smithi029.stdout: create 185/39 file after the snapshot 2024-04-17T19:51:17.985 INFO:tasks.workunit.client.0.smithi029.stdout: create 186/.snap/snap-subdir-test 2024-04-17T19:51:18.019 INFO:tasks.workunit.client.0.smithi029.stdout: create 186/30 file after the snapshot 2024-04-17T19:51:18.021 INFO:tasks.workunit.client.0.smithi029.stdout: create 186/31 file after the snapshot 2024-04-17T19:51:18.025 INFO:tasks.workunit.client.0.smithi029.stdout: create 186/32 file after the snapshot 2024-04-17T19:51:18.029 INFO:tasks.workunit.client.0.smithi029.stdout: create 186/33 file after the snapshot 2024-04-17T19:51:18.034 INFO:tasks.workunit.client.0.smithi029.stdout: create 186/34 file after the snapshot 2024-04-17T19:51:18.647 INFO:tasks.workunit.client.0.smithi029.stdout: create 186/35 file after the snapshot 2024-04-17T19:51:18.901 INFO:tasks.workunit.client.0.smithi029.stdout: create 186/36 file after the snapshot 2024-04-17T19:51:18.903 INFO:tasks.workunit.client.0.smithi029.stdout: create 186/37 file after the snapshot 2024-04-17T19:51:18.905 INFO:tasks.workunit.client.0.smithi029.stdout: create 186/38 file after the snapshot 2024-04-17T19:51:18.906 INFO:tasks.workunit.client.0.smithi029.stdout: create 186/39 file after the snapshot 2024-04-17T19:51:18.908 INFO:tasks.workunit.client.0.smithi029.stdout: create 187/.snap/snap-subdir-test 2024-04-17T19:51:18.938 INFO:tasks.workunit.client.0.smithi029.stdout: create 187/30 file after the snapshot 2024-04-17T19:51:18.940 INFO:tasks.workunit.client.0.smithi029.stdout: create 187/31 file after the snapshot 2024-04-17T19:51:18.943 INFO:tasks.workunit.client.0.smithi029.stdout: create 187/32 file after the snapshot 2024-04-17T19:51:18.946 INFO:tasks.workunit.client.0.smithi029.stdout: create 187/33 file after the snapshot 2024-04-17T19:51:18.953 INFO:tasks.workunit.client.0.smithi029.stdout: create 187/34 file after the snapshot 2024-04-17T19:51:18.959 INFO:tasks.workunit.client.0.smithi029.stdout: create 187/35 file after the snapshot 2024-04-17T19:51:18.960 INFO:tasks.workunit.client.0.smithi029.stdout: create 187/36 file after the snapshot 2024-04-17T19:51:18.961 INFO:tasks.workunit.client.0.smithi029.stdout: create 187/37 file after the snapshot 2024-04-17T19:51:18.963 INFO:tasks.workunit.client.0.smithi029.stdout: create 187/38 file after the snapshot 2024-04-17T19:51:18.964 INFO:tasks.workunit.client.0.smithi029.stdout: create 187/39 file after the snapshot 2024-04-17T19:51:18.965 INFO:tasks.workunit.client.0.smithi029.stdout: create 188/.snap/snap-subdir-test 2024-04-17T19:51:18.993 INFO:tasks.workunit.client.0.smithi029.stdout: create 188/30 file after the snapshot 2024-04-17T19:51:19.247 INFO:tasks.workunit.client.0.smithi029.stdout: create 188/31 file after the snapshot 2024-04-17T19:51:19.259 INFO:tasks.workunit.client.0.smithi029.stdout: create 188/32 file after the snapshot 2024-04-17T19:51:19.260 INFO:tasks.workunit.client.0.smithi029.stdout: create 188/33 file after the snapshot 2024-04-17T19:51:19.261 INFO:tasks.workunit.client.0.smithi029.stdout: create 188/34 file after the snapshot 2024-04-17T19:51:19.262 INFO:tasks.workunit.client.0.smithi029.stdout: create 188/35 file after the snapshot 2024-04-17T19:51:19.263 INFO:tasks.workunit.client.0.smithi029.stdout: create 188/36 file after the snapshot 2024-04-17T19:51:19.264 INFO:tasks.workunit.client.0.smithi029.stdout: create 188/37 file after the snapshot 2024-04-17T19:51:19.265 INFO:tasks.workunit.client.0.smithi029.stdout: create 188/38 file after the snapshot 2024-04-17T19:51:19.266 INFO:tasks.workunit.client.0.smithi029.stdout: create 188/39 file after the snapshot 2024-04-17T19:51:19.267 INFO:tasks.workunit.client.0.smithi029.stdout: create 189/.snap/snap-subdir-test 2024-04-17T19:51:19.296 INFO:tasks.workunit.client.0.smithi029.stdout: create 189/30 file after the snapshot 2024-04-17T19:51:19.297 INFO:tasks.workunit.client.0.smithi029.stdout: create 189/31 file after the snapshot 2024-04-17T19:51:19.298 INFO:tasks.workunit.client.0.smithi029.stdout: create 189/32 file after the snapshot 2024-04-17T19:51:19.299 INFO:tasks.workunit.client.0.smithi029.stdout: create 189/33 file after the snapshot 2024-04-17T19:51:19.301 INFO:tasks.workunit.client.0.smithi029.stdout: create 189/34 file after the snapshot 2024-04-17T19:51:19.302 INFO:tasks.workunit.client.0.smithi029.stdout: create 189/35 file after the snapshot 2024-04-17T19:51:19.305 INFO:tasks.workunit.client.0.smithi029.stdout: create 189/36 file after the snapshot 2024-04-17T19:51:19.310 INFO:tasks.workunit.client.0.smithi029.stdout: create 189/37 file after the snapshot 2024-04-17T19:51:19.317 INFO:tasks.workunit.client.0.smithi029.stdout: create 189/38 file after the snapshot 2024-04-17T19:51:19.318 INFO:tasks.workunit.client.0.smithi029.stdout: create 189/39 file after the snapshot 2024-04-17T19:51:19.319 INFO:tasks.workunit.client.0.smithi029.stdout: create 190/.snap/snap-subdir-test 2024-04-17T19:51:19.345 INFO:tasks.workunit.client.0.smithi029.stdout: create 190/30 file after the snapshot 2024-04-17T19:51:19.348 INFO:tasks.workunit.client.0.smithi029.stdout: create 190/31 file after the snapshot 2024-04-17T19:51:19.352 INFO:tasks.workunit.client.0.smithi029.stdout: create 190/32 file after the snapshot 2024-04-17T19:51:19.357 INFO:tasks.workunit.client.0.smithi029.stdout: create 190/33 file after the snapshot 2024-04-17T19:51:19.364 INFO:tasks.workunit.client.0.smithi029.stdout: create 190/34 file after the snapshot 2024-04-17T19:51:19.365 INFO:tasks.workunit.client.0.smithi029.stdout: create 190/35 file after the snapshot 2024-04-17T19:51:19.366 INFO:tasks.workunit.client.0.smithi029.stdout: create 190/36 file after the snapshot 2024-04-17T19:51:19.367 INFO:tasks.workunit.client.0.smithi029.stdout: create 190/37 file after the snapshot 2024-04-17T19:51:19.368 INFO:tasks.workunit.client.0.smithi029.stdout: create 190/38 file after the snapshot 2024-04-17T19:51:19.368 INFO:tasks.workunit.client.0.smithi029.stdout: create 190/39 file after the snapshot 2024-04-17T19:51:19.635 INFO:tasks.workunit.client.0.smithi029.stdout: create 191/.snap/snap-subdir-test 2024-04-17T19:51:20.122 INFO:tasks.workunit.client.0.smithi029.stdout: create 191/30 file after the snapshot 2024-04-17T19:51:20.123 INFO:tasks.workunit.client.0.smithi029.stdout: create 191/31 file after the snapshot 2024-04-17T19:51:20.124 INFO:tasks.workunit.client.0.smithi029.stdout: create 191/32 file after the snapshot 2024-04-17T19:51:20.125 INFO:tasks.workunit.client.0.smithi029.stdout: create 191/33 file after the snapshot 2024-04-17T19:51:20.129 INFO:tasks.workunit.client.0.smithi029.stdout: create 191/34 file after the snapshot 2024-04-17T19:51:20.138 INFO:tasks.workunit.client.0.smithi029.stdout: create 191/35 file after the snapshot 2024-04-17T19:51:20.142 INFO:tasks.workunit.client.0.smithi029.stdout: create 191/36 file after the snapshot 2024-04-17T19:51:20.143 INFO:tasks.workunit.client.0.smithi029.stdout: create 191/37 file after the snapshot 2024-04-17T19:51:20.145 INFO:tasks.workunit.client.0.smithi029.stdout: create 191/38 file after the snapshot 2024-04-17T19:51:20.146 INFO:tasks.workunit.client.0.smithi029.stdout: create 191/39 file after the snapshot 2024-04-17T19:51:20.148 INFO:tasks.workunit.client.0.smithi029.stdout: create 192/.snap/snap-subdir-test 2024-04-17T19:51:20.601 INFO:tasks.workunit.client.0.smithi029.stdout: create 192/30 file after the snapshot 2024-04-17T19:51:20.603 INFO:tasks.workunit.client.0.smithi029.stdout: create 192/31 file after the snapshot 2024-04-17T19:51:20.605 INFO:tasks.workunit.client.0.smithi029.stdout: create 192/32 file after the snapshot 2024-04-17T19:51:20.608 INFO:tasks.workunit.client.0.smithi029.stdout: create 192/33 file after the snapshot 2024-04-17T19:51:20.616 INFO:tasks.workunit.client.0.smithi029.stdout: create 192/34 file after the snapshot 2024-04-17T19:51:20.622 INFO:tasks.workunit.client.0.smithi029.stdout: create 192/35 file after the snapshot 2024-04-17T19:51:20.623 INFO:tasks.workunit.client.0.smithi029.stdout: create 192/36 file after the snapshot 2024-04-17T19:51:20.624 INFO:tasks.workunit.client.0.smithi029.stdout: create 192/37 file after the snapshot 2024-04-17T19:51:20.625 INFO:tasks.workunit.client.0.smithi029.stdout: create 192/38 file after the snapshot 2024-04-17T19:51:20.627 INFO:tasks.workunit.client.0.smithi029.stdout: create 192/39 file after the snapshot 2024-04-17T19:51:20.628 INFO:tasks.workunit.client.0.smithi029.stdout: create 193/.snap/snap-subdir-test 2024-04-17T19:51:20.648 INFO:tasks.workunit.client.0.smithi029.stdout: create 193/30 file after the snapshot 2024-04-17T19:51:20.649 INFO:tasks.workunit.client.0.smithi029.stdout: create 193/31 file after the snapshot 2024-04-17T19:51:20.651 INFO:tasks.workunit.client.0.smithi029.stdout: create 193/32 file after the snapshot 2024-04-17T19:51:20.653 INFO:tasks.workunit.client.0.smithi029.stdout: create 193/33 file after the snapshot 2024-04-17T19:51:20.659 INFO:tasks.workunit.client.0.smithi029.stdout: create 193/34 file after the snapshot 2024-04-17T19:51:21.127 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:51:21.131 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:51:21.603 INFO:tasks.workunit.client.0.smithi029.stdout: create 193/35 file after the snapshot 2024-04-17T19:51:21.604 INFO:tasks.workunit.client.0.smithi029.stdout: create 193/36 file after the snapshot 2024-04-17T19:51:21.606 INFO:tasks.workunit.client.0.smithi029.stdout: create 193/37 file after the snapshot 2024-04-17T19:51:21.608 INFO:tasks.workunit.client.0.smithi029.stdout: create 193/38 file after the snapshot 2024-04-17T19:51:21.610 INFO:tasks.workunit.client.0.smithi029.stdout: create 193/39 file after the snapshot 2024-04-17T19:51:21.611 INFO:tasks.workunit.client.0.smithi029.stdout: create 194/.snap/snap-subdir-test 2024-04-17T19:51:21.646 INFO:tasks.workunit.client.0.smithi029.stdout: create 194/30 file after the snapshot 2024-04-17T19:51:21.648 INFO:tasks.workunit.client.0.smithi029.stdout: create 194/31 file after the snapshot 2024-04-17T19:51:21.650 INFO:tasks.workunit.client.0.smithi029.stdout: create 194/32 file after the snapshot 2024-04-17T19:51:21.657 INFO:tasks.workunit.client.0.smithi029.stdout: create 194/33 file after the snapshot 2024-04-17T19:51:21.672 INFO:tasks.workunit.client.0.smithi029.stdout: create 194/34 file after the snapshot 2024-04-17T19:51:21.674 INFO:tasks.workunit.client.0.smithi029.stdout: create 194/35 file after the snapshot 2024-04-17T19:51:21.675 INFO:tasks.workunit.client.0.smithi029.stdout: create 194/36 file after the snapshot 2024-04-17T19:51:21.677 INFO:tasks.workunit.client.0.smithi029.stdout: create 194/37 file after the snapshot 2024-04-17T19:51:21.678 INFO:tasks.workunit.client.0.smithi029.stdout: create 194/38 file after the snapshot 2024-04-17T19:51:21.680 INFO:tasks.workunit.client.0.smithi029.stdout: create 194/39 file after the snapshot 2024-04-17T19:51:21.682 INFO:tasks.workunit.client.0.smithi029.stdout: create 195/.snap/snap-subdir-test 2024-04-17T19:51:21.709 INFO:tasks.workunit.client.0.smithi029.stdout: create 195/30 file after the snapshot 2024-04-17T19:51:21.712 INFO:tasks.workunit.client.0.smithi029.stdout: create 195/31 file after the snapshot 2024-04-17T19:51:21.714 INFO:tasks.workunit.client.0.smithi029.stdout: create 195/32 file after the snapshot 2024-04-17T19:51:21.716 INFO:tasks.workunit.client.0.smithi029.stdout: create 195/33 file after the snapshot 2024-04-17T19:51:21.719 INFO:tasks.workunit.client.0.smithi029.stdout: create 195/34 file after the snapshot 2024-04-17T19:51:21.723 INFO:tasks.workunit.client.0.smithi029.stdout: create 195/35 file after the snapshot 2024-04-17T19:51:21.727 INFO:tasks.workunit.client.0.smithi029.stdout: create 195/36 file after the snapshot 2024-04-17T19:51:21.728 INFO:tasks.workunit.client.0.smithi029.stdout: create 195/37 file after the snapshot 2024-04-17T19:51:21.729 INFO:tasks.workunit.client.0.smithi029.stdout: create 195/38 file after the snapshot 2024-04-17T19:51:21.730 INFO:tasks.workunit.client.0.smithi029.stdout: create 195/39 file after the snapshot 2024-04-17T19:51:21.731 INFO:tasks.workunit.client.0.smithi029.stdout: create 196/.snap/snap-subdir-test 2024-04-17T19:51:21.754 INFO:tasks.workunit.client.0.smithi029.stdout: create 196/30 file after the snapshot 2024-04-17T19:51:21.756 INFO:tasks.workunit.client.0.smithi029.stdout: create 196/31 file after the snapshot 2024-04-17T19:51:21.757 INFO:tasks.workunit.client.0.smithi029.stdout: create 196/32 file after the snapshot 2024-04-17T19:51:21.758 INFO:tasks.workunit.client.0.smithi029.stdout: create 196/33 file after the snapshot 2024-04-17T19:51:21.762 INFO:tasks.workunit.client.0.smithi029.stdout: create 196/34 file after the snapshot 2024-04-17T19:51:21.770 INFO:tasks.workunit.client.0.smithi029.stdout: create 196/35 file after the snapshot 2024-04-17T19:51:21.774 INFO:tasks.workunit.client.0.smithi029.stdout: create 196/36 file after the snapshot 2024-04-17T19:51:21.774 INFO:tasks.workunit.client.0.smithi029.stdout: create 196/37 file after the snapshot 2024-04-17T19:51:21.775 INFO:tasks.workunit.client.0.smithi029.stdout: create 196/38 file after the snapshot 2024-04-17T19:51:21.776 INFO:tasks.workunit.client.0.smithi029.stdout: create 196/39 file after the snapshot 2024-04-17T19:51:21.777 INFO:tasks.workunit.client.0.smithi029.stdout: create 197/.snap/snap-subdir-test 2024-04-17T19:51:21.799 INFO:tasks.workunit.client.0.smithi029.stdout: create 197/30 file after the snapshot 2024-04-17T19:51:21.801 INFO:tasks.workunit.client.0.smithi029.stdout: create 197/31 file after the snapshot 2024-04-17T19:51:21.802 INFO:tasks.workunit.client.0.smithi029.stdout: create 197/32 file after the snapshot 2024-04-17T19:51:21.804 INFO:tasks.workunit.client.0.smithi029.stdout: create 197/33 file after the snapshot 2024-04-17T19:51:21.810 INFO:tasks.workunit.client.0.smithi029.stdout: create 197/34 file after the snapshot 2024-04-17T19:51:21.818 INFO:tasks.workunit.client.0.smithi029.stdout: create 197/35 file after the snapshot 2024-04-17T19:51:21.819 INFO:tasks.workunit.client.0.smithi029.stdout: create 197/36 file after the snapshot 2024-04-17T19:51:21.820 INFO:tasks.workunit.client.0.smithi029.stdout: create 197/37 file after the snapshot 2024-04-17T19:51:21.821 INFO:tasks.workunit.client.0.smithi029.stdout: create 197/38 file after the snapshot 2024-04-17T19:51:21.822 INFO:tasks.workunit.client.0.smithi029.stdout: create 197/39 file after the snapshot 2024-04-17T19:51:21.823 INFO:tasks.workunit.client.0.smithi029.stdout: create 198/.snap/snap-subdir-test 2024-04-17T19:51:22.270 INFO:tasks.workunit.client.0.smithi029.stdout: create 198/30 file after the snapshot 2024-04-17T19:51:22.272 INFO:tasks.workunit.client.0.smithi029.stdout: create 198/31 file after the snapshot 2024-04-17T19:51:22.273 INFO:tasks.workunit.client.0.smithi029.stdout: create 198/32 file after the snapshot 2024-04-17T19:51:22.278 INFO:tasks.workunit.client.0.smithi029.stdout: create 198/33 file after the snapshot 2024-04-17T19:51:22.284 INFO:tasks.workunit.client.0.smithi029.stdout: create 198/34 file after the snapshot 2024-04-17T19:51:22.288 INFO:tasks.workunit.client.0.smithi029.stdout: create 198/35 file after the snapshot 2024-04-17T19:51:22.289 INFO:tasks.workunit.client.0.smithi029.stdout: create 198/36 file after the snapshot 2024-04-17T19:51:22.290 INFO:tasks.workunit.client.0.smithi029.stdout: create 198/37 file after the snapshot 2024-04-17T19:51:22.291 INFO:tasks.workunit.client.0.smithi029.stdout: create 198/38 file after the snapshot 2024-04-17T19:51:22.292 INFO:tasks.workunit.client.0.smithi029.stdout: create 198/39 file after the snapshot 2024-04-17T19:51:22.293 INFO:tasks.workunit.client.0.smithi029.stdout: create 199/.snap/snap-subdir-test 2024-04-17T19:51:22.324 INFO:tasks.workunit.client.0.smithi029.stdout: create 199/30 file after the snapshot 2024-04-17T19:51:22.325 INFO:tasks.workunit.client.0.smithi029.stdout: create 199/31 file after the snapshot 2024-04-17T19:51:22.327 INFO:tasks.workunit.client.0.smithi029.stdout: create 199/32 file after the snapshot 2024-04-17T19:51:22.330 INFO:tasks.workunit.client.0.smithi029.stdout: create 199/33 file after the snapshot 2024-04-17T19:51:22.333 INFO:tasks.workunit.client.0.smithi029.stdout: create 199/34 file after the snapshot 2024-04-17T19:51:22.338 INFO:tasks.workunit.client.0.smithi029.stdout: create 199/35 file after the snapshot 2024-04-17T19:51:22.343 INFO:tasks.workunit.client.0.smithi029.stdout: create 199/36 file after the snapshot 2024-04-17T19:51:22.344 INFO:tasks.workunit.client.0.smithi029.stdout: create 199/37 file after the snapshot 2024-04-17T19:51:22.643 INFO:tasks.workunit.client.0.smithi029.stdout: create 199/38 file after the snapshot 2024-04-17T19:51:23.195 INFO:tasks.workunit.client.0.smithi029.stdout: create 199/39 file after the snapshot 2024-04-17T19:51:23.196 INFO:tasks.workunit.client.0.smithi029.stdout: create 200/.snap/snap-subdir-test 2024-04-17T19:51:23.221 INFO:tasks.workunit.client.0.smithi029.stdout: create 200/30 file after the snapshot 2024-04-17T19:51:23.223 INFO:tasks.workunit.client.0.smithi029.stdout: create 200/31 file after the snapshot 2024-04-17T19:51:23.224 INFO:tasks.workunit.client.0.smithi029.stdout: create 200/32 file after the snapshot 2024-04-17T19:51:23.225 INFO:tasks.workunit.client.0.smithi029.stdout: create 200/33 file after the snapshot 2024-04-17T19:51:23.226 INFO:tasks.workunit.client.0.smithi029.stdout: create 200/34 file after the snapshot 2024-04-17T19:51:23.227 INFO:tasks.workunit.client.0.smithi029.stdout: create 200/35 file after the snapshot 2024-04-17T19:51:23.228 INFO:tasks.workunit.client.0.smithi029.stdout: create 200/36 file after the snapshot 2024-04-17T19:51:23.229 INFO:tasks.workunit.client.0.smithi029.stdout: create 200/37 file after the snapshot 2024-04-17T19:51:23.230 INFO:tasks.workunit.client.0.smithi029.stdout: create 200/38 file after the snapshot 2024-04-17T19:51:23.231 INFO:tasks.workunit.client.0.smithi029.stdout: create 200/39 file after the snapshot 2024-04-17T19:51:23.232 INFO:tasks.workunit.client.0.smithi029.stdout: create 201/.snap/snap-subdir-test 2024-04-17T19:51:23.249 INFO:tasks.workunit.client.0.smithi029.stdout: create 201/30 file after the snapshot 2024-04-17T19:51:23.250 INFO:tasks.workunit.client.0.smithi029.stdout: create 201/31 file after the snapshot 2024-04-17T19:51:23.251 INFO:tasks.workunit.client.0.smithi029.stdout: create 201/32 file after the snapshot 2024-04-17T19:51:23.252 INFO:tasks.workunit.client.0.smithi029.stdout: create 201/33 file after the snapshot 2024-04-17T19:51:23.253 INFO:tasks.workunit.client.0.smithi029.stdout: create 201/34 file after the snapshot 2024-04-17T19:51:23.254 INFO:tasks.workunit.client.0.smithi029.stdout: create 201/35 file after the snapshot 2024-04-17T19:51:23.255 INFO:tasks.workunit.client.0.smithi029.stdout: create 201/36 file after the snapshot 2024-04-17T19:51:23.256 INFO:tasks.workunit.client.0.smithi029.stdout: create 201/37 file after the snapshot 2024-04-17T19:51:23.257 INFO:tasks.workunit.client.0.smithi029.stdout: create 201/38 file after the snapshot 2024-04-17T19:51:23.258 INFO:tasks.workunit.client.0.smithi029.stdout: create 201/39 file after the snapshot 2024-04-17T19:51:23.259 INFO:tasks.workunit.client.0.smithi029.stdout: create 202/.snap/snap-subdir-test 2024-04-17T19:51:23.277 INFO:tasks.workunit.client.0.smithi029.stdout: create 202/30 file after the snapshot 2024-04-17T19:51:23.527 INFO:tasks.workunit.client.0.smithi029.stdout: create 202/31 file after the snapshot 2024-04-17T19:51:23.529 INFO:tasks.workunit.client.0.smithi029.stdout: create 202/32 file after the snapshot 2024-04-17T19:51:23.530 INFO:tasks.workunit.client.0.smithi029.stdout: create 202/33 file after the snapshot 2024-04-17T19:51:23.531 INFO:tasks.workunit.client.0.smithi029.stdout: create 202/34 file after the snapshot 2024-04-17T19:51:23.532 INFO:tasks.workunit.client.0.smithi029.stdout: create 202/35 file after the snapshot 2024-04-17T19:51:23.533 INFO:tasks.workunit.client.0.smithi029.stdout: create 202/36 file after the snapshot 2024-04-17T19:51:23.534 INFO:tasks.workunit.client.0.smithi029.stdout: create 202/37 file after the snapshot 2024-04-17T19:51:23.535 INFO:tasks.workunit.client.0.smithi029.stdout: create 202/38 file after the snapshot 2024-04-17T19:51:23.536 INFO:tasks.workunit.client.0.smithi029.stdout: create 202/39 file after the snapshot 2024-04-17T19:51:23.537 INFO:tasks.workunit.client.0.smithi029.stdout: create 203/.snap/snap-subdir-test 2024-04-17T19:51:23.555 INFO:tasks.workunit.client.0.smithi029.stdout: create 203/30 file after the snapshot 2024-04-17T19:51:23.556 INFO:tasks.workunit.client.0.smithi029.stdout: create 203/31 file after the snapshot 2024-04-17T19:51:23.558 INFO:tasks.workunit.client.0.smithi029.stdout: create 203/32 file after the snapshot 2024-04-17T19:51:23.560 INFO:tasks.workunit.client.0.smithi029.stdout: create 203/33 file after the snapshot 2024-04-17T19:51:23.561 INFO:tasks.workunit.client.0.smithi029.stdout: create 203/34 file after the snapshot 2024-04-17T19:51:23.563 INFO:tasks.workunit.client.0.smithi029.stdout: create 203/35 file after the snapshot 2024-04-17T19:51:23.564 INFO:tasks.workunit.client.0.smithi029.stdout: create 203/36 file after the snapshot 2024-04-17T19:51:23.566 INFO:tasks.workunit.client.0.smithi029.stdout: create 203/37 file after the snapshot 2024-04-17T19:51:23.567 INFO:tasks.workunit.client.0.smithi029.stdout: create 203/38 file after the snapshot 2024-04-17T19:51:23.568 INFO:tasks.workunit.client.0.smithi029.stdout: create 203/39 file after the snapshot 2024-04-17T19:51:23.569 INFO:tasks.workunit.client.0.smithi029.stdout: create 204/.snap/snap-subdir-test 2024-04-17T19:51:23.585 INFO:tasks.workunit.client.0.smithi029.stdout: create 204/30 file after the snapshot 2024-04-17T19:51:23.586 INFO:tasks.workunit.client.0.smithi029.stdout: create 204/31 file after the snapshot 2024-04-17T19:51:23.587 INFO:tasks.workunit.client.0.smithi029.stdout: create 204/32 file after the snapshot 2024-04-17T19:51:23.588 INFO:tasks.workunit.client.0.smithi029.stdout: create 204/33 file after the snapshot 2024-04-17T19:51:23.589 INFO:tasks.workunit.client.0.smithi029.stdout: create 204/34 file after the snapshot 2024-04-17T19:51:23.590 INFO:tasks.workunit.client.0.smithi029.stdout: create 204/35 file after the snapshot 2024-04-17T19:51:23.591 INFO:tasks.workunit.client.0.smithi029.stdout: create 204/36 file after the snapshot 2024-04-17T19:51:23.592 INFO:tasks.workunit.client.0.smithi029.stdout: create 204/37 file after the snapshot 2024-04-17T19:51:23.593 INFO:tasks.workunit.client.0.smithi029.stdout: create 204/38 file after the snapshot 2024-04-17T19:51:23.594 INFO:tasks.workunit.client.0.smithi029.stdout: create 204/39 file after the snapshot 2024-04-17T19:51:23.595 INFO:tasks.workunit.client.0.smithi029.stdout: create 205/.snap/snap-subdir-test 2024-04-17T19:51:23.611 INFO:tasks.workunit.client.0.smithi029.stdout: create 205/30 file after the snapshot 2024-04-17T19:51:23.612 INFO:tasks.workunit.client.0.smithi029.stdout: create 205/31 file after the snapshot 2024-04-17T19:51:23.613 INFO:tasks.workunit.client.0.smithi029.stdout: create 205/32 file after the snapshot 2024-04-17T19:51:23.614 INFO:tasks.workunit.client.0.smithi029.stdout: create 205/33 file after the snapshot 2024-04-17T19:51:23.615 INFO:tasks.workunit.client.0.smithi029.stdout: create 205/34 file after the snapshot 2024-04-17T19:51:23.616 INFO:tasks.workunit.client.0.smithi029.stdout: create 205/35 file after the snapshot 2024-04-17T19:51:23.617 INFO:tasks.workunit.client.0.smithi029.stdout: create 205/36 file after the snapshot 2024-04-17T19:51:23.618 INFO:tasks.workunit.client.0.smithi029.stdout: create 205/37 file after the snapshot 2024-04-17T19:51:23.619 INFO:tasks.workunit.client.0.smithi029.stdout: create 205/38 file after the snapshot 2024-04-17T19:51:23.620 INFO:tasks.workunit.client.0.smithi029.stdout: create 205/39 file after the snapshot 2024-04-17T19:51:23.621 INFO:tasks.workunit.client.0.smithi029.stdout: create 206/.snap/snap-subdir-test 2024-04-17T19:51:23.636 INFO:tasks.workunit.client.0.smithi029.stdout: create 206/30 file after the snapshot 2024-04-17T19:51:23.637 INFO:tasks.workunit.client.0.smithi029.stdout: create 206/31 file after the snapshot 2024-04-17T19:51:23.638 INFO:tasks.workunit.client.0.smithi029.stdout: create 206/32 file after the snapshot 2024-04-17T19:51:23.639 INFO:tasks.workunit.client.0.smithi029.stdout: create 206/33 file after the snapshot 2024-04-17T19:51:23.640 INFO:tasks.workunit.client.0.smithi029.stdout: create 206/34 file after the snapshot 2024-04-17T19:51:23.641 INFO:tasks.workunit.client.0.smithi029.stdout: create 206/35 file after the snapshot 2024-04-17T19:51:23.642 INFO:tasks.workunit.client.0.smithi029.stdout: create 206/36 file after the snapshot 2024-04-17T19:51:23.643 INFO:tasks.workunit.client.0.smithi029.stdout: create 206/37 file after the snapshot 2024-04-17T19:51:23.644 INFO:tasks.workunit.client.0.smithi029.stdout: create 206/38 file after the snapshot 2024-04-17T19:51:23.645 INFO:tasks.workunit.client.0.smithi029.stdout: create 206/39 file after the snapshot 2024-04-17T19:51:23.646 INFO:tasks.workunit.client.0.smithi029.stdout: create 207/.snap/snap-subdir-test 2024-04-17T19:51:23.662 INFO:tasks.workunit.client.0.smithi029.stdout: create 207/30 file after the snapshot 2024-04-17T19:51:23.664 INFO:tasks.workunit.client.0.smithi029.stdout: create 207/31 file after the snapshot 2024-04-17T19:51:23.665 INFO:tasks.workunit.client.0.smithi029.stdout: create 207/32 file after the snapshot 2024-04-17T19:51:23.667 INFO:tasks.workunit.client.0.smithi029.stdout: create 207/33 file after the snapshot 2024-04-17T19:51:23.668 INFO:tasks.workunit.client.0.smithi029.stdout: create 207/34 file after the snapshot 2024-04-17T19:51:23.670 INFO:tasks.workunit.client.0.smithi029.stdout: create 207/35 file after the snapshot 2024-04-17T19:51:23.671 INFO:tasks.workunit.client.0.smithi029.stdout: create 207/36 file after the snapshot 2024-04-17T19:51:23.673 INFO:tasks.workunit.client.0.smithi029.stdout: create 207/37 file after the snapshot 2024-04-17T19:51:23.674 INFO:tasks.workunit.client.0.smithi029.stdout: create 207/38 file after the snapshot 2024-04-17T19:51:23.676 INFO:tasks.workunit.client.0.smithi029.stdout: create 207/39 file after the snapshot 2024-04-17T19:51:23.677 INFO:tasks.workunit.client.0.smithi029.stdout: create 208/.snap/snap-subdir-test 2024-04-17T19:51:23.693 INFO:tasks.workunit.client.0.smithi029.stdout: create 208/30 file after the snapshot 2024-04-17T19:51:23.694 INFO:tasks.workunit.client.0.smithi029.stdout: create 208/31 file after the snapshot 2024-04-17T19:51:23.695 INFO:tasks.workunit.client.0.smithi029.stdout: create 208/32 file after the snapshot 2024-04-17T19:51:23.696 INFO:tasks.workunit.client.0.smithi029.stdout: create 208/33 file after the snapshot 2024-04-17T19:51:23.697 INFO:tasks.workunit.client.0.smithi029.stdout: create 208/34 file after the snapshot 2024-04-17T19:51:23.698 INFO:tasks.workunit.client.0.smithi029.stdout: create 208/35 file after the snapshot 2024-04-17T19:51:23.699 INFO:tasks.workunit.client.0.smithi029.stdout: create 208/36 file after the snapshot 2024-04-17T19:51:23.700 INFO:tasks.workunit.client.0.smithi029.stdout: create 208/37 file after the snapshot 2024-04-17T19:51:23.700 INFO:tasks.workunit.client.0.smithi029.stdout: create 208/38 file after the snapshot 2024-04-17T19:51:23.701 INFO:tasks.workunit.client.0.smithi029.stdout: create 208/39 file after the snapshot 2024-04-17T19:51:23.702 INFO:tasks.workunit.client.0.smithi029.stdout: create 209/.snap/snap-subdir-test 2024-04-17T19:51:23.718 INFO:tasks.workunit.client.0.smithi029.stdout: create 209/30 file after the snapshot 2024-04-17T19:51:23.719 INFO:tasks.workunit.client.0.smithi029.stdout: create 209/31 file after the snapshot 2024-04-17T19:51:23.720 INFO:tasks.workunit.client.0.smithi029.stdout: create 209/32 file after the snapshot 2024-04-17T19:51:23.722 INFO:tasks.workunit.client.0.smithi029.stdout: create 209/33 file after the snapshot 2024-04-17T19:51:23.723 INFO:tasks.workunit.client.0.smithi029.stdout: create 209/34 file after the snapshot 2024-04-17T19:51:23.725 INFO:tasks.workunit.client.0.smithi029.stdout: create 209/35 file after the snapshot 2024-04-17T19:51:23.726 INFO:tasks.workunit.client.0.smithi029.stdout: create 209/36 file after the snapshot 2024-04-17T19:51:23.728 INFO:tasks.workunit.client.0.smithi029.stdout: create 209/37 file after the snapshot 2024-04-17T19:51:23.729 INFO:tasks.workunit.client.0.smithi029.stdout: create 209/38 file after the snapshot 2024-04-17T19:51:23.730 INFO:tasks.workunit.client.0.smithi029.stdout: create 209/39 file after the snapshot 2024-04-17T19:51:23.732 INFO:tasks.workunit.client.0.smithi029.stdout: create 210/.snap/snap-subdir-test 2024-04-17T19:51:23.748 INFO:tasks.workunit.client.0.smithi029.stdout: create 210/30 file after the snapshot 2024-04-17T19:51:23.750 INFO:tasks.workunit.client.0.smithi029.stdout: create 210/31 file after the snapshot 2024-04-17T19:51:23.751 INFO:tasks.workunit.client.0.smithi029.stdout: create 210/32 file after the snapshot 2024-04-17T19:51:23.753 INFO:tasks.workunit.client.0.smithi029.stdout: create 210/33 file after the snapshot 2024-04-17T19:51:23.754 INFO:tasks.workunit.client.0.smithi029.stdout: create 210/34 file after the snapshot 2024-04-17T19:51:23.755 INFO:tasks.workunit.client.0.smithi029.stdout: create 210/35 file after the snapshot 2024-04-17T19:51:23.757 INFO:tasks.workunit.client.0.smithi029.stdout: create 210/36 file after the snapshot 2024-04-17T19:51:23.758 INFO:tasks.workunit.client.0.smithi029.stdout: create 210/37 file after the snapshot 2024-04-17T19:51:23.760 INFO:tasks.workunit.client.0.smithi029.stdout: create 210/38 file after the snapshot 2024-04-17T19:51:23.761 INFO:tasks.workunit.client.0.smithi029.stdout: create 210/39 file after the snapshot 2024-04-17T19:51:23.763 INFO:tasks.workunit.client.0.smithi029.stdout: create 211/.snap/snap-subdir-test 2024-04-17T19:51:23.779 INFO:tasks.workunit.client.0.smithi029.stdout: create 211/30 file after the snapshot 2024-04-17T19:51:23.780 INFO:tasks.workunit.client.0.smithi029.stdout: create 211/31 file after the snapshot 2024-04-17T19:51:23.781 INFO:tasks.workunit.client.0.smithi029.stdout: create 211/32 file after the snapshot 2024-04-17T19:51:23.783 INFO:tasks.workunit.client.0.smithi029.stdout: create 211/33 file after the snapshot 2024-04-17T19:51:23.784 INFO:tasks.workunit.client.0.smithi029.stdout: create 211/34 file after the snapshot 2024-04-17T19:51:23.786 INFO:tasks.workunit.client.0.smithi029.stdout: create 211/35 file after the snapshot 2024-04-17T19:51:23.787 INFO:tasks.workunit.client.0.smithi029.stdout: create 211/36 file after the snapshot 2024-04-17T19:51:23.789 INFO:tasks.workunit.client.0.smithi029.stdout: create 211/37 file after the snapshot 2024-04-17T19:51:23.790 INFO:tasks.workunit.client.0.smithi029.stdout: create 211/38 file after the snapshot 2024-04-17T19:51:23.791 INFO:tasks.workunit.client.0.smithi029.stdout: create 211/39 file after the snapshot 2024-04-17T19:51:23.793 INFO:tasks.workunit.client.0.smithi029.stdout: create 212/.snap/snap-subdir-test 2024-04-17T19:51:23.810 INFO:tasks.workunit.client.0.smithi029.stdout: create 212/30 file after the snapshot 2024-04-17T19:51:23.811 INFO:tasks.workunit.client.0.smithi029.stdout: create 212/31 file after the snapshot 2024-04-17T19:51:23.811 INFO:tasks.workunit.client.0.smithi029.stdout: create 212/32 file after the snapshot 2024-04-17T19:51:23.812 INFO:tasks.workunit.client.0.smithi029.stdout: create 212/33 file after the snapshot 2024-04-17T19:51:23.813 INFO:tasks.workunit.client.0.smithi029.stdout: create 212/34 file after the snapshot 2024-04-17T19:51:23.814 INFO:tasks.workunit.client.0.smithi029.stdout: create 212/35 file after the snapshot 2024-04-17T19:51:23.815 INFO:tasks.workunit.client.0.smithi029.stdout: create 212/36 file after the snapshot 2024-04-17T19:51:23.816 INFO:tasks.workunit.client.0.smithi029.stdout: create 212/37 file after the snapshot 2024-04-17T19:51:23.817 INFO:tasks.workunit.client.0.smithi029.stdout: create 212/38 file after the snapshot 2024-04-17T19:51:23.818 INFO:tasks.workunit.client.0.smithi029.stdout: create 212/39 file after the snapshot 2024-04-17T19:51:23.819 INFO:tasks.workunit.client.0.smithi029.stdout: create 213/.snap/snap-subdir-test 2024-04-17T19:51:23.834 INFO:tasks.workunit.client.0.smithi029.stdout: create 213/30 file after the snapshot 2024-04-17T19:51:23.835 INFO:tasks.workunit.client.0.smithi029.stdout: create 213/31 file after the snapshot 2024-04-17T19:51:23.836 INFO:tasks.workunit.client.0.smithi029.stdout: create 213/32 file after the snapshot 2024-04-17T19:51:23.837 INFO:tasks.workunit.client.0.smithi029.stdout: create 213/33 file after the snapshot 2024-04-17T19:51:23.837 INFO:tasks.workunit.client.0.smithi029.stdout: create 213/34 file after the snapshot 2024-04-17T19:51:23.838 INFO:tasks.workunit.client.0.smithi029.stdout: create 213/35 file after the snapshot 2024-04-17T19:51:23.839 INFO:tasks.workunit.client.0.smithi029.stdout: create 213/36 file after the snapshot 2024-04-17T19:51:23.840 INFO:tasks.workunit.client.0.smithi029.stdout: create 213/37 file after the snapshot 2024-04-17T19:51:23.841 INFO:tasks.workunit.client.0.smithi029.stdout: create 213/38 file after the snapshot 2024-04-17T19:51:23.842 INFO:tasks.workunit.client.0.smithi029.stdout: create 213/39 file after the snapshot 2024-04-17T19:51:23.843 INFO:tasks.workunit.client.0.smithi029.stdout: create 214/.snap/snap-subdir-test 2024-04-17T19:51:23.857 INFO:tasks.workunit.client.0.smithi029.stdout: create 214/30 file after the snapshot 2024-04-17T19:51:23.859 INFO:tasks.workunit.client.0.smithi029.stdout: create 214/31 file after the snapshot 2024-04-17T19:51:23.859 INFO:tasks.workunit.client.0.smithi029.stdout: create 214/32 file after the snapshot 2024-04-17T19:51:23.860 INFO:tasks.workunit.client.0.smithi029.stdout: create 214/33 file after the snapshot 2024-04-17T19:51:23.861 INFO:tasks.workunit.client.0.smithi029.stdout: create 214/34 file after the snapshot 2024-04-17T19:51:23.862 INFO:tasks.workunit.client.0.smithi029.stdout: create 214/35 file after the snapshot 2024-04-17T19:51:23.863 INFO:tasks.workunit.client.0.smithi029.stdout: create 214/36 file after the snapshot 2024-04-17T19:51:23.864 INFO:tasks.workunit.client.0.smithi029.stdout: create 214/37 file after the snapshot 2024-04-17T19:51:23.865 INFO:tasks.workunit.client.0.smithi029.stdout: create 214/38 file after the snapshot 2024-04-17T19:51:23.866 INFO:tasks.workunit.client.0.smithi029.stdout: create 214/39 file after the snapshot 2024-04-17T19:51:23.867 INFO:tasks.workunit.client.0.smithi029.stdout: create 215/.snap/snap-subdir-test 2024-04-17T19:51:23.882 INFO:tasks.workunit.client.0.smithi029.stdout: create 215/30 file after the snapshot 2024-04-17T19:51:23.883 INFO:tasks.workunit.client.0.smithi029.stdout: create 215/31 file after the snapshot 2024-04-17T19:51:23.884 INFO:tasks.workunit.client.0.smithi029.stdout: create 215/32 file after the snapshot 2024-04-17T19:51:23.885 INFO:tasks.workunit.client.0.smithi029.stdout: create 215/33 file after the snapshot 2024-04-17T19:51:23.886 INFO:tasks.workunit.client.0.smithi029.stdout: create 215/34 file after the snapshot 2024-04-17T19:51:23.887 INFO:tasks.workunit.client.0.smithi029.stdout: create 215/35 file after the snapshot 2024-04-17T19:51:23.888 INFO:tasks.workunit.client.0.smithi029.stdout: create 215/36 file after the snapshot 2024-04-17T19:51:23.888 INFO:tasks.workunit.client.0.smithi029.stdout: create 215/37 file after the snapshot 2024-04-17T19:51:23.889 INFO:tasks.workunit.client.0.smithi029.stdout: create 215/38 file after the snapshot 2024-04-17T19:51:23.890 INFO:tasks.workunit.client.0.smithi029.stdout: create 215/39 file after the snapshot 2024-04-17T19:51:23.891 INFO:tasks.workunit.client.0.smithi029.stdout: create 216/.snap/snap-subdir-test 2024-04-17T19:51:23.907 INFO:tasks.workunit.client.0.smithi029.stdout: create 216/30 file after the snapshot 2024-04-17T19:51:23.908 INFO:tasks.workunit.client.0.smithi029.stdout: create 216/31 file after the snapshot 2024-04-17T19:51:23.909 INFO:tasks.workunit.client.0.smithi029.stdout: create 216/32 file after the snapshot 2024-04-17T19:51:23.910 INFO:tasks.workunit.client.0.smithi029.stdout: create 216/33 file after the snapshot 2024-04-17T19:51:23.911 INFO:tasks.workunit.client.0.smithi029.stdout: create 216/34 file after the snapshot 2024-04-17T19:51:23.912 INFO:tasks.workunit.client.0.smithi029.stdout: create 216/35 file after the snapshot 2024-04-17T19:51:23.913 INFO:tasks.workunit.client.0.smithi029.stdout: create 216/36 file after the snapshot 2024-04-17T19:51:23.913 INFO:tasks.workunit.client.0.smithi029.stdout: create 216/37 file after the snapshot 2024-04-17T19:51:23.915 INFO:tasks.workunit.client.0.smithi029.stdout: create 216/38 file after the snapshot 2024-04-17T19:51:23.915 INFO:tasks.workunit.client.0.smithi029.stdout: create 216/39 file after the snapshot 2024-04-17T19:51:23.916 INFO:tasks.workunit.client.0.smithi029.stdout: create 217/.snap/snap-subdir-test 2024-04-17T19:51:23.934 INFO:tasks.workunit.client.0.smithi029.stdout: create 217/30 file after the snapshot 2024-04-17T19:51:23.934 INFO:tasks.workunit.client.0.smithi029.stdout: create 217/31 file after the snapshot 2024-04-17T19:51:23.935 INFO:tasks.workunit.client.0.smithi029.stdout: create 217/32 file after the snapshot 2024-04-17T19:51:23.936 INFO:tasks.workunit.client.0.smithi029.stdout: create 217/33 file after the snapshot 2024-04-17T19:51:23.937 INFO:tasks.workunit.client.0.smithi029.stdout: create 217/34 file after the snapshot 2024-04-17T19:51:23.938 INFO:tasks.workunit.client.0.smithi029.stdout: create 217/35 file after the snapshot 2024-04-17T19:51:23.939 INFO:tasks.workunit.client.0.smithi029.stdout: create 217/36 file after the snapshot 2024-04-17T19:51:23.940 INFO:tasks.workunit.client.0.smithi029.stdout: create 217/37 file after the snapshot 2024-04-17T19:51:23.941 INFO:tasks.workunit.client.0.smithi029.stdout: create 217/38 file after the snapshot 2024-04-17T19:51:23.942 INFO:tasks.workunit.client.0.smithi029.stdout: create 217/39 file after the snapshot 2024-04-17T19:51:23.943 INFO:tasks.workunit.client.0.smithi029.stdout: create 218/.snap/snap-subdir-test 2024-04-17T19:51:23.960 INFO:tasks.workunit.client.0.smithi029.stdout: create 218/30 file after the snapshot 2024-04-17T19:51:23.961 INFO:tasks.workunit.client.0.smithi029.stdout: create 218/31 file after the snapshot 2024-04-17T19:51:23.962 INFO:tasks.workunit.client.0.smithi029.stdout: create 218/32 file after the snapshot 2024-04-17T19:51:23.963 INFO:tasks.workunit.client.0.smithi029.stdout: create 218/33 file after the snapshot 2024-04-17T19:51:23.964 INFO:tasks.workunit.client.0.smithi029.stdout: create 218/34 file after the snapshot 2024-04-17T19:51:23.965 INFO:tasks.workunit.client.0.smithi029.stdout: create 218/35 file after the snapshot 2024-04-17T19:51:23.966 INFO:tasks.workunit.client.0.smithi029.stdout: create 218/36 file after the snapshot 2024-04-17T19:51:23.967 INFO:tasks.workunit.client.0.smithi029.stdout: create 218/37 file after the snapshot 2024-04-17T19:51:23.968 INFO:tasks.workunit.client.0.smithi029.stdout: create 218/38 file after the snapshot 2024-04-17T19:51:23.970 INFO:tasks.workunit.client.0.smithi029.stdout: create 218/39 file after the snapshot 2024-04-17T19:51:23.971 INFO:tasks.workunit.client.0.smithi029.stdout: create 219/.snap/snap-subdir-test 2024-04-17T19:51:23.986 INFO:tasks.workunit.client.0.smithi029.stdout: create 219/30 file after the snapshot 2024-04-17T19:51:23.987 INFO:tasks.workunit.client.0.smithi029.stdout: create 219/31 file after the snapshot 2024-04-17T19:51:23.988 INFO:tasks.workunit.client.0.smithi029.stdout: create 219/32 file after the snapshot 2024-04-17T19:51:23.989 INFO:tasks.workunit.client.0.smithi029.stdout: create 219/33 file after the snapshot 2024-04-17T19:51:23.990 INFO:tasks.workunit.client.0.smithi029.stdout: create 219/34 file after the snapshot 2024-04-17T19:51:23.990 INFO:tasks.workunit.client.0.smithi029.stdout: create 219/35 file after the snapshot 2024-04-17T19:51:23.991 INFO:tasks.workunit.client.0.smithi029.stdout: create 219/36 file after the snapshot 2024-04-17T19:51:23.992 INFO:tasks.workunit.client.0.smithi029.stdout: create 219/37 file after the snapshot 2024-04-17T19:51:23.993 INFO:tasks.workunit.client.0.smithi029.stdout: create 219/38 file after the snapshot 2024-04-17T19:51:23.994 INFO:tasks.workunit.client.0.smithi029.stdout: create 219/39 file after the snapshot 2024-04-17T19:51:23.995 INFO:tasks.workunit.client.0.smithi029.stdout: create 220/.snap/snap-subdir-test 2024-04-17T19:51:24.011 INFO:tasks.workunit.client.0.smithi029.stdout: create 220/30 file after the snapshot 2024-04-17T19:51:24.012 INFO:tasks.workunit.client.0.smithi029.stdout: create 220/31 file after the snapshot 2024-04-17T19:51:24.013 INFO:tasks.workunit.client.0.smithi029.stdout: create 220/32 file after the snapshot 2024-04-17T19:51:24.014 INFO:tasks.workunit.client.0.smithi029.stdout: create 220/33 file after the snapshot 2024-04-17T19:51:24.015 INFO:tasks.workunit.client.0.smithi029.stdout: create 220/34 file after the snapshot 2024-04-17T19:51:24.016 INFO:tasks.workunit.client.0.smithi029.stdout: create 220/35 file after the snapshot 2024-04-17T19:51:24.017 INFO:tasks.workunit.client.0.smithi029.stdout: create 220/36 file after the snapshot 2024-04-17T19:51:24.018 INFO:tasks.workunit.client.0.smithi029.stdout: create 220/37 file after the snapshot 2024-04-17T19:51:24.019 INFO:tasks.workunit.client.0.smithi029.stdout: create 220/38 file after the snapshot 2024-04-17T19:51:24.019 INFO:tasks.workunit.client.0.smithi029.stdout: create 220/39 file after the snapshot 2024-04-17T19:51:24.021 INFO:tasks.workunit.client.0.smithi029.stdout: create 221/.snap/snap-subdir-test 2024-04-17T19:51:24.037 INFO:tasks.workunit.client.0.smithi029.stdout: create 221/30 file after the snapshot 2024-04-17T19:51:24.038 INFO:tasks.workunit.client.0.smithi029.stdout: create 221/31 file after the snapshot 2024-04-17T19:51:24.039 INFO:tasks.workunit.client.0.smithi029.stdout: create 221/32 file after the snapshot 2024-04-17T19:51:24.040 INFO:tasks.workunit.client.0.smithi029.stdout: create 221/33 file after the snapshot 2024-04-17T19:51:24.041 INFO:tasks.workunit.client.0.smithi029.stdout: create 221/34 file after the snapshot 2024-04-17T19:51:24.042 INFO:tasks.workunit.client.0.smithi029.stdout: create 221/35 file after the snapshot 2024-04-17T19:51:24.043 INFO:tasks.workunit.client.0.smithi029.stdout: create 221/36 file after the snapshot 2024-04-17T19:51:24.044 INFO:tasks.workunit.client.0.smithi029.stdout: create 221/37 file after the snapshot 2024-04-17T19:51:24.044 INFO:tasks.workunit.client.0.smithi029.stdout: create 221/38 file after the snapshot 2024-04-17T19:51:24.046 INFO:tasks.workunit.client.0.smithi029.stdout: create 221/39 file after the snapshot 2024-04-17T19:51:24.046 INFO:tasks.workunit.client.0.smithi029.stdout: create 222/.snap/snap-subdir-test 2024-04-17T19:51:24.063 INFO:tasks.workunit.client.0.smithi029.stdout: create 222/30 file after the snapshot 2024-04-17T19:51:24.064 INFO:tasks.workunit.client.0.smithi029.stdout: create 222/31 file after the snapshot 2024-04-17T19:51:24.065 INFO:tasks.workunit.client.0.smithi029.stdout: create 222/32 file after the snapshot 2024-04-17T19:51:24.066 INFO:tasks.workunit.client.0.smithi029.stdout: create 222/33 file after the snapshot 2024-04-17T19:51:24.067 INFO:tasks.workunit.client.0.smithi029.stdout: create 222/34 file after the snapshot 2024-04-17T19:51:24.068 INFO:tasks.workunit.client.0.smithi029.stdout: create 222/35 file after the snapshot 2024-04-17T19:51:24.068 INFO:tasks.workunit.client.0.smithi029.stdout: create 222/36 file after the snapshot 2024-04-17T19:51:24.069 INFO:tasks.workunit.client.0.smithi029.stdout: create 222/37 file after the snapshot 2024-04-17T19:51:24.070 INFO:tasks.workunit.client.0.smithi029.stdout: create 222/38 file after the snapshot 2024-04-17T19:51:24.071 INFO:tasks.workunit.client.0.smithi029.stdout: create 222/39 file after the snapshot 2024-04-17T19:51:24.072 INFO:tasks.workunit.client.0.smithi029.stdout: create 223/.snap/snap-subdir-test 2024-04-17T19:51:24.087 INFO:tasks.workunit.client.0.smithi029.stdout: create 223/30 file after the snapshot 2024-04-17T19:51:24.088 INFO:tasks.workunit.client.0.smithi029.stdout: create 223/31 file after the snapshot 2024-04-17T19:51:24.089 INFO:tasks.workunit.client.0.smithi029.stdout: create 223/32 file after the snapshot 2024-04-17T19:51:24.090 INFO:tasks.workunit.client.0.smithi029.stdout: create 223/33 file after the snapshot 2024-04-17T19:51:24.091 INFO:tasks.workunit.client.0.smithi029.stdout: create 223/34 file after the snapshot 2024-04-17T19:51:24.092 INFO:tasks.workunit.client.0.smithi029.stdout: create 223/35 file after the snapshot 2024-04-17T19:51:24.093 INFO:tasks.workunit.client.0.smithi029.stdout: create 223/36 file after the snapshot 2024-04-17T19:51:24.094 INFO:tasks.workunit.client.0.smithi029.stdout: create 223/37 file after the snapshot 2024-04-17T19:51:24.095 INFO:tasks.workunit.client.0.smithi029.stdout: create 223/38 file after the snapshot 2024-04-17T19:51:24.096 INFO:tasks.workunit.client.0.smithi029.stdout: create 223/39 file after the snapshot 2024-04-17T19:51:24.096 INFO:tasks.workunit.client.0.smithi029.stdout: create 224/.snap/snap-subdir-test 2024-04-17T19:51:24.112 INFO:tasks.workunit.client.0.smithi029.stdout: create 224/30 file after the snapshot 2024-04-17T19:51:24.113 INFO:tasks.workunit.client.0.smithi029.stdout: create 224/31 file after the snapshot 2024-04-17T19:51:24.114 INFO:tasks.workunit.client.0.smithi029.stdout: create 224/32 file after the snapshot 2024-04-17T19:51:24.115 INFO:tasks.workunit.client.0.smithi029.stdout: create 224/33 file after the snapshot 2024-04-17T19:51:24.116 INFO:tasks.workunit.client.0.smithi029.stdout: create 224/34 file after the snapshot 2024-04-17T19:51:24.117 INFO:tasks.workunit.client.0.smithi029.stdout: create 224/35 file after the snapshot 2024-04-17T19:51:24.118 INFO:tasks.workunit.client.0.smithi029.stdout: create 224/36 file after the snapshot 2024-04-17T19:51:24.119 INFO:tasks.workunit.client.0.smithi029.stdout: create 224/37 file after the snapshot 2024-04-17T19:51:24.120 INFO:tasks.workunit.client.0.smithi029.stdout: create 224/38 file after the snapshot 2024-04-17T19:51:24.121 INFO:tasks.workunit.client.0.smithi029.stdout: create 224/39 file after the snapshot 2024-04-17T19:51:24.121 INFO:tasks.workunit.client.0.smithi029.stdout: create 225/.snap/snap-subdir-test 2024-04-17T19:51:24.137 INFO:tasks.workunit.client.0.smithi029.stdout: create 225/30 file after the snapshot 2024-04-17T19:51:24.138 INFO:tasks.workunit.client.0.smithi029.stdout: create 225/31 file after the snapshot 2024-04-17T19:51:24.139 INFO:tasks.workunit.client.0.smithi029.stdout: create 225/32 file after the snapshot 2024-04-17T19:51:24.140 INFO:tasks.workunit.client.0.smithi029.stdout: create 225/33 file after the snapshot 2024-04-17T19:51:24.141 INFO:tasks.workunit.client.0.smithi029.stdout: create 225/34 file after the snapshot 2024-04-17T19:51:24.141 INFO:tasks.workunit.client.0.smithi029.stdout: create 225/35 file after the snapshot 2024-04-17T19:51:24.142 INFO:tasks.workunit.client.0.smithi029.stdout: create 225/36 file after the snapshot 2024-04-17T19:51:24.143 INFO:tasks.workunit.client.0.smithi029.stdout: create 225/37 file after the snapshot 2024-04-17T19:51:24.144 INFO:tasks.workunit.client.0.smithi029.stdout: create 225/38 file after the snapshot 2024-04-17T19:51:24.145 INFO:tasks.workunit.client.0.smithi029.stdout: create 225/39 file after the snapshot 2024-04-17T19:51:24.146 INFO:tasks.workunit.client.0.smithi029.stdout: create 226/.snap/snap-subdir-test 2024-04-17T19:51:24.161 INFO:tasks.workunit.client.0.smithi029.stdout: create 226/30 file after the snapshot 2024-04-17T19:51:24.162 INFO:tasks.workunit.client.0.smithi029.stdout: create 226/31 file after the snapshot 2024-04-17T19:51:24.164 INFO:tasks.workunit.client.0.smithi029.stdout: create 226/32 file after the snapshot 2024-04-17T19:51:24.164 INFO:tasks.workunit.client.0.smithi029.stdout: create 226/33 file after the snapshot 2024-04-17T19:51:24.165 INFO:tasks.workunit.client.0.smithi029.stdout: create 226/34 file after the snapshot 2024-04-17T19:51:24.166 INFO:tasks.workunit.client.0.smithi029.stdout: create 226/35 file after the snapshot 2024-04-17T19:51:24.167 INFO:tasks.workunit.client.0.smithi029.stdout: create 226/36 file after the snapshot 2024-04-17T19:51:24.168 INFO:tasks.workunit.client.0.smithi029.stdout: create 226/37 file after the snapshot 2024-04-17T19:51:24.169 INFO:tasks.workunit.client.0.smithi029.stdout: create 226/38 file after the snapshot 2024-04-17T19:51:24.170 INFO:tasks.workunit.client.0.smithi029.stdout: create 226/39 file after the snapshot 2024-04-17T19:51:24.171 INFO:tasks.workunit.client.0.smithi029.stdout: create 227/.snap/snap-subdir-test 2024-04-17T19:51:24.186 INFO:tasks.workunit.client.0.smithi029.stdout: create 227/30 file after the snapshot 2024-04-17T19:51:24.187 INFO:tasks.workunit.client.0.smithi029.stdout: create 227/31 file after the snapshot 2024-04-17T19:51:24.188 INFO:tasks.workunit.client.0.smithi029.stdout: create 227/32 file after the snapshot 2024-04-17T19:51:24.189 INFO:tasks.workunit.client.0.smithi029.stdout: create 227/33 file after the snapshot 2024-04-17T19:51:24.190 INFO:tasks.workunit.client.0.smithi029.stdout: create 227/34 file after the snapshot 2024-04-17T19:51:24.191 INFO:tasks.workunit.client.0.smithi029.stdout: create 227/35 file after the snapshot 2024-04-17T19:51:24.192 INFO:tasks.workunit.client.0.smithi029.stdout: create 227/36 file after the snapshot 2024-04-17T19:51:24.193 INFO:tasks.workunit.client.0.smithi029.stdout: create 227/37 file after the snapshot 2024-04-17T19:51:24.194 INFO:tasks.workunit.client.0.smithi029.stdout: create 227/38 file after the snapshot 2024-04-17T19:51:24.195 INFO:tasks.workunit.client.0.smithi029.stdout: create 227/39 file after the snapshot 2024-04-17T19:51:24.196 INFO:tasks.workunit.client.0.smithi029.stdout: create 228/.snap/snap-subdir-test 2024-04-17T19:51:24.211 INFO:tasks.workunit.client.0.smithi029.stdout: create 228/30 file after the snapshot 2024-04-17T19:51:24.212 INFO:tasks.workunit.client.0.smithi029.stdout: create 228/31 file after the snapshot 2024-04-17T19:51:24.212 INFO:tasks.workunit.client.0.smithi029.stdout: create 228/32 file after the snapshot 2024-04-17T19:51:24.627 INFO:tasks.workunit.client.0.smithi029.stdout: create 228/33 file after the snapshot 2024-04-17T19:51:24.772 INFO:tasks.workunit.client.0.smithi029.stdout: create 228/34 file after the snapshot 2024-04-17T19:51:24.773 INFO:tasks.workunit.client.0.smithi029.stdout: create 228/35 file after the snapshot 2024-04-17T19:51:24.774 INFO:tasks.workunit.client.0.smithi029.stdout: create 228/36 file after the snapshot 2024-04-17T19:51:24.775 INFO:tasks.workunit.client.0.smithi029.stdout: create 228/37 file after the snapshot 2024-04-17T19:51:24.776 INFO:tasks.workunit.client.0.smithi029.stdout: create 228/38 file after the snapshot 2024-04-17T19:51:24.777 INFO:tasks.workunit.client.0.smithi029.stdout: create 228/39 file after the snapshot 2024-04-17T19:51:24.778 INFO:tasks.workunit.client.0.smithi029.stdout: create 229/.snap/snap-subdir-test 2024-04-17T19:51:24.798 INFO:tasks.workunit.client.0.smithi029.stdout: create 229/30 file after the snapshot 2024-04-17T19:51:24.799 INFO:tasks.workunit.client.0.smithi029.stdout: create 229/31 file after the snapshot 2024-04-17T19:51:24.800 INFO:tasks.workunit.client.0.smithi029.stdout: create 229/32 file after the snapshot 2024-04-17T19:51:24.801 INFO:tasks.workunit.client.0.smithi029.stdout: create 229/33 file after the snapshot 2024-04-17T19:51:24.802 INFO:tasks.workunit.client.0.smithi029.stdout: create 229/34 file after the snapshot 2024-04-17T19:51:24.803 INFO:tasks.workunit.client.0.smithi029.stdout: create 229/35 file after the snapshot 2024-04-17T19:51:24.804 INFO:tasks.workunit.client.0.smithi029.stdout: create 229/36 file after the snapshot 2024-04-17T19:51:24.805 INFO:tasks.workunit.client.0.smithi029.stdout: create 229/37 file after the snapshot 2024-04-17T19:51:24.806 INFO:tasks.workunit.client.0.smithi029.stdout: create 229/38 file after the snapshot 2024-04-17T19:51:24.807 INFO:tasks.workunit.client.0.smithi029.stdout: create 229/39 file after the snapshot 2024-04-17T19:51:24.808 INFO:tasks.workunit.client.0.smithi029.stdout: create 230/.snap/snap-subdir-test 2024-04-17T19:51:24.828 INFO:tasks.workunit.client.0.smithi029.stdout: create 230/30 file after the snapshot 2024-04-17T19:51:24.829 INFO:tasks.workunit.client.0.smithi029.stdout: create 230/31 file after the snapshot 2024-04-17T19:51:25.626 INFO:tasks.workunit.client.0.smithi029.stdout: create 230/32 file after the snapshot 2024-04-17T19:51:25.643 INFO:tasks.workunit.client.0.smithi029.stdout: create 230/33 file after the snapshot 2024-04-17T19:51:25.673 INFO:tasks.workunit.client.0.smithi029.stdout: create 230/34 file after the snapshot 2024-04-17T19:51:25.688 INFO:tasks.workunit.client.0.smithi029.stdout: create 230/35 file after the snapshot 2024-04-17T19:51:25.709 INFO:tasks.workunit.client.0.smithi029.stdout: create 230/36 file after the snapshot 2024-04-17T19:51:25.715 INFO:tasks.workunit.client.0.smithi029.stdout: create 230/37 file after the snapshot 2024-04-17T19:51:25.726 INFO:tasks.workunit.client.0.smithi029.stdout: create 230/38 file after the snapshot 2024-04-17T19:51:25.741 INFO:tasks.workunit.client.0.smithi029.stdout: create 230/39 file after the snapshot 2024-04-17T19:51:25.767 INFO:tasks.workunit.client.0.smithi029.stdout: create 231/.snap/snap-subdir-test 2024-04-17T19:51:28.706 INFO:tasks.workunit.client.0.smithi029.stdout: create 231/30 file after the snapshot 2024-04-17T19:51:28.708 INFO:tasks.workunit.client.0.smithi029.stdout: create 231/31 file after the snapshot 2024-04-17T19:51:28.709 INFO:tasks.workunit.client.0.smithi029.stdout: create 231/32 file after the snapshot 2024-04-17T19:51:28.710 INFO:tasks.workunit.client.0.smithi029.stdout: create 231/33 file after the snapshot 2024-04-17T19:51:28.711 INFO:tasks.workunit.client.0.smithi029.stdout: create 231/34 file after the snapshot 2024-04-17T19:51:28.712 INFO:tasks.workunit.client.0.smithi029.stdout: create 231/35 file after the snapshot 2024-04-17T19:51:28.713 INFO:tasks.workunit.client.0.smithi029.stdout: create 231/36 file after the snapshot 2024-04-17T19:51:28.714 INFO:tasks.workunit.client.0.smithi029.stdout: create 231/37 file after the snapshot 2024-04-17T19:51:28.715 INFO:tasks.workunit.client.0.smithi029.stdout: create 231/38 file after the snapshot 2024-04-17T19:51:28.717 INFO:tasks.workunit.client.0.smithi029.stdout: create 231/39 file after the snapshot 2024-04-17T19:51:28.718 INFO:tasks.workunit.client.0.smithi029.stdout: create 232/.snap/snap-subdir-test 2024-04-17T19:51:29.369 INFO:tasks.workunit.client.0.smithi029.stdout: create 232/30 file after the snapshot 2024-04-17T19:51:29.370 INFO:tasks.workunit.client.0.smithi029.stdout: create 232/31 file after the snapshot 2024-04-17T19:51:29.371 INFO:tasks.workunit.client.0.smithi029.stdout: create 232/32 file after the snapshot 2024-04-17T19:51:29.372 INFO:tasks.workunit.client.0.smithi029.stdout: create 232/33 file after the snapshot 2024-04-17T19:51:30.132 INFO:tasks.workunit.client.0.smithi029.stdout: create 232/34 file after the snapshot 2024-04-17T19:51:30.133 INFO:tasks.workunit.client.0.smithi029.stdout: create 232/35 file after the snapshot 2024-04-17T19:51:30.134 INFO:tasks.workunit.client.0.smithi029.stdout: create 232/36 file after the snapshot 2024-04-17T19:51:30.135 INFO:tasks.workunit.client.0.smithi029.stdout: create 232/37 file after the snapshot 2024-04-17T19:51:30.136 INFO:tasks.workunit.client.0.smithi029.stdout: create 232/38 file after the snapshot 2024-04-17T19:51:30.137 INFO:tasks.workunit.client.0.smithi029.stdout: create 232/39 file after the snapshot 2024-04-17T19:51:30.138 INFO:tasks.workunit.client.0.smithi029.stdout: create 233/.snap/snap-subdir-test 2024-04-17T19:51:30.159 INFO:tasks.workunit.client.0.smithi029.stdout: create 233/30 file after the snapshot 2024-04-17T19:51:30.160 INFO:tasks.workunit.client.0.smithi029.stdout: create 233/31 file after the snapshot 2024-04-17T19:51:30.161 INFO:tasks.workunit.client.0.smithi029.stdout: create 233/32 file after the snapshot 2024-04-17T19:51:30.162 INFO:tasks.workunit.client.0.smithi029.stdout: create 233/33 file after the snapshot 2024-04-17T19:51:30.163 INFO:tasks.workunit.client.0.smithi029.stdout: create 233/34 file after the snapshot 2024-04-17T19:51:30.164 INFO:tasks.workunit.client.0.smithi029.stdout: create 233/35 file after the snapshot 2024-04-17T19:51:30.165 INFO:tasks.workunit.client.0.smithi029.stdout: create 233/36 file after the snapshot 2024-04-17T19:51:30.166 INFO:tasks.workunit.client.0.smithi029.stdout: create 233/37 file after the snapshot 2024-04-17T19:51:30.167 INFO:tasks.workunit.client.0.smithi029.stdout: create 233/38 file after the snapshot 2024-04-17T19:51:30.168 INFO:tasks.workunit.client.0.smithi029.stdout: create 233/39 file after the snapshot 2024-04-17T19:51:30.169 INFO:tasks.workunit.client.0.smithi029.stdout: create 234/.snap/snap-subdir-test 2024-04-17T19:51:30.191 INFO:tasks.workunit.client.0.smithi029.stdout: create 234/30 file after the snapshot 2024-04-17T19:51:30.192 INFO:tasks.workunit.client.0.smithi029.stdout: create 234/31 file after the snapshot 2024-04-17T19:51:30.193 INFO:tasks.workunit.client.0.smithi029.stdout: create 234/32 file after the snapshot 2024-04-17T19:51:30.194 INFO:tasks.workunit.client.0.smithi029.stdout: create 234/33 file after the snapshot 2024-04-17T19:51:30.195 INFO:tasks.workunit.client.0.smithi029.stdout: create 234/34 file after the snapshot 2024-04-17T19:51:30.196 INFO:tasks.workunit.client.0.smithi029.stdout: create 234/35 file after the snapshot 2024-04-17T19:51:30.197 INFO:tasks.workunit.client.0.smithi029.stdout: create 234/36 file after the snapshot 2024-04-17T19:51:30.198 INFO:tasks.workunit.client.0.smithi029.stdout: create 234/37 file after the snapshot 2024-04-17T19:51:30.199 INFO:tasks.workunit.client.0.smithi029.stdout: create 234/38 file after the snapshot 2024-04-17T19:51:30.200 INFO:tasks.workunit.client.0.smithi029.stdout: create 234/39 file after the snapshot 2024-04-17T19:51:30.201 INFO:tasks.workunit.client.0.smithi029.stdout: create 235/.snap/snap-subdir-test 2024-04-17T19:51:30.682 INFO:tasks.workunit.client.0.smithi029.stdout: create 235/30 file after the snapshot 2024-04-17T19:51:30.683 INFO:tasks.workunit.client.0.smithi029.stdout: create 235/31 file after the snapshot 2024-04-17T19:51:30.684 INFO:tasks.workunit.client.0.smithi029.stdout: create 235/32 file after the snapshot 2024-04-17T19:51:30.685 INFO:tasks.workunit.client.0.smithi029.stdout: create 235/33 file after the snapshot 2024-04-17T19:51:30.687 INFO:tasks.workunit.client.0.smithi029.stdout: create 235/34 file after the snapshot 2024-04-17T19:51:30.688 INFO:tasks.workunit.client.0.smithi029.stdout: create 235/35 file after the snapshot 2024-04-17T19:51:30.689 INFO:tasks.workunit.client.0.smithi029.stdout: create 235/36 file after the snapshot 2024-04-17T19:51:30.690 INFO:tasks.workunit.client.0.smithi029.stdout: create 235/37 file after the snapshot 2024-04-17T19:51:30.691 INFO:tasks.workunit.client.0.smithi029.stdout: create 235/38 file after the snapshot 2024-04-17T19:51:30.692 INFO:tasks.workunit.client.0.smithi029.stdout: create 235/39 file after the snapshot 2024-04-17T19:51:30.693 INFO:tasks.workunit.client.0.smithi029.stdout: create 236/.snap/snap-subdir-test 2024-04-17T19:51:31.671 INFO:tasks.workunit.client.0.smithi029.stdout: create 236/30 file after the snapshot 2024-04-17T19:51:31.672 INFO:tasks.workunit.client.0.smithi029.stdout: create 236/31 file after the snapshot 2024-04-17T19:51:31.673 INFO:tasks.workunit.client.0.smithi029.stdout: create 236/32 file after the snapshot 2024-04-17T19:51:31.674 INFO:tasks.workunit.client.0.smithi029.stdout: create 236/33 file after the snapshot 2024-04-17T19:51:31.675 INFO:tasks.workunit.client.0.smithi029.stdout: create 236/34 file after the snapshot 2024-04-17T19:51:31.676 INFO:tasks.workunit.client.0.smithi029.stdout: create 236/35 file after the snapshot 2024-04-17T19:51:31.678 INFO:tasks.workunit.client.0.smithi029.stdout: create 236/36 file after the snapshot 2024-04-17T19:51:31.679 INFO:tasks.workunit.client.0.smithi029.stdout: create 236/37 file after the snapshot 2024-04-17T19:51:31.680 INFO:tasks.workunit.client.0.smithi029.stdout: create 236/38 file after the snapshot 2024-04-17T19:51:31.681 INFO:tasks.workunit.client.0.smithi029.stdout: create 236/39 file after the snapshot 2024-04-17T19:51:31.682 INFO:tasks.workunit.client.0.smithi029.stdout: create 237/.snap/snap-subdir-test 2024-04-17T19:51:31.704 INFO:tasks.workunit.client.0.smithi029.stdout: create 237/30 file after the snapshot 2024-04-17T19:51:31.705 INFO:tasks.workunit.client.0.smithi029.stdout: create 237/31 file after the snapshot 2024-04-17T19:51:31.706 INFO:tasks.workunit.client.0.smithi029.stdout: create 237/32 file after the snapshot 2024-04-17T19:51:31.707 INFO:tasks.workunit.client.0.smithi029.stdout: create 237/33 file after the snapshot 2024-04-17T19:51:31.708 INFO:tasks.workunit.client.0.smithi029.stdout: create 237/34 file after the snapshot 2024-04-17T19:51:31.709 INFO:tasks.workunit.client.0.smithi029.stdout: create 237/35 file after the snapshot 2024-04-17T19:51:31.710 INFO:tasks.workunit.client.0.smithi029.stdout: create 237/36 file after the snapshot 2024-04-17T19:51:31.711 INFO:tasks.workunit.client.0.smithi029.stdout: create 237/37 file after the snapshot 2024-04-17T19:51:31.712 INFO:tasks.workunit.client.0.smithi029.stdout: create 237/38 file after the snapshot 2024-04-17T19:51:31.713 INFO:tasks.workunit.client.0.smithi029.stdout: create 237/39 file after the snapshot 2024-04-17T19:51:31.714 INFO:tasks.workunit.client.0.smithi029.stdout: create 238/.snap/snap-subdir-test 2024-04-17T19:51:31.733 INFO:tasks.workunit.client.0.smithi029.stdout: create 238/30 file after the snapshot 2024-04-17T19:51:31.734 INFO:tasks.workunit.client.0.smithi029.stdout: create 238/31 file after the snapshot 2024-04-17T19:51:31.735 INFO:tasks.workunit.client.0.smithi029.stdout: create 238/32 file after the snapshot 2024-04-17T19:51:31.736 INFO:tasks.workunit.client.0.smithi029.stdout: create 238/33 file after the snapshot 2024-04-17T19:51:31.737 INFO:tasks.workunit.client.0.smithi029.stdout: create 238/34 file after the snapshot 2024-04-17T19:51:31.738 INFO:tasks.workunit.client.0.smithi029.stdout: create 238/35 file after the snapshot 2024-04-17T19:51:31.739 INFO:tasks.workunit.client.0.smithi029.stdout: create 238/36 file after the snapshot 2024-04-17T19:51:31.740 INFO:tasks.workunit.client.0.smithi029.stdout: create 238/37 file after the snapshot 2024-04-17T19:51:31.741 INFO:tasks.workunit.client.0.smithi029.stdout: create 238/38 file after the snapshot 2024-04-17T19:51:31.742 INFO:tasks.workunit.client.0.smithi029.stdout: create 238/39 file after the snapshot 2024-04-17T19:51:31.743 INFO:tasks.workunit.client.0.smithi029.stdout: create 239/.snap/snap-subdir-test 2024-04-17T19:51:31.761 INFO:tasks.workunit.client.0.smithi029.stdout: create 239/30 file after the snapshot 2024-04-17T19:51:31.762 INFO:tasks.workunit.client.0.smithi029.stdout: create 239/31 file after the snapshot 2024-04-17T19:51:31.763 INFO:tasks.workunit.client.0.smithi029.stdout: create 239/32 file after the snapshot 2024-04-17T19:51:31.764 INFO:tasks.workunit.client.0.smithi029.stdout: create 239/33 file after the snapshot 2024-04-17T19:51:31.765 INFO:tasks.workunit.client.0.smithi029.stdout: create 239/34 file after the snapshot 2024-04-17T19:51:31.766 INFO:tasks.workunit.client.0.smithi029.stdout: create 239/35 file after the snapshot 2024-04-17T19:51:31.767 INFO:tasks.workunit.client.0.smithi029.stdout: create 239/36 file after the snapshot 2024-04-17T19:51:31.768 INFO:tasks.workunit.client.0.smithi029.stdout: create 239/37 file after the snapshot 2024-04-17T19:51:31.769 INFO:tasks.workunit.client.0.smithi029.stdout: create 239/38 file after the snapshot 2024-04-17T19:51:31.770 INFO:tasks.workunit.client.0.smithi029.stdout: create 239/39 file after the snapshot 2024-04-17T19:51:31.771 INFO:tasks.workunit.client.0.smithi029.stdout: create 240/.snap/snap-subdir-test 2024-04-17T19:51:31.790 INFO:tasks.workunit.client.0.smithi029.stdout: create 240/30 file after the snapshot 2024-04-17T19:51:31.791 INFO:tasks.workunit.client.0.smithi029.stdout: create 240/31 file after the snapshot 2024-04-17T19:51:31.792 INFO:tasks.workunit.client.0.smithi029.stdout: create 240/32 file after the snapshot 2024-04-17T19:51:31.793 INFO:tasks.workunit.client.0.smithi029.stdout: create 240/33 file after the snapshot 2024-04-17T19:51:31.794 INFO:tasks.workunit.client.0.smithi029.stdout: create 240/34 file after the snapshot 2024-04-17T19:51:31.795 INFO:tasks.workunit.client.0.smithi029.stdout: create 240/35 file after the snapshot 2024-04-17T19:51:31.796 INFO:tasks.workunit.client.0.smithi029.stdout: create 240/36 file after the snapshot 2024-04-17T19:51:31.797 INFO:tasks.workunit.client.0.smithi029.stdout: create 240/37 file after the snapshot 2024-04-17T19:51:31.798 INFO:tasks.workunit.client.0.smithi029.stdout: create 240/38 file after the snapshot 2024-04-17T19:51:31.798 INFO:tasks.workunit.client.0.smithi029.stdout: create 240/39 file after the snapshot 2024-04-17T19:51:31.799 INFO:tasks.workunit.client.0.smithi029.stdout: create 241/.snap/snap-subdir-test 2024-04-17T19:51:31.818 INFO:tasks.workunit.client.0.smithi029.stdout: create 241/30 file after the snapshot 2024-04-17T19:51:31.820 INFO:tasks.workunit.client.0.smithi029.stdout: create 241/31 file after the snapshot 2024-04-17T19:51:31.821 INFO:tasks.workunit.client.0.smithi029.stdout: create 241/32 file after the snapshot 2024-04-17T19:51:31.823 INFO:tasks.workunit.client.0.smithi029.stdout: create 241/33 file after the snapshot 2024-04-17T19:51:31.824 INFO:tasks.workunit.client.0.smithi029.stdout: create 241/34 file after the snapshot 2024-04-17T19:51:31.826 INFO:tasks.workunit.client.0.smithi029.stdout: create 241/35 file after the snapshot 2024-04-17T19:51:31.827 INFO:tasks.workunit.client.0.smithi029.stdout: create 241/36 file after the snapshot 2024-04-17T19:51:31.829 INFO:tasks.workunit.client.0.smithi029.stdout: create 241/37 file after the snapshot 2024-04-17T19:51:31.830 INFO:tasks.workunit.client.0.smithi029.stdout: create 241/38 file after the snapshot 2024-04-17T19:51:31.832 INFO:tasks.workunit.client.0.smithi029.stdout: create 241/39 file after the snapshot 2024-04-17T19:51:31.833 INFO:tasks.workunit.client.0.smithi029.stdout: create 242/.snap/snap-subdir-test 2024-04-17T19:51:33.303 INFO:tasks.workunit.client.0.smithi029.stdout: create 242/30 file after the snapshot 2024-04-17T19:51:33.307 INFO:tasks.workunit.client.0.smithi029.stdout: create 242/31 file after the snapshot 2024-04-17T19:51:33.307 INFO:tasks.workunit.client.0.smithi029.stdout: create 242/32 file after the snapshot 2024-04-17T19:51:33.309 INFO:tasks.workunit.client.0.smithi029.stdout: create 242/33 file after the snapshot 2024-04-17T19:51:33.310 INFO:tasks.workunit.client.0.smithi029.stdout: create 242/34 file after the snapshot 2024-04-17T19:51:33.311 INFO:tasks.workunit.client.0.smithi029.stdout: create 242/35 file after the snapshot 2024-04-17T19:51:33.312 INFO:tasks.workunit.client.0.smithi029.stdout: create 242/36 file after the snapshot 2024-04-17T19:51:33.313 INFO:tasks.workunit.client.0.smithi029.stdout: create 242/37 file after the snapshot 2024-04-17T19:51:33.314 INFO:tasks.workunit.client.0.smithi029.stdout: create 242/38 file after the snapshot 2024-04-17T19:51:33.316 INFO:tasks.workunit.client.0.smithi029.stdout: create 242/39 file after the snapshot 2024-04-17T19:51:33.317 INFO:tasks.workunit.client.0.smithi029.stdout: create 243/.snap/snap-subdir-test 2024-04-17T19:51:33.342 INFO:tasks.workunit.client.0.smithi029.stdout: create 243/30 file after the snapshot 2024-04-17T19:51:33.343 INFO:tasks.workunit.client.0.smithi029.stdout: create 243/31 file after the snapshot 2024-04-17T19:51:33.344 INFO:tasks.workunit.client.0.smithi029.stdout: create 243/32 file after the snapshot 2024-04-17T19:51:33.345 INFO:tasks.workunit.client.0.smithi029.stdout: create 243/33 file after the snapshot 2024-04-17T19:51:33.346 INFO:tasks.workunit.client.0.smithi029.stdout: create 243/34 file after the snapshot 2024-04-17T19:51:33.347 INFO:tasks.workunit.client.0.smithi029.stdout: create 243/35 file after the snapshot 2024-04-17T19:51:33.348 INFO:tasks.workunit.client.0.smithi029.stdout: create 243/36 file after the snapshot 2024-04-17T19:51:33.349 INFO:tasks.workunit.client.0.smithi029.stdout: create 243/37 file after the snapshot 2024-04-17T19:51:33.350 INFO:tasks.workunit.client.0.smithi029.stdout: create 243/38 file after the snapshot 2024-04-17T19:51:33.351 INFO:tasks.workunit.client.0.smithi029.stdout: create 243/39 file after the snapshot 2024-04-17T19:51:33.352 INFO:tasks.workunit.client.0.smithi029.stdout: create 244/.snap/snap-subdir-test 2024-04-17T19:51:34.170 INFO:tasks.workunit.client.0.smithi029.stdout: create 244/30 file after the snapshot 2024-04-17T19:51:34.172 INFO:tasks.workunit.client.0.smithi029.stdout: create 244/31 file after the snapshot 2024-04-17T19:51:34.173 INFO:tasks.workunit.client.0.smithi029.stdout: create 244/32 file after the snapshot 2024-04-17T19:51:34.175 INFO:tasks.workunit.client.0.smithi029.stdout: create 244/33 file after the snapshot 2024-04-17T19:51:34.177 INFO:tasks.workunit.client.0.smithi029.stdout: create 244/34 file after the snapshot 2024-04-17T19:51:34.178 INFO:tasks.workunit.client.0.smithi029.stdout: create 244/35 file after the snapshot 2024-04-17T19:51:34.180 INFO:tasks.workunit.client.0.smithi029.stdout: create 244/36 file after the snapshot 2024-04-17T19:51:34.182 INFO:tasks.workunit.client.0.smithi029.stdout: create 244/37 file after the snapshot 2024-04-17T19:51:34.184 INFO:tasks.workunit.client.0.smithi029.stdout: create 244/38 file after the snapshot 2024-04-17T19:51:34.185 INFO:tasks.workunit.client.0.smithi029.stdout: create 244/39 file after the snapshot 2024-04-17T19:51:34.187 INFO:tasks.workunit.client.0.smithi029.stdout: create 245/.snap/snap-subdir-test 2024-04-17T19:51:34.207 INFO:tasks.workunit.client.0.smithi029.stdout: create 245/30 file after the snapshot 2024-04-17T19:51:34.208 INFO:tasks.workunit.client.0.smithi029.stdout: create 245/31 file after the snapshot 2024-04-17T19:51:34.209 INFO:tasks.workunit.client.0.smithi029.stdout: create 245/32 file after the snapshot 2024-04-17T19:51:34.210 INFO:tasks.workunit.client.0.smithi029.stdout: create 245/33 file after the snapshot 2024-04-17T19:51:34.211 INFO:tasks.workunit.client.0.smithi029.stdout: create 245/34 file after the snapshot 2024-04-17T19:51:34.212 INFO:tasks.workunit.client.0.smithi029.stdout: create 245/35 file after the snapshot 2024-04-17T19:51:34.213 INFO:tasks.workunit.client.0.smithi029.stdout: create 245/36 file after the snapshot 2024-04-17T19:51:34.214 INFO:tasks.workunit.client.0.smithi029.stdout: create 245/37 file after the snapshot 2024-04-17T19:51:34.215 INFO:tasks.workunit.client.0.smithi029.stdout: create 245/38 file after the snapshot 2024-04-17T19:51:34.216 INFO:tasks.workunit.client.0.smithi029.stdout: create 245/39 file after the snapshot 2024-04-17T19:51:34.217 INFO:tasks.workunit.client.0.smithi029.stdout: create 246/.snap/snap-subdir-test 2024-04-17T19:51:34.236 INFO:tasks.workunit.client.0.smithi029.stdout: create 246/30 file after the snapshot 2024-04-17T19:51:34.237 INFO:tasks.workunit.client.0.smithi029.stdout: create 246/31 file after the snapshot 2024-04-17T19:51:34.239 INFO:tasks.workunit.client.0.smithi029.stdout: create 246/32 file after the snapshot 2024-04-17T19:51:34.239 INFO:tasks.workunit.client.0.smithi029.stdout: create 246/33 file after the snapshot 2024-04-17T19:51:34.240 INFO:tasks.workunit.client.0.smithi029.stdout: create 246/34 file after the snapshot 2024-04-17T19:51:34.241 INFO:tasks.workunit.client.0.smithi029.stdout: create 246/35 file after the snapshot 2024-04-17T19:51:34.242 INFO:tasks.workunit.client.0.smithi029.stdout: create 246/36 file after the snapshot 2024-04-17T19:51:34.243 INFO:tasks.workunit.client.0.smithi029.stdout: create 246/37 file after the snapshot 2024-04-17T19:51:34.245 INFO:tasks.workunit.client.0.smithi029.stdout: create 246/38 file after the snapshot 2024-04-17T19:51:34.245 INFO:tasks.workunit.client.0.smithi029.stdout: create 246/39 file after the snapshot 2024-04-17T19:51:34.247 INFO:tasks.workunit.client.0.smithi029.stdout: create 247/.snap/snap-subdir-test 2024-04-17T19:51:34.265 INFO:tasks.workunit.client.0.smithi029.stdout: create 247/30 file after the snapshot 2024-04-17T19:51:34.266 INFO:tasks.workunit.client.0.smithi029.stdout: create 247/31 file after the snapshot 2024-04-17T19:51:34.267 INFO:tasks.workunit.client.0.smithi029.stdout: create 247/32 file after the snapshot 2024-04-17T19:51:34.268 INFO:tasks.workunit.client.0.smithi029.stdout: create 247/33 file after the snapshot 2024-04-17T19:51:34.269 INFO:tasks.workunit.client.0.smithi029.stdout: create 247/34 file after the snapshot 2024-04-17T19:51:34.270 INFO:tasks.workunit.client.0.smithi029.stdout: create 247/35 file after the snapshot 2024-04-17T19:51:34.271 INFO:tasks.workunit.client.0.smithi029.stdout: create 247/36 file after the snapshot 2024-04-17T19:51:34.272 INFO:tasks.workunit.client.0.smithi029.stdout: create 247/37 file after the snapshot 2024-04-17T19:51:34.273 INFO:tasks.workunit.client.0.smithi029.stdout: create 247/38 file after the snapshot 2024-04-17T19:51:34.273 INFO:tasks.workunit.client.0.smithi029.stdout: create 247/39 file after the snapshot 2024-04-17T19:51:34.274 INFO:tasks.workunit.client.0.smithi029.stdout: create 248/.snap/snap-subdir-test 2024-04-17T19:51:34.293 INFO:tasks.workunit.client.0.smithi029.stdout: create 248/30 file after the snapshot 2024-04-17T19:51:34.294 INFO:tasks.workunit.client.0.smithi029.stdout: create 248/31 file after the snapshot 2024-04-17T19:51:34.295 INFO:tasks.workunit.client.0.smithi029.stdout: create 248/32 file after the snapshot 2024-04-17T19:51:34.296 INFO:tasks.workunit.client.0.smithi029.stdout: create 248/33 file after the snapshot 2024-04-17T19:51:34.297 INFO:tasks.workunit.client.0.smithi029.stdout: create 248/34 file after the snapshot 2024-04-17T19:51:34.298 INFO:tasks.workunit.client.0.smithi029.stdout: create 248/35 file after the snapshot 2024-04-17T19:51:34.299 INFO:tasks.workunit.client.0.smithi029.stdout: create 248/36 file after the snapshot 2024-04-17T19:51:34.300 INFO:tasks.workunit.client.0.smithi029.stdout: create 248/37 file after the snapshot 2024-04-17T19:51:34.301 INFO:tasks.workunit.client.0.smithi029.stdout: create 248/38 file after the snapshot 2024-04-17T19:51:34.302 INFO:tasks.workunit.client.0.smithi029.stdout: create 248/39 file after the snapshot 2024-04-17T19:51:34.303 INFO:tasks.workunit.client.0.smithi029.stdout: create 249/.snap/snap-subdir-test 2024-04-17T19:51:34.322 INFO:tasks.workunit.client.0.smithi029.stdout: create 249/30 file after the snapshot 2024-04-17T19:51:34.323 INFO:tasks.workunit.client.0.smithi029.stdout: create 249/31 file after the snapshot 2024-04-17T19:51:34.324 INFO:tasks.workunit.client.0.smithi029.stdout: create 249/32 file after the snapshot 2024-04-17T19:51:34.325 INFO:tasks.workunit.client.0.smithi029.stdout: create 249/33 file after the snapshot 2024-04-17T19:51:34.326 INFO:tasks.workunit.client.0.smithi029.stdout: create 249/34 file after the snapshot 2024-04-17T19:51:34.327 INFO:tasks.workunit.client.0.smithi029.stdout: create 249/35 file after the snapshot 2024-04-17T19:51:34.328 INFO:tasks.workunit.client.0.smithi029.stdout: create 249/36 file after the snapshot 2024-04-17T19:51:34.329 INFO:tasks.workunit.client.0.smithi029.stdout: create 249/37 file after the snapshot 2024-04-17T19:51:34.330 INFO:tasks.workunit.client.0.smithi029.stdout: create 249/38 file after the snapshot 2024-04-17T19:51:34.331 INFO:tasks.workunit.client.0.smithi029.stdout: create 249/39 file after the snapshot 2024-04-17T19:51:34.332 INFO:tasks.workunit.client.0.smithi029.stdout: create 250/.snap/snap-subdir-test 2024-04-17T19:51:34.350 INFO:tasks.workunit.client.0.smithi029.stdout: create 250/30 file after the snapshot 2024-04-17T19:51:34.351 INFO:tasks.workunit.client.0.smithi029.stdout: create 250/31 file after the snapshot 2024-04-17T19:51:34.352 INFO:tasks.workunit.client.0.smithi029.stdout: create 250/32 file after the snapshot 2024-04-17T19:51:34.353 INFO:tasks.workunit.client.0.smithi029.stdout: create 250/33 file after the snapshot 2024-04-17T19:51:34.353 INFO:tasks.workunit.client.0.smithi029.stdout: create 250/34 file after the snapshot 2024-04-17T19:51:34.355 INFO:tasks.workunit.client.0.smithi029.stdout: create 250/35 file after the snapshot 2024-04-17T19:51:34.355 INFO:tasks.workunit.client.0.smithi029.stdout: create 250/36 file after the snapshot 2024-04-17T19:51:34.357 INFO:tasks.workunit.client.0.smithi029.stdout: create 250/37 file after the snapshot 2024-04-17T19:51:34.357 INFO:tasks.workunit.client.0.smithi029.stdout: create 250/38 file after the snapshot 2024-04-17T19:51:34.358 INFO:tasks.workunit.client.0.smithi029.stdout: create 250/39 file after the snapshot 2024-04-17T19:51:34.359 INFO:tasks.workunit.client.0.smithi029.stdout: create 251/.snap/snap-subdir-test 2024-04-17T19:51:34.378 INFO:tasks.workunit.client.0.smithi029.stdout: create 251/30 file after the snapshot 2024-04-17T19:51:34.379 INFO:tasks.workunit.client.0.smithi029.stdout: create 251/31 file after the snapshot 2024-04-17T19:51:34.380 INFO:tasks.workunit.client.0.smithi029.stdout: create 251/32 file after the snapshot 2024-04-17T19:51:34.381 INFO:tasks.workunit.client.0.smithi029.stdout: create 251/33 file after the snapshot 2024-04-17T19:51:34.382 INFO:tasks.workunit.client.0.smithi029.stdout: create 251/34 file after the snapshot 2024-04-17T19:51:34.383 INFO:tasks.workunit.client.0.smithi029.stdout: create 251/35 file after the snapshot 2024-04-17T19:51:34.384 INFO:tasks.workunit.client.0.smithi029.stdout: create 251/36 file after the snapshot 2024-04-17T19:51:34.385 INFO:tasks.workunit.client.0.smithi029.stdout: create 251/37 file after the snapshot 2024-04-17T19:51:34.386 INFO:tasks.workunit.client.0.smithi029.stdout: create 251/38 file after the snapshot 2024-04-17T19:51:34.387 INFO:tasks.workunit.client.0.smithi029.stdout: create 251/39 file after the snapshot 2024-04-17T19:51:34.388 INFO:tasks.workunit.client.0.smithi029.stdout: create 252/.snap/snap-subdir-test 2024-04-17T19:51:34.407 INFO:tasks.workunit.client.0.smithi029.stdout: create 252/30 file after the snapshot 2024-04-17T19:51:34.408 INFO:tasks.workunit.client.0.smithi029.stdout: create 252/31 file after the snapshot 2024-04-17T19:51:34.409 INFO:tasks.workunit.client.0.smithi029.stdout: create 252/32 file after the snapshot 2024-04-17T19:51:34.410 INFO:tasks.workunit.client.0.smithi029.stdout: create 252/33 file after the snapshot 2024-04-17T19:51:34.411 INFO:tasks.workunit.client.0.smithi029.stdout: create 252/34 file after the snapshot 2024-04-17T19:51:34.663 INFO:tasks.workunit.client.0.smithi029.stdout: create 252/35 file after the snapshot 2024-04-17T19:51:34.664 INFO:tasks.workunit.client.0.smithi029.stdout: create 252/36 file after the snapshot 2024-04-17T19:51:34.665 INFO:tasks.workunit.client.0.smithi029.stdout: create 252/37 file after the snapshot 2024-04-17T19:51:34.666 INFO:tasks.workunit.client.0.smithi029.stdout: create 252/38 file after the snapshot 2024-04-17T19:51:34.667 INFO:tasks.workunit.client.0.smithi029.stdout: create 252/39 file after the snapshot 2024-04-17T19:51:34.668 INFO:tasks.workunit.client.0.smithi029.stdout: create 253/.snap/snap-subdir-test 2024-04-17T19:51:34.687 INFO:tasks.workunit.client.0.smithi029.stdout: create 253/30 file after the snapshot 2024-04-17T19:51:34.688 INFO:tasks.workunit.client.0.smithi029.stdout: create 253/31 file after the snapshot 2024-04-17T19:51:34.689 INFO:tasks.workunit.client.0.smithi029.stdout: create 253/32 file after the snapshot 2024-04-17T19:51:34.690 INFO:tasks.workunit.client.0.smithi029.stdout: create 253/33 file after the snapshot 2024-04-17T19:51:34.691 INFO:tasks.workunit.client.0.smithi029.stdout: create 253/34 file after the snapshot 2024-04-17T19:51:34.692 INFO:tasks.workunit.client.0.smithi029.stdout: create 253/35 file after the snapshot 2024-04-17T19:51:34.693 INFO:tasks.workunit.client.0.smithi029.stdout: create 253/36 file after the snapshot 2024-04-17T19:51:34.693 INFO:tasks.workunit.client.0.smithi029.stdout: create 253/37 file after the snapshot 2024-04-17T19:51:34.695 INFO:tasks.workunit.client.0.smithi029.stdout: create 253/38 file after the snapshot 2024-04-17T19:51:34.696 INFO:tasks.workunit.client.0.smithi029.stdout: create 253/39 file after the snapshot 2024-04-17T19:51:34.697 INFO:tasks.workunit.client.0.smithi029.stdout: create 254/.snap/snap-subdir-test 2024-04-17T19:51:34.715 INFO:tasks.workunit.client.0.smithi029.stdout: create 254/30 file after the snapshot 2024-04-17T19:51:34.716 INFO:tasks.workunit.client.0.smithi029.stdout: create 254/31 file after the snapshot 2024-04-17T19:51:34.717 INFO:tasks.workunit.client.0.smithi029.stdout: create 254/32 file after the snapshot 2024-04-17T19:51:34.718 INFO:tasks.workunit.client.0.smithi029.stdout: create 254/33 file after the snapshot 2024-04-17T19:51:34.719 INFO:tasks.workunit.client.0.smithi029.stdout: create 254/34 file after the snapshot 2024-04-17T19:51:34.719 INFO:tasks.workunit.client.0.smithi029.stdout: create 254/35 file after the snapshot 2024-04-17T19:51:34.720 INFO:tasks.workunit.client.0.smithi029.stdout: create 254/36 file after the snapshot 2024-04-17T19:51:34.721 INFO:tasks.workunit.client.0.smithi029.stdout: create 254/37 file after the snapshot 2024-04-17T19:51:34.722 INFO:tasks.workunit.client.0.smithi029.stdout: create 254/38 file after the snapshot 2024-04-17T19:51:34.723 INFO:tasks.workunit.client.0.smithi029.stdout: create 254/39 file after the snapshot 2024-04-17T19:51:34.724 INFO:tasks.workunit.client.0.smithi029.stdout: create 255/.snap/snap-subdir-test 2024-04-17T19:51:34.743 INFO:tasks.workunit.client.0.smithi029.stdout: create 255/30 file after the snapshot 2024-04-17T19:51:34.744 INFO:tasks.workunit.client.0.smithi029.stdout: create 255/31 file after the snapshot 2024-04-17T19:51:34.745 INFO:tasks.workunit.client.0.smithi029.stdout: create 255/32 file after the snapshot 2024-04-17T19:51:34.746 INFO:tasks.workunit.client.0.smithi029.stdout: create 255/33 file after the snapshot 2024-04-17T19:51:34.747 INFO:tasks.workunit.client.0.smithi029.stdout: create 255/34 file after the snapshot 2024-04-17T19:51:34.748 INFO:tasks.workunit.client.0.smithi029.stdout: create 255/35 file after the snapshot 2024-04-17T19:51:34.748 INFO:tasks.workunit.client.0.smithi029.stdout: create 255/36 file after the snapshot 2024-04-17T19:51:34.749 INFO:tasks.workunit.client.0.smithi029.stdout: create 255/37 file after the snapshot 2024-04-17T19:51:34.750 INFO:tasks.workunit.client.0.smithi029.stdout: create 255/38 file after the snapshot 2024-04-17T19:51:34.751 INFO:tasks.workunit.client.0.smithi029.stdout: create 255/39 file after the snapshot 2024-04-17T19:51:34.752 INFO:tasks.workunit.client.0.smithi029.stdout: create 256/.snap/snap-subdir-test 2024-04-17T19:51:34.771 INFO:tasks.workunit.client.0.smithi029.stdout: create 256/30 file after the snapshot 2024-04-17T19:51:34.772 INFO:tasks.workunit.client.0.smithi029.stdout: create 256/31 file after the snapshot 2024-04-17T19:51:34.773 INFO:tasks.workunit.client.0.smithi029.stdout: create 256/32 file after the snapshot 2024-04-17T19:51:34.774 INFO:tasks.workunit.client.0.smithi029.stdout: create 256/33 file after the snapshot 2024-04-17T19:51:34.775 INFO:tasks.workunit.client.0.smithi029.stdout: create 256/34 file after the snapshot 2024-04-17T19:51:34.775 INFO:tasks.workunit.client.0.smithi029.stdout: create 256/35 file after the snapshot 2024-04-17T19:51:34.776 INFO:tasks.workunit.client.0.smithi029.stdout: create 256/36 file after the snapshot 2024-04-17T19:51:34.777 INFO:tasks.workunit.client.0.smithi029.stdout: create 256/37 file after the snapshot 2024-04-17T19:51:34.778 INFO:tasks.workunit.client.0.smithi029.stdout: create 256/38 file after the snapshot 2024-04-17T19:51:34.779 INFO:tasks.workunit.client.0.smithi029.stdout: create 256/39 file after the snapshot 2024-04-17T19:51:34.780 INFO:tasks.workunit.client.0.smithi029.stdout: create 257/.snap/snap-subdir-test 2024-04-17T19:51:34.799 INFO:tasks.workunit.client.0.smithi029.stdout: create 257/30 file after the snapshot 2024-04-17T19:51:34.799 INFO:tasks.workunit.client.0.smithi029.stdout: create 257/31 file after the snapshot 2024-04-17T19:51:34.800 INFO:tasks.workunit.client.0.smithi029.stdout: create 257/32 file after the snapshot 2024-04-17T19:51:34.801 INFO:tasks.workunit.client.0.smithi029.stdout: create 257/33 file after the snapshot 2024-04-17T19:51:34.802 INFO:tasks.workunit.client.0.smithi029.stdout: create 257/34 file after the snapshot 2024-04-17T19:51:34.803 INFO:tasks.workunit.client.0.smithi029.stdout: create 257/35 file after the snapshot 2024-04-17T19:51:34.804 INFO:tasks.workunit.client.0.smithi029.stdout: create 257/36 file after the snapshot 2024-04-17T19:51:34.805 INFO:tasks.workunit.client.0.smithi029.stdout: create 257/37 file after the snapshot 2024-04-17T19:51:34.806 INFO:tasks.workunit.client.0.smithi029.stdout: create 257/38 file after the snapshot 2024-04-17T19:51:34.807 INFO:tasks.workunit.client.0.smithi029.stdout: create 257/39 file after the snapshot 2024-04-17T19:51:35.213 INFO:tasks.workunit.client.0.smithi029.stdout: create 258/.snap/snap-subdir-test 2024-04-17T19:51:35.230 INFO:tasks.workunit.client.0.smithi029.stdout: create 258/30 file after the snapshot 2024-04-17T19:51:35.231 INFO:tasks.workunit.client.0.smithi029.stdout: create 258/31 file after the snapshot 2024-04-17T19:51:35.232 INFO:tasks.workunit.client.0.smithi029.stdout: create 258/32 file after the snapshot 2024-04-17T19:51:35.233 INFO:tasks.workunit.client.0.smithi029.stdout: create 258/33 file after the snapshot 2024-04-17T19:51:35.234 INFO:tasks.workunit.client.0.smithi029.stdout: create 258/34 file after the snapshot 2024-04-17T19:51:35.235 INFO:tasks.workunit.client.0.smithi029.stdout: create 258/35 file after the snapshot 2024-04-17T19:51:35.237 INFO:tasks.workunit.client.0.smithi029.stdout: create 258/36 file after the snapshot 2024-04-17T19:51:35.238 INFO:tasks.workunit.client.0.smithi029.stdout: create 258/37 file after the snapshot 2024-04-17T19:51:35.240 INFO:tasks.workunit.client.0.smithi029.stdout: create 258/38 file after the snapshot 2024-04-17T19:51:35.241 INFO:tasks.workunit.client.0.smithi029.stdout: create 258/39 file after the snapshot 2024-04-17T19:51:35.242 INFO:tasks.workunit.client.0.smithi029.stdout: create 259/.snap/snap-subdir-test 2024-04-17T19:51:35.265 INFO:tasks.workunit.client.0.smithi029.stdout: create 259/30 file after the snapshot 2024-04-17T19:51:35.266 INFO:tasks.workunit.client.0.smithi029.stdout: create 259/31 file after the snapshot 2024-04-17T19:51:35.267 INFO:tasks.workunit.client.0.smithi029.stdout: create 259/32 file after the snapshot 2024-04-17T19:51:35.268 INFO:tasks.workunit.client.0.smithi029.stdout: create 259/33 file after the snapshot 2024-04-17T19:51:35.269 INFO:tasks.workunit.client.0.smithi029.stdout: create 259/34 file after the snapshot 2024-04-17T19:51:35.270 INFO:tasks.workunit.client.0.smithi029.stdout: create 259/35 file after the snapshot 2024-04-17T19:51:35.271 INFO:tasks.workunit.client.0.smithi029.stdout: create 259/36 file after the snapshot 2024-04-17T19:51:35.271 INFO:tasks.workunit.client.0.smithi029.stdout: create 259/37 file after the snapshot 2024-04-17T19:51:35.272 INFO:tasks.workunit.client.0.smithi029.stdout: create 259/38 file after the snapshot 2024-04-17T19:51:35.273 INFO:tasks.workunit.client.0.smithi029.stdout: create 259/39 file after the snapshot 2024-04-17T19:51:35.274 INFO:tasks.workunit.client.0.smithi029.stdout: create 260/.snap/snap-subdir-test 2024-04-17T19:51:35.294 INFO:tasks.workunit.client.0.smithi029.stdout: create 260/30 file after the snapshot 2024-04-17T19:51:35.296 INFO:tasks.workunit.client.0.smithi029.stdout: create 260/31 file after the snapshot 2024-04-17T19:51:35.297 INFO:tasks.workunit.client.0.smithi029.stdout: create 260/32 file after the snapshot 2024-04-17T19:51:35.299 INFO:tasks.workunit.client.0.smithi029.stdout: create 260/33 file after the snapshot 2024-04-17T19:51:35.300 INFO:tasks.workunit.client.0.smithi029.stdout: create 260/34 file after the snapshot 2024-04-17T19:51:35.301 INFO:tasks.workunit.client.0.smithi029.stdout: create 260/35 file after the snapshot 2024-04-17T19:51:35.303 INFO:tasks.workunit.client.0.smithi029.stdout: create 260/36 file after the snapshot 2024-04-17T19:51:35.305 INFO:tasks.workunit.client.0.smithi029.stdout: create 260/37 file after the snapshot 2024-04-17T19:51:35.306 INFO:tasks.workunit.client.0.smithi029.stdout: create 260/38 file after the snapshot 2024-04-17T19:51:35.308 INFO:tasks.workunit.client.0.smithi029.stdout: create 260/39 file after the snapshot 2024-04-17T19:51:35.309 INFO:tasks.workunit.client.0.smithi029.stdout: create 261/.snap/snap-subdir-test 2024-04-17T19:51:35.328 INFO:tasks.workunit.client.0.smithi029.stdout: create 261/30 file after the snapshot 2024-04-17T19:51:35.329 INFO:tasks.workunit.client.0.smithi029.stdout: create 261/31 file after the snapshot 2024-04-17T19:51:35.329 INFO:tasks.workunit.client.0.smithi029.stdout: create 261/32 file after the snapshot 2024-04-17T19:51:35.330 INFO:tasks.workunit.client.0.smithi029.stdout: create 261/33 file after the snapshot 2024-04-17T19:51:35.331 INFO:tasks.workunit.client.0.smithi029.stdout: create 261/34 file after the snapshot 2024-04-17T19:51:35.332 INFO:tasks.workunit.client.0.smithi029.stdout: create 261/35 file after the snapshot 2024-04-17T19:51:35.333 INFO:tasks.workunit.client.0.smithi029.stdout: create 261/36 file after the snapshot 2024-04-17T19:51:35.334 INFO:tasks.workunit.client.0.smithi029.stdout: create 261/37 file after the snapshot 2024-04-17T19:51:35.335 INFO:tasks.workunit.client.0.smithi029.stdout: create 261/38 file after the snapshot 2024-04-17T19:51:35.336 INFO:tasks.workunit.client.0.smithi029.stdout: create 261/39 file after the snapshot 2024-04-17T19:51:35.337 INFO:tasks.workunit.client.0.smithi029.stdout: create 262/.snap/snap-subdir-test 2024-04-17T19:51:35.355 INFO:tasks.workunit.client.0.smithi029.stdout: create 262/30 file after the snapshot 2024-04-17T19:51:35.356 INFO:tasks.workunit.client.0.smithi029.stdout: create 262/31 file after the snapshot 2024-04-17T19:51:35.356 INFO:tasks.workunit.client.0.smithi029.stdout: create 262/32 file after the snapshot 2024-04-17T19:51:35.358 INFO:tasks.workunit.client.0.smithi029.stdout: create 262/33 file after the snapshot 2024-04-17T19:51:35.359 INFO:tasks.workunit.client.0.smithi029.stdout: create 262/34 file after the snapshot 2024-04-17T19:51:35.359 INFO:tasks.workunit.client.0.smithi029.stdout: create 262/35 file after the snapshot 2024-04-17T19:51:35.360 INFO:tasks.workunit.client.0.smithi029.stdout: create 262/36 file after the snapshot 2024-04-17T19:51:35.361 INFO:tasks.workunit.client.0.smithi029.stdout: create 262/37 file after the snapshot 2024-04-17T19:51:35.362 INFO:tasks.workunit.client.0.smithi029.stdout: create 262/38 file after the snapshot 2024-04-17T19:51:35.363 INFO:tasks.workunit.client.0.smithi029.stdout: create 262/39 file after the snapshot 2024-04-17T19:51:35.364 INFO:tasks.workunit.client.0.smithi029.stdout: create 263/.snap/snap-subdir-test 2024-04-17T19:51:35.382 INFO:tasks.workunit.client.0.smithi029.stdout: create 263/30 file after the snapshot 2024-04-17T19:51:35.383 INFO:tasks.workunit.client.0.smithi029.stdout: create 263/31 file after the snapshot 2024-04-17T19:51:35.384 INFO:tasks.workunit.client.0.smithi029.stdout: create 263/32 file after the snapshot 2024-04-17T19:51:35.385 INFO:tasks.workunit.client.0.smithi029.stdout: create 263/33 file after the snapshot 2024-04-17T19:51:35.386 INFO:tasks.workunit.client.0.smithi029.stdout: create 263/34 file after the snapshot 2024-04-17T19:51:35.387 INFO:tasks.workunit.client.0.smithi029.stdout: create 263/35 file after the snapshot 2024-04-17T19:51:35.388 INFO:tasks.workunit.client.0.smithi029.stdout: create 263/36 file after the snapshot 2024-04-17T19:51:35.389 INFO:tasks.workunit.client.0.smithi029.stdout: create 263/37 file after the snapshot 2024-04-17T19:51:35.390 INFO:tasks.workunit.client.0.smithi029.stdout: create 263/38 file after the snapshot 2024-04-17T19:51:35.391 INFO:tasks.workunit.client.0.smithi029.stdout: create 263/39 file after the snapshot 2024-04-17T19:51:35.392 INFO:tasks.workunit.client.0.smithi029.stdout: create 264/.snap/snap-subdir-test 2024-04-17T19:51:35.410 INFO:tasks.workunit.client.0.smithi029.stdout: create 264/30 file after the snapshot 2024-04-17T19:51:35.410 INFO:tasks.workunit.client.0.smithi029.stdout: create 264/31 file after the snapshot 2024-04-17T19:51:35.411 INFO:tasks.workunit.client.0.smithi029.stdout: create 264/32 file after the snapshot 2024-04-17T19:51:35.413 INFO:tasks.workunit.client.0.smithi029.stdout: create 264/33 file after the snapshot 2024-04-17T19:51:35.413 INFO:tasks.workunit.client.0.smithi029.stdout: create 264/34 file after the snapshot 2024-04-17T19:51:35.414 INFO:tasks.workunit.client.0.smithi029.stdout: create 264/35 file after the snapshot 2024-04-17T19:51:35.415 INFO:tasks.workunit.client.0.smithi029.stdout: create 264/36 file after the snapshot 2024-04-17T19:51:35.416 INFO:tasks.workunit.client.0.smithi029.stdout: create 264/37 file after the snapshot 2024-04-17T19:51:35.417 INFO:tasks.workunit.client.0.smithi029.stdout: create 264/38 file after the snapshot 2024-04-17T19:51:35.418 INFO:tasks.workunit.client.0.smithi029.stdout: create 264/39 file after the snapshot 2024-04-17T19:51:35.419 INFO:tasks.workunit.client.0.smithi029.stdout: create 265/.snap/snap-subdir-test 2024-04-17T19:51:35.438 INFO:tasks.workunit.client.0.smithi029.stdout: create 265/30 file after the snapshot 2024-04-17T19:51:35.440 INFO:tasks.workunit.client.0.smithi029.stdout: create 265/31 file after the snapshot 2024-04-17T19:51:35.441 INFO:tasks.workunit.client.0.smithi029.stdout: create 265/32 file after the snapshot 2024-04-17T19:51:35.443 INFO:tasks.workunit.client.0.smithi029.stdout: create 265/33 file after the snapshot 2024-04-17T19:51:35.444 INFO:tasks.workunit.client.0.smithi029.stdout: create 265/34 file after the snapshot 2024-04-17T19:51:35.446 INFO:tasks.workunit.client.0.smithi029.stdout: create 265/35 file after the snapshot 2024-04-17T19:51:35.447 INFO:tasks.workunit.client.0.smithi029.stdout: create 265/36 file after the snapshot 2024-04-17T19:51:35.448 INFO:tasks.workunit.client.0.smithi029.stdout: create 265/37 file after the snapshot 2024-04-17T19:51:35.450 INFO:tasks.workunit.client.0.smithi029.stdout: create 265/38 file after the snapshot 2024-04-17T19:51:35.451 INFO:tasks.workunit.client.0.smithi029.stdout: create 265/39 file after the snapshot 2024-04-17T19:51:35.453 INFO:tasks.workunit.client.0.smithi029.stdout: create 266/.snap/snap-subdir-test 2024-04-17T19:51:35.473 INFO:tasks.workunit.client.0.smithi029.stdout: create 266/30 file after the snapshot 2024-04-17T19:51:35.474 INFO:tasks.workunit.client.0.smithi029.stdout: create 266/31 file after the snapshot 2024-04-17T19:51:35.475 INFO:tasks.workunit.client.0.smithi029.stdout: create 266/32 file after the snapshot 2024-04-17T19:51:35.476 INFO:tasks.workunit.client.0.smithi029.stdout: create 266/33 file after the snapshot 2024-04-17T19:51:35.476 INFO:tasks.workunit.client.0.smithi029.stdout: create 266/34 file after the snapshot 2024-04-17T19:51:35.477 INFO:tasks.workunit.client.0.smithi029.stdout: create 266/35 file after the snapshot 2024-04-17T19:51:35.478 INFO:tasks.workunit.client.0.smithi029.stdout: create 266/36 file after the snapshot 2024-04-17T19:51:35.479 INFO:tasks.workunit.client.0.smithi029.stdout: create 266/37 file after the snapshot 2024-04-17T19:51:35.480 INFO:tasks.workunit.client.0.smithi029.stdout: create 266/38 file after the snapshot 2024-04-17T19:51:35.481 INFO:tasks.workunit.client.0.smithi029.stdout: create 266/39 file after the snapshot 2024-04-17T19:51:35.482 INFO:tasks.workunit.client.0.smithi029.stdout: create 267/.snap/snap-subdir-test 2024-04-17T19:51:35.501 INFO:tasks.workunit.client.0.smithi029.stdout: create 267/30 file after the snapshot 2024-04-17T19:51:35.502 INFO:tasks.workunit.client.0.smithi029.stdout: create 267/31 file after the snapshot 2024-04-17T19:51:35.503 INFO:tasks.workunit.client.0.smithi029.stdout: create 267/32 file after the snapshot 2024-04-17T19:51:35.504 INFO:tasks.workunit.client.0.smithi029.stdout: create 267/33 file after the snapshot 2024-04-17T19:51:35.505 INFO:tasks.workunit.client.0.smithi029.stdout: create 267/34 file after the snapshot 2024-04-17T19:51:35.506 INFO:tasks.workunit.client.0.smithi029.stdout: create 267/35 file after the snapshot 2024-04-17T19:51:35.506 INFO:tasks.workunit.client.0.smithi029.stdout: create 267/36 file after the snapshot 2024-04-17T19:51:35.507 INFO:tasks.workunit.client.0.smithi029.stdout: create 267/37 file after the snapshot 2024-04-17T19:51:35.508 INFO:tasks.workunit.client.0.smithi029.stdout: create 267/38 file after the snapshot 2024-04-17T19:51:35.509 INFO:tasks.workunit.client.0.smithi029.stdout: create 267/39 file after the snapshot 2024-04-17T19:51:35.510 INFO:tasks.workunit.client.0.smithi029.stdout: create 268/.snap/snap-subdir-test 2024-04-17T19:51:35.530 INFO:tasks.workunit.client.0.smithi029.stdout: create 268/30 file after the snapshot 2024-04-17T19:51:35.531 INFO:tasks.workunit.client.0.smithi029.stdout: create 268/31 file after the snapshot 2024-04-17T19:51:35.533 INFO:tasks.workunit.client.0.smithi029.stdout: create 268/32 file after the snapshot 2024-04-17T19:51:35.534 INFO:tasks.workunit.client.0.smithi029.stdout: create 268/33 file after the snapshot 2024-04-17T19:51:35.536 INFO:tasks.workunit.client.0.smithi029.stdout: create 268/34 file after the snapshot 2024-04-17T19:51:35.537 INFO:tasks.workunit.client.0.smithi029.stdout: create 268/35 file after the snapshot 2024-04-17T19:51:35.539 INFO:tasks.workunit.client.0.smithi029.stdout: create 268/36 file after the snapshot 2024-04-17T19:51:35.540 INFO:tasks.workunit.client.0.smithi029.stdout: create 268/37 file after the snapshot 2024-04-17T19:51:35.542 INFO:tasks.workunit.client.0.smithi029.stdout: create 268/38 file after the snapshot 2024-04-17T19:51:35.543 INFO:tasks.workunit.client.0.smithi029.stdout: create 268/39 file after the snapshot 2024-04-17T19:51:35.545 INFO:tasks.workunit.client.0.smithi029.stdout: create 269/.snap/snap-subdir-test 2024-04-17T19:51:35.563 INFO:tasks.workunit.client.0.smithi029.stdout: create 269/30 file after the snapshot 2024-04-17T19:51:35.565 INFO:tasks.workunit.client.0.smithi029.stdout: create 269/31 file after the snapshot 2024-04-17T19:51:35.566 INFO:tasks.workunit.client.0.smithi029.stdout: create 269/32 file after the snapshot 2024-04-17T19:51:35.568 INFO:tasks.workunit.client.0.smithi029.stdout: create 269/33 file after the snapshot 2024-04-17T19:51:35.569 INFO:tasks.workunit.client.0.smithi029.stdout: create 269/34 file after the snapshot 2024-04-17T19:51:35.571 INFO:tasks.workunit.client.0.smithi029.stdout: create 269/35 file after the snapshot 2024-04-17T19:51:35.572 INFO:tasks.workunit.client.0.smithi029.stdout: create 269/36 file after the snapshot 2024-04-17T19:51:35.573 INFO:tasks.workunit.client.0.smithi029.stdout: create 269/37 file after the snapshot 2024-04-17T19:51:35.575 INFO:tasks.workunit.client.0.smithi029.stdout: create 269/38 file after the snapshot 2024-04-17T19:51:35.576 INFO:tasks.workunit.client.0.smithi029.stdout: create 269/39 file after the snapshot 2024-04-17T19:51:35.578 INFO:tasks.workunit.client.0.smithi029.stdout: create 270/.snap/snap-subdir-test 2024-04-17T19:51:35.596 INFO:tasks.workunit.client.0.smithi029.stdout: create 270/30 file after the snapshot 2024-04-17T19:51:35.597 INFO:tasks.workunit.client.0.smithi029.stdout: create 270/31 file after the snapshot 2024-04-17T19:51:35.598 INFO:tasks.workunit.client.0.smithi029.stdout: create 270/32 file after the snapshot 2024-04-17T19:51:35.599 INFO:tasks.workunit.client.0.smithi029.stdout: create 270/33 file after the snapshot 2024-04-17T19:51:35.600 INFO:tasks.workunit.client.0.smithi029.stdout: create 270/34 file after the snapshot 2024-04-17T19:51:35.601 INFO:tasks.workunit.client.0.smithi029.stdout: create 270/35 file after the snapshot 2024-04-17T19:51:35.602 INFO:tasks.workunit.client.0.smithi029.stdout: create 270/36 file after the snapshot 2024-04-17T19:51:35.603 INFO:tasks.workunit.client.0.smithi029.stdout: create 270/37 file after the snapshot 2024-04-17T19:51:35.604 INFO:tasks.workunit.client.0.smithi029.stdout: create 270/38 file after the snapshot 2024-04-17T19:51:35.604 INFO:tasks.workunit.client.0.smithi029.stdout: create 270/39 file after the snapshot 2024-04-17T19:51:35.605 INFO:tasks.workunit.client.0.smithi029.stdout: create 271/.snap/snap-subdir-test 2024-04-17T19:51:35.623 INFO:tasks.workunit.client.0.smithi029.stdout: create 271/30 file after the snapshot 2024-04-17T19:51:35.624 INFO:tasks.workunit.client.0.smithi029.stdout: create 271/31 file after the snapshot 2024-04-17T19:51:35.625 INFO:tasks.workunit.client.0.smithi029.stdout: create 271/32 file after the snapshot 2024-04-17T19:51:35.626 INFO:tasks.workunit.client.0.smithi029.stdout: create 271/33 file after the snapshot 2024-04-17T19:51:35.627 INFO:tasks.workunit.client.0.smithi029.stdout: create 271/34 file after the snapshot 2024-04-17T19:51:35.628 INFO:tasks.workunit.client.0.smithi029.stdout: create 271/35 file after the snapshot 2024-04-17T19:51:35.629 INFO:tasks.workunit.client.0.smithi029.stdout: create 271/36 file after the snapshot 2024-04-17T19:51:35.629 INFO:tasks.workunit.client.0.smithi029.stdout: create 271/37 file after the snapshot 2024-04-17T19:51:35.630 INFO:tasks.workunit.client.0.smithi029.stdout: create 271/38 file after the snapshot 2024-04-17T19:51:35.631 INFO:tasks.workunit.client.0.smithi029.stdout: create 271/39 file after the snapshot 2024-04-17T19:51:35.632 INFO:tasks.workunit.client.0.smithi029.stdout: create 272/.snap/snap-subdir-test 2024-04-17T19:51:35.650 INFO:tasks.workunit.client.0.smithi029.stdout: create 272/30 file after the snapshot 2024-04-17T19:51:35.651 INFO:tasks.workunit.client.0.smithi029.stdout: create 272/31 file after the snapshot 2024-04-17T19:51:35.652 INFO:tasks.workunit.client.0.smithi029.stdout: create 272/32 file after the snapshot 2024-04-17T19:51:35.652 INFO:tasks.workunit.client.0.smithi029.stdout: create 272/33 file after the snapshot 2024-04-17T19:51:35.653 INFO:tasks.workunit.client.0.smithi029.stdout: create 272/34 file after the snapshot 2024-04-17T19:51:35.654 INFO:tasks.workunit.client.0.smithi029.stdout: create 272/35 file after the snapshot 2024-04-17T19:51:35.655 INFO:tasks.workunit.client.0.smithi029.stdout: create 272/36 file after the snapshot 2024-04-17T19:51:35.656 INFO:tasks.workunit.client.0.smithi029.stdout: create 272/37 file after the snapshot 2024-04-17T19:51:35.657 INFO:tasks.workunit.client.0.smithi029.stdout: create 272/38 file after the snapshot 2024-04-17T19:51:35.658 INFO:tasks.workunit.client.0.smithi029.stdout: create 272/39 file after the snapshot 2024-04-17T19:51:35.659 INFO:tasks.workunit.client.0.smithi029.stdout: create 273/.snap/snap-subdir-test 2024-04-17T19:51:35.678 INFO:tasks.workunit.client.0.smithi029.stdout: create 273/30 file after the snapshot 2024-04-17T19:51:35.680 INFO:tasks.workunit.client.0.smithi029.stdout: create 273/31 file after the snapshot 2024-04-17T19:51:35.681 INFO:tasks.workunit.client.0.smithi029.stdout: create 273/32 file after the snapshot 2024-04-17T19:51:35.682 INFO:tasks.workunit.client.0.smithi029.stdout: create 273/33 file after the snapshot 2024-04-17T19:51:35.684 INFO:tasks.workunit.client.0.smithi029.stdout: create 273/34 file after the snapshot 2024-04-17T19:51:35.685 INFO:tasks.workunit.client.0.smithi029.stdout: create 273/35 file after the snapshot 2024-04-17T19:51:35.686 INFO:tasks.workunit.client.0.smithi029.stdout: create 273/36 file after the snapshot 2024-04-17T19:51:35.688 INFO:tasks.workunit.client.0.smithi029.stdout: create 273/37 file after the snapshot 2024-04-17T19:51:35.689 INFO:tasks.workunit.client.0.smithi029.stdout: create 273/38 file after the snapshot 2024-04-17T19:51:35.691 INFO:tasks.workunit.client.0.smithi029.stdout: create 273/39 file after the snapshot 2024-04-17T19:51:35.693 INFO:tasks.workunit.client.0.smithi029.stdout: create 274/.snap/snap-subdir-test 2024-04-17T19:51:35.711 INFO:tasks.workunit.client.0.smithi029.stdout: create 274/30 file after the snapshot 2024-04-17T19:51:35.712 INFO:tasks.workunit.client.0.smithi029.stdout: create 274/31 file after the snapshot 2024-04-17T19:51:35.714 INFO:tasks.workunit.client.0.smithi029.stdout: create 274/32 file after the snapshot 2024-04-17T19:51:35.715 INFO:tasks.workunit.client.0.smithi029.stdout: create 274/33 file after the snapshot 2024-04-17T19:51:35.717 INFO:tasks.workunit.client.0.smithi029.stdout: create 274/34 file after the snapshot 2024-04-17T19:51:35.718 INFO:tasks.workunit.client.0.smithi029.stdout: create 274/35 file after the snapshot 2024-04-17T19:51:35.720 INFO:tasks.workunit.client.0.smithi029.stdout: create 274/36 file after the snapshot 2024-04-17T19:51:35.721 INFO:tasks.workunit.client.0.smithi029.stdout: create 274/37 file after the snapshot 2024-04-17T19:51:35.723 INFO:tasks.workunit.client.0.smithi029.stdout: create 274/38 file after the snapshot 2024-04-17T19:51:35.724 INFO:tasks.workunit.client.0.smithi029.stdout: create 274/39 file after the snapshot 2024-04-17T19:51:35.726 INFO:tasks.workunit.client.0.smithi029.stdout: create 275/.snap/snap-subdir-test 2024-04-17T19:51:35.745 INFO:tasks.workunit.client.0.smithi029.stdout: create 275/30 file after the snapshot 2024-04-17T19:51:35.746 INFO:tasks.workunit.client.0.smithi029.stdout: create 275/31 file after the snapshot 2024-04-17T19:51:35.747 INFO:tasks.workunit.client.0.smithi029.stdout: create 275/32 file after the snapshot 2024-04-17T19:51:35.748 INFO:tasks.workunit.client.0.smithi029.stdout: create 275/33 file after the snapshot 2024-04-17T19:51:35.749 INFO:tasks.workunit.client.0.smithi029.stdout: create 275/34 file after the snapshot 2024-04-17T19:51:35.750 INFO:tasks.workunit.client.0.smithi029.stdout: create 275/35 file after the snapshot 2024-04-17T19:51:35.751 INFO:tasks.workunit.client.0.smithi029.stdout: create 275/36 file after the snapshot 2024-04-17T19:51:35.752 INFO:tasks.workunit.client.0.smithi029.stdout: create 275/37 file after the snapshot 2024-04-17T19:51:35.753 INFO:tasks.workunit.client.0.smithi029.stdout: create 275/38 file after the snapshot 2024-04-17T19:51:35.754 INFO:tasks.workunit.client.0.smithi029.stdout: create 275/39 file after the snapshot 2024-04-17T19:51:35.755 INFO:tasks.workunit.client.0.smithi029.stdout: create 276/.snap/snap-subdir-test 2024-04-17T19:51:35.777 INFO:tasks.workunit.client.0.smithi029.stdout: create 276/30 file after the snapshot 2024-04-17T19:51:35.778 INFO:tasks.workunit.client.0.smithi029.stdout: create 276/31 file after the snapshot 2024-04-17T19:51:35.779 INFO:tasks.workunit.client.0.smithi029.stdout: create 276/32 file after the snapshot 2024-04-17T19:51:35.780 INFO:tasks.workunit.client.0.smithi029.stdout: create 276/33 file after the snapshot 2024-04-17T19:51:35.781 INFO:tasks.workunit.client.0.smithi029.stdout: create 276/34 file after the snapshot 2024-04-17T19:51:35.781 INFO:tasks.workunit.client.0.smithi029.stdout: create 276/35 file after the snapshot 2024-04-17T19:51:35.782 INFO:tasks.workunit.client.0.smithi029.stdout: create 276/36 file after the snapshot 2024-04-17T19:51:35.783 INFO:tasks.workunit.client.0.smithi029.stdout: create 276/37 file after the snapshot 2024-04-17T19:51:35.784 INFO:tasks.workunit.client.0.smithi029.stdout: create 276/38 file after the snapshot 2024-04-17T19:51:35.785 INFO:tasks.workunit.client.0.smithi029.stdout: create 276/39 file after the snapshot 2024-04-17T19:51:35.786 INFO:tasks.workunit.client.0.smithi029.stdout: create 277/.snap/snap-subdir-test 2024-04-17T19:51:35.807 INFO:tasks.workunit.client.0.smithi029.stdout: create 277/30 file after the snapshot 2024-04-17T19:51:35.808 INFO:tasks.workunit.client.0.smithi029.stdout: create 277/31 file after the snapshot 2024-04-17T19:51:35.809 INFO:tasks.workunit.client.0.smithi029.stdout: create 277/32 file after the snapshot 2024-04-17T19:51:35.810 INFO:tasks.workunit.client.0.smithi029.stdout: create 277/33 file after the snapshot 2024-04-17T19:51:35.811 INFO:tasks.workunit.client.0.smithi029.stdout: create 277/34 file after the snapshot 2024-04-17T19:51:35.811 INFO:tasks.workunit.client.0.smithi029.stdout: create 277/35 file after the snapshot 2024-04-17T19:51:35.812 INFO:tasks.workunit.client.0.smithi029.stdout: create 277/36 file after the snapshot 2024-04-17T19:51:35.813 INFO:tasks.workunit.client.0.smithi029.stdout: create 277/37 file after the snapshot 2024-04-17T19:51:35.814 INFO:tasks.workunit.client.0.smithi029.stdout: create 277/38 file after the snapshot 2024-04-17T19:51:35.816 INFO:tasks.workunit.client.0.smithi029.stdout: create 277/39 file after the snapshot 2024-04-17T19:51:35.817 INFO:tasks.workunit.client.0.smithi029.stdout: create 278/.snap/snap-subdir-test 2024-04-17T19:51:35.837 INFO:tasks.workunit.client.0.smithi029.stdout: create 278/30 file after the snapshot 2024-04-17T19:51:35.837 INFO:tasks.workunit.client.0.smithi029.stdout: create 278/31 file after the snapshot 2024-04-17T19:51:36.565 INFO:tasks.workunit.client.0.smithi029.stdout: create 278/32 file after the snapshot 2024-04-17T19:51:36.566 INFO:tasks.workunit.client.0.smithi029.stdout: create 278/33 file after the snapshot 2024-04-17T19:51:36.567 INFO:tasks.workunit.client.0.smithi029.stdout: create 278/34 file after the snapshot 2024-04-17T19:51:36.568 INFO:tasks.workunit.client.0.smithi029.stdout: create 278/35 file after the snapshot 2024-04-17T19:51:36.569 INFO:tasks.workunit.client.0.smithi029.stdout: create 278/36 file after the snapshot 2024-04-17T19:51:36.571 INFO:tasks.workunit.client.0.smithi029.stdout: create 278/37 file after the snapshot 2024-04-17T19:51:36.572 INFO:tasks.workunit.client.0.smithi029.stdout: create 278/38 file after the snapshot 2024-04-17T19:51:36.573 INFO:tasks.workunit.client.0.smithi029.stdout: create 278/39 file after the snapshot 2024-04-17T19:51:36.574 INFO:tasks.workunit.client.0.smithi029.stdout: create 279/.snap/snap-subdir-test 2024-04-17T19:51:36.599 INFO:tasks.workunit.client.0.smithi029.stdout: create 279/30 file after the snapshot 2024-04-17T19:51:36.600 INFO:tasks.workunit.client.0.smithi029.stdout: create 279/31 file after the snapshot 2024-04-17T19:51:36.601 INFO:tasks.workunit.client.0.smithi029.stdout: create 279/32 file after the snapshot 2024-04-17T19:51:36.602 INFO:tasks.workunit.client.0.smithi029.stdout: create 279/33 file after the snapshot 2024-04-17T19:51:36.603 INFO:tasks.workunit.client.0.smithi029.stdout: create 279/34 file after the snapshot 2024-04-17T19:51:36.604 INFO:tasks.workunit.client.0.smithi029.stdout: create 279/35 file after the snapshot 2024-04-17T19:51:36.605 INFO:tasks.workunit.client.0.smithi029.stdout: create 279/36 file after the snapshot 2024-04-17T19:51:36.606 INFO:tasks.workunit.client.0.smithi029.stdout: create 279/37 file after the snapshot 2024-04-17T19:51:36.607 INFO:tasks.workunit.client.0.smithi029.stdout: create 279/38 file after the snapshot 2024-04-17T19:51:36.608 INFO:tasks.workunit.client.0.smithi029.stdout: create 279/39 file after the snapshot 2024-04-17T19:51:36.609 INFO:tasks.workunit.client.0.smithi029.stdout: create 280/.snap/snap-subdir-test 2024-04-17T19:51:36.629 INFO:tasks.workunit.client.0.smithi029.stdout: create 280/30 file after the snapshot 2024-04-17T19:51:36.630 INFO:tasks.workunit.client.0.smithi029.stdout: create 280/31 file after the snapshot 2024-04-17T19:51:36.632 INFO:tasks.workunit.client.0.smithi029.stdout: create 280/32 file after the snapshot 2024-04-17T19:51:36.633 INFO:tasks.workunit.client.0.smithi029.stdout: create 280/33 file after the snapshot 2024-04-17T19:51:36.635 INFO:tasks.workunit.client.0.smithi029.stdout: create 280/34 file after the snapshot 2024-04-17T19:51:36.636 INFO:tasks.workunit.client.0.smithi029.stdout: create 280/35 file after the snapshot 2024-04-17T19:51:36.638 INFO:tasks.workunit.client.0.smithi029.stdout: create 280/36 file after the snapshot 2024-04-17T19:51:36.640 INFO:tasks.workunit.client.0.smithi029.stdout: create 280/37 file after the snapshot 2024-04-17T19:51:36.641 INFO:tasks.workunit.client.0.smithi029.stdout: create 280/38 file after the snapshot 2024-04-17T19:51:36.642 INFO:tasks.workunit.client.0.smithi029.stdout: create 280/39 file after the snapshot 2024-04-17T19:51:36.644 INFO:tasks.workunit.client.0.smithi029.stdout: create 281/.snap/snap-subdir-test 2024-04-17T19:51:36.665 INFO:tasks.workunit.client.0.smithi029.stdout: create 281/30 file after the snapshot 2024-04-17T19:51:36.667 INFO:tasks.workunit.client.0.smithi029.stdout: create 281/31 file after the snapshot 2024-04-17T19:51:36.668 INFO:tasks.workunit.client.0.smithi029.stdout: create 281/32 file after the snapshot 2024-04-17T19:51:36.669 INFO:tasks.workunit.client.0.smithi029.stdout: create 281/33 file after the snapshot 2024-04-17T19:51:36.671 INFO:tasks.workunit.client.0.smithi029.stdout: create 281/34 file after the snapshot 2024-04-17T19:51:36.672 INFO:tasks.workunit.client.0.smithi029.stdout: create 281/35 file after the snapshot 2024-04-17T19:51:36.673 INFO:tasks.workunit.client.0.smithi029.stdout: create 281/36 file after the snapshot 2024-04-17T19:51:36.673 INFO:tasks.workunit.client.0.smithi029.stdout: create 281/37 file after the snapshot 2024-04-17T19:51:36.674 INFO:tasks.workunit.client.0.smithi029.stdout: create 281/38 file after the snapshot 2024-04-17T19:51:36.676 INFO:tasks.workunit.client.0.smithi029.stdout: create 281/39 file after the snapshot 2024-04-17T19:51:36.676 INFO:tasks.workunit.client.0.smithi029.stdout: create 282/.snap/snap-subdir-test 2024-04-17T19:51:36.696 INFO:tasks.workunit.client.0.smithi029.stdout: create 282/30 file after the snapshot 2024-04-17T19:51:36.697 INFO:tasks.workunit.client.0.smithi029.stdout: create 282/31 file after the snapshot 2024-04-17T19:51:36.698 INFO:tasks.workunit.client.0.smithi029.stdout: create 282/32 file after the snapshot 2024-04-17T19:51:36.699 INFO:tasks.workunit.client.0.smithi029.stdout: create 282/33 file after the snapshot 2024-04-17T19:51:36.700 INFO:tasks.workunit.client.0.smithi029.stdout: create 282/34 file after the snapshot 2024-04-17T19:51:36.701 INFO:tasks.workunit.client.0.smithi029.stdout: create 282/35 file after the snapshot 2024-04-17T19:51:36.702 INFO:tasks.workunit.client.0.smithi029.stdout: create 282/36 file after the snapshot 2024-04-17T19:51:36.703 INFO:tasks.workunit.client.0.smithi029.stdout: create 282/37 file after the snapshot 2024-04-17T19:51:36.704 INFO:tasks.workunit.client.0.smithi029.stdout: create 282/38 file after the snapshot 2024-04-17T19:51:36.705 INFO:tasks.workunit.client.0.smithi029.stdout: create 282/39 file after the snapshot 2024-04-17T19:51:36.706 INFO:tasks.workunit.client.0.smithi029.stdout: create 283/.snap/snap-subdir-test 2024-04-17T19:51:36.725 INFO:tasks.workunit.client.0.smithi029.stdout: create 283/30 file after the snapshot 2024-04-17T19:51:36.726 INFO:tasks.workunit.client.0.smithi029.stdout: create 283/31 file after the snapshot 2024-04-17T19:51:36.727 INFO:tasks.workunit.client.0.smithi029.stdout: create 283/32 file after the snapshot 2024-04-17T19:51:36.728 INFO:tasks.workunit.client.0.smithi029.stdout: create 283/33 file after the snapshot 2024-04-17T19:51:36.728 INFO:tasks.workunit.client.0.smithi029.stdout: create 283/34 file after the snapshot 2024-04-17T19:51:36.729 INFO:tasks.workunit.client.0.smithi029.stdout: create 283/35 file after the snapshot 2024-04-17T19:51:36.730 INFO:tasks.workunit.client.0.smithi029.stdout: create 283/36 file after the snapshot 2024-04-17T19:51:36.731 INFO:tasks.workunit.client.0.smithi029.stdout: create 283/37 file after the snapshot 2024-04-17T19:51:36.732 INFO:tasks.workunit.client.0.smithi029.stdout: create 283/38 file after the snapshot 2024-04-17T19:51:36.733 INFO:tasks.workunit.client.0.smithi029.stdout: create 283/39 file after the snapshot 2024-04-17T19:51:36.734 INFO:tasks.workunit.client.0.smithi029.stdout: create 284/.snap/snap-subdir-test 2024-04-17T19:51:36.753 INFO:tasks.workunit.client.0.smithi029.stdout: create 284/30 file after the snapshot 2024-04-17T19:51:36.754 INFO:tasks.workunit.client.0.smithi029.stdout: create 284/31 file after the snapshot 2024-04-17T19:51:36.755 INFO:tasks.workunit.client.0.smithi029.stdout: create 284/32 file after the snapshot 2024-04-17T19:51:36.756 INFO:tasks.workunit.client.0.smithi029.stdout: create 284/33 file after the snapshot 2024-04-17T19:51:36.757 INFO:tasks.workunit.client.0.smithi029.stdout: create 284/34 file after the snapshot 2024-04-17T19:51:36.758 INFO:tasks.workunit.client.0.smithi029.stdout: create 284/35 file after the snapshot 2024-04-17T19:51:36.759 INFO:tasks.workunit.client.0.smithi029.stdout: create 284/36 file after the snapshot 2024-04-17T19:51:36.759 INFO:tasks.workunit.client.0.smithi029.stdout: create 284/37 file after the snapshot 2024-04-17T19:51:36.760 INFO:tasks.workunit.client.0.smithi029.stdout: create 284/38 file after the snapshot 2024-04-17T19:51:36.761 INFO:tasks.workunit.client.0.smithi029.stdout: create 284/39 file after the snapshot 2024-04-17T19:51:36.762 INFO:tasks.workunit.client.0.smithi029.stdout: create 285/.snap/snap-subdir-test 2024-04-17T19:51:36.782 INFO:tasks.workunit.client.0.smithi029.stdout: create 285/30 file after the snapshot 2024-04-17T19:51:36.783 INFO:tasks.workunit.client.0.smithi029.stdout: create 285/31 file after the snapshot 2024-04-17T19:51:36.783 INFO:tasks.workunit.client.0.smithi029.stdout: create 285/32 file after the snapshot 2024-04-17T19:51:36.784 INFO:tasks.workunit.client.0.smithi029.stdout: create 285/33 file after the snapshot 2024-04-17T19:51:36.785 INFO:tasks.workunit.client.0.smithi029.stdout: create 285/34 file after the snapshot 2024-04-17T19:51:36.786 INFO:tasks.workunit.client.0.smithi029.stdout: create 285/35 file after the snapshot 2024-04-17T19:51:36.787 INFO:tasks.workunit.client.0.smithi029.stdout: create 285/36 file after the snapshot 2024-04-17T19:51:36.788 INFO:tasks.workunit.client.0.smithi029.stdout: create 285/37 file after the snapshot 2024-04-17T19:51:36.789 INFO:tasks.workunit.client.0.smithi029.stdout: create 285/38 file after the snapshot 2024-04-17T19:51:36.790 INFO:tasks.workunit.client.0.smithi029.stdout: create 285/39 file after the snapshot 2024-04-17T19:51:36.791 INFO:tasks.workunit.client.0.smithi029.stdout: create 286/.snap/snap-subdir-test 2024-04-17T19:51:36.809 INFO:tasks.workunit.client.0.smithi029.stdout: create 286/30 file after the snapshot 2024-04-17T19:51:36.810 INFO:tasks.workunit.client.0.smithi029.stdout: create 286/31 file after the snapshot 2024-04-17T19:51:36.811 INFO:tasks.workunit.client.0.smithi029.stdout: create 286/32 file after the snapshot 2024-04-17T19:51:36.812 INFO:tasks.workunit.client.0.smithi029.stdout: create 286/33 file after the snapshot 2024-04-17T19:51:36.813 INFO:tasks.workunit.client.0.smithi029.stdout: create 286/34 file after the snapshot 2024-04-17T19:51:36.814 INFO:tasks.workunit.client.0.smithi029.stdout: create 286/35 file after the snapshot 2024-04-17T19:51:36.815 INFO:tasks.workunit.client.0.smithi029.stdout: create 286/36 file after the snapshot 2024-04-17T19:51:36.816 INFO:tasks.workunit.client.0.smithi029.stdout: create 286/37 file after the snapshot 2024-04-17T19:51:36.817 INFO:tasks.workunit.client.0.smithi029.stdout: create 286/38 file after the snapshot 2024-04-17T19:51:36.818 INFO:tasks.workunit.client.0.smithi029.stdout: create 286/39 file after the snapshot 2024-04-17T19:51:36.819 INFO:tasks.workunit.client.0.smithi029.stdout: create 287/.snap/snap-subdir-test 2024-04-17T19:51:36.837 INFO:tasks.workunit.client.0.smithi029.stdout: create 287/30 file after the snapshot 2024-04-17T19:51:36.838 INFO:tasks.workunit.client.0.smithi029.stdout: create 287/31 file after the snapshot 2024-04-17T19:51:36.839 INFO:tasks.workunit.client.0.smithi029.stdout: create 287/32 file after the snapshot 2024-04-17T19:51:36.840 INFO:tasks.workunit.client.0.smithi029.stdout: create 287/33 file after the snapshot 2024-04-17T19:51:36.841 INFO:tasks.workunit.client.0.smithi029.stdout: create 287/34 file after the snapshot 2024-04-17T19:51:36.842 INFO:tasks.workunit.client.0.smithi029.stdout: create 287/35 file after the snapshot 2024-04-17T19:51:36.843 INFO:tasks.workunit.client.0.smithi029.stdout: create 287/36 file after the snapshot 2024-04-17T19:51:36.843 INFO:tasks.workunit.client.0.smithi029.stdout: create 287/37 file after the snapshot 2024-04-17T19:51:36.845 INFO:tasks.workunit.client.0.smithi029.stdout: create 287/38 file after the snapshot 2024-04-17T19:51:36.845 INFO:tasks.workunit.client.0.smithi029.stdout: create 287/39 file after the snapshot 2024-04-17T19:51:36.846 INFO:tasks.workunit.client.0.smithi029.stdout: create 288/.snap/snap-subdir-test 2024-04-17T19:51:36.866 INFO:tasks.workunit.client.0.smithi029.stdout: create 288/30 file after the snapshot 2024-04-17T19:51:36.867 INFO:tasks.workunit.client.0.smithi029.stdout: create 288/31 file after the snapshot 2024-04-17T19:51:36.868 INFO:tasks.workunit.client.0.smithi029.stdout: create 288/32 file after the snapshot 2024-04-17T19:51:36.869 INFO:tasks.workunit.client.0.smithi029.stdout: create 288/33 file after the snapshot 2024-04-17T19:51:36.870 INFO:tasks.workunit.client.0.smithi029.stdout: create 288/34 file after the snapshot 2024-04-17T19:51:36.870 INFO:tasks.workunit.client.0.smithi029.stdout: create 288/35 file after the snapshot 2024-04-17T19:51:36.871 INFO:tasks.workunit.client.0.smithi029.stdout: create 288/36 file after the snapshot 2024-04-17T19:51:36.872 INFO:tasks.workunit.client.0.smithi029.stdout: create 288/37 file after the snapshot 2024-04-17T19:51:36.873 INFO:tasks.workunit.client.0.smithi029.stdout: create 288/38 file after the snapshot 2024-04-17T19:51:36.874 INFO:tasks.workunit.client.0.smithi029.stdout: create 288/39 file after the snapshot 2024-04-17T19:51:36.875 INFO:tasks.workunit.client.0.smithi029.stdout: create 289/.snap/snap-subdir-test 2024-04-17T19:51:36.895 INFO:tasks.workunit.client.0.smithi029.stdout: create 289/30 file after the snapshot 2024-04-17T19:51:36.896 INFO:tasks.workunit.client.0.smithi029.stdout: create 289/31 file after the snapshot 2024-04-17T19:51:36.897 INFO:tasks.workunit.client.0.smithi029.stdout: create 289/32 file after the snapshot 2024-04-17T19:51:36.897 INFO:tasks.workunit.client.0.smithi029.stdout: create 289/33 file after the snapshot 2024-04-17T19:51:36.898 INFO:tasks.workunit.client.0.smithi029.stdout: create 289/34 file after the snapshot 2024-04-17T19:51:36.899 INFO:tasks.workunit.client.0.smithi029.stdout: create 289/35 file after the snapshot 2024-04-17T19:51:36.900 INFO:tasks.workunit.client.0.smithi029.stdout: create 289/36 file after the snapshot 2024-04-17T19:51:36.901 INFO:tasks.workunit.client.0.smithi029.stdout: create 289/37 file after the snapshot 2024-04-17T19:51:36.902 INFO:tasks.workunit.client.0.smithi029.stdout: create 289/38 file after the snapshot 2024-04-17T19:51:36.903 INFO:tasks.workunit.client.0.smithi029.stdout: create 289/39 file after the snapshot 2024-04-17T19:51:36.904 INFO:tasks.workunit.client.0.smithi029.stdout: create 290/.snap/snap-subdir-test 2024-04-17T19:51:36.922 INFO:tasks.workunit.client.0.smithi029.stdout: create 290/30 file after the snapshot 2024-04-17T19:51:36.923 INFO:tasks.workunit.client.0.smithi029.stdout: create 290/31 file after the snapshot 2024-04-17T19:51:36.924 INFO:tasks.workunit.client.0.smithi029.stdout: create 290/32 file after the snapshot 2024-04-17T19:51:36.925 INFO:tasks.workunit.client.0.smithi029.stdout: create 290/33 file after the snapshot 2024-04-17T19:51:36.926 INFO:tasks.workunit.client.0.smithi029.stdout: create 290/34 file after the snapshot 2024-04-17T19:51:36.927 INFO:tasks.workunit.client.0.smithi029.stdout: create 290/35 file after the snapshot 2024-04-17T19:51:36.928 INFO:tasks.workunit.client.0.smithi029.stdout: create 290/36 file after the snapshot 2024-04-17T19:51:36.928 INFO:tasks.workunit.client.0.smithi029.stdout: create 290/37 file after the snapshot 2024-04-17T19:51:36.929 INFO:tasks.workunit.client.0.smithi029.stdout: create 290/38 file after the snapshot 2024-04-17T19:51:36.930 INFO:tasks.workunit.client.0.smithi029.stdout: create 290/39 file after the snapshot 2024-04-17T19:51:36.933 INFO:tasks.workunit.client.0.smithi029.stdout: create 291/.snap/snap-subdir-test 2024-04-17T19:51:36.951 INFO:tasks.workunit.client.0.smithi029.stdout: create 291/30 file after the snapshot 2024-04-17T19:51:36.953 INFO:tasks.workunit.client.0.smithi029.stdout: create 291/31 file after the snapshot 2024-04-17T19:51:36.953 INFO:tasks.workunit.client.0.smithi029.stdout: create 291/32 file after the snapshot 2024-04-17T19:51:36.954 INFO:tasks.workunit.client.0.smithi029.stdout: create 291/33 file after the snapshot 2024-04-17T19:51:36.955 INFO:tasks.workunit.client.0.smithi029.stdout: create 291/34 file after the snapshot 2024-04-17T19:51:36.956 INFO:tasks.workunit.client.0.smithi029.stdout: create 291/35 file after the snapshot 2024-04-17T19:51:36.957 INFO:tasks.workunit.client.0.smithi029.stdout: create 291/36 file after the snapshot 2024-04-17T19:51:36.958 INFO:tasks.workunit.client.0.smithi029.stdout: create 291/37 file after the snapshot 2024-04-17T19:51:36.959 INFO:tasks.workunit.client.0.smithi029.stdout: create 291/38 file after the snapshot 2024-04-17T19:51:36.960 INFO:tasks.workunit.client.0.smithi029.stdout: create 291/39 file after the snapshot 2024-04-17T19:51:36.961 INFO:tasks.workunit.client.0.smithi029.stdout: create 292/.snap/snap-subdir-test 2024-04-17T19:51:36.981 INFO:tasks.workunit.client.0.smithi029.stdout: create 292/30 file after the snapshot 2024-04-17T19:51:37.067 INFO:tasks.workunit.client.0.smithi029.stdout: create 292/31 file after the snapshot 2024-04-17T19:51:37.068 INFO:tasks.workunit.client.0.smithi029.stdout: create 292/32 file after the snapshot 2024-04-17T19:51:37.069 INFO:tasks.workunit.client.0.smithi029.stdout: create 292/33 file after the snapshot 2024-04-17T19:51:37.069 INFO:tasks.workunit.client.0.smithi029.stdout: create 292/34 file after the snapshot 2024-04-17T19:51:37.071 INFO:tasks.workunit.client.0.smithi029.stdout: create 292/35 file after the snapshot 2024-04-17T19:51:37.071 INFO:tasks.workunit.client.0.smithi029.stdout: create 292/36 file after the snapshot 2024-04-17T19:51:37.072 INFO:tasks.workunit.client.0.smithi029.stdout: create 292/37 file after the snapshot 2024-04-17T19:51:37.073 INFO:tasks.workunit.client.0.smithi029.stdout: create 292/38 file after the snapshot 2024-04-17T19:51:37.074 INFO:tasks.workunit.client.0.smithi029.stdout: create 292/39 file after the snapshot 2024-04-17T19:51:37.075 INFO:tasks.workunit.client.0.smithi029.stdout: create 293/.snap/snap-subdir-test 2024-04-17T19:51:37.094 INFO:tasks.workunit.client.0.smithi029.stdout: create 293/30 file after the snapshot 2024-04-17T19:51:37.095 INFO:tasks.workunit.client.0.smithi029.stdout: create 293/31 file after the snapshot 2024-04-17T19:51:37.096 INFO:tasks.workunit.client.0.smithi029.stdout: create 293/32 file after the snapshot 2024-04-17T19:51:37.097 INFO:tasks.workunit.client.0.smithi029.stdout: create 293/33 file after the snapshot 2024-04-17T19:51:37.098 INFO:tasks.workunit.client.0.smithi029.stdout: create 293/34 file after the snapshot 2024-04-17T19:51:37.099 INFO:tasks.workunit.client.0.smithi029.stdout: create 293/35 file after the snapshot 2024-04-17T19:51:37.100 INFO:tasks.workunit.client.0.smithi029.stdout: create 293/36 file after the snapshot 2024-04-17T19:51:37.100 INFO:tasks.workunit.client.0.smithi029.stdout: create 293/37 file after the snapshot 2024-04-17T19:51:37.101 INFO:tasks.workunit.client.0.smithi029.stdout: create 293/38 file after the snapshot 2024-04-17T19:51:37.102 INFO:tasks.workunit.client.0.smithi029.stdout: create 293/39 file after the snapshot 2024-04-17T19:51:37.103 INFO:tasks.workunit.client.0.smithi029.stdout: create 294/.snap/snap-subdir-test 2024-04-17T19:51:37.123 INFO:tasks.workunit.client.0.smithi029.stdout: create 294/30 file after the snapshot 2024-04-17T19:51:37.124 INFO:tasks.workunit.client.0.smithi029.stdout: create 294/31 file after the snapshot 2024-04-17T19:51:37.124 INFO:tasks.workunit.client.0.smithi029.stdout: create 294/32 file after the snapshot 2024-04-17T19:51:37.125 INFO:tasks.workunit.client.0.smithi029.stdout: create 294/33 file after the snapshot 2024-04-17T19:51:37.126 INFO:tasks.workunit.client.0.smithi029.stdout: create 294/34 file after the snapshot 2024-04-17T19:51:37.127 INFO:tasks.workunit.client.0.smithi029.stdout: create 294/35 file after the snapshot 2024-04-17T19:51:37.128 INFO:tasks.workunit.client.0.smithi029.stdout: create 294/36 file after the snapshot 2024-04-17T19:51:37.129 INFO:tasks.workunit.client.0.smithi029.stdout: create 294/37 file after the snapshot 2024-04-17T19:51:37.130 INFO:tasks.workunit.client.0.smithi029.stdout: create 294/38 file after the snapshot 2024-04-17T19:51:37.383 INFO:tasks.workunit.client.0.smithi029.stdout: create 294/39 file after the snapshot 2024-04-17T19:51:37.384 INFO:tasks.workunit.client.0.smithi029.stdout: create 295/.snap/snap-subdir-test 2024-04-17T19:51:37.405 INFO:tasks.workunit.client.0.smithi029.stdout: create 295/30 file after the snapshot 2024-04-17T19:51:37.406 INFO:tasks.workunit.client.0.smithi029.stdout: create 295/31 file after the snapshot 2024-04-17T19:51:37.407 INFO:tasks.workunit.client.0.smithi029.stdout: create 295/32 file after the snapshot 2024-04-17T19:51:37.407 INFO:tasks.workunit.client.0.smithi029.stdout: create 295/33 file after the snapshot 2024-04-17T19:51:37.408 INFO:tasks.workunit.client.0.smithi029.stdout: create 295/34 file after the snapshot 2024-04-17T19:51:37.410 INFO:tasks.workunit.client.0.smithi029.stdout: create 295/35 file after the snapshot 2024-04-17T19:51:37.411 INFO:tasks.workunit.client.0.smithi029.stdout: create 295/36 file after the snapshot 2024-04-17T19:51:37.411 INFO:tasks.workunit.client.0.smithi029.stdout: create 295/37 file after the snapshot 2024-04-17T19:51:37.412 INFO:tasks.workunit.client.0.smithi029.stdout: create 295/38 file after the snapshot 2024-04-17T19:51:37.413 INFO:tasks.workunit.client.0.smithi029.stdout: create 295/39 file after the snapshot 2024-04-17T19:51:37.414 INFO:tasks.workunit.client.0.smithi029.stdout: create 296/.snap/snap-subdir-test 2024-04-17T19:51:37.435 INFO:tasks.workunit.client.0.smithi029.stdout: create 296/30 file after the snapshot 2024-04-17T19:51:37.436 INFO:tasks.workunit.client.0.smithi029.stdout: create 296/31 file after the snapshot 2024-04-17T19:51:37.438 INFO:tasks.workunit.client.0.smithi029.stdout: create 296/32 file after the snapshot 2024-04-17T19:51:37.439 INFO:tasks.workunit.client.0.smithi029.stdout: create 296/33 file after the snapshot 2024-04-17T19:51:37.441 INFO:tasks.workunit.client.0.smithi029.stdout: create 296/34 file after the snapshot 2024-04-17T19:51:37.442 INFO:tasks.workunit.client.0.smithi029.stdout: create 296/35 file after the snapshot 2024-04-17T19:51:37.444 INFO:tasks.workunit.client.0.smithi029.stdout: create 296/36 file after the snapshot 2024-04-17T19:51:37.445 INFO:tasks.workunit.client.0.smithi029.stdout: create 296/37 file after the snapshot 2024-04-17T19:51:37.447 INFO:tasks.workunit.client.0.smithi029.stdout: create 296/38 file after the snapshot 2024-04-17T19:51:37.448 INFO:tasks.workunit.client.0.smithi029.stdout: create 296/39 file after the snapshot 2024-04-17T19:51:37.450 INFO:tasks.workunit.client.0.smithi029.stdout: create 297/.snap/snap-subdir-test 2024-04-17T19:51:37.470 INFO:tasks.workunit.client.0.smithi029.stdout: create 297/30 file after the snapshot 2024-04-17T19:51:37.470 INFO:tasks.workunit.client.0.smithi029.stdout: create 297/31 file after the snapshot 2024-04-17T19:51:37.471 INFO:tasks.workunit.client.0.smithi029.stdout: create 297/32 file after the snapshot 2024-04-17T19:51:37.472 INFO:tasks.workunit.client.0.smithi029.stdout: create 297/33 file after the snapshot 2024-04-17T19:51:37.473 INFO:tasks.workunit.client.0.smithi029.stdout: create 297/34 file after the snapshot 2024-04-17T19:51:37.474 INFO:tasks.workunit.client.0.smithi029.stdout: create 297/35 file after the snapshot 2024-04-17T19:51:37.475 INFO:tasks.workunit.client.0.smithi029.stdout: create 297/36 file after the snapshot 2024-04-17T19:51:37.476 INFO:tasks.workunit.client.0.smithi029.stdout: create 297/37 file after the snapshot 2024-04-17T19:51:37.477 INFO:tasks.workunit.client.0.smithi029.stdout: create 297/38 file after the snapshot 2024-04-17T19:51:37.478 INFO:tasks.workunit.client.0.smithi029.stdout: create 297/39 file after the snapshot 2024-04-17T19:51:37.479 INFO:tasks.workunit.client.0.smithi029.stdout: create 298/.snap/snap-subdir-test 2024-04-17T19:51:37.500 INFO:tasks.workunit.client.0.smithi029.stdout: create 298/30 file after the snapshot 2024-04-17T19:51:37.501 INFO:tasks.workunit.client.0.smithi029.stdout: create 298/31 file after the snapshot 2024-04-17T19:51:37.502 INFO:tasks.workunit.client.0.smithi029.stdout: create 298/32 file after the snapshot 2024-04-17T19:51:37.503 INFO:tasks.workunit.client.0.smithi029.stdout: create 298/33 file after the snapshot 2024-04-17T19:51:37.504 INFO:tasks.workunit.client.0.smithi029.stdout: create 298/34 file after the snapshot 2024-04-17T19:51:37.505 INFO:tasks.workunit.client.0.smithi029.stdout: create 298/35 file after the snapshot 2024-04-17T19:51:37.505 INFO:tasks.workunit.client.0.smithi029.stdout: create 298/36 file after the snapshot 2024-04-17T19:51:37.507 INFO:tasks.workunit.client.0.smithi029.stdout: create 298/37 file after the snapshot 2024-04-17T19:51:37.507 INFO:tasks.workunit.client.0.smithi029.stdout: create 298/38 file after the snapshot 2024-04-17T19:51:37.508 INFO:tasks.workunit.client.0.smithi029.stdout: create 298/39 file after the snapshot 2024-04-17T19:51:37.509 INFO:tasks.workunit.client.0.smithi029.stdout: create 299/.snap/snap-subdir-test 2024-04-17T19:51:37.532 INFO:tasks.workunit.client.0.smithi029.stdout: create 299/30 file after the snapshot 2024-04-17T19:51:37.533 INFO:tasks.workunit.client.0.smithi029.stdout: create 299/31 file after the snapshot 2024-04-17T19:51:37.533 INFO:tasks.workunit.client.0.smithi029.stdout: create 299/32 file after the snapshot 2024-04-17T19:51:37.534 INFO:tasks.workunit.client.0.smithi029.stdout: create 299/33 file after the snapshot 2024-04-17T19:51:37.535 INFO:tasks.workunit.client.0.smithi029.stdout: create 299/34 file after the snapshot 2024-04-17T19:51:37.536 INFO:tasks.workunit.client.0.smithi029.stdout: create 299/35 file after the snapshot 2024-04-17T19:51:37.537 INFO:tasks.workunit.client.0.smithi029.stdout: create 299/36 file after the snapshot 2024-04-17T19:51:37.538 INFO:tasks.workunit.client.0.smithi029.stdout: create 299/37 file after the snapshot 2024-04-17T19:51:37.539 INFO:tasks.workunit.client.0.smithi029.stdout: create 299/38 file after the snapshot 2024-04-17T19:51:37.540 INFO:tasks.workunit.client.0.smithi029.stdout: create 299/39 file after the snapshot 2024-04-17T19:51:37.541 INFO:tasks.workunit.client.0.smithi029.stdout:Take second snapshot .snap/test2 2024-04-17T19:51:37.563 INFO:tasks.workunit.client.0.smithi029.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2024-04-17T19:51:40.248 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2024-04-17T19:51:40.248 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2024-04-17T19:51:40.248 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2024-04-17T19:51:40.248 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2024-04-17T19:51:40.248 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2024-04-17T19:51:40.248 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2024-04-17T19:51:40.248 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2024-04-17T19:51:40.248 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2024-04-17T19:51:40.248 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2024-04-17T19:51:40.249 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2024-04-17T19:51:40.249 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2024-04-17T19:51:40.249 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2024-04-17T19:51:40.249 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2024-04-17T19:51:40.249 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2024-04-17T19:51:40.249 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2024-04-17T19:51:40.249 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2024-04-17T19:51:40.249 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2024-04-17T19:51:40.249 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2024-04-17T19:51:40.249 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2024-04-17T19:51:40.249 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2024-04-17T19:51:40.249 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2024-04-17T19:51:40.249 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2024-04-17T19:51:40.249 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2024-04-17T19:51:40.250 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2024-04-17T19:51:40.250 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2024-04-17T19:51:40.250 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2024-04-17T19:51:40.250 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2024-04-17T19:51:40.250 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2024-04-17T19:51:40.250 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2024-04-17T19:51:40.250 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2024-04-17T19:51:40.250 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2024-04-17T19:51:40.250 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2024-04-17T19:51:40.250 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2024-04-17T19:51:40.250 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2024-04-17T19:51:40.250 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2024-04-17T19:51:40.250 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2024-04-17T19:51:40.251 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2024-04-17T19:51:40.251 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2024-04-17T19:51:40.251 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2024-04-17T19:51:40.251 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2024-04-17T19:51:40.251 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2024-04-17T19:51:40.251 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2024-04-17T19:51:40.251 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2024-04-17T19:51:40.251 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2024-04-17T19:51:40.251 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2024-04-17T19:51:40.251 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2024-04-17T19:51:40.251 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2024-04-17T19:51:40.251 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2024-04-17T19:51:40.251 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2024-04-17T19:51:40.251 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2024-04-17T19:51:40.251 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2024-04-17T19:51:40.252 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2024-04-17T19:51:40.252 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2024-04-17T19:51:40.252 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2024-04-17T19:51:40.252 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2024-04-17T19:51:40.252 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2024-04-17T19:51:40.252 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2024-04-17T19:51:40.252 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2024-04-17T19:51:40.252 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2024-04-17T19:51:40.252 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2024-04-17T19:51:40.252 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2024-04-17T19:51:40.252 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2024-04-17T19:51:40.252 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2024-04-17T19:51:40.252 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2024-04-17T19:51:40.252 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2024-04-17T19:51:40.252 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2024-04-17T19:51:40.252 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2024-04-17T19:51:40.253 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2024-04-17T19:51:40.253 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2024-04-17T19:51:40.253 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2024-04-17T19:51:40.253 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2024-04-17T19:51:40.253 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2024-04-17T19:51:40.253 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2024-04-17T19:51:40.253 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2024-04-17T19:51:40.253 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2024-04-17T19:51:40.253 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2024-04-17T19:51:40.253 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2024-04-17T19:51:40.253 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2024-04-17T19:51:40.253 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2024-04-17T19:51:40.253 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2024-04-17T19:51:40.253 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2024-04-17T19:51:42.605 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2024-04-17T19:51:42.605 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2024-04-17T19:51:42.605 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2024-04-17T19:51:42.605 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2024-04-17T19:51:42.605 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2024-04-17T19:51:42.605 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2024-04-17T19:51:42.605 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2024-04-17T19:51:42.606 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2024-04-17T19:51:42.606 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2024-04-17T19:51:42.606 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2024-04-17T19:51:42.606 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2024-04-17T19:51:42.606 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2024-04-17T19:51:42.606 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2024-04-17T19:51:42.606 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2024-04-17T19:51:42.606 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2024-04-17T19:51:42.606 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2024-04-17T19:51:42.606 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2024-04-17T19:51:42.606 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2024-04-17T19:51:42.606 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2024-04-17T19:51:42.606 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2024-04-17T19:51:42.606 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2024-04-17T19:51:42.606 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2024-04-17T19:51:42.607 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2024-04-17T19:51:42.607 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2024-04-17T19:51:42.607 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2024-04-17T19:51:42.607 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2024-04-17T19:51:42.607 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2024-04-17T19:51:42.607 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2024-04-17T19:51:42.607 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2024-04-17T19:51:42.607 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2024-04-17T19:51:42.607 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2024-04-17T19:51:42.607 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2024-04-17T19:51:42.607 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2024-04-17T19:51:42.607 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2024-04-17T19:51:42.607 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2024-04-17T19:51:42.607 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2024-04-17T19:51:42.607 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2024-04-17T19:51:42.607 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2024-04-17T19:51:42.608 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2024-04-17T19:51:42.608 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2024-04-17T19:51:42.608 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2024-04-17T19:51:42.608 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2024-04-17T19:51:42.608 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2024-04-17T19:51:42.608 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2024-04-17T19:51:42.608 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2024-04-17T19:51:42.608 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2024-04-17T19:51:42.608 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2024-04-17T19:51:42.608 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2024-04-17T19:51:42.608 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2024-04-17T19:51:42.608 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2024-04-17T19:51:42.608 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2024-04-17T19:51:42.608 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2024-04-17T19:51:42.608 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2024-04-17T19:51:42.608 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2024-04-17T19:51:42.608 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2024-04-17T19:51:42.609 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2024-04-17T19:51:42.609 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2024-04-17T19:51:42.609 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2024-04-17T19:51:42.609 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2024-04-17T19:51:42.609 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2024-04-17T19:51:42.609 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2024-04-17T19:51:42.609 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2024-04-17T19:51:42.609 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2024-04-17T19:51:42.609 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2024-04-17T19:51:42.609 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2024-04-17T19:51:42.609 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2024-04-17T19:51:42.609 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2024-04-17T19:51:42.609 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2024-04-17T19:51:42.609 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2024-04-17T19:51:42.609 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2024-04-17T19:51:42.609 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2024-04-17T19:51:42.610 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2024-04-17T19:51:42.610 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2024-04-17T19:51:42.610 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2024-04-17T19:51:42.610 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2024-04-17T19:51:42.610 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2024-04-17T19:51:42.610 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2024-04-17T19:51:42.610 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2024-04-17T19:51:42.610 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2024-04-17T19:51:42.610 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2024-04-17T19:51:42.610 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2024-04-17T19:51:43.100 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2024-04-17T19:51:43.100 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2024-04-17T19:51:43.100 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2024-04-17T19:51:43.100 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2024-04-17T19:51:43.100 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2024-04-17T19:51:43.101 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2024-04-17T19:51:43.101 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2024-04-17T19:51:43.101 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2024-04-17T19:51:43.101 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2024-04-17T19:51:43.101 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2024-04-17T19:51:43.101 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2024-04-17T19:51:43.101 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2024-04-17T19:51:43.101 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2024-04-17T19:51:43.101 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2024-04-17T19:51:43.101 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2024-04-17T19:51:43.101 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2024-04-17T19:51:43.101 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2024-04-17T19:51:43.101 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2024-04-17T19:51:43.101 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2024-04-17T19:51:43.102 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2024-04-17T19:51:43.102 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2024-04-17T19:51:43.102 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2024-04-17T19:51:43.102 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2024-04-17T19:51:43.102 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2024-04-17T19:51:43.102 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2024-04-17T19:51:43.102 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2024-04-17T19:51:43.102 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2024-04-17T19:51:43.102 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2024-04-17T19:51:43.102 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2024-04-17T19:51:43.102 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2024-04-17T19:51:43.102 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2024-04-17T19:51:43.102 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2024-04-17T19:51:43.102 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2024-04-17T19:51:43.102 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2024-04-17T19:51:43.103 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2024-04-17T19:51:43.103 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2024-04-17T19:51:43.103 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2024-04-17T19:51:43.103 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2024-04-17T19:51:43.103 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2024-04-17T19:51:43.103 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2024-04-17T19:51:43.103 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2024-04-17T19:51:43.103 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2024-04-17T19:51:43.103 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2024-04-17T19:51:43.103 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2024-04-17T19:51:43.103 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2024-04-17T19:51:43.103 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2024-04-17T19:51:43.103 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2024-04-17T19:51:43.103 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2024-04-17T19:51:43.103 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2024-04-17T19:51:43.103 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2024-04-17T19:51:43.104 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2024-04-17T19:51:43.104 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2024-04-17T19:51:43.104 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2024-04-17T19:51:43.104 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2024-04-17T19:51:43.104 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2024-04-17T19:51:43.104 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2024-04-17T19:51:43.104 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2024-04-17T19:51:43.104 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2024-04-17T19:51:43.104 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2024-04-17T19:51:43.104 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2024-04-17T19:51:43.104 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2024-04-17T19:51:43.104 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2024-04-17T19:51:43.104 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2024-04-17T19:51:43.104 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2024-04-17T19:51:43.104 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2024-04-17T19:51:43.104 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2024-04-17T19:51:43.104 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2024-04-17T19:51:43.105 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2024-04-17T19:51:43.105 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2024-04-17T19:51:43.105 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2024-04-17T19:51:43.105 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2024-04-17T19:51:43.105 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2024-04-17T19:51:43.105 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2024-04-17T19:51:43.105 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2024-04-17T19:51:43.105 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2024-04-17T19:51:43.105 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2024-04-17T19:51:43.105 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2024-04-17T19:51:43.105 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2024-04-17T19:51:43.105 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2024-04-17T19:51:43.105 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2024-04-17T19:51:43.105 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2024-04-17T19:51:44.361 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2024-04-17T19:51:44.361 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2024-04-17T19:51:44.361 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2024-04-17T19:51:44.361 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2024-04-17T19:51:44.361 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2024-04-17T19:51:44.361 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2024-04-17T19:51:44.361 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2024-04-17T19:51:44.361 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2024-04-17T19:51:44.361 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2024-04-17T19:51:44.361 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2024-04-17T19:51:44.362 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2024-04-17T19:51:44.362 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2024-04-17T19:51:44.362 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2024-04-17T19:51:44.362 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2024-04-17T19:51:44.362 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2024-04-17T19:51:44.362 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2024-04-17T19:51:44.362 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2024-04-17T19:51:44.362 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2024-04-17T19:51:44.362 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2024-04-17T19:51:44.362 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2024-04-17T19:51:44.362 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2024-04-17T19:51:44.362 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2024-04-17T19:51:44.362 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2024-04-17T19:51:44.362 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2024-04-17T19:51:44.362 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2024-04-17T19:51:44.363 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2024-04-17T19:51:44.363 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2024-04-17T19:51:44.363 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2024-04-17T19:51:44.363 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2024-04-17T19:51:44.363 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2024-04-17T19:51:44.363 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2024-04-17T19:51:44.363 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2024-04-17T19:51:44.363 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2024-04-17T19:51:44.363 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2024-04-17T19:51:44.363 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2024-04-17T19:51:44.363 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2024-04-17T19:51:44.363 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2024-04-17T19:51:44.363 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2024-04-17T19:51:44.363 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2024-04-17T19:51:44.363 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2024-04-17T19:51:44.363 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2024-04-17T19:51:44.363 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2024-04-17T19:51:44.364 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2024-04-17T19:51:44.364 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2024-04-17T19:51:44.364 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2024-04-17T19:51:44.364 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2024-04-17T19:51:44.364 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2024-04-17T19:51:44.364 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2024-04-17T19:51:44.364 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2024-04-17T19:51:44.364 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2024-04-17T19:51:44.364 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2024-04-17T19:51:44.364 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2024-04-17T19:51:44.364 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2024-04-17T19:51:44.364 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2024-04-17T19:51:44.364 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2024-04-17T19:51:44.364 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2024-04-17T19:51:44.364 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2024-04-17T19:51:44.364 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2024-04-17T19:51:44.365 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2024-04-17T19:51:44.365 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2024-04-17T19:51:44.365 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2024-04-17T19:51:44.365 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2024-04-17T19:51:44.365 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2024-04-17T19:51:44.365 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2024-04-17T19:51:44.365 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2024-04-17T19:51:44.365 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2024-04-17T19:51:44.365 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2024-04-17T19:51:44.365 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2024-04-17T19:51:44.365 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2024-04-17T19:51:44.365 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2024-04-17T19:51:44.365 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2024-04-17T19:51:44.365 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2024-04-17T19:51:44.365 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2024-04-17T19:51:44.365 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2024-04-17T19:51:44.366 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2024-04-17T19:51:44.366 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2024-04-17T19:51:44.366 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2024-04-17T19:51:44.366 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2024-04-17T19:51:44.366 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2024-04-17T19:51:44.366 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2024-04-17T19:51:44.366 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2024-04-17T19:51:44.852 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2024-04-17T19:51:44.852 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2024-04-17T19:51:44.852 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2024-04-17T19:51:44.852 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2024-04-17T19:51:44.852 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2024-04-17T19:51:44.852 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2024-04-17T19:51:44.852 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2024-04-17T19:51:44.852 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2024-04-17T19:51:44.852 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2024-04-17T19:51:44.852 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2024-04-17T19:51:44.852 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2024-04-17T19:51:44.852 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2024-04-17T19:51:44.853 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2024-04-17T19:51:44.853 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2024-04-17T19:51:44.853 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2024-04-17T19:51:44.853 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2024-04-17T19:51:44.853 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2024-04-17T19:51:44.853 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2024-04-17T19:51:44.853 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2024-04-17T19:51:44.853 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2024-04-17T19:51:44.853 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2024-04-17T19:51:44.853 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2024-04-17T19:51:44.853 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2024-04-17T19:51:44.853 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2024-04-17T19:51:44.853 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2024-04-17T19:51:44.853 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2024-04-17T19:51:44.853 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2024-04-17T19:51:44.854 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2024-04-17T19:51:44.854 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2024-04-17T19:51:44.854 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2024-04-17T19:51:44.854 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2024-04-17T19:51:44.854 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2024-04-17T19:51:44.854 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2024-04-17T19:51:44.854 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2024-04-17T19:51:44.854 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2024-04-17T19:51:44.854 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2024-04-17T19:51:44.854 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2024-04-17T19:51:44.854 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2024-04-17T19:51:44.854 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2024-04-17T19:51:44.854 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2024-04-17T19:51:44.854 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2024-04-17T19:51:44.855 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2024-04-17T19:51:44.855 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2024-04-17T19:51:44.855 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2024-04-17T19:51:44.855 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2024-04-17T19:51:44.855 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2024-04-17T19:51:44.855 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2024-04-17T19:51:44.855 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2024-04-17T19:51:44.855 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2024-04-17T19:51:44.855 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2024-04-17T19:51:44.855 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2024-04-17T19:51:44.855 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2024-04-17T19:51:44.855 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2024-04-17T19:51:44.855 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2024-04-17T19:51:44.855 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2024-04-17T19:51:44.856 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2024-04-17T19:51:44.856 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2024-04-17T19:51:44.856 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2024-04-17T19:51:44.856 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2024-04-17T19:51:44.856 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2024-04-17T19:51:44.856 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2024-04-17T19:51:44.856 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2024-04-17T19:51:44.856 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2024-04-17T19:51:44.856 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2024-04-17T19:51:44.856 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2024-04-17T19:51:44.856 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2024-04-17T19:51:44.856 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2024-04-17T19:51:44.856 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2024-04-17T19:51:44.856 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2024-04-17T19:51:44.856 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2024-04-17T19:51:44.856 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2024-04-17T19:51:44.857 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2024-04-17T19:51:44.857 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2024-04-17T19:51:44.857 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2024-04-17T19:51:44.857 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2024-04-17T19:51:44.857 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2024-04-17T19:51:44.857 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2024-04-17T19:51:44.857 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2024-04-17T19:51:44.857 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2024-04-17T19:51:44.857 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2024-04-17T19:51:44.857 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2024-04-17T19:51:47.996 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2024-04-17T19:51:47.996 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2024-04-17T19:51:47.996 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2024-04-17T19:51:47.996 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2024-04-17T19:51:47.996 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2024-04-17T19:51:47.997 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2024-04-17T19:51:47.997 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2024-04-17T19:51:47.997 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2024-04-17T19:51:47.997 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2024-04-17T19:51:47.997 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2024-04-17T19:51:47.997 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2024-04-17T19:51:47.997 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2024-04-17T19:51:47.997 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2024-04-17T19:51:47.997 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2024-04-17T19:51:47.997 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2024-04-17T19:51:47.997 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2024-04-17T19:51:47.997 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2024-04-17T19:51:47.997 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2024-04-17T19:51:47.997 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2024-04-17T19:51:47.997 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2024-04-17T19:51:47.997 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2024-04-17T19:51:47.998 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2024-04-17T19:51:47.998 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2024-04-17T19:51:47.998 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2024-04-17T19:51:47.998 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2024-04-17T19:51:47.998 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2024-04-17T19:51:47.998 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2024-04-17T19:51:47.998 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2024-04-17T19:51:47.998 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2024-04-17T19:51:47.998 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2024-04-17T19:51:47.998 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2024-04-17T19:51:47.998 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2024-04-17T19:51:47.998 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2024-04-17T19:51:47.998 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2024-04-17T19:51:47.998 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2024-04-17T19:51:47.999 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2024-04-17T19:51:47.999 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2024-04-17T19:51:47.999 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2024-04-17T19:51:47.999 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2024-04-17T19:51:47.999 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2024-04-17T19:51:47.999 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2024-04-17T19:51:47.999 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2024-04-17T19:51:47.999 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2024-04-17T19:51:47.999 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2024-04-17T19:51:47.999 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2024-04-17T19:51:47.999 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2024-04-17T19:51:47.999 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2024-04-17T19:51:47.999 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2024-04-17T19:51:47.999 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2024-04-17T19:51:47.999 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2024-04-17T19:51:47.999 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2024-04-17T19:51:47.999 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2024-04-17T19:51:48.000 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2024-04-17T19:51:48.000 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2024-04-17T19:51:48.000 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2024-04-17T19:51:48.000 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2024-04-17T19:51:48.000 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2024-04-17T19:51:48.000 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2024-04-17T19:51:48.000 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2024-04-17T19:51:48.000 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2024-04-17T19:51:48.000 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2024-04-17T19:51:48.000 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2024-04-17T19:51:48.000 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2024-04-17T19:51:48.000 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2024-04-17T19:51:48.000 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2024-04-17T19:51:48.000 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2024-04-17T19:51:48.000 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2024-04-17T19:51:48.001 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2024-04-17T19:51:48.001 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2024-04-17T19:51:48.001 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2024-04-17T19:51:48.001 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2024-04-17T19:51:48.001 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2024-04-17T19:51:48.001 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2024-04-17T19:51:48.001 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2024-04-17T19:51:48.001 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2024-04-17T19:51:48.001 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2024-04-17T19:51:48.001 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2024-04-17T19:51:48.001 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2024-04-17T19:51:48.001 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2024-04-17T19:51:48.001 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2024-04-17T19:51:48.001 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2024-04-17T19:51:48.487 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2024-04-17T19:51:48.487 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2024-04-17T19:51:48.487 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2024-04-17T19:51:48.487 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2024-04-17T19:51:48.487 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2024-04-17T19:51:48.487 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2024-04-17T19:51:48.487 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2024-04-17T19:51:48.487 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2024-04-17T19:51:48.487 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2024-04-17T19:51:48.487 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2024-04-17T19:51:48.487 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2024-04-17T19:51:48.488 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2024-04-17T19:51:48.488 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2024-04-17T19:51:48.488 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2024-04-17T19:51:48.488 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2024-04-17T19:51:48.488 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2024-04-17T19:51:48.488 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2024-04-17T19:51:48.488 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2024-04-17T19:51:48.488 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2024-04-17T19:51:48.488 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2024-04-17T19:51:48.488 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2024-04-17T19:51:48.488 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2024-04-17T19:51:48.488 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2024-04-17T19:51:48.488 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2024-04-17T19:51:48.488 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2024-04-17T19:51:48.488 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2024-04-17T19:51:48.488 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2024-04-17T19:51:48.489 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2024-04-17T19:51:48.489 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2024-04-17T19:51:48.489 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2024-04-17T19:51:48.489 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2024-04-17T19:51:48.489 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2024-04-17T19:51:48.489 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2024-04-17T19:51:48.489 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2024-04-17T19:51:48.489 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2024-04-17T19:51:48.489 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2024-04-17T19:51:48.489 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2024-04-17T19:51:48.489 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2024-04-17T19:51:48.489 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2024-04-17T19:51:48.489 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2024-04-17T19:51:48.489 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2024-04-17T19:51:48.489 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2024-04-17T19:51:48.490 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2024-04-17T19:51:48.490 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2024-04-17T19:51:48.490 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2024-04-17T19:51:48.490 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2024-04-17T19:51:48.490 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2024-04-17T19:51:48.490 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2024-04-17T19:51:48.490 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2024-04-17T19:51:48.490 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2024-04-17T19:51:48.490 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2024-04-17T19:51:48.490 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2024-04-17T19:51:48.490 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2024-04-17T19:51:48.490 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2024-04-17T19:51:48.490 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2024-04-17T19:51:48.490 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2024-04-17T19:51:48.491 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2024-04-17T19:51:48.491 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2024-04-17T19:51:48.491 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2024-04-17T19:51:48.491 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2024-04-17T19:51:48.491 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2024-04-17T19:51:48.491 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2024-04-17T19:51:48.491 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2024-04-17T19:51:48.491 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2024-04-17T19:51:48.491 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2024-04-17T19:51:48.491 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2024-04-17T19:51:48.491 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2024-04-17T19:51:48.491 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2024-04-17T19:51:48.491 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2024-04-17T19:51:48.491 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2024-04-17T19:51:48.491 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2024-04-17T19:51:48.492 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2024-04-17T19:51:48.492 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2024-04-17T19:51:48.492 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2024-04-17T19:51:48.492 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2024-04-17T19:51:48.492 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2024-04-17T19:51:48.492 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2024-04-17T19:51:48.492 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2024-04-17T19:51:48.492 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2024-04-17T19:51:48.492 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2024-04-17T19:51:48.492 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2024-04-17T19:51:48.492 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2024-04-17T19:51:50.784 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2024-04-17T19:51:50.784 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2024-04-17T19:51:50.784 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2024-04-17T19:51:50.785 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2024-04-17T19:51:50.785 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2024-04-17T19:51:50.785 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2024-04-17T19:51:50.785 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2024-04-17T19:51:50.785 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2024-04-17T19:51:50.785 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2024-04-17T19:51:50.785 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2024-04-17T19:51:50.785 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2024-04-17T19:51:50.785 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2024-04-17T19:51:50.785 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2024-04-17T19:51:50.786 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2024-04-17T19:51:50.786 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2024-04-17T19:51:50.786 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2024-04-17T19:51:50.786 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2024-04-17T19:51:50.786 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2024-04-17T19:51:50.786 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2024-04-17T19:51:50.786 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2024-04-17T19:51:50.786 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2024-04-17T19:51:50.786 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2024-04-17T19:51:50.786 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2024-04-17T19:51:50.787 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2024-04-17T19:51:50.787 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2024-04-17T19:51:50.787 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2024-04-17T19:51:50.787 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2024-04-17T19:51:50.787 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2024-04-17T19:51:50.787 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2024-04-17T19:51:50.787 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2024-04-17T19:51:50.787 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2024-04-17T19:51:50.787 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2024-04-17T19:51:50.787 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2024-04-17T19:51:50.787 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2024-04-17T19:51:50.788 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2024-04-17T19:51:50.788 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2024-04-17T19:51:50.788 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2024-04-17T19:51:50.788 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2024-04-17T19:51:50.788 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2024-04-17T19:51:50.788 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2024-04-17T19:51:50.788 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2024-04-17T19:51:50.788 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2024-04-17T19:51:50.788 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2024-04-17T19:51:50.788 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2024-04-17T19:51:50.788 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2024-04-17T19:51:50.789 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2024-04-17T19:51:50.789 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2024-04-17T19:51:50.789 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2024-04-17T19:51:50.789 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2024-04-17T19:51:50.789 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2024-04-17T19:51:50.789 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2024-04-17T19:51:50.789 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2024-04-17T19:51:50.789 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2024-04-17T19:51:50.789 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2024-04-17T19:51:50.790 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2024-04-17T19:51:50.790 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2024-04-17T19:51:50.790 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2024-04-17T19:51:50.790 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2024-04-17T19:51:50.790 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2024-04-17T19:51:50.790 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2024-04-17T19:51:50.790 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2024-04-17T19:51:50.790 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2024-04-17T19:51:50.790 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2024-04-17T19:51:50.790 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2024-04-17T19:51:50.791 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2024-04-17T19:51:50.791 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2024-04-17T19:51:50.791 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2024-04-17T19:51:50.791 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2024-04-17T19:51:50.791 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2024-04-17T19:51:50.791 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2024-04-17T19:51:50.791 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2024-04-17T19:51:50.791 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2024-04-17T19:51:50.791 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2024-04-17T19:51:50.791 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2024-04-17T19:51:50.791 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2024-04-17T19:51:50.792 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2024-04-17T19:51:50.792 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2024-04-17T19:51:50.792 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2024-04-17T19:51:50.792 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2024-04-17T19:51:50.792 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2024-04-17T19:51:50.792 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2024-04-17T19:51:51.192 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:51:51.196 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:51:51.223 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2024-04-17T19:51:51.223 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2024-04-17T19:51:51.223 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2024-04-17T19:51:51.223 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2024-04-17T19:51:51.223 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2024-04-17T19:51:51.224 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2024-04-17T19:51:51.224 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2024-04-17T19:51:51.224 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2024-04-17T19:51:51.224 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2024-04-17T19:51:51.224 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2024-04-17T19:51:51.224 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2024-04-17T19:51:51.224 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2024-04-17T19:51:51.224 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2024-04-17T19:51:51.224 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2024-04-17T19:51:51.224 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2024-04-17T19:51:51.224 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2024-04-17T19:51:51.225 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2024-04-17T19:51:51.225 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2024-04-17T19:51:51.225 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2024-04-17T19:51:51.225 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2024-04-17T19:51:51.225 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2024-04-17T19:51:51.225 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2024-04-17T19:51:51.225 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2024-04-17T19:51:51.225 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2024-04-17T19:51:51.225 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2024-04-17T19:51:51.225 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2024-04-17T19:51:51.226 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2024-04-17T19:51:51.226 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2024-04-17T19:51:51.226 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2024-04-17T19:51:51.226 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2024-04-17T19:51:51.226 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2024-04-17T19:51:51.226 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2024-04-17T19:51:51.226 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2024-04-17T19:51:51.226 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2024-04-17T19:51:51.226 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2024-04-17T19:51:51.226 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2024-04-17T19:51:51.227 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2024-04-17T19:51:51.227 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2024-04-17T19:51:51.227 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2024-04-17T19:51:51.227 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2024-04-17T19:51:51.227 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2024-04-17T19:51:51.227 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2024-04-17T19:51:51.227 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2024-04-17T19:51:51.227 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2024-04-17T19:51:51.227 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2024-04-17T19:51:51.227 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2024-04-17T19:51:51.227 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2024-04-17T19:51:51.227 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2024-04-17T19:51:51.228 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2024-04-17T19:51:51.228 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2024-04-17T19:51:51.228 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2024-04-17T19:51:51.228 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2024-04-17T19:51:51.228 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2024-04-17T19:51:51.228 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2024-04-17T19:51:51.228 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2024-04-17T19:51:51.228 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2024-04-17T19:51:51.228 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2024-04-17T19:51:51.228 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2024-04-17T19:51:51.228 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2024-04-17T19:51:51.229 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2024-04-17T19:51:51.229 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2024-04-17T19:51:51.229 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2024-04-17T19:51:51.229 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2024-04-17T19:51:51.229 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2024-04-17T19:51:51.229 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2024-04-17T19:51:51.229 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2024-04-17T19:51:51.229 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2024-04-17T19:51:51.229 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2024-04-17T19:51:51.229 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2024-04-17T19:51:51.229 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2024-04-17T19:51:51.230 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2024-04-17T19:51:51.230 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2024-04-17T19:51:51.230 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2024-04-17T19:51:51.230 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2024-04-17T19:51:51.230 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2024-04-17T19:51:51.230 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2024-04-17T19:51:51.230 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2024-04-17T19:51:51.230 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2024-04-17T19:51:51.230 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2024-04-17T19:51:51.230 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2024-04-17T19:51:51.230 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2024-04-17T19:51:52.447 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2024-04-17T19:51:52.448 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2024-04-17T19:51:52.448 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2024-04-17T19:51:52.448 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2024-04-17T19:51:52.448 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2024-04-17T19:51:52.448 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2024-04-17T19:51:52.448 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2024-04-17T19:51:52.448 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2024-04-17T19:51:52.448 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2024-04-17T19:51:52.448 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2024-04-17T19:51:52.448 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2024-04-17T19:51:52.448 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2024-04-17T19:51:52.448 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2024-04-17T19:51:52.448 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2024-04-17T19:51:52.448 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2024-04-17T19:51:52.448 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2024-04-17T19:51:52.449 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2024-04-17T19:51:52.449 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2024-04-17T19:51:52.449 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2024-04-17T19:51:52.449 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2024-04-17T19:51:52.449 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2024-04-17T19:51:52.449 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2024-04-17T19:51:52.449 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2024-04-17T19:51:52.449 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2024-04-17T19:51:52.449 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2024-04-17T19:51:52.449 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2024-04-17T19:51:52.449 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2024-04-17T19:51:52.449 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2024-04-17T19:51:52.449 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2024-04-17T19:51:52.449 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2024-04-17T19:51:52.449 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2024-04-17T19:51:52.450 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2024-04-17T19:51:52.450 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2024-04-17T19:51:52.450 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2024-04-17T19:51:52.450 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2024-04-17T19:51:52.450 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2024-04-17T19:51:52.450 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2024-04-17T19:51:52.450 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2024-04-17T19:51:52.450 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2024-04-17T19:51:52.450 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2024-04-17T19:51:52.450 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2024-04-17T19:51:52.450 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2024-04-17T19:51:52.450 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2024-04-17T19:51:52.450 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2024-04-17T19:51:52.450 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2024-04-17T19:51:52.450 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2024-04-17T19:51:52.451 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2024-04-17T19:51:52.451 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2024-04-17T19:51:52.451 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2024-04-17T19:51:52.451 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2024-04-17T19:51:52.451 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2024-04-17T19:51:52.451 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2024-04-17T19:51:52.451 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2024-04-17T19:51:52.451 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2024-04-17T19:51:52.451 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2024-04-17T19:51:52.451 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2024-04-17T19:51:52.451 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2024-04-17T19:51:52.451 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2024-04-17T19:51:52.451 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2024-04-17T19:51:52.451 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2024-04-17T19:51:52.452 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2024-04-17T19:51:52.452 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2024-04-17T19:51:52.452 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2024-04-17T19:51:52.452 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2024-04-17T19:51:52.452 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2024-04-17T19:51:52.452 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2024-04-17T19:51:52.452 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2024-04-17T19:51:52.452 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2024-04-17T19:51:52.452 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2024-04-17T19:51:52.452 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2024-04-17T19:51:52.452 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2024-04-17T19:51:52.452 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2024-04-17T19:51:52.452 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2024-04-17T19:51:52.452 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2024-04-17T19:51:52.452 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2024-04-17T19:51:52.453 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2024-04-17T19:51:52.453 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2024-04-17T19:51:52.453 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2024-04-17T19:51:52.453 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2024-04-17T19:51:52.453 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2024-04-17T19:51:52.453 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2024-04-17T19:51:52.726 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2024-04-17T19:51:52.726 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2024-04-17T19:51:52.726 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2024-04-17T19:51:52.726 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2024-04-17T19:51:52.726 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2024-04-17T19:51:52.726 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2024-04-17T19:51:52.726 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2024-04-17T19:51:52.726 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2024-04-17T19:51:52.726 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2024-04-17T19:51:52.726 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2024-04-17T19:51:52.726 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2024-04-17T19:51:52.726 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2024-04-17T19:51:52.727 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2024-04-17T19:51:52.727 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2024-04-17T19:51:52.727 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2024-04-17T19:51:52.727 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2024-04-17T19:51:52.727 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2024-04-17T19:51:52.727 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2024-04-17T19:51:52.727 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2024-04-17T19:51:52.727 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2024-04-17T19:51:52.727 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2024-04-17T19:51:52.727 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2024-04-17T19:51:52.727 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2024-04-17T19:51:52.727 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2024-04-17T19:51:52.727 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2024-04-17T19:51:52.727 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2024-04-17T19:51:52.727 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2024-04-17T19:51:52.727 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2024-04-17T19:51:52.728 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2024-04-17T19:51:52.728 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2024-04-17T19:51:52.728 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2024-04-17T19:51:52.728 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2024-04-17T19:51:52.728 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2024-04-17T19:51:52.728 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2024-04-17T19:51:52.728 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2024-04-17T19:51:52.728 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2024-04-17T19:51:52.728 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2024-04-17T19:51:52.728 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2024-04-17T19:51:52.728 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2024-04-17T19:51:52.728 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2024-04-17T19:51:52.728 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2024-04-17T19:51:52.728 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2024-04-17T19:51:52.728 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2024-04-17T19:51:52.728 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2024-04-17T19:51:52.728 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2024-04-17T19:51:52.729 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2024-04-17T19:51:52.729 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2024-04-17T19:51:52.729 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2024-04-17T19:51:52.729 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2024-04-17T19:51:52.729 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2024-04-17T19:51:52.729 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2024-04-17T19:51:52.729 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2024-04-17T19:51:52.729 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2024-04-17T19:51:52.729 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2024-04-17T19:51:52.729 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2024-04-17T19:51:52.729 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2024-04-17T19:51:52.729 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2024-04-17T19:51:52.729 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2024-04-17T19:51:52.730 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2024-04-17T19:51:52.730 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2024-04-17T19:51:52.730 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2024-04-17T19:51:52.730 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2024-04-17T19:51:52.730 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2024-04-17T19:51:52.730 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2024-04-17T19:51:52.730 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2024-04-17T19:51:52.730 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2024-04-17T19:51:52.730 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2024-04-17T19:51:52.730 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2024-04-17T19:51:52.730 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2024-04-17T19:51:52.730 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2024-04-17T19:51:52.730 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2024-04-17T19:51:52.730 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2024-04-17T19:51:52.730 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2024-04-17T19:51:52.730 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2024-04-17T19:51:52.731 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2024-04-17T19:51:52.731 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2024-04-17T19:51:52.731 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2024-04-17T19:51:52.731 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2024-04-17T19:51:52.731 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2024-04-17T19:51:52.731 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2024-04-17T19:51:52.731 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2024-04-17T19:51:53.210 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2024-04-17T19:51:53.211 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2024-04-17T19:51:53.211 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2024-04-17T19:51:53.211 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2024-04-17T19:51:53.211 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2024-04-17T19:51:53.211 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2024-04-17T19:51:53.211 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2024-04-17T19:51:53.211 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2024-04-17T19:51:53.211 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2024-04-17T19:51:53.211 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2024-04-17T19:51:53.211 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2024-04-17T19:51:53.212 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2024-04-17T19:51:53.212 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2024-04-17T19:51:53.212 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2024-04-17T19:51:53.212 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2024-04-17T19:51:53.212 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2024-04-17T19:51:53.212 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2024-04-17T19:51:53.212 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2024-04-17T19:51:53.212 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2024-04-17T19:51:53.212 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2024-04-17T19:51:53.212 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2024-04-17T19:51:53.212 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2024-04-17T19:51:53.212 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2024-04-17T19:51:53.212 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2024-04-17T19:51:53.213 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2024-04-17T19:51:53.213 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2024-04-17T19:51:53.213 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2024-04-17T19:51:53.213 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2024-04-17T19:51:53.213 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2024-04-17T19:51:53.213 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2024-04-17T19:51:53.213 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2024-04-17T19:51:53.213 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2024-04-17T19:51:53.213 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2024-04-17T19:51:53.213 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2024-04-17T19:51:53.213 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2024-04-17T19:51:53.213 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2024-04-17T19:51:53.213 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2024-04-17T19:51:53.214 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2024-04-17T19:51:53.214 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2024-04-17T19:51:53.214 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2024-04-17T19:51:53.214 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2024-04-17T19:51:53.214 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2024-04-17T19:51:53.214 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2024-04-17T19:51:53.214 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2024-04-17T19:51:53.214 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2024-04-17T19:51:53.214 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2024-04-17T19:51:53.214 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2024-04-17T19:51:53.214 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2024-04-17T19:51:53.215 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2024-04-17T19:51:53.215 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2024-04-17T19:51:53.215 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2024-04-17T19:51:53.215 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2024-04-17T19:51:53.215 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2024-04-17T19:51:53.215 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2024-04-17T19:51:53.215 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2024-04-17T19:51:53.215 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2024-04-17T19:51:53.216 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2024-04-17T19:51:53.216 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2024-04-17T19:51:53.216 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2024-04-17T19:51:53.216 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2024-04-17T19:51:53.216 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2024-04-17T19:51:53.216 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2024-04-17T19:51:53.216 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2024-04-17T19:51:53.216 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2024-04-17T19:51:53.216 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2024-04-17T19:51:53.216 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2024-04-17T19:51:53.216 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2024-04-17T19:51:53.216 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2024-04-17T19:51:53.217 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2024-04-17T19:51:53.217 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2024-04-17T19:51:53.217 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2024-04-17T19:51:53.217 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2024-04-17T19:51:53.217 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2024-04-17T19:51:53.217 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2024-04-17T19:51:53.217 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2024-04-17T19:51:53.217 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2024-04-17T19:51:53.217 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2024-04-17T19:51:53.217 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2024-04-17T19:51:53.217 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2024-04-17T19:51:53.217 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2024-04-17T19:51:53.218 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2024-04-17T19:51:54.398 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2024-04-17T19:51:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2024-04-17T19:51:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2024-04-17T19:51:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2024-04-17T19:51:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2024-04-17T19:51:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2024-04-17T19:51:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2024-04-17T19:51:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2024-04-17T19:51:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2024-04-17T19:51:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2024-04-17T19:51:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2024-04-17T19:51:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2024-04-17T19:51:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2024-04-17T19:51:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2024-04-17T19:51:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2024-04-17T19:51:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2024-04-17T19:51:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2024-04-17T19:51:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2024-04-17T19:51:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2024-04-17T19:51:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2024-04-17T19:51:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2024-04-17T19:51:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2024-04-17T19:51:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2024-04-17T19:51:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2024-04-17T19:51:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2024-04-17T19:51:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2024-04-17T19:51:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2024-04-17T19:51:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2024-04-17T19:51:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2024-04-17T19:51:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2024-04-17T19:51:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2024-04-17T19:51:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2024-04-17T19:51:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2024-04-17T19:51:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2024-04-17T19:51:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2024-04-17T19:51:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2024-04-17T19:51:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2024-04-17T19:51:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2024-04-17T19:51:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2024-04-17T19:51:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2024-04-17T19:51:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2024-04-17T19:51:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2024-04-17T19:51:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2024-04-17T19:51:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2024-04-17T19:51:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2024-04-17T19:51:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2024-04-17T19:51:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2024-04-17T19:51:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2024-04-17T19:51:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2024-04-17T19:51:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2024-04-17T19:51:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2024-04-17T19:51:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2024-04-17T19:51:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2024-04-17T19:51:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2024-04-17T19:51:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2024-04-17T19:51:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2024-04-17T19:51:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2024-04-17T19:51:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2024-04-17T19:51:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2024-04-17T19:51:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2024-04-17T19:51:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2024-04-17T19:51:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2024-04-17T19:51:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2024-04-17T19:51:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2024-04-17T19:51:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2024-04-17T19:51:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2024-04-17T19:51:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2024-04-17T19:51:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2024-04-17T19:51:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2024-04-17T19:51:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2024-04-17T19:51:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2024-04-17T19:51:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2024-04-17T19:51:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2024-04-17T19:51:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2024-04-17T19:51:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2024-04-17T19:51:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2024-04-17T19:51:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2024-04-17T19:51:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2024-04-17T19:51:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2024-04-17T19:51:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2024-04-17T19:51:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2024-04-17T19:51:54.645 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2024-04-17T19:51:54.645 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2024-04-17T19:51:54.645 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2024-04-17T19:51:54.645 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2024-04-17T19:51:54.645 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2024-04-17T19:51:54.645 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2024-04-17T19:51:54.645 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2024-04-17T19:51:54.646 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2024-04-17T19:51:54.646 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2024-04-17T19:51:54.646 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2024-04-17T19:51:54.646 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2024-04-17T19:51:54.646 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2024-04-17T19:51:54.646 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2024-04-17T19:51:54.646 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2024-04-17T19:51:54.646 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2024-04-17T19:51:54.646 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2024-04-17T19:51:54.646 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2024-04-17T19:51:54.646 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2024-04-17T19:51:54.646 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2024-04-17T19:51:54.646 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2024-04-17T19:51:54.647 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2024-04-17T19:51:54.647 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2024-04-17T19:51:54.647 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2024-04-17T19:51:54.647 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2024-04-17T19:51:54.647 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2024-04-17T19:51:54.647 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2024-04-17T19:51:54.647 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2024-04-17T19:51:54.647 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2024-04-17T19:51:54.647 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2024-04-17T19:51:54.647 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2024-04-17T19:51:54.647 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2024-04-17T19:51:54.647 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2024-04-17T19:51:54.647 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2024-04-17T19:51:54.648 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2024-04-17T19:51:54.648 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2024-04-17T19:51:54.648 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2024-04-17T19:51:54.648 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2024-04-17T19:51:54.648 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2024-04-17T19:51:54.648 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2024-04-17T19:51:54.648 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2024-04-17T19:51:54.648 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2024-04-17T19:51:54.648 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2024-04-17T19:51:54.648 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2024-04-17T19:51:54.648 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2024-04-17T19:51:54.648 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2024-04-17T19:51:54.648 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2024-04-17T19:51:54.649 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2024-04-17T19:51:54.649 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2024-04-17T19:51:54.649 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2024-04-17T19:51:54.649 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2024-04-17T19:51:54.649 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2024-04-17T19:51:54.649 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2024-04-17T19:51:54.649 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2024-04-17T19:51:54.649 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2024-04-17T19:51:54.649 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2024-04-17T19:51:54.649 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2024-04-17T19:51:54.649 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2024-04-17T19:51:54.649 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2024-04-17T19:51:54.649 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2024-04-17T19:51:54.649 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2024-04-17T19:51:54.650 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2024-04-17T19:51:54.650 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2024-04-17T19:51:54.650 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2024-04-17T19:51:54.650 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2024-04-17T19:51:54.650 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2024-04-17T19:51:54.650 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2024-04-17T19:51:54.650 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2024-04-17T19:51:54.650 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2024-04-17T19:51:54.650 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2024-04-17T19:51:54.650 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2024-04-17T19:51:54.650 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2024-04-17T19:51:54.650 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2024-04-17T19:51:54.651 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2024-04-17T19:51:54.651 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2024-04-17T19:51:54.651 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2024-04-17T19:51:54.651 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2024-04-17T19:51:54.651 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2024-04-17T19:51:54.651 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2024-04-17T19:51:54.651 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2024-04-17T19:51:54.651 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2024-04-17T19:51:54.651 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2024-04-17T19:51:55.288 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2024-04-17T19:51:55.288 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2024-04-17T19:51:55.288 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2024-04-17T19:51:55.288 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2024-04-17T19:51:55.288 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2024-04-17T19:51:55.288 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2024-04-17T19:51:55.288 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2024-04-17T19:51:55.288 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2024-04-17T19:51:55.288 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2024-04-17T19:51:55.288 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2024-04-17T19:51:55.289 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2024-04-17T19:51:55.289 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2024-04-17T19:51:55.289 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2024-04-17T19:51:55.289 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2024-04-17T19:51:55.289 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2024-04-17T19:51:55.289 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2024-04-17T19:51:55.289 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2024-04-17T19:51:55.289 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2024-04-17T19:51:55.289 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2024-04-17T19:51:55.289 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2024-04-17T19:51:55.289 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2024-04-17T19:51:55.289 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2024-04-17T19:51:55.290 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2024-04-17T19:51:55.290 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2024-04-17T19:51:55.290 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2024-04-17T19:51:55.290 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2024-04-17T19:51:55.290 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2024-04-17T19:51:55.290 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2024-04-17T19:51:55.290 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2024-04-17T19:51:55.290 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2024-04-17T19:51:55.290 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2024-04-17T19:51:55.291 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2024-04-17T19:51:55.291 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2024-04-17T19:51:55.291 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2024-04-17T19:51:55.291 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2024-04-17T19:51:55.291 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2024-04-17T19:51:55.291 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2024-04-17T19:51:55.291 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2024-04-17T19:51:55.291 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2024-04-17T19:51:55.291 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2024-04-17T19:51:55.291 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2024-04-17T19:51:55.291 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2024-04-17T19:51:55.291 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2024-04-17T19:51:55.291 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2024-04-17T19:51:55.292 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2024-04-17T19:51:55.292 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2024-04-17T19:51:55.292 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2024-04-17T19:51:55.292 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2024-04-17T19:51:55.292 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2024-04-17T19:51:55.292 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2024-04-17T19:51:55.292 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2024-04-17T19:51:55.292 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2024-04-17T19:51:55.292 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2024-04-17T19:51:55.292 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2024-04-17T19:51:55.292 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2024-04-17T19:51:55.292 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2024-04-17T19:51:55.292 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2024-04-17T19:51:55.293 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2024-04-17T19:51:55.293 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2024-04-17T19:51:55.293 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2024-04-17T19:51:55.293 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2024-04-17T19:51:55.293 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2024-04-17T19:51:55.293 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2024-04-17T19:51:55.293 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2024-04-17T19:51:55.293 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2024-04-17T19:51:55.293 INFO:tasks.workunit.client.0.smithi029.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2024-04-17T19:51:55.294 INFO:tasks.workunit.client.0.smithi029.stdout:Take third snapshot .snap/test3 2024-04-17T19:51:55.319 INFO:tasks.workunit.client.0.smithi029.stdout:Delete the snapshots... 2024-04-17T19:51:57.802 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:51:57.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.991+0000 7f9f8e600640 1 Processor -- start 2024-04-17T19:51:57.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.991+0000 7f9f8e600640 1 -- start start 2024-04-17T19:51:57.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.991+0000 7f9f8e600640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9f88153d40 0x7f9f88067ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:51:57.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.991+0000 7f9f8e600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f88159c70 con 0x7f9f880cac60 2024-04-17T19:51:57.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.991+0000 7f9f8e600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f88159f50 con 0x7f9f88068220 2024-04-17T19:51:57.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.991+0000 7f9f8e600640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9f8815a230 con 0x7f9f88153d40 2024-04-17T19:51:57.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.991+0000 7f9f8dc00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9f880cac60 0x7f9f880cb070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59824/0 (socket says 172.21.15.29:59824) 2024-04-17T19:51:57.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.991+0000 7f9f8dc00640 1 -- 172.21.15.29:0/2582990981 learned_addr learned my addr 172.21.15.29:0/2582990981 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:51:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.991+0000 7f9f8d200640 1 --2- 172.21.15.29:0/2582990981 >> v2:172.21.15.37:3300/0 conn(0x7f9f88153d40 0x7f9f88067ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:51:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87400640 1 -- 172.21.15.29:0/2582990981 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3380736080 0 0) 0x7f9f88159c70 con 0x7f9f880cac60 2024-04-17T19:51:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87400640 1 -- 172.21.15.29:0/2582990981 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f70003530 con 0x7f9f880cac60 2024-04-17T19:51:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87400640 1 -- 172.21.15.29:0/2582990981 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4105337023 0 0) 0x7f9f70003530 con 0x7f9f880cac60 2024-04-17T19:51:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87400640 1 -- 172.21.15.29:0/2582990981 >> v2:172.21.15.37:3300/0 conn(0x7f9f88153d40 msgr2=0x7f9f88067ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:51:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87400640 1 --2- 172.21.15.29:0/2582990981 >> v2:172.21.15.37:3300/0 conn(0x7f9f88153d40 0x7f9f88067ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87400640 1 -- 172.21.15.29:0/2582990981 >> v1:172.21.15.37:6789/0 conn(0x7f9f88068220 legacy=0x7f9f880c89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:51:57.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87400640 1 -- 172.21.15.29:0/2582990981 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8815a510 con 0x7f9f880cac60 2024-04-17T19:51:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87400640 1 -- 172.21.15.29:0/2582990981 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f9f80001a00 con 0x7f9f880cac60 2024-04-17T19:51:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87400640 1 -- 172.21.15.29:0/2582990981 >> v1:172.21.15.29:6789/0 conn(0x7f9f880cac60 legacy=0x7f9f880cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:51:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87400640 1 --2- 172.21.15.29:0/2582990981 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f70003a30 0x7f9f70003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:51:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87400640 1 --2- 172.21.15.29:0/2582990981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f70004810 0x7f9f70004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:51:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87400640 1 --2- 172.21.15.29:0/2582990981 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f70005230 0x7f9f7000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:51:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87400640 1 -- 172.21.15.29:0/2582990981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9f8815a230 con 0x7f9f70004810 2024-04-17T19:51:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87400640 1 -- 172.21.15.29:0/2582990981 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7000c1b0 con 0x7f9f70005230 2024-04-17T19:51:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87400640 1 -- 172.21.15.29:0/2582990981 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9f7000c3b0 con 0x7f9f70003a30 2024-04-17T19:51:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87e00640 1 --2- 172.21.15.29:0/2582990981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f70004810 0x7f9f70004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:51:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f8dc00640 1 --2- 172.21.15.29:0/2582990981 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f70005230 0x7f9f7000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:51:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f8d200640 1 --2- 172.21.15.29:0/2582990981 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f70003a30 0x7f9f70003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:51:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87e00640 1 -- 172.21.15.29:0/2582990981 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f70003a30 msgr2=0x7f9f70003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:51:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87e00640 1 --2- 172.21.15.29:0/2582990981 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f70003a30 0x7f9f70003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87e00640 1 -- 172.21.15.29:0/2582990981 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f70005230 msgr2=0x7f9f7000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:51:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87e00640 1 --2- 172.21.15.29:0/2582990981 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f70005230 0x7f9f7000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:57.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f87e00640 1 -- 172.21.15.29:0/2582990981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f70003530 con 0x7f9f70004810 2024-04-17T19:51:57.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.995+0000 7f9f8dc00640 1 -- 172.21.15.29:0/2582990981 reap_dead start 2024-04-17T19:51:57.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.999+0000 7f9f87e00640 1 --2- 172.21.15.29:0/2582990981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f70004810 0x7f9f70004c20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9f74001780 tx=0x7f9f7401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c8986501f7e70e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:51:57.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.999+0000 7f9f87400640 1 -- 172.21.15.29:0/2582990981 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f740013a0 con 0x7f9f70004810 2024-04-17T19:51:57.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.999+0000 7f9f87400640 1 -- 172.21.15.29:0/2582990981 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f740052f0 con 0x7f9f70004810 2024-04-17T19:51:57.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.999+0000 7f9f87400640 1 -- 172.21.15.29:0/2582990981 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f74003840 con 0x7f9f70004810 2024-04-17T19:51:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.999+0000 7f9f8e600640 1 -- 172.21.15.29:0/2582990981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f70004810 msgr2=0x7f9f70004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:51:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.999+0000 7f9f8e600640 1 --2- 172.21.15.29:0/2582990981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f70004810 0x7f9f70004c20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f9f74001780 tx=0x7f9f7401c400 comp rx=0 tx=0).stop 2024-04-17T19:51:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.999+0000 7f9f8e600640 1 -- 172.21.15.29:0/2582990981 shutdown_connections 2024-04-17T19:51:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.999+0000 7f9f8e600640 1 --2- 172.21.15.29:0/2582990981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f70004810 0x7f9f70004c20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.999+0000 7f9f8e600640 1 -- 172.21.15.29:0/2582990981 >> 172.21.15.29:0/2582990981 conn(0x7f9f88072db0 msgr2=0x7f9f880731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:51:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.999+0000 7f9f8e600640 1 -- 172.21.15.29:0/2582990981 shutdown_connections 2024-04-17T19:51:57.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.999+0000 7f9f8e600640 1 -- 172.21.15.29:0/2582990981 wait complete. 2024-04-17T19:51:57.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.999+0000 7f9f8e600640 1 Processor -- start 2024-04-17T19:51:57.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:57.999+0000 7f9f8e600640 1 -- start start 2024-04-17T19:51:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8e600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f70004810 0x7f9f8814f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:51:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8e600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f880cd110 0x7f9f8815d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:51:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8e600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f880cd520 0x7f9f8815d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:51:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8e600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9f88067ce0 con 0x7f9f70004810 2024-04-17T19:51:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8e600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9f88154150 con 0x7f9f880cd110 2024-04-17T19:51:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8e600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9f88067e60 con 0x7f9f880cd520 2024-04-17T19:51:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8d200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f70004810 0x7f9f8814f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:51:57.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8d200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f70004810 0x7f9f8814f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49040/0 (socket says 172.21.15.29:49040) 2024-04-17T19:51:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8d200640 1 -- 172.21.15.29:0/1914992820 learned_addr learned my addr 172.21.15.29:0/1914992820 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:51:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f87e00640 1 --2- 172.21.15.29:0/1914992820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f880cd110 0x7f9f8815d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:51:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8dc00640 1 --2- 172.21.15.29:0/1914992820 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f880cd520 0x7f9f8815d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:51:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8d200640 1 -- 172.21.15.29:0/1914992820 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f880cd520 msgr2=0x7f9f8815d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:51:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8d200640 1 --2- 172.21.15.29:0/1914992820 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f880cd520 0x7f9f8815d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8d200640 1 -- 172.21.15.29:0/1914992820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f880cd110 msgr2=0x7f9f8815d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:51:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8d200640 1 --2- 172.21.15.29:0/1914992820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f880cd110 0x7f9f8815d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8d200640 1 -- 172.21.15.29:0/1914992820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f8814fe90 con 0x7f9f70004810 2024-04-17T19:51:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8dc00640 1 --2- 172.21.15.29:0/1914992820 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f880cd520 0x7f9f8815d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T19:51:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8d200640 1 --2- 172.21.15.29:0/1914992820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f70004810 0x7f9f8814f950 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c011530 tx=0x7f9f7c0119f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:51:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f84c00640 1 -- 172.21.15.29:0/1914992820 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c015870 con 0x7f9f70004810 2024-04-17T19:51:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f84c00640 1 -- 172.21.15.29:0/1914992820 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f7c015a10 con 0x7f9f70004810 2024-04-17T19:51:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f84c00640 1 -- 172.21.15.29:0/1914992820 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c015dc0 con 0x7f9f70004810 2024-04-17T19:51:57.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8e600640 1 -- 172.21.15.29:0/1914992820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f8814e8b0 con 0x7f9f70004810 2024-04-17T19:51:57.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8e600640 1 -- 172.21.15.29:0/1914992820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f8814ee70 con 0x7f9f70004810 2024-04-17T19:51:57.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.003+0000 7f9f8e600640 1 -- 172.21.15.29:0/1914992820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f8804ff10 con 0x7f9f70004810 2024-04-17T19:51:57.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.007+0000 7f9f84c00640 1 -- 172.21.15.29:0/1914992820 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9f7c013400 con 0x7f9f70004810 2024-04-17T19:51:58.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.007+0000 7f9f84c00640 1 --2- 172.21.15.29:0/1914992820 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9f680683b0 0x7f9f6806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:51:58.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.007+0000 7f9f84c00640 1 -- 172.21.15.29:0/1914992820 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6877+0+0 (secure 0 0 0) 0x7f9f7c087310 con 0x7f9f70004810 2024-04-17T19:51:58.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.007+0000 7f9f87e00640 1 --2- 172.21.15.29:0/1914992820 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9f680683b0 0x7f9f6806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:51:58.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.007+0000 7f9f87e00640 1 --2- 172.21.15.29:0/1914992820 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9f680683b0 0x7f9f6806a8c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9f74004000 tx=0x7f9f740011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:51:58.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.011+0000 7f9f84c00640 1 -- 172.21.15.29:0/1914992820 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9f7c019030 con 0x7f9f70004810 2024-04-17T19:51:58.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.215+0000 7f9f8e600640 1 -- 172.21.15.29:0/1914992820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f88068740 con 0x7f9f70004810 2024-04-17T19:51:58.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.215+0000 7f9f84c00640 1 -- 172.21.15.29:0/1914992820 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6904 (secure 0 0 0) 0x7f9f7c057430 con 0x7f9f70004810 2024-04-17T19:51:58.209 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:51:58.209 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5212,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:34.757701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:active","state_seq":47,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229,4223]},"id":1}]} 2024-04-17T19:51:58.209 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2024-04-17T19:51:58.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.219+0000 7f9f8e600640 1 -- 172.21.15.29:0/1914992820 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9f680683b0 msgr2=0x7f9f6806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:51:58.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.219+0000 7f9f8e600640 1 --2- 172.21.15.29:0/1914992820 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9f680683b0 0x7f9f6806a8c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9f74004000 tx=0x7f9f740011a0 comp rx=0 tx=0).stop 2024-04-17T19:51:58.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.219+0000 7f9f8e600640 1 -- 172.21.15.29:0/1914992820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f70004810 msgr2=0x7f9f8814f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:51:58.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.219+0000 7f9f8e600640 1 --2- 172.21.15.29:0/1914992820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f70004810 0x7f9f8814f950 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c011530 tx=0x7f9f7c0119f0 comp rx=0 tx=0).stop 2024-04-17T19:51:58.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.219+0000 7f9f8e600640 1 -- 172.21.15.29:0/1914992820 shutdown_connections 2024-04-17T19:51:58.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.219+0000 7f9f8e600640 1 --2- 172.21.15.29:0/1914992820 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9f680683b0 0x7f9f6806a8c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:58.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.219+0000 7f9f8e600640 1 --2- 172.21.15.29:0/1914992820 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f880cd520 0x7f9f8815d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:58.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.219+0000 7f9f8e600640 1 --2- 172.21.15.29:0/1914992820 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f880cd110 0x7f9f8815d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:58.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.219+0000 7f9f8e600640 1 --2- 172.21.15.29:0/1914992820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f70004810 0x7f9f8814f950 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:58.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.219+0000 7f9f8e600640 1 -- 172.21.15.29:0/1914992820 >> 172.21.15.29:0/1914992820 conn(0x7f9f88072db0 msgr2=0x7f9f88153020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:51:58.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.219+0000 7f9f8e600640 1 -- 172.21.15.29:0/1914992820 shutdown_connections 2024-04-17T19:51:58.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.219+0000 7f9f8e600640 1 -- 172.21.15.29:0/1914992820 wait complete. 2024-04-17T19:51:58.231 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=4) 2024-04-17T19:51:58.231 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T19:51:58.231 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T19:51:58.248 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T19:51:58.248 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:51:58.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bbd400640 1 Processor -- start 2024-04-17T19:51:58.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bbd400640 1 -- start start 2024-04-17T19:51:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bbd400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3bb8159c90 0x7f3bb80ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:51:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bbd400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3bb80ca7f0 0x7f3bb80c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:51:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bbd400640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bb815a0a0 con 0x7f3bb80c74b0 2024-04-17T19:51:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bbd400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f3bb815abe0 con 0x7f3bb8159c90 2024-04-17T19:51:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bbd400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3bb815a340 con 0x7f3bb80ca7f0 2024-04-17T19:51:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb7200640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3bb80c74b0 0x7f3bb806c660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59838/0 (socket says 172.21.15.29:59838) 2024-04-17T19:51:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb7200640 1 -- 172.21.15.29:0/3744286531 learned_addr learned my addr 172.21.15.29:0/3744286531 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:51:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb6800640 1 --2- 172.21.15.29:0/3744286531 >> v2:172.21.15.37:3300/0 conn(0x7f3bb8159c90 0x7f3bb80ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:51:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bafe00640 1 --2- 172.21.15.29:0/3744286531 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3bb80ca7f0 0x7f3bb80c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:51:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb5e00640 1 -- 172.21.15.29:0/3744286531 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2482667768 0 0) 0x7f3bb815a0a0 con 0x7f3bb80c74b0 2024-04-17T19:51:58.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb5e00640 1 -- 172.21.15.29:0/3744286531 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b9c003530 con 0x7f3bb80c74b0 2024-04-17T19:51:58.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb5e00640 1 -- 172.21.15.29:0/3744286531 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1476361736 0 0) 0x7f3b9c003530 con 0x7f3bb80c74b0 2024-04-17T19:51:58.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb5e00640 1 -- 172.21.15.29:0/3744286531 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3bb80ca7f0 msgr2=0x7f3bb80c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:51:58.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb5e00640 1 --2- 172.21.15.29:0/3744286531 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3bb80ca7f0 0x7f3bb80c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:58.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb5e00640 1 -- 172.21.15.29:0/3744286531 >> v2:172.21.15.37:3300/0 conn(0x7f3bb8159c90 msgr2=0x7f3bb80ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:51:58.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb5e00640 1 --2- 172.21.15.29:0/3744286531 >> v2:172.21.15.37:3300/0 conn(0x7f3bb8159c90 0x7f3bb80ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:58.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb5e00640 1 -- 172.21.15.29:0/3744286531 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bb815a620 con 0x7f3bb80c74b0 2024-04-17T19:51:58.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb5e00640 1 -- 172.21.15.29:0/3744286531 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f3bb0001a20 con 0x7f3bb80c74b0 2024-04-17T19:51:58.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb6800640 1 --2- 172.21.15.29:0/3744286531 >> v2:172.21.15.37:3300/0 conn(0x7f3bb8159c90 0x7f3bb80ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:51:58.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb5e00640 1 -- 172.21.15.29:0/3744286531 >> v1:172.21.15.29:6789/0 conn(0x7f3bb80c74b0 legacy=0x7f3bb806c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:51:58.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb5e00640 1 --2- 172.21.15.29:0/3744286531 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b9c003a30 0x7f3b9c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:51:58.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb5e00640 1 --2- 172.21.15.29:0/3744286531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c004810 0x7f3b9c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:51:58.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb5e00640 1 --2- 172.21.15.29:0/3744286531 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3bb80ca7f0 0x7f3b9c005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:51:58.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb5e00640 1 -- 172.21.15.29:0/3744286531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb815abe0 con 0x7f3b9c004810 2024-04-17T19:51:58.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb5e00640 1 -- 172.21.15.29:0/3744286531 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb815a340 con 0x7f3b9c003a30 2024-04-17T19:51:58.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb5e00640 1 -- 172.21.15.29:0/3744286531 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3b9c00ba80 con 0x7f3bb80ca7f0 2024-04-17T19:51:58.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bafe00640 1 --2- 172.21.15.29:0/3744286531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c004810 0x7f3b9c004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:51:58.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb6800640 1 --2- 172.21.15.29:0/3744286531 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b9c003a30 0x7f3b9c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:51:58.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bb7200640 1 --2- 172.21.15.29:0/3744286531 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3bb80ca7f0 0x7f3b9c005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:51:58.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bafe00640 1 -- 172.21.15.29:0/3744286531 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3bb80ca7f0 msgr2=0x7f3b9c005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:51:58.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bafe00640 1 --2- 172.21.15.29:0/3744286531 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3bb80ca7f0 0x7f3b9c005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:58.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bafe00640 1 -- 172.21.15.29:0/3744286531 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b9c003a30 msgr2=0x7f3b9c003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:51:58.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bafe00640 1 --2- 172.21.15.29:0/3744286531 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b9c003a30 0x7f3b9c003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:58.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.407+0000 7f3bafe00640 1 -- 172.21.15.29:0/3744286531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b9c003530 con 0x7f3b9c004810 2024-04-17T19:51:58.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.411+0000 7f3bafe00640 1 --2- 172.21.15.29:0/3744286531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c004810 0x7f3b9c004c20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0009ae0 tx=0x7f3ba001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c663212812b5639 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:51:58.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.411+0000 7f3bb5e00640 1 -- 172.21.15.29:0/3744286531 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba00b2070 con 0x7f3b9c004810 2024-04-17T19:51:58.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.411+0000 7f3bb5e00640 1 -- 172.21.15.29:0/3744286531 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ba00a5640 con 0x7f3b9c004810 2024-04-17T19:51:58.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.411+0000 7f3bb5e00640 1 -- 172.21.15.29:0/3744286531 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba00096b0 con 0x7f3b9c004810 2024-04-17T19:51:58.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.411+0000 7f3bbd400640 1 -- 172.21.15.29:0/3744286531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c004810 msgr2=0x7f3b9c004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:51:58.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.411+0000 7f3bbd400640 1 --2- 172.21.15.29:0/3744286531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c004810 0x7f3b9c004c20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3ba0009ae0 tx=0x7f3ba001ca50 comp rx=0 tx=0).stop 2024-04-17T19:51:58.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.411+0000 7f3bb7200640 1 -- 172.21.15.29:0/3744286531 reap_dead start 2024-04-17T19:51:58.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.411+0000 7f3bbd400640 1 -- 172.21.15.29:0/3744286531 shutdown_connections 2024-04-17T19:51:58.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.411+0000 7f3bbd400640 1 -- 172.21.15.29:0/3744286531 >> 172.21.15.29:0/3744286531 conn(0x7f3bb806fde0 msgr2=0x7f3bb80701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:51:58.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.411+0000 7f3bbd400640 1 -- 172.21.15.29:0/3744286531 shutdown_connections 2024-04-17T19:51:58.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.411+0000 7f3bbd400640 1 -- 172.21.15.29:0/3744286531 wait complete. 2024-04-17T19:51:58.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.415+0000 7f3bbd400640 1 Processor -- start 2024-04-17T19:51:58.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.415+0000 7f3bbd400640 1 -- start start 2024-04-17T19:51:58.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.415+0000 7f3bbd400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b9c004810 0x7f3bb81f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:51:58.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.415+0000 7f3bbd400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb81e4d30 0x7f3bb8148fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:51:58.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.415+0000 7f3bbd400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3bb81e5b70 0x7f3bb81e5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:51:58.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.415+0000 7f3bbd400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb806c140 con 0x7f3bb81e4d30 2024-04-17T19:51:58.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.415+0000 7f3bbd400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb806be60 con 0x7f3b9c004810 2024-04-17T19:51:58.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.415+0000 7f3bbd400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3bb806cd70 con 0x7f3bb81e5b70 2024-04-17T19:51:58.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.415+0000 7f3bafe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb81e4d30 0x7f3bb8148fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:51:58.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.415+0000 7f3bafe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb81e4d30 0x7f3bb8148fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49058/0 (socket says 172.21.15.29:49058) 2024-04-17T19:51:58.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.415+0000 7f3bafe00640 1 -- 172.21.15.29:0/2963446487 learned_addr learned my addr 172.21.15.29:0/2963446487 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:51:58.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.415+0000 7f3bb6800640 1 --2- 172.21.15.29:0/2963446487 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b9c004810 0x7f3bb81f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:51:58.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.415+0000 7f3bb7200640 1 --2- 172.21.15.29:0/2963446487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3bb81e5b70 0x7f3bb81e5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:51:58.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.415+0000 7f3bafe00640 1 -- 172.21.15.29:0/2963446487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3bb81e5b70 msgr2=0x7f3bb81e5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:51:58.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.415+0000 7f3bafe00640 1 --2- 172.21.15.29:0/2963446487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3bb81e5b70 0x7f3bb81e5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:58.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.415+0000 7f3bafe00640 1 -- 172.21.15.29:0/2963446487 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b9c004810 msgr2=0x7f3bb81f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:51:58.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.415+0000 7f3bafe00640 1 --2- 172.21.15.29:0/2963446487 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b9c004810 0x7f3bb81f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:58.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.415+0000 7f3bafe00640 1 -- 172.21.15.29:0/2963446487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bb81f2f10 con 0x7f3bb81e4d30 2024-04-17T19:51:58.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.419+0000 7f3bafe00640 1 --2- 172.21.15.29:0/2963446487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb81e4d30 0x7f3bb8148fa0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3ba00b0040 tx=0x7f3b9c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:51:58.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.419+0000 7f3baea00640 1 -- 172.21.15.29:0/2963446487 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba00b2040 con 0x7f3bb81e4d30 2024-04-17T19:51:58.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.419+0000 7f3baea00640 1 -- 172.21.15.29:0/2963446487 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ba000f000 con 0x7f3bb81e4d30 2024-04-17T19:51:58.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.419+0000 7f3baea00640 1 -- 172.21.15.29:0/2963446487 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ba0004260 con 0x7f3bb81e4d30 2024-04-17T19:51:58.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.419+0000 7f3bbd400640 1 -- 172.21.15.29:0/2963446487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bb815aef0 con 0x7f3bb81e4d30 2024-04-17T19:51:58.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.419+0000 7f3bbd400640 1 -- 172.21.15.29:0/2963446487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bb815b470 con 0x7f3bb81e4d30 2024-04-17T19:51:58.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.419+0000 7f3bbd400640 1 -- 172.21.15.29:0/2963446487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bb8069b90 con 0x7f3bb81e4d30 2024-04-17T19:51:58.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.419+0000 7f3baea00640 1 -- 172.21.15.29:0/2963446487 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3ba00c0030 con 0x7f3bb81e4d30 2024-04-17T19:51:58.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.419+0000 7f3baea00640 1 --2- 172.21.15.29:0/2963446487 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3b94068380 0x7f3b9406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:51:58.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.419+0000 7f3baea00640 1 -- 172.21.15.29:0/2963446487 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6877+0+0 (secure 0 0 0) 0x7f3ba01208d0 con 0x7f3bb81e4d30 2024-04-17T19:51:58.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.423+0000 7f3bb6800640 1 --2- 172.21.15.29:0/2963446487 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3b94068380 0x7f3b9406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:51:58.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.423+0000 7f3bb6800640 1 --2- 172.21.15.29:0/2963446487 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3b94068380 0x7f3b9406a890 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3bb80c8e90 tx=0x7f3ba8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:51:58.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.423+0000 7f3baea00640 1 -- 172.21.15.29:0/2963446487 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3ba00167a0 con 0x7f3bb81e4d30 2024-04-17T19:51:58.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.639+0000 7f3bbd400640 1 -- 172.21.15.29:0/2963446487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bb80cb780 con 0x7f3bb81e4d30 2024-04-17T19:51:58.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.639+0000 7f3baea00640 1 -- 172.21.15.29:0/2963446487 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6904 (secure 0 0 0) 0x7f3ba00f03a0 con 0x7f3bb81e4d30 2024-04-17T19:51:58.632 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:51:58.632 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5212,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:34.757701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:active","state_seq":47,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229,4223]},"id":1}]} 2024-04-17T19:51:58.633 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2024-04-17T19:51:58.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.639+0000 7f3bbd400640 1 -- 172.21.15.29:0/2963446487 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3b94068380 msgr2=0x7f3b9406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:51:58.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.639+0000 7f3bbd400640 1 --2- 172.21.15.29:0/2963446487 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3b94068380 0x7f3b9406a890 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3bb80c8e90 tx=0x7f3ba8009190 comp rx=0 tx=0).stop 2024-04-17T19:51:58.634 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.639+0000 7f3bbd400640 1 -- 172.21.15.29:0/2963446487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb81e4d30 msgr2=0x7f3bb8148fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:51:58.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.639+0000 7f3bbd400640 1 --2- 172.21.15.29:0/2963446487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb81e4d30 0x7f3bb8148fa0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3ba00b0040 tx=0x7f3b9c005840 comp rx=0 tx=0).stop 2024-04-17T19:51:58.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.639+0000 7f3bbd400640 1 -- 172.21.15.29:0/2963446487 shutdown_connections 2024-04-17T19:51:58.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.639+0000 7f3bbd400640 1 --2- 172.21.15.29:0/2963446487 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3b94068380 0x7f3b9406a890 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:58.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.639+0000 7f3bbd400640 1 --2- 172.21.15.29:0/2963446487 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3bb81e5b70 0x7f3bb81e5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:58.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.639+0000 7f3bbd400640 1 --2- 172.21.15.29:0/2963446487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb81e4d30 0x7f3bb8148fa0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:58.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.639+0000 7f3bbd400640 1 --2- 172.21.15.29:0/2963446487 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3b9c004810 0x7f3bb81f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:51:58.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.639+0000 7f3bbd400640 1 -- 172.21.15.29:0/2963446487 >> 172.21.15.29:0/2963446487 conn(0x7f3bb806fde0 msgr2=0x7f3bb80c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:51:58.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.639+0000 7f3bbd400640 1 -- 172.21.15.29:0/2963446487 shutdown_connections 2024-04-17T19:51:58.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:51:58.639+0000 7f3bbd400640 1 -- 172.21.15.29:0/2963446487 wait complete. 2024-04-17T19:51:58.654 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-17T19:52:00.655 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcfec00640 1 Processor -- start 2024-04-17T19:52:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcfec00640 1 -- start start 2024-04-17T19:52:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcfec00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcf80cccd0 0x7fdcf80ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcfec00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcf80cd0e0 con 0x7fdcf80ca870 2024-04-17T19:52:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcfec00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcf80c88b0 con 0x7fdcf80cc8c0 2024-04-17T19:52:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcfec00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fdcf80c6ce0 con 0x7fdcf80cccd0 2024-04-17T19:52:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcfcc00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdcf80ca870 0x7fdcf81635e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41320/0 (socket says 172.21.15.29:41320) 2024-04-17T19:52:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcfcc00640 1 -- 172.21.15.29:0/2699625218 learned_addr learned my addr 172.21.15.29:0/2699625218 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:52:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf7400640 1 --2- 172.21.15.29:0/2699625218 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcf80cccd0 0x7fdcf80ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf6a00640 1 -- 172.21.15.29:0/2699625218 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1267647768 0 0) 0x7fdcf80cd0e0 con 0x7fdcf80ca870 2024-04-17T19:52:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf6a00640 1 -- 172.21.15.29:0/2699625218 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdcdc003530 con 0x7fdcf80ca870 2024-04-17T19:52:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf6a00640 1 -- 172.21.15.29:0/2699625218 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3983801327 0 0) 0x7fdcf80c88b0 con 0x7fdcf80cc8c0 2024-04-17T19:52:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf6a00640 1 -- 172.21.15.29:0/2699625218 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdcf80cd0e0 con 0x7fdcf80cc8c0 2024-04-17T19:52:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf6a00640 1 -- 172.21.15.29:0/2699625218 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1887367964 0 0) 0x7fdcdc003530 con 0x7fdcf80ca870 2024-04-17T19:52:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf6a00640 1 -- 172.21.15.29:0/2699625218 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcf80cccd0 msgr2=0x7fdcf80ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf6a00640 1 --2- 172.21.15.29:0/2699625218 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcf80cccd0 0x7fdcf80ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf6a00640 1 -- 172.21.15.29:0/2699625218 >> v1:172.21.15.37:6789/0 conn(0x7fdcf80cc8c0 legacy=0x7fdcf8159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf6a00640 1 -- 172.21.15.29:0/2699625218 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf80c6fc0 con 0x7fdcf80ca870 2024-04-17T19:52:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf6a00640 1 -- 172.21.15.29:0/2699625218 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fdcf0001a00 con 0x7fdcf80ca870 2024-04-17T19:52:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf6a00640 1 -- 172.21.15.29:0/2699625218 >> v1:172.21.15.29:6789/0 conn(0x7fdcf80ca870 legacy=0x7fdcf81635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf6a00640 1 --2- 172.21.15.29:0/2699625218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcdc003bf0 0x7fdcdc004040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf6a00640 1 --2- 172.21.15.29:0/2699625218 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcdc004810 0x7fdcdc004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf6a00640 1 --2- 172.21.15.29:0/2699625218 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcf80cccd0 0x7fdcdc0052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf6a00640 1 -- 172.21.15.29:0/2699625218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf80c6ce0 con 0x7fdcdc003bf0 2024-04-17T19:52:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf6a00640 1 -- 172.21.15.29:0/2699625218 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fdcdc00ddc0 con 0x7fdcdc004810 2024-04-17T19:52:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf6a00640 1 -- 172.21.15.29:0/2699625218 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fdcdc00dfc0 con 0x7fdcf80cccd0 2024-04-17T19:52:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.819+0000 7fdcf7e00640 1 --2- 172.21.15.29:0/2699625218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcdc003bf0 0x7fdcdc004040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.823+0000 7fdcf7400640 1 --2- 172.21.15.29:0/2699625218 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcdc004810 0x7fdcdc004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.823+0000 7fdcfcc00640 1 --2- 172.21.15.29:0/2699625218 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcf80cccd0 0x7fdcdc0052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.823+0000 7fdcf7e00640 1 -- 172.21.15.29:0/2699625218 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcf80cccd0 msgr2=0x7fdcdc0052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:00.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.823+0000 7fdcf7e00640 1 --2- 172.21.15.29:0/2699625218 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcf80cccd0 0x7fdcdc0052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:00.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.823+0000 7fdcf7e00640 1 -- 172.21.15.29:0/2699625218 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcdc004810 msgr2=0x7fdcdc004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:00.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.823+0000 7fdcf7e00640 1 --2- 172.21.15.29:0/2699625218 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcdc004810 0x7fdcdc004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:00.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.823+0000 7fdcf7e00640 1 -- 172.21.15.29:0/2699625218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcdc003530 con 0x7fdcdc003bf0 2024-04-17T19:52:00.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.823+0000 7fdcfcc00640 1 --2- 172.21.15.29:0/2699625218 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcf80cccd0 0x7fdcdc0052d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:52:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.823+0000 7fdcf7e00640 1 --2- 172.21.15.29:0/2699625218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcdc003bf0 0x7fdcdc004040 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdce80018d0 tx=0x7fdce801c580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf65ed4c1be7e132 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.823+0000 7fdcf6a00640 1 -- 172.21.15.29:0/2699625218 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce8001520 con 0x7fdcdc003bf0 2024-04-17T19:52:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.823+0000 7fdcf6a00640 1 -- 172.21.15.29:0/2699625218 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdce80052f0 con 0x7fdcdc003bf0 2024-04-17T19:52:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.823+0000 7fdcf6a00640 1 -- 172.21.15.29:0/2699625218 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce8003840 con 0x7fdcdc003bf0 2024-04-17T19:52:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.827+0000 7fdcfec00640 1 -- 172.21.15.29:0/2699625218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcdc003bf0 msgr2=0x7fdcdc004040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.827+0000 7fdcfec00640 1 --2- 172.21.15.29:0/2699625218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcdc003bf0 0x7fdcdc004040 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fdce80018d0 tx=0x7fdce801c580 comp rx=0 tx=0).stop 2024-04-17T19:52:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.827+0000 7fdcfcc00640 1 -- 172.21.15.29:0/2699625218 reap_dead start 2024-04-17T19:52:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.827+0000 7fdcfec00640 1 -- 172.21.15.29:0/2699625218 shutdown_connections 2024-04-17T19:52:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.827+0000 7fdcfec00640 1 -- 172.21.15.29:0/2699625218 >> 172.21.15.29:0/2699625218 conn(0x7fdcf806fde0 msgr2=0x7fdcf80701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.827+0000 7fdcfec00640 1 -- 172.21.15.29:0/2699625218 shutdown_connections 2024-04-17T19:52:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.827+0000 7fdcfec00640 1 -- 172.21.15.29:0/2699625218 wait complete. 2024-04-17T19:52:00.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.827+0000 7fdcfec00640 1 Processor -- start 2024-04-17T19:52:00.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.827+0000 7fdcfec00640 1 -- start start 2024-04-17T19:52:00.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcfec00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcdc003bf0 0x7fdcf8159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:00.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcfec00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcf81f9ca0 0x7fdcf8140960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:00.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcfec00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcf815ea60 0x7fdcf8155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:00.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcfec00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf8159c20 con 0x7fdcf815ea60 2024-04-17T19:52:00.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcfec00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fdcf8159aa0 con 0x7fdcdc003bf0 2024-04-17T19:52:00.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcfec00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fdcf80c6880 con 0x7fdcf81f9ca0 2024-04-17T19:52:00.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcfcc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcf815ea60 0x7fdcf8155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:00.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcfcc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcf815ea60 0x7fdcf8155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57532/0 (socket says 172.21.15.29:57532) 2024-04-17T19:52:00.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcfcc00640 1 -- 172.21.15.29:0/255754324 learned_addr learned my addr 172.21.15.29:0/255754324 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:00.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcf7e00640 1 --2- 172.21.15.29:0/255754324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcdc003bf0 0x7fdcf8159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:00.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcf7400640 1 --2- 172.21.15.29:0/255754324 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcf81f9ca0 0x7fdcf8140960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:00.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcfcc00640 1 -- 172.21.15.29:0/255754324 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcf81f9ca0 msgr2=0x7fdcf8140960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:00.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcfcc00640 1 --2- 172.21.15.29:0/255754324 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcf81f9ca0 0x7fdcf8140960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:00.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcfcc00640 1 -- 172.21.15.29:0/255754324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcdc003bf0 msgr2=0x7fdcf8159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:00.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcfcc00640 1 --2- 172.21.15.29:0/255754324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcdc003bf0 0x7fdcf8159e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:00.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcfcc00640 1 -- 172.21.15.29:0/255754324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcf81412d0 con 0x7fdcf815ea60 2024-04-17T19:52:00.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcf7e00640 1 --2- 172.21.15.29:0/255754324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcdc003bf0 0x7fdcf8159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:52:00.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcf7400640 1 --2- 172.21.15.29:0/255754324 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcf81f9ca0 0x7fdcf8140960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:52:00.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcfcc00640 1 --2- 172.21.15.29:0/255754324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcf815ea60 0x7fdcf8155c30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8070990 tx=0x7fdcf0007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:00.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcefe00640 1 -- 172.21.15.29:0/255754324 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf0019070 con 0x7fdcf815ea60 2024-04-17T19:52:00.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcefe00640 1 -- 172.21.15.29:0/255754324 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcf000ea80 con 0x7fdcf815ea60 2024-04-17T19:52:00.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcefe00640 1 -- 172.21.15.29:0/255754324 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf000ee90 con 0x7fdcf815ea60 2024-04-17T19:52:00.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcfec00640 1 -- 172.21.15.29:0/255754324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcf81f91c0 con 0x7fdcf815ea60 2024-04-17T19:52:00.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcfec00640 1 -- 172.21.15.29:0/255754324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcf81f5e50 con 0x7fdcf815ea60 2024-04-17T19:52:00.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcfec00640 1 -- 172.21.15.29:0/255754324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcf8141040 con 0x7fdcf815ea60 2024-04-17T19:52:00.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.831+0000 7fdcefe00640 1 -- 172.21.15.29:0/255754324 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdcf001c070 con 0x7fdcf815ea60 2024-04-17T19:52:00.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.835+0000 7fdcefe00640 1 --2- 172.21.15.29:0/255754324 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fdcd8068520 0x7fdcd806aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:00.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.835+0000 7fdcefe00640 1 -- 172.21.15.29:0/255754324 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6877+0+0 (secure 0 0 0) 0x7fdcf0086620 con 0x7fdcf815ea60 2024-04-17T19:52:00.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.835+0000 7fdcf7e00640 1 --2- 172.21.15.29:0/255754324 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fdcd8068520 0x7fdcd806aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:00.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.835+0000 7fdcf7e00640 1 --2- 172.21.15.29:0/255754324 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fdcd8068520 0x7fdcd806aa30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdce80ac000 tx=0x7fdce80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:00.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:00.839+0000 7fdcefe00640 1 -- 172.21.15.29:0/255754324 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdcf00567b0 con 0x7fdcf815ea60 2024-04-17T19:52:01.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:01.039+0000 7fdcfec00640 1 -- 172.21.15.29:0/255754324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdcf81f60b0 con 0x7fdcf815ea60 2024-04-17T19:52:01.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:01.039+0000 7fdcefe00640 1 -- 172.21.15.29:0/255754324 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6904 (secure 0 0 0) 0x7fdcf00560b0 con 0x7fdcf815ea60 2024-04-17T19:52:01.034 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:01.034 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5212,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:34.757701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:active","state_seq":47,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229,4223]},"id":1}]} 2024-04-17T19:52:01.035 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2024-04-17T19:52:01.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:01.043+0000 7fdcfec00640 1 -- 172.21.15.29:0/255754324 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fdcd8068520 msgr2=0x7fdcd806aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:01.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:01.043+0000 7fdcfec00640 1 --2- 172.21.15.29:0/255754324 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fdcd8068520 0x7fdcd806aa30 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdce80ac000 tx=0x7fdce80a6000 comp rx=0 tx=0).stop 2024-04-17T19:52:01.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:01.043+0000 7fdcfec00640 1 -- 172.21.15.29:0/255754324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcf815ea60 msgr2=0x7fdcf8155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:01.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:01.043+0000 7fdcfec00640 1 --2- 172.21.15.29:0/255754324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcf815ea60 0x7fdcf8155c30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fdcf8070990 tx=0x7fdcf0007bc0 comp rx=0 tx=0).stop 2024-04-17T19:52:01.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:01.043+0000 7fdcfec00640 1 -- 172.21.15.29:0/255754324 shutdown_connections 2024-04-17T19:52:01.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:01.043+0000 7fdcfec00640 1 --2- 172.21.15.29:0/255754324 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fdcd8068520 0x7fdcd806aa30 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:01.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:01.043+0000 7fdcfec00640 1 --2- 172.21.15.29:0/255754324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcf815ea60 0x7fdcf8155c30 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:01.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:01.043+0000 7fdcfec00640 1 --2- 172.21.15.29:0/255754324 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcf81f9ca0 0x7fdcf8140960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:01.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:01.043+0000 7fdcfec00640 1 --2- 172.21.15.29:0/255754324 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcdc003bf0 0x7fdcf8159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:01.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:01.043+0000 7fdcfec00640 1 -- 172.21.15.29:0/255754324 >> 172.21.15.29:0/255754324 conn(0x7fdcf806fde0 msgr2=0x7fdcf8073460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:01.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:01.043+0000 7fdcfec00640 1 -- 172.21.15.29:0/255754324 shutdown_connections 2024-04-17T19:52:01.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:01.043+0000 7fdcfec00640 1 -- 172.21.15.29:0/255754324 wait complete. 2024-04-17T19:52:01.057 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-17T19:52:03.123 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:03.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.295+0000 7fc230400640 1 Processor -- start 2024-04-17T19:52:03.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.295+0000 7fc230400640 1 -- start start 2024-04-17T19:52:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.295+0000 7fc230400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc228159c90 0x7fc2280ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:03.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.295+0000 7fc230400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2280ca7f0 0x7fc2280c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:03.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.295+0000 7fc230400640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc22815a0a0 con 0x7fc2280c74b0 2024-04-17T19:52:03.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.295+0000 7fc230400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc22815abe0 con 0x7fc228159c90 2024-04-17T19:52:03.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.295+0000 7fc230400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc22815a340 con 0x7fc2280ca7f0 2024-04-17T19:52:03.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.295+0000 7fc22d000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2280ca7f0 0x7fc2280c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:03.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.295+0000 7fc22da00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc228159c90 0x7fc2280ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:03.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.295+0000 7fc22d000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2280ca7f0 0x7fc2280c6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:35436/0 (socket says 172.21.15.29:35436) 2024-04-17T19:52:03.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.295+0000 7fc22d000640 1 -- 172.21.15.29:0/1709068677 learned_addr learned my addr 172.21.15.29:0/1709068677 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:03.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.295+0000 7fc22e400640 1 --1- 172.21.15.29:0/1709068677 >> v1:172.21.15.37:6789/0 conn(0x7fc2280c74b0 0x7fc22806c660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:51732/0 (socket says 172.21.15.29:51732) 2024-04-17T19:52:03.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.299+0000 7fc22d000640 1 -- 172.21.15.29:0/1709068677 >> v2:172.21.15.37:3300/0 conn(0x7fc228159c90 msgr2=0x7fc2280ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:03.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.299+0000 7fc22d000640 1 --2- 172.21.15.29:0/1709068677 >> v2:172.21.15.37:3300/0 conn(0x7fc228159c90 0x7fc2280ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:03.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.299+0000 7fc22d000640 1 -- 172.21.15.29:0/1709068677 >> v1:172.21.15.37:6789/0 conn(0x7fc2280c74b0 legacy=0x7fc22806c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:03.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.299+0000 7fc22d000640 1 -- 172.21.15.29:0/1709068677 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc22815a620 con 0x7fc2280ca7f0 2024-04-17T19:52:03.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.299+0000 7fc22da00640 1 --2- 172.21.15.29:0/1709068677 >> v2:172.21.15.37:3300/0 conn(0x7fc228159c90 0x7fc2280ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:52:03.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.299+0000 7fc22d000640 1 --2- 172.21.15.29:0/1709068677 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2280ca7f0 0x7fc2280c6f70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc214009870 tx=0x7fc21401c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ac386934a64cb6e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:03.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.299+0000 7fc227e00640 1 -- 172.21.15.29:0/1709068677 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2140a7070 con 0x7fc2280ca7f0 2024-04-17T19:52:03.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.299+0000 7fc227e00640 1 -- 172.21.15.29:0/1709068677 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2140b0070 con 0x7fc2280ca7f0 2024-04-17T19:52:03.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.299+0000 7fc227e00640 1 -- 172.21.15.29:0/1709068677 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2140ab890 con 0x7fc2280ca7f0 2024-04-17T19:52:03.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.303+0000 7fc230400640 1 -- 172.21.15.29:0/1709068677 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2280ca7f0 msgr2=0x7fc2280c6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:03.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.303+0000 7fc230400640 1 --2- 172.21.15.29:0/1709068677 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2280ca7f0 0x7fc2280c6f70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc214009870 tx=0x7fc21401c560 comp rx=0 tx=0).stop 2024-04-17T19:52:03.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.303+0000 7fc230400640 1 -- 172.21.15.29:0/1709068677 shutdown_connections 2024-04-17T19:52:03.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.303+0000 7fc230400640 1 --2- 172.21.15.29:0/1709068677 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2280ca7f0 0x7fc2280c6f70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:03.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.303+0000 7fc230400640 1 --2- 172.21.15.29:0/1709068677 >> v2:172.21.15.37:3300/0 conn(0x7fc228159c90 0x7fc2280ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:03.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.303+0000 7fc230400640 1 -- 172.21.15.29:0/1709068677 >> 172.21.15.29:0/1709068677 conn(0x7fc22806fde0 msgr2=0x7fc2280701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:03.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.303+0000 7fc230400640 1 -- 172.21.15.29:0/1709068677 shutdown_connections 2024-04-17T19:52:03.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.303+0000 7fc230400640 1 -- 172.21.15.29:0/1709068677 wait complete. 2024-04-17T19:52:03.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.303+0000 7fc230400640 1 Processor -- start 2024-04-17T19:52:03.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.303+0000 7fc230400640 1 -- start start 2024-04-17T19:52:03.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc230400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2280c74b0 0x7fc228148ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc230400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2280ca7f0 0x7fc22814e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc230400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc228159c90 0x7fc22814fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc230400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc228166fb0 con 0x7fc2280ca7f0 2024-04-17T19:52:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc230400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc228166e30 con 0x7fc228159c90 2024-04-17T19:52:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc230400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc228167130 con 0x7fc2280c74b0 2024-04-17T19:52:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc22d000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2280ca7f0 0x7fc22814e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc22d000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2280ca7f0 0x7fc22814e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57548/0 (socket says 172.21.15.29:57548) 2024-04-17T19:52:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc22d000640 1 -- 172.21.15.29:0/2136014469 learned_addr learned my addr 172.21.15.29:0/2136014469 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc22da00640 1 --2- 172.21.15.29:0/2136014469 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2280c74b0 0x7fc228148ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc22e400640 1 --2- 172.21.15.29:0/2136014469 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc228159c90 0x7fc22814fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc22d000640 1 -- 172.21.15.29:0/2136014469 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2280c74b0 msgr2=0x7fc228148ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc22d000640 1 --2- 172.21.15.29:0/2136014469 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2280c74b0 0x7fc228148ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:03.301 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc22d000640 1 -- 172.21.15.29:0/2136014469 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc228159c90 msgr2=0x7fc22814fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:03.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc22d000640 1 --2- 172.21.15.29:0/2136014469 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc228159c90 0x7fc22814fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:03.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc22d000640 1 -- 172.21.15.29:0/2136014469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc228149010 con 0x7fc2280ca7f0 2024-04-17T19:52:03.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc22e400640 1 --2- 172.21.15.29:0/2136014469 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc228159c90 0x7fc22814fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:52:03.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc22d000640 1 --2- 172.21.15.29:0/2136014469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2280ca7f0 0x7fc22814e200 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc2140ad040 tx=0x7fc21401ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:03.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc225600640 1 -- 172.21.15.29:0/2136014469 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2140be5f0 con 0x7fc2280ca7f0 2024-04-17T19:52:03.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc225600640 1 -- 172.21.15.29:0/2136014469 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2140be7b0 con 0x7fc2280ca7f0 2024-04-17T19:52:03.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc230400640 1 -- 172.21.15.29:0/2136014469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2281476c0 con 0x7fc2280ca7f0 2024-04-17T19:52:03.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc225600640 1 -- 172.21.15.29:0/2136014469 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2140beba0 con 0x7fc2280ca7f0 2024-04-17T19:52:03.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc230400640 1 -- 172.21.15.29:0/2136014469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc22814e740 con 0x7fc2280ca7f0 2024-04-17T19:52:03.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.307+0000 7fc230400640 1 -- 172.21.15.29:0/2136014469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc228069b90 con 0x7fc2280ca7f0 2024-04-17T19:52:03.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.311+0000 7fc225600640 1 -- 172.21.15.29:0/2136014469 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc2140b43f0 con 0x7fc2280ca7f0 2024-04-17T19:52:03.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.311+0000 7fc225600640 1 --2- 172.21.15.29:0/2136014469 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc2080683b0 0x7fc20806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:03.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.311+0000 7fc225600640 1 -- 172.21.15.29:0/2136014469 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6877+0+0 (secure 0 0 0) 0x7fc2141219a0 con 0x7fc2280ca7f0 2024-04-17T19:52:03.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.311+0000 7fc22da00640 1 --2- 172.21.15.29:0/2136014469 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc2080683b0 0x7fc20806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:03.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.311+0000 7fc22da00640 1 --2- 172.21.15.29:0/2136014469 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc2080683b0 0x7fc20806a8c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc21c009580 tx=0x7fc21c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:03.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.315+0000 7fc225600640 1 -- 172.21.15.29:0/2136014469 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc2140c4060 con 0x7fc2280ca7f0 2024-04-17T19:52:03.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.515+0000 7fc230400640 1 -- 172.21.15.29:0/2136014469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc228163dd0 con 0x7fc2280ca7f0 2024-04-17T19:52:03.508 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.515+0000 7fc225600640 1 -- 172.21.15.29:0/2136014469 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6904 (secure 0 0 0) 0x7fc214016150 con 0x7fc2280ca7f0 2024-04-17T19:52:03.509 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:03.509 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5212,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:34.757701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:active","state_seq":47,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229,4223]},"id":1}]} 2024-04-17T19:52:03.509 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2024-04-17T19:52:03.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.515+0000 7fc230400640 1 -- 172.21.15.29:0/2136014469 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc2080683b0 msgr2=0x7fc20806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:03.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.515+0000 7fc230400640 1 --2- 172.21.15.29:0/2136014469 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc2080683b0 0x7fc20806a8c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc21c009580 tx=0x7fc21c009190 comp rx=0 tx=0).stop 2024-04-17T19:52:03.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.515+0000 7fc230400640 1 -- 172.21.15.29:0/2136014469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2280ca7f0 msgr2=0x7fc22814e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:03.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.515+0000 7fc230400640 1 --2- 172.21.15.29:0/2136014469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2280ca7f0 0x7fc22814e200 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc2140ad040 tx=0x7fc21401ada0 comp rx=0 tx=0).stop 2024-04-17T19:52:03.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.515+0000 7fc230400640 1 -- 172.21.15.29:0/2136014469 shutdown_connections 2024-04-17T19:52:03.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.515+0000 7fc230400640 1 --2- 172.21.15.29:0/2136014469 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc2080683b0 0x7fc20806a8c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:03.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.515+0000 7fc230400640 1 --2- 172.21.15.29:0/2136014469 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc228159c90 0x7fc22814fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:03.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.515+0000 7fc230400640 1 --2- 172.21.15.29:0/2136014469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2280ca7f0 0x7fc22814e200 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:03.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.515+0000 7fc230400640 1 --2- 172.21.15.29:0/2136014469 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc2280c74b0 0x7fc228148ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:03.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.515+0000 7fc230400640 1 -- 172.21.15.29:0/2136014469 >> 172.21.15.29:0/2136014469 conn(0x7fc22806fde0 msgr2=0x7fc2280701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:03.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.515+0000 7fc230400640 1 -- 172.21.15.29:0/2136014469 shutdown_connections 2024-04-17T19:52:03.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:03.515+0000 7fc230400640 1 -- 172.21.15.29:0/2136014469 wait complete. 2024-04-17T19:52:03.531 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-17T19:52:05.532 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:05.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.703+0000 7f0f14e00640 1 Processor -- start 2024-04-17T19:52:05.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.703+0000 7f0f14e00640 1 -- start start 2024-04-17T19:52:05.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.707+0000 7f0f14e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0f100cc8c0 0x7f0f100cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:05.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.707+0000 7f0f14e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f101597a0 0x7f0f101635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:05.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.707+0000 7f0f14e00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f100c88b0 con 0x7f0f10159390 2024-04-17T19:52:05.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.707+0000 7f0f14e00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f0f100c6ce0 con 0x7f0f101597a0 2024-04-17T19:52:05.699 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.707+0000 7f0f14e00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f0f100c6fc0 con 0x7f0f100cc8c0 2024-04-17T19:52:05.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.707+0000 7f0f0e800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f101597a0 0x7f0f101635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:05.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.707+0000 7f0f0e800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f101597a0 0x7f0f101635e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57550/0 (socket says 172.21.15.29:57550) 2024-04-17T19:52:05.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.707+0000 7f0f0e800640 1 -- 172.21.15.29:0/2855770782 learned_addr learned my addr 172.21.15.29:0/2855770782 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:05.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.707+0000 7f0f0de00640 1 --2- 172.21.15.29:0/2855770782 >> v2:172.21.15.37:3300/0 conn(0x7f0f100cc8c0 0x7f0f100cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.707+0000 7f0f0e800640 1 -- 172.21.15.29:0/2855770782 >> v2:172.21.15.37:3300/0 conn(0x7f0f100cc8c0 msgr2=0x7f0f100cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.707+0000 7f0f0e800640 1 --2- 172.21.15.29:0/2855770782 >> v2:172.21.15.37:3300/0 conn(0x7f0f100cc8c0 0x7f0f100cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.707+0000 7f0f0e800640 1 -- 172.21.15.29:0/2855770782 >> v1:172.21.15.37:6789/0 conn(0x7f0f10159390 legacy=0x7f0f100ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:05.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.707+0000 7f0f0e800640 1 -- 172.21.15.29:0/2855770782 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f100c72a0 con 0x7f0f101597a0 2024-04-17T19:52:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.711+0000 7f0f0e800640 1 --2- 172.21.15.29:0/2855770782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f101597a0 0x7f0f101635e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0f100721b0 tx=0x7f0f0801c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea3cc252cd19b04 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.711+0000 7f0f0d400640 1 -- 172.21.15.29:0/2855770782 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f080a5670 con 0x7f0f101597a0 2024-04-17T19:52:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.711+0000 7f0f0d400640 1 -- 172.21.15.29:0/2855770782 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f080a5810 con 0x7f0f101597a0 2024-04-17T19:52:05.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.711+0000 7f0f0d400640 1 -- 172.21.15.29:0/2855770782 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f080a5be0 con 0x7f0f101597a0 2024-04-17T19:52:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.711+0000 7f0f14e00640 1 -- 172.21.15.29:0/2855770782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f101597a0 msgr2=0x7f0f101635e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.711+0000 7f0f14e00640 1 --2- 172.21.15.29:0/2855770782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f101597a0 0x7f0f101635e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f0f100721b0 tx=0x7f0f0801c9b0 comp rx=0 tx=0).stop 2024-04-17T19:52:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.711+0000 7f0f14e00640 1 -- 172.21.15.29:0/2855770782 shutdown_connections 2024-04-17T19:52:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.711+0000 7f0f14e00640 1 --2- 172.21.15.29:0/2855770782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f101597a0 0x7f0f101635e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.711+0000 7f0f14e00640 1 --2- 172.21.15.29:0/2855770782 >> v2:172.21.15.37:3300/0 conn(0x7f0f100cc8c0 0x7f0f100cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.711+0000 7f0f14e00640 1 -- 172.21.15.29:0/2855770782 >> 172.21.15.29:0/2855770782 conn(0x7f0f1006fde0 msgr2=0x7f0f100701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.711+0000 7f0f14e00640 1 -- 172.21.15.29:0/2855770782 shutdown_connections 2024-04-17T19:52:05.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.711+0000 7f0f14e00640 1 -- 172.21.15.29:0/2855770782 wait complete. 2024-04-17T19:52:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.711+0000 7f0f14e00640 1 Processor -- start 2024-04-17T19:52:05.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.711+0000 7f0f14e00640 1 -- start start 2024-04-17T19:52:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f14e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f100cc8c0 0x7f0f1015b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f14e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f10159390 0x7f0f101f6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f14e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f101597a0 0x7f0f10140d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f14e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0f100cabc0 con 0x7f0f100cc8c0 2024-04-17T19:52:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f14e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0f100caa40 con 0x7f0f10159390 2024-04-17T19:52:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f14e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0f100c6880 con 0x7f0f101597a0 2024-04-17T19:52:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f0de00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f100cc8c0 0x7f0f1015b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f0de00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f100cc8c0 0x7f0f1015b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57554/0 (socket says 172.21.15.29:57554) 2024-04-17T19:52:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f0de00640 1 -- 172.21.15.29:0/1152368622 learned_addr learned my addr 172.21.15.29:0/1152368622 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:05.709 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f07e00640 1 --2- 172.21.15.29:0/1152368622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f10159390 0x7f0f101f6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f0e800640 1 --2- 172.21.15.29:0/1152368622 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f101597a0 0x7f0f10140d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f0de00640 1 -- 172.21.15.29:0/1152368622 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f101597a0 msgr2=0x7f0f10140d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f0de00640 1 --2- 172.21.15.29:0/1152368622 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f101597a0 0x7f0f10140d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f0de00640 1 -- 172.21.15.29:0/1152368622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f10159390 msgr2=0x7f0f101f6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f0de00640 1 --2- 172.21.15.29:0/1152368622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f10159390 0x7f0f101f6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f0de00640 1 -- 172.21.15.29:0/1152368622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f100ca830 con 0x7f0f100cc8c0 2024-04-17T19:52:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f0de00640 1 --2- 172.21.15.29:0/1152368622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f100cc8c0 0x7f0f1015b020 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0f0000d620 tx=0x7f0f0000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f06000640 1 -- 172.21.15.29:0/1152368622 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0000b620 con 0x7f0f100cc8c0 2024-04-17T19:52:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f06000640 1 -- 172.21.15.29:0/1152368622 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f00019070 con 0x7f0f100cc8c0 2024-04-17T19:52:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f06000640 1 -- 172.21.15.29:0/1152368622 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0000b9e0 con 0x7f0f100cc8c0 2024-04-17T19:52:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f14e00640 1 -- 172.21.15.29:0/1152368622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f101f66d0 con 0x7f0f100cc8c0 2024-04-17T19:52:05.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f14e00640 1 -- 172.21.15.29:0/1152368622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f101f68e0 con 0x7f0f100cc8c0 2024-04-17T19:52:05.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.715+0000 7f0f14e00640 1 -- 172.21.15.29:0/1152368622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f10069b90 con 0x7f0f100cc8c0 2024-04-17T19:52:05.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.719+0000 7f0f06000640 1 -- 172.21.15.29:0/1152368622 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0f0000bb80 con 0x7f0f100cc8c0 2024-04-17T19:52:05.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.719+0000 7f0f06000640 1 --2- 172.21.15.29:0/1152368622 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0eec068380 0x7f0eec06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:05.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.719+0000 7f0f06000640 1 -- 172.21.15.29:0/1152368622 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6877+0+0 (secure 0 0 0) 0x7f0f00087350 con 0x7f0f100cc8c0 2024-04-17T19:52:05.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.719+0000 7f0f07e00640 1 --2- 172.21.15.29:0/1152368622 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0eec068380 0x7f0eec06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:05.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.719+0000 7f0f07e00640 1 --2- 172.21.15.29:0/1152368622 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0eec068380 0x7f0eec06a890 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0f10163670 tx=0x7f0ef8009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:05.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.723+0000 7f0f06000640 1 -- 172.21.15.29:0/1152368622 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0f00057350 con 0x7f0f100cc8c0 2024-04-17T19:52:05.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.927+0000 7f0f14e00640 1 -- 172.21.15.29:0/1152368622 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f100c5ac0 con 0x7f0f100cc8c0 2024-04-17T19:52:05.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.927+0000 7f0f06000640 1 -- 172.21.15.29:0/1152368622 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6904 (secure 0 0 0) 0x7f0f00056c50 con 0x7f0f100cc8c0 2024-04-17T19:52:05.923 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:05.923 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5212,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:34.757701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:active","state_seq":47,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229,4223]},"id":1}]} 2024-04-17T19:52:05.923 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2024-04-17T19:52:05.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.931+0000 7f0f14e00640 1 -- 172.21.15.29:0/1152368622 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0eec068380 msgr2=0x7f0eec06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:05.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.931+0000 7f0f14e00640 1 --2- 172.21.15.29:0/1152368622 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0eec068380 0x7f0eec06a890 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0f10163670 tx=0x7f0ef8009060 comp rx=0 tx=0).stop 2024-04-17T19:52:05.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.931+0000 7f0f14e00640 1 -- 172.21.15.29:0/1152368622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f100cc8c0 msgr2=0x7f0f1015b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:05.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.931+0000 7f0f14e00640 1 --2- 172.21.15.29:0/1152368622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f100cc8c0 0x7f0f1015b020 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0f0000d620 tx=0x7f0f0000dae0 comp rx=0 tx=0).stop 2024-04-17T19:52:05.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.931+0000 7f0f14e00640 1 -- 172.21.15.29:0/1152368622 shutdown_connections 2024-04-17T19:52:05.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.931+0000 7f0f14e00640 1 --2- 172.21.15.29:0/1152368622 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0eec068380 0x7f0eec06a890 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:05.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.931+0000 7f0f14e00640 1 --2- 172.21.15.29:0/1152368622 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0f101597a0 0x7f0f10140d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:05.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.931+0000 7f0f14e00640 1 --2- 172.21.15.29:0/1152368622 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0f10159390 0x7f0f101f6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:05.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.931+0000 7f0f14e00640 1 --2- 172.21.15.29:0/1152368622 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f100cc8c0 0x7f0f1015b020 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:05.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.931+0000 7f0f14e00640 1 -- 172.21.15.29:0/1152368622 >> 172.21.15.29:0/1152368622 conn(0x7f0f1006fde0 msgr2=0x7f0f1013ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:05.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.931+0000 7f0f14e00640 1 -- 172.21.15.29:0/1152368622 shutdown_connections 2024-04-17T19:52:05.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:05.931+0000 7f0f14e00640 1 -- 172.21.15.29:0/1152368622 wait complete. 2024-04-17T19:52:05.945 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-17T19:52:07.946 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:08.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.139+0000 7f95efe00640 1 Processor -- start 2024-04-17T19:52:08.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.139+0000 7f95efe00640 1 -- start start 2024-04-17T19:52:08.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.139+0000 7f95efe00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f95f0152d90 0x7f95f00c2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:08.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.139+0000 7f95efe00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f95f015c830 0x7f95f01634b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:08.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.139+0000 7f95efe00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95f00c95e0 con 0x7f95f00c3360 2024-04-17T19:52:08.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.139+0000 7f95efe00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f95f0071e30 con 0x7f95f0152d90 2024-04-17T19:52:08.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.139+0000 7f95efe00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f95f015ce00 con 0x7f95f015c830 2024-04-17T19:52:08.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.139+0000 7f95eea00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f95f0152d90 0x7f95f00c2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.139+0000 7f95ee000640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f95f00c3360 0x7f95f015c120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41324/0 (socket says 172.21.15.29:41324) 2024-04-17T19:52:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.139+0000 7f95ee000640 1 -- 172.21.15.29:0/2903934054 learned_addr learned my addr 172.21.15.29:0/2903934054 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:52:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.139+0000 7f95ef400640 1 --2- 172.21.15.29:0/2903934054 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f95f015c830 0x7f95f01634b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.139+0000 7f95ed600640 1 -- 172.21.15.29:0/2903934054 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2591612538 0 0) 0x7f95f00c95e0 con 0x7f95f00c3360 2024-04-17T19:52:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.139+0000 7f95ed600640 1 -- 172.21.15.29:0/2903934054 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95d4003530 con 0x7f95f00c3360 2024-04-17T19:52:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.143+0000 7f95eea00640 1 -- 172.21.15.29:0/2903934054 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f95f015c830 msgr2=0x7f95f01634b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.143+0000 7f95eea00640 1 --2- 172.21.15.29:0/2903934054 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f95f015c830 0x7f95f01634b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.143+0000 7f95eea00640 1 -- 172.21.15.29:0/2903934054 >> v1:172.21.15.29:6789/0 conn(0x7f95f00c3360 legacy=0x7f95f015c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.143+0000 7f95eea00640 1 -- 172.21.15.29:0/2903934054 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f0073eb0 con 0x7f95f0152d90 2024-04-17T19:52:08.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.143+0000 7f95eea00640 1 --2- 172.21.15.29:0/2903934054 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95f0152d90 0x7f95f00c2e20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f95e00097c0 tx=0x7f95e001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9693fc0440bf8ccd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:08.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.147+0000 7f95ed600640 1 -- 172.21.15.29:0/2903934054 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e00ab680 con 0x7f95f0152d90 2024-04-17T19:52:08.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.147+0000 7f95ed600640 1 -- 172.21.15.29:0/2903934054 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95e00b0070 con 0x7f95f0152d90 2024-04-17T19:52:08.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.147+0000 7f95ed600640 1 -- 172.21.15.29:0/2903934054 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e00aba20 con 0x7f95f0152d90 2024-04-17T19:52:08.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.147+0000 7f95efe00640 1 -- 172.21.15.29:0/2903934054 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95f0152d90 msgr2=0x7f95f00c2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:08.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.147+0000 7f95efe00640 1 --2- 172.21.15.29:0/2903934054 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95f0152d90 0x7f95f00c2e20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f95e00097c0 tx=0x7f95e001c4a0 comp rx=0 tx=0).stop 2024-04-17T19:52:08.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.147+0000 7f95efe00640 1 -- 172.21.15.29:0/2903934054 shutdown_connections 2024-04-17T19:52:08.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.147+0000 7f95efe00640 1 --2- 172.21.15.29:0/2903934054 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f95f015c830 0x7f95f01634b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:08.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.147+0000 7f95efe00640 1 --2- 172.21.15.29:0/2903934054 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95f0152d90 0x7f95f00c2e20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:08.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.147+0000 7f95efe00640 1 -- 172.21.15.29:0/2903934054 >> 172.21.15.29:0/2903934054 conn(0x7f95f00ccf80 msgr2=0x7f95f00cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:08.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.147+0000 7f95efe00640 1 -- 172.21.15.29:0/2903934054 shutdown_connections 2024-04-17T19:52:08.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.147+0000 7f95efe00640 1 -- 172.21.15.29:0/2903934054 wait complete. 2024-04-17T19:52:08.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.147+0000 7f95efe00640 1 Processor -- start 2024-04-17T19:52:08.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.147+0000 7f95efe00640 1 -- start start 2024-04-17T19:52:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95efe00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f95f00c3360 0x7f95f01f5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95efe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95f0152d90 0x7f95f01f8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95efe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95f015c830 0x7f95f00c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95efe00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f95f006d390 con 0x7f95f015c830 2024-04-17T19:52:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95efe00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f95f006d210 con 0x7f95f0152d90 2024-04-17T19:52:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95efe00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f95f00c1fa0 con 0x7f95f00c3360 2024-04-17T19:52:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95ef400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95f015c830 0x7f95f00c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95ef400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95f015c830 0x7f95f00c59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57562/0 (socket says 172.21.15.29:57562) 2024-04-17T19:52:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95ef400640 1 -- 172.21.15.29:0/4283345923 learned_addr learned my addr 172.21.15.29:0/4283345923 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95ee000640 1 --2- 172.21.15.29:0/4283345923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95f0152d90 0x7f95f01f8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95eea00640 1 --2- 172.21.15.29:0/4283345923 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f95f00c3360 0x7f95f01f5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95ef400640 1 -- 172.21.15.29:0/4283345923 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f95f00c3360 msgr2=0x7f95f01f5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95ef400640 1 --2- 172.21.15.29:0/4283345923 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f95f00c3360 0x7f95f01f5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95ef400640 1 -- 172.21.15.29:0/4283345923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95f0152d90 msgr2=0x7f95f01f8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95ef400640 1 --2- 172.21.15.29:0/4283345923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95f0152d90 0x7f95f01f8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95ef400640 1 -- 172.21.15.29:0/4283345923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f00c5f30 con 0x7f95f015c830 2024-04-17T19:52:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95ee000640 1 --2- 172.21.15.29:0/4283345923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95f0152d90 0x7f95f01f8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:52:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95ef400640 1 --2- 172.21.15.29:0/4283345923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95f015c830 0x7f95f00c59f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f95e8009cb0 tx=0x7f95e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95e7400640 1 -- 172.21.15.29:0/4283345923 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e8018020 con 0x7f95f015c830 2024-04-17T19:52:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95e7400640 1 -- 172.21.15.29:0/4283345923 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95e8014480 con 0x7f95f015c830 2024-04-17T19:52:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95efe00640 1 -- 172.21.15.29:0/4283345923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95f00c67e0 con 0x7f95f015c830 2024-04-17T19:52:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95e7400640 1 -- 172.21.15.29:0/4283345923 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e8014870 con 0x7f95f015c830 2024-04-17T19:52:08.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95efe00640 1 -- 172.21.15.29:0/4283345923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95f01f4e90 con 0x7f95f015c830 2024-04-17T19:52:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.151+0000 7f95efe00640 1 -- 172.21.15.29:0/4283345923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95f004ff10 con 0x7f95f015c830 2024-04-17T19:52:08.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.155+0000 7f95e7400640 1 -- 172.21.15.29:0/4283345923 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f95e801b020 con 0x7f95f015c830 2024-04-17T19:52:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.155+0000 7f95e7400640 1 --2- 172.21.15.29:0/4283345923 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f95d00683b0 0x7f95d006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.155+0000 7f95e7400640 1 -- 172.21.15.29:0/4283345923 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6877+0+0 (secure 0 0 0) 0x7f95e8087c00 con 0x7f95f015c830 2024-04-17T19:52:08.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.155+0000 7f95eea00640 1 --2- 172.21.15.29:0/4283345923 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f95d00683b0 0x7f95d006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:08.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.155+0000 7f95eea00640 1 --2- 172.21.15.29:0/4283345923 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f95d00683b0 0x7f95d006a8c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f95e00af000 tx=0x7f95e00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:08.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.159+0000 7f95e7400640 1 -- 172.21.15.29:0/4283345923 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f95e8022020 con 0x7f95f015c830 2024-04-17T19:52:08.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.359+0000 7f95efe00640 1 -- 172.21.15.29:0/4283345923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95f006d880 con 0x7f95f015c830 2024-04-17T19:52:08.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.359+0000 7f95e7400640 1 -- 172.21.15.29:0/4283345923 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6904 (secure 0 0 0) 0x7f95e8010040 con 0x7f95f015c830 2024-04-17T19:52:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:08.352 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5212,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:34.757701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:active","state_seq":47,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229,4223]},"id":1}]} 2024-04-17T19:52:08.352 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2024-04-17T19:52:08.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.359+0000 7f95efe00640 1 -- 172.21.15.29:0/4283345923 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f95d00683b0 msgr2=0x7f95d006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:08.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.359+0000 7f95efe00640 1 --2- 172.21.15.29:0/4283345923 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f95d00683b0 0x7f95d006a8c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f95e00af000 tx=0x7f95e00a7000 comp rx=0 tx=0).stop 2024-04-17T19:52:08.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.359+0000 7f95efe00640 1 -- 172.21.15.29:0/4283345923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95f015c830 msgr2=0x7f95f00c59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:08.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.359+0000 7f95efe00640 1 --2- 172.21.15.29:0/4283345923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95f015c830 0x7f95f00c59f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f95e8009cb0 tx=0x7f95e800c490 comp rx=0 tx=0).stop 2024-04-17T19:52:08.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.359+0000 7f95efe00640 1 -- 172.21.15.29:0/4283345923 shutdown_connections 2024-04-17T19:52:08.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.359+0000 7f95efe00640 1 --2- 172.21.15.29:0/4283345923 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f95d00683b0 0x7f95d006a8c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:08.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.359+0000 7f95efe00640 1 --2- 172.21.15.29:0/4283345923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95f015c830 0x7f95f00c59f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:08.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.359+0000 7f95efe00640 1 --2- 172.21.15.29:0/4283345923 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f95f0152d90 0x7f95f01f8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:08.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.359+0000 7f95efe00640 1 --2- 172.21.15.29:0/4283345923 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f95f00c3360 0x7f95f01f5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:08.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.359+0000 7f95efe00640 1 -- 172.21.15.29:0/4283345923 >> 172.21.15.29:0/4283345923 conn(0x7f95f00ccf80 msgr2=0x7f95f0151550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:08.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.359+0000 7f95efe00640 1 -- 172.21.15.29:0/4283345923 shutdown_connections 2024-04-17T19:52:08.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:08.359+0000 7f95efe00640 1 -- 172.21.15.29:0/4283345923 wait complete. 2024-04-17T19:52:08.375 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-17T19:52:10.376 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:10.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.547+0000 7f9f39c00640 1 Processor -- start 2024-04-17T19:52:10.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.547+0000 7f9f39c00640 1 -- start start 2024-04-17T19:52:10.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.551+0000 7f9f39c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9f340cc8c0 0x7f9f340cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:10.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.551+0000 7f9f39c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f34159390 0x7f9f341597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:10.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.551+0000 7f9f39c00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9f340ca330 0x7f9f340ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:10.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.551+0000 7f9f39c00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f9f340c88b0 con 0x7f9f340cc8c0 2024-04-17T19:52:10.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.551+0000 7f9f39c00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9f340c6ce0 con 0x7f9f340ca330 2024-04-17T19:52:10.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.551+0000 7f9f39c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9f340c6fc0 con 0x7f9f34159390 2024-04-17T19:52:10.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.551+0000 7f9f32a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9f340cc8c0 0x7f9f340cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:10.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.551+0000 7f9f32a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9f340cc8c0 0x7f9f340cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47424/0 (socket says 172.21.15.29:47424) 2024-04-17T19:52:10.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.551+0000 7f9f32a00640 1 -- 172.21.15.29:0/687476099 learned_addr learned my addr 172.21.15.29:0/687476099 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:10.544 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.551+0000 7f9f33400640 1 --2- 172.21.15.29:0/687476099 >> v2:172.21.15.37:3300/0 conn(0x7f9f340ca330 0x7f9f340ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:10.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.551+0000 7f9f2a000640 1 --2- 172.21.15.29:0/687476099 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f34159390 0x7f9f341597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:10.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.551+0000 7f9f32a00640 1 -- 172.21.15.29:0/687476099 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f34159390 msgr2=0x7f9f341597a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:10.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.551+0000 7f9f32a00640 1 --2- 172.21.15.29:0/687476099 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f34159390 0x7f9f341597a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:10.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.551+0000 7f9f32a00640 1 -- 172.21.15.29:0/687476099 >> v2:172.21.15.37:3300/0 conn(0x7f9f340ca330 msgr2=0x7f9f340ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:10.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.551+0000 7f9f32a00640 1 --2- 172.21.15.29:0/687476099 >> v2:172.21.15.37:3300/0 conn(0x7f9f340ca330 0x7f9f340ca740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:10.545 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.551+0000 7f9f32a00640 1 -- 172.21.15.29:0/687476099 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f340c72a0 con 0x7f9f340cc8c0 2024-04-17T19:52:10.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.555+0000 7f9f32a00640 1 --2- 172.21.15.29:0/687476099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f340cc8c0 0x7f9f340cccd0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9f34071e60 tx=0x7f9f2401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6be9087ef0b728a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:10.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.555+0000 7f9f32000640 1 -- 172.21.15.29:0/687476099 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f240b0070 con 0x7f9f340cc8c0 2024-04-17T19:52:10.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.555+0000 7f9f32000640 1 -- 172.21.15.29:0/687476099 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f240ab600 con 0x7f9f340cc8c0 2024-04-17T19:52:10.547 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.555+0000 7f9f32000640 1 -- 172.21.15.29:0/687476099 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f240ab9d0 con 0x7f9f340cc8c0 2024-04-17T19:52:10.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.555+0000 7f9f39c00640 1 -- 172.21.15.29:0/687476099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f340cc8c0 msgr2=0x7f9f340cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:10.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.555+0000 7f9f39c00640 1 --2- 172.21.15.29:0/687476099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f340cc8c0 0x7f9f340cccd0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9f34071e60 tx=0x7f9f2401c520 comp rx=0 tx=0).stop 2024-04-17T19:52:10.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.555+0000 7f9f39c00640 1 -- 172.21.15.29:0/687476099 shutdown_connections 2024-04-17T19:52:10.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.555+0000 7f9f39c00640 1 --2- 172.21.15.29:0/687476099 >> v2:172.21.15.37:3300/0 conn(0x7f9f340ca330 0x7f9f340ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:10.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.555+0000 7f9f39c00640 1 --2- 172.21.15.29:0/687476099 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f34159390 0x7f9f341597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:10.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.555+0000 7f9f39c00640 1 --2- 172.21.15.29:0/687476099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f340cc8c0 0x7f9f340cccd0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:10.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.555+0000 7f9f39c00640 1 -- 172.21.15.29:0/687476099 >> 172.21.15.29:0/687476099 conn(0x7f9f3406fde0 msgr2=0x7f9f340701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:10.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.555+0000 7f9f39c00640 1 -- 172.21.15.29:0/687476099 shutdown_connections 2024-04-17T19:52:10.549 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.555+0000 7f9f39c00640 1 -- 172.21.15.29:0/687476099 wait complete. 2024-04-17T19:52:10.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.555+0000 7f9f39c00640 1 Processor -- start 2024-04-17T19:52:10.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.555+0000 7f9f39c00640 1 -- start start 2024-04-17T19:52:10.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f39c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f340ca330 0x7f9f341365c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:10.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f39c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f340cc8c0 0x7f9f34141310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:10.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f39c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f34159390 0x7f9f34141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:10.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f39c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9f3415ca60 con 0x7f9f34159390 2024-04-17T19:52:10.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f39c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9f340c6a90 con 0x7f9f340cc8c0 2024-04-17T19:52:10.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f39c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9f3415cbe0 con 0x7f9f340ca330 2024-04-17T19:52:10.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f33400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f34159390 0x7f9f34141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:10.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f33400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f34159390 0x7f9f34141850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47436/0 (socket says 172.21.15.29:47436) 2024-04-17T19:52:10.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f33400640 1 -- 172.21.15.29:0/3287097898 learned_addr learned my addr 172.21.15.29:0/3287097898 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:10.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f2a000640 1 --2- 172.21.15.29:0/3287097898 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f340cc8c0 0x7f9f34141310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:10.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f32a00640 1 --2- 172.21.15.29:0/3287097898 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f340ca330 0x7f9f341365c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:10.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f33400640 1 -- 172.21.15.29:0/3287097898 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f340ca330 msgr2=0x7f9f341365c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:10.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f33400640 1 --2- 172.21.15.29:0/3287097898 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f340ca330 0x7f9f341365c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:10.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f33400640 1 -- 172.21.15.29:0/3287097898 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f340cc8c0 msgr2=0x7f9f34141310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:10.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f33400640 1 --2- 172.21.15.29:0/3287097898 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f340cc8c0 0x7f9f34141310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:10.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f33400640 1 -- 172.21.15.29:0/3287097898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f341f5f00 con 0x7f9f34159390 2024-04-17T19:52:10.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f32a00640 1 --2- 172.21.15.29:0/3287097898 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f340ca330 0x7f9f341365c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:52:10.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f33400640 1 --2- 172.21.15.29:0/3287097898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f34159390 0x7f9f34141850 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c007d10 tx=0x7f9f2c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:10.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f2b400640 1 -- 172.21.15.29:0/3287097898 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c014420 con 0x7f9f34159390 2024-04-17T19:52:10.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f2b400640 1 -- 172.21.15.29:0/3287097898 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f2c019070 con 0x7f9f34159390 2024-04-17T19:52:10.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f2b400640 1 -- 172.21.15.29:0/3287097898 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c014800 con 0x7f9f34159390 2024-04-17T19:52:10.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f39c00640 1 -- 172.21.15.29:0/3287097898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f3413fc60 con 0x7f9f34159390 2024-04-17T19:52:10.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f39c00640 1 -- 172.21.15.29:0/3287097898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f341f5420 con 0x7f9f34159390 2024-04-17T19:52:10.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.559+0000 7f9f39c00640 1 -- 172.21.15.29:0/3287097898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f34069b90 con 0x7f9f34159390 2024-04-17T19:52:10.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.563+0000 7f9f2b400640 1 -- 172.21.15.29:0/3287097898 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9f2c0149a0 con 0x7f9f34159390 2024-04-17T19:52:10.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.563+0000 7f9f2b400640 1 --2- 172.21.15.29:0/3287097898 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9f100683d0 0x7f9f1006a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:10.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.563+0000 7f9f2b400640 1 -- 172.21.15.29:0/3287097898 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6877+0+0 (secure 0 0 0) 0x7f9f2c087590 con 0x7f9f34159390 2024-04-17T19:52:10.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.563+0000 7f9f32a00640 1 --2- 172.21.15.29:0/3287097898 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9f100683d0 0x7f9f1006a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:10.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.563+0000 7f9f32a00640 1 --2- 172.21.15.29:0/3287097898 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9f100683d0 0x7f9f1006a8e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9f240ad000 tx=0x7f9f24008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:10.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.567+0000 7f9f2b400640 1 -- 172.21.15.29:0/3287097898 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9f2c0576a0 con 0x7f9f34159390 2024-04-17T19:52:10.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.767+0000 7f9f39c00640 1 -- 172.21.15.29:0/3287097898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9f3413f9f0 con 0x7f9f34159390 2024-04-17T19:52:10.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.767+0000 7f9f2b400640 1 -- 172.21.15.29:0/3287097898 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6904 (secure 0 0 0) 0x7f9f2c056fa0 con 0x7f9f34159390 2024-04-17T19:52:10.760 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:10.760 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5212,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:34.757701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:active","state_seq":47,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229,4223]},"id":1}]} 2024-04-17T19:52:10.761 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2024-04-17T19:52:10.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.767+0000 7f9f39c00640 1 -- 172.21.15.29:0/3287097898 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9f100683d0 msgr2=0x7f9f1006a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:10.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.767+0000 7f9f39c00640 1 --2- 172.21.15.29:0/3287097898 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9f100683d0 0x7f9f1006a8e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9f240ad000 tx=0x7f9f24008040 comp rx=0 tx=0).stop 2024-04-17T19:52:10.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.767+0000 7f9f39c00640 1 -- 172.21.15.29:0/3287097898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f34159390 msgr2=0x7f9f34141850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:10.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.767+0000 7f9f39c00640 1 --2- 172.21.15.29:0/3287097898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f34159390 0x7f9f34141850 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c007d10 tx=0x7f9f2c00e4b0 comp rx=0 tx=0).stop 2024-04-17T19:52:10.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.767+0000 7f9f39c00640 1 -- 172.21.15.29:0/3287097898 shutdown_connections 2024-04-17T19:52:10.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.767+0000 7f9f39c00640 1 --2- 172.21.15.29:0/3287097898 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9f100683d0 0x7f9f1006a8e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:10.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.767+0000 7f9f39c00640 1 --2- 172.21.15.29:0/3287097898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f34159390 0x7f9f34141850 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:10.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.767+0000 7f9f39c00640 1 --2- 172.21.15.29:0/3287097898 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9f340cc8c0 0x7f9f34141310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:10.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.767+0000 7f9f39c00640 1 --2- 172.21.15.29:0/3287097898 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9f340ca330 0x7f9f341365c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:10.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.767+0000 7f9f39c00640 1 -- 172.21.15.29:0/3287097898 >> 172.21.15.29:0/3287097898 conn(0x7f9f3406fde0 msgr2=0x7f9f3406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:10.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.767+0000 7f9f39c00640 1 -- 172.21.15.29:0/3287097898 shutdown_connections 2024-04-17T19:52:10.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:10.767+0000 7f9f39c00640 1 -- 172.21.15.29:0/3287097898 wait complete. 2024-04-17T19:52:10.782 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-17T19:52:12.784 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.959+0000 7fce6e000640 1 Processor -- start 2024-04-17T19:52:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.959+0000 7fce6e000640 1 -- start start 2024-04-17T19:52:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.959+0000 7fce6e000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce68159c90 0x7fce680ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.959+0000 7fce6e000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fce680ca7f0 0x7fce680c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.959+0000 7fce6e000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce6815a0a0 con 0x7fce680c74b0 2024-04-17T19:52:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.959+0000 7fce6e000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fce6815abe0 con 0x7fce680ca7f0 2024-04-17T19:52:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.959+0000 7fce6e000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fce6815a340 con 0x7fce68159c90 2024-04-17T19:52:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.959+0000 7fce67400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce68159c90 0x7fce680ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.959+0000 7fce5ea00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fce680ca7f0 0x7fce680c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.959+0000 7fce67e00640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fce680c74b0 0x7fce6806c660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:60834/0 (socket says 172.21.15.29:60834) 2024-04-17T19:52:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.959+0000 7fce67e00640 1 -- 172.21.15.29:0/3542502053 learned_addr learned my addr 172.21.15.29:0/3542502053 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:52:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.963+0000 7fce66a00640 1 -- 172.21.15.29:0/3542502053 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2189915994 0 0) 0x7fce6815a0a0 con 0x7fce680c74b0 2024-04-17T19:52:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.963+0000 7fce66a00640 1 -- 172.21.15.29:0/3542502053 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce4c003530 con 0x7fce680c74b0 2024-04-17T19:52:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.963+0000 7fce67400640 1 -- 172.21.15.29:0/3542502053 >> v2:172.21.15.37:3300/0 conn(0x7fce680ca7f0 msgr2=0x7fce680c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.963+0000 7fce67400640 1 --2- 172.21.15.29:0/3542502053 >> v2:172.21.15.37:3300/0 conn(0x7fce680ca7f0 0x7fce680c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.963+0000 7fce67400640 1 -- 172.21.15.29:0/3542502053 >> v1:172.21.15.37:6789/0 conn(0x7fce680c74b0 legacy=0x7fce6806c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.963+0000 7fce67400640 1 -- 172.21.15.29:0/3542502053 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce6815a620 con 0x7fce68159c90 2024-04-17T19:52:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.963+0000 7fce5ea00640 1 --2- 172.21.15.29:0/3542502053 >> v2:172.21.15.37:3300/0 conn(0x7fce680ca7f0 0x7fce680c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:52:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.963+0000 7fce67400640 1 --2- 172.21.15.29:0/3542502053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce68159c90 0x7fce680ca2b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fce580097c0 tx=0x7fce5801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fff09462177d24a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.963+0000 7fce66a00640 1 -- 172.21.15.29:0/3542502053 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce580b1070 con 0x7fce68159c90 2024-04-17T19:52:12.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.967+0000 7fce66a00640 1 -- 172.21.15.29:0/3542502053 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce580ac410 con 0x7fce68159c90 2024-04-17T19:52:12.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.967+0000 7fce66a00640 1 -- 172.21.15.29:0/3542502053 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce580ac7c0 con 0x7fce68159c90 2024-04-17T19:52:12.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.967+0000 7fce6e000640 1 -- 172.21.15.29:0/3542502053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce68159c90 msgr2=0x7fce680ca2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:12.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.967+0000 7fce6e000640 1 --2- 172.21.15.29:0/3542502053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce68159c90 0x7fce680ca2b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fce580097c0 tx=0x7fce5801c4a0 comp rx=0 tx=0).stop 2024-04-17T19:52:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.967+0000 7fce6e000640 1 -- 172.21.15.29:0/3542502053 shutdown_connections 2024-04-17T19:52:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.967+0000 7fce6e000640 1 --2- 172.21.15.29:0/3542502053 >> v2:172.21.15.37:3300/0 conn(0x7fce680ca7f0 0x7fce680c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.967+0000 7fce6e000640 1 --2- 172.21.15.29:0/3542502053 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce68159c90 0x7fce680ca2b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.967+0000 7fce6e000640 1 -- 172.21.15.29:0/3542502053 >> 172.21.15.29:0/3542502053 conn(0x7fce6806fde0 msgr2=0x7fce680701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.967+0000 7fce6e000640 1 -- 172.21.15.29:0/3542502053 shutdown_connections 2024-04-17T19:52:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.967+0000 7fce6e000640 1 -- 172.21.15.29:0/3542502053 wait complete. 2024-04-17T19:52:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.967+0000 7fce6e000640 1 Processor -- start 2024-04-17T19:52:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.967+0000 7fce6e000640 1 -- start start 2024-04-17T19:52:12.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce6e000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce680c74b0 0x7fce6814bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:12.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce6e000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680ca7f0 0x7fce680c4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:12.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce6e000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce68159c90 0x7fce6814e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:12.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce6e000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fce68166fb0 con 0x7fce680c74b0 2024-04-17T19:52:12.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce6e000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fce68166e30 con 0x7fce68159c90 2024-04-17T19:52:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce6e000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fce68167130 con 0x7fce680ca7f0 2024-04-17T19:52:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce67400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce680c74b0 0x7fce6814bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce67400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce680c74b0 0x7fce6814bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47444/0 (socket says 172.21.15.29:47444) 2024-04-17T19:52:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce67400640 1 -- 172.21.15.29:0/1765173861 learned_addr learned my addr 172.21.15.29:0/1765173861 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce5ea00640 1 --2- 172.21.15.29:0/1765173861 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680ca7f0 0x7fce680c4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce67e00640 1 --2- 172.21.15.29:0/1765173861 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce68159c90 0x7fce6814e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce67400640 1 -- 172.21.15.29:0/1765173861 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680ca7f0 msgr2=0x7fce680c4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce67400640 1 --2- 172.21.15.29:0/1765173861 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680ca7f0 0x7fce680c4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce67400640 1 -- 172.21.15.29:0/1765173861 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce68159c90 msgr2=0x7fce6814e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce67400640 1 --2- 172.21.15.29:0/1765173861 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce68159c90 0x7fce6814e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce67400640 1 -- 172.21.15.29:0/1765173861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce680c5020 con 0x7fce680c74b0 2024-04-17T19:52:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce67400640 1 --2- 172.21.15.29:0/1765173861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce680c74b0 0x7fce6814bae0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fce58000c10 tx=0x7fce58019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:12.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce5fe00640 1 -- 172.21.15.29:0/1765173861 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce580bb070 con 0x7fce680c74b0 2024-04-17T19:52:12.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce5fe00640 1 -- 172.21.15.29:0/1765173861 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce58003620 con 0x7fce680c74b0 2024-04-17T19:52:12.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce5fe00640 1 -- 172.21.15.29:0/1765173861 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce580b1040 con 0x7fce680c74b0 2024-04-17T19:52:12.966 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce6e000640 1 -- 172.21.15.29:0/1765173861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce680c5eb0 con 0x7fce680c74b0 2024-04-17T19:52:12.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce6e000640 1 -- 172.21.15.29:0/1765173861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce680c6170 con 0x7fce680c74b0 2024-04-17T19:52:12.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.971+0000 7fce6e000640 1 -- 172.21.15.29:0/1765173861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce68069b90 con 0x7fce680c74b0 2024-04-17T19:52:12.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.975+0000 7fce5fe00640 1 -- 172.21.15.29:0/1765173861 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fce580039d0 con 0x7fce680c74b0 2024-04-17T19:52:12.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.975+0000 7fce5fe00640 1 --2- 172.21.15.29:0/1765173861 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fce44068380 0x7fce4406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:12.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.975+0000 7fce5fe00640 1 -- 172.21.15.29:0/1765173861 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6877+0+0 (secure 0 0 0) 0x7fce5811fb70 con 0x7fce680c74b0 2024-04-17T19:52:12.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.975+0000 7fce5ea00640 1 --2- 172.21.15.29:0/1765173861 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fce44068380 0x7fce4406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:12.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.975+0000 7fce5ea00640 1 --2- 172.21.15.29:0/1765173861 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fce44068380 0x7fce4406a890 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fce680cb7c0 tx=0x7fce50009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:12.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:12.979+0000 7fce5fe00640 1 -- 172.21.15.29:0/1765173861 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fce58016090 con 0x7fce680c74b0 2024-04-17T19:52:13.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:13.179+0000 7fce6e000640 1 -- 172.21.15.29:0/1765173861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce68074300 con 0x7fce680c74b0 2024-04-17T19:52:13.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:13.179+0000 7fce5fe00640 1 -- 172.21.15.29:0/1765173861 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6904 (secure 0 0 0) 0x7fce580ef5b0 con 0x7fce680c74b0 2024-04-17T19:52:13.173 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:13.174 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5212,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:50:34.757701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":4223},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4223":{"gid":4223,"name":"d","rank":4,"incarnation":20,"state":"up:active","state_seq":47,"addr":"172.21.15.37:6837/274798483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":274798483},{"type":"v1","addr":"172.21.15.37:6837","nonce":274798483}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229,4223]},"id":1}]} 2024-04-17T19:52:13.174 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 25 2024-04-17T19:52:13.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:13.183+0000 7fce6e000640 1 -- 172.21.15.29:0/1765173861 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fce44068380 msgr2=0x7fce4406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:13.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:13.183+0000 7fce6e000640 1 --2- 172.21.15.29:0/1765173861 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fce44068380 0x7fce4406a890 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fce680cb7c0 tx=0x7fce50009190 comp rx=0 tx=0).stop 2024-04-17T19:52:13.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:13.183+0000 7fce6e000640 1 -- 172.21.15.29:0/1765173861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce680c74b0 msgr2=0x7fce6814bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:13.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:13.183+0000 7fce6e000640 1 --2- 172.21.15.29:0/1765173861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce680c74b0 0x7fce6814bae0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fce58000c10 tx=0x7fce58019240 comp rx=0 tx=0).stop 2024-04-17T19:52:13.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:13.183+0000 7fce6e000640 1 -- 172.21.15.29:0/1765173861 shutdown_connections 2024-04-17T19:52:13.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:13.183+0000 7fce6e000640 1 --2- 172.21.15.29:0/1765173861 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fce44068380 0x7fce4406a890 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:13.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:13.183+0000 7fce6e000640 1 --2- 172.21.15.29:0/1765173861 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fce68159c90 0x7fce6814e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:13.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:13.183+0000 7fce6e000640 1 --2- 172.21.15.29:0/1765173861 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fce680ca7f0 0x7fce680c4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:13.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:13.183+0000 7fce6e000640 1 --2- 172.21.15.29:0/1765173861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce680c74b0 0x7fce6814bae0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:13.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:13.183+0000 7fce6e000640 1 -- 172.21.15.29:0/1765173861 >> 172.21.15.29:0/1765173861 conn(0x7fce6806fde0 msgr2=0x7fce68158e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:13.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:13.183+0000 7fce6e000640 1 -- 172.21.15.29:0/1765173861 shutdown_connections 2024-04-17T19:52:13.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:13.183+0000 7fce6e000640 1 -- 172.21.15.29:0/1765173861 wait complete. 2024-04-17T19:52:13.196 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-17T19:52:15.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:15.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7fedabc00640 1 Processor -- start 2024-04-17T19:52:15.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7fedabc00640 1 -- start start 2024-04-17T19:52:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7fedabc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feda41563a0 0x7feda415c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7fedabc00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7feda40cba40 0x7feda4166980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7fedabc00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feda406c080 con 0x7feda40cb630 2024-04-17T19:52:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7fedabc00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7feda40ca700 con 0x7feda40cba40 2024-04-17T19:52:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7fedabc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7feda40ca9e0 con 0x7feda41563a0 2024-04-17T19:52:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7feda3e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7feda40cb630 0x7feda40cc2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59254/0 (socket says 172.21.15.29:59254) 2024-04-17T19:52:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7feda3e00640 1 -- 172.21.15.29:0/1804960957 learned_addr learned my addr 172.21.15.29:0/1804960957 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:52:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7feda9c00640 1 --2- 172.21.15.29:0/1804960957 >> v2:172.21.15.37:3300/0 conn(0x7feda40cba40 0x7feda4166980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7feda9200640 1 --2- 172.21.15.29:0/1804960957 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feda41563a0 0x7feda415c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:15.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7feda3400640 1 -- 172.21.15.29:0/1804960957 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4160366982 0 0) 0x7feda406c080 con 0x7feda40cb630 2024-04-17T19:52:15.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7feda3400640 1 -- 172.21.15.29:0/1804960957 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed8c003530 con 0x7feda40cb630 2024-04-17T19:52:15.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7feda3400640 1 -- 172.21.15.29:0/1804960957 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3983878844 0 0) 0x7fed8c003530 con 0x7feda40cb630 2024-04-17T19:52:15.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7feda9200640 1 -- 172.21.15.29:0/1804960957 >> v2:172.21.15.37:3300/0 conn(0x7feda40cba40 msgr2=0x7feda4166980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:15.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7feda9200640 1 --2- 172.21.15.29:0/1804960957 >> v2:172.21.15.37:3300/0 conn(0x7feda40cba40 0x7feda4166980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7feda9200640 1 -- 172.21.15.29:0/1804960957 >> v1:172.21.15.29:6789/0 conn(0x7feda40cb630 legacy=0x7feda40cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:15.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7feda9200640 1 -- 172.21.15.29:0/1804960957 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feda415f650 con 0x7feda41563a0 2024-04-17T19:52:15.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.359+0000 7feda9c00640 1 --2- 172.21.15.29:0/1804960957 >> v2:172.21.15.37:3300/0 conn(0x7feda40cba40 0x7feda4166980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:52:15.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.363+0000 7feda9200640 1 --2- 172.21.15.29:0/1804960957 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feda41563a0 0x7feda415c0e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fed980097c0 tx=0x7fed9801ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=491c7cf7b63017a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:15.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.363+0000 7feda3400640 1 -- 172.21.15.29:0/1804960957 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed980b5070 con 0x7feda41563a0 2024-04-17T19:52:15.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.363+0000 7feda3400640 1 -- 172.21.15.29:0/1804960957 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed980a85f0 con 0x7feda41563a0 2024-04-17T19:52:15.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.363+0000 7feda3400640 1 -- 172.21.15.29:0/1804960957 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed90001680 con 0x7feda41563a0 2024-04-17T19:52:15.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.363+0000 7fedabc00640 1 -- 172.21.15.29:0/1804960957 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feda41563a0 msgr2=0x7feda415c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:15.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.363+0000 7fedabc00640 1 --2- 172.21.15.29:0/1804960957 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feda41563a0 0x7feda415c0e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fed980097c0 tx=0x7fed9801ebb0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.363+0000 7fedabc00640 1 -- 172.21.15.29:0/1804960957 shutdown_connections 2024-04-17T19:52:15.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.363+0000 7fedabc00640 1 --2- 172.21.15.29:0/1804960957 >> v2:172.21.15.37:3300/0 conn(0x7feda40cba40 0x7feda4166980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.363+0000 7fedabc00640 1 --2- 172.21.15.29:0/1804960957 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feda41563a0 0x7feda415c0e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.363+0000 7fedabc00640 1 -- 172.21.15.29:0/1804960957 >> 172.21.15.29:0/1804960957 conn(0x7feda406e200 msgr2=0x7feda406e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:15.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.367+0000 7fedabc00640 1 -- 172.21.15.29:0/1804960957 shutdown_connections 2024-04-17T19:52:15.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.367+0000 7fedabc00640 1 -- 172.21.15.29:0/1804960957 wait complete. 2024-04-17T19:52:15.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.367+0000 7fedabc00640 1 Processor -- start 2024-04-17T19:52:15.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.367+0000 7fedabc00640 1 -- start start 2024-04-17T19:52:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.367+0000 7fedabc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feda40cb630 0x7feda41423d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.367+0000 7fedabc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda40cba40 0x7feda4147c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.367+0000 7fedabc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feda41563a0 0x7feda4153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.367+0000 7fedabc00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feda415fef0 con 0x7feda40cba40 2024-04-17T19:52:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.367+0000 7fedabc00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7feda415fc10 con 0x7feda40cb630 2024-04-17T19:52:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.367+0000 7fedabc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7feda41567b0 con 0x7feda41563a0 2024-04-17T19:52:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.367+0000 7feda3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda40cba40 0x7feda4147c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.367+0000 7feda3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda40cba40 0x7feda4147c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47456/0 (socket says 172.21.15.29:47456) 2024-04-17T19:52:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.367+0000 7feda3e00640 1 -- 172.21.15.29:0/3664050780 learned_addr learned my addr 172.21.15.29:0/3664050780 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.371+0000 7feda9200640 1 --2- 172.21.15.29:0/3664050780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feda40cb630 0x7feda41423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.371+0000 7feda9c00640 1 --2- 172.21.15.29:0/3664050780 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feda41563a0 0x7feda4153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:15.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.371+0000 7feda3e00640 1 -- 172.21.15.29:0/3664050780 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feda41563a0 msgr2=0x7feda4153210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:15.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.371+0000 7feda3e00640 1 --2- 172.21.15.29:0/3664050780 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feda41563a0 0x7feda4153210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.371+0000 7feda3e00640 1 -- 172.21.15.29:0/3664050780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feda40cb630 msgr2=0x7feda41423d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:15.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.371+0000 7feda3e00640 1 --2- 172.21.15.29:0/3664050780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feda40cb630 0x7feda41423d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.371+0000 7feda3e00640 1 -- 172.21.15.29:0/3664050780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feda4148180 con 0x7feda40cba40 2024-04-17T19:52:15.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.371+0000 7feda9c00640 1 --2- 172.21.15.29:0/3664050780 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feda41563a0 0x7feda4153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:52:15.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.371+0000 7feda3e00640 1 --2- 172.21.15.29:0/3664050780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda40cba40 0x7feda4147c40 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fed900017f0 tx=0x7fed9000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:15.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.371+0000 7feda0c00640 1 -- 172.21.15.29:0/3664050780 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed90005ae0 con 0x7feda40cba40 2024-04-17T19:52:15.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.371+0000 7feda0c00640 1 -- 172.21.15.29:0/3664050780 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed90005ca0 con 0x7feda40cba40 2024-04-17T19:52:15.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.371+0000 7fedabc00640 1 -- 172.21.15.29:0/3664050780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feda4146920 con 0x7feda40cba40 2024-04-17T19:52:15.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.371+0000 7feda0c00640 1 -- 172.21.15.29:0/3664050780 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed900072b0 con 0x7feda40cba40 2024-04-17T19:52:15.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.371+0000 7fedabc00640 1 -- 172.21.15.29:0/3664050780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feda415eef0 con 0x7feda40cba40 2024-04-17T19:52:15.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.371+0000 7fedabc00640 1 -- 172.21.15.29:0/3664050780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feda4069b90 con 0x7feda40cba40 2024-04-17T19:52:15.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.371+0000 7feda0c00640 1 -- 172.21.15.29:0/3664050780 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fed90007450 con 0x7feda40cba40 2024-04-17T19:52:15.366 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.371+0000 7feda0c00640 1 --2- 172.21.15.29:0/3664050780 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fed84068380 0x7fed8406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:15.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.375+0000 7feda0c00640 1 -- 172.21.15.29:0/3664050780 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6907+0+0 (secure 0 0 0) 0x7fed90086b60 con 0x7feda40cba40 2024-04-17T19:52:15.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.375+0000 7feda9200640 1 --2- 172.21.15.29:0/3664050780 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fed84068380 0x7fed8406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:15.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.375+0000 7feda9200640 1 --2- 172.21.15.29:0/3664050780 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fed84068380 0x7fed8406a890 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fed980b2000 tx=0x7fed980aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:15.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.379+0000 7feda0c00640 1 -- 172.21.15.29:0/3664050780 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fed90056c60 con 0x7feda40cba40 2024-04-17T19:52:15.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.579+0000 7fedabc00640 1 -- 172.21.15.29:0/3664050780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feda40cad20 con 0x7feda40cba40 2024-04-17T19:52:15.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.579+0000 7feda0c00640 1 -- 172.21.15.29:0/3664050780 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6077 (secure 0 0 0) 0x7fed90056560 con 0x7feda40cba40 2024-04-17T19:52:15.571 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:15.572 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:13.837746+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:52:15.572 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 26 2024-04-17T19:52:15.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.579+0000 7fedabc00640 1 -- 172.21.15.29:0/3664050780 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fed84068380 msgr2=0x7fed8406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:15.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.579+0000 7fedabc00640 1 --2- 172.21.15.29:0/3664050780 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fed84068380 0x7fed8406a890 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fed980b2000 tx=0x7fed980aa070 comp rx=0 tx=0).stop 2024-04-17T19:52:15.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.579+0000 7fedabc00640 1 -- 172.21.15.29:0/3664050780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda40cba40 msgr2=0x7feda4147c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:15.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.579+0000 7fedabc00640 1 --2- 172.21.15.29:0/3664050780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda40cba40 0x7feda4147c40 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fed900017f0 tx=0x7fed9000cbe0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.579+0000 7fedabc00640 1 -- 172.21.15.29:0/3664050780 shutdown_connections 2024-04-17T19:52:15.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.579+0000 7fedabc00640 1 --2- 172.21.15.29:0/3664050780 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fed84068380 0x7fed8406a890 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.579+0000 7fedabc00640 1 --2- 172.21.15.29:0/3664050780 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feda41563a0 0x7feda4153210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.579+0000 7fedabc00640 1 --2- 172.21.15.29:0/3664050780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feda40cba40 0x7feda4147c40 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.579+0000 7fedabc00640 1 --2- 172.21.15.29:0/3664050780 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feda40cb630 0x7feda41423d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.579+0000 7fedabc00640 1 -- 172.21.15.29:0/3664050780 >> 172.21.15.29:0/3664050780 conn(0x7feda406e200 msgr2=0x7feda4154fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:15.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.579+0000 7fedabc00640 1 -- 172.21.15.29:0/3664050780 shutdown_connections 2024-04-17T19:52:15.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.579+0000 7fedabc00640 1 -- 172.21.15.29:0/3664050780 wait complete. 2024-04-17T19:52:15.594 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2024-04-17T19:52:15.594 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-17T19:52:15.594 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:15.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.751+0000 7f54f8000640 1 Processor -- start 2024-04-17T19:52:15.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.751+0000 7f54f8000640 1 -- start start 2024-04-17T19:52:15.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.751+0000 7f54f8000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f54f00cc8c0 0x7f54f00cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:15.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.751+0000 7f54f8000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54f01597a0 0x7f54f01635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:15.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.751+0000 7f54f8000640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54f00c88b0 con 0x7f54f0159390 2024-04-17T19:52:15.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.751+0000 7f54f8000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f54f00c6ce0 con 0x7f54f01597a0 2024-04-17T19:52:15.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.751+0000 7f54f8000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f54f00c6fc0 con 0x7f54f00cc8c0 2024-04-17T19:52:15.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.751+0000 7f54f6000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54f01597a0 0x7f54f01635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:15.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.751+0000 7f54f4c00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f54f0159390 0x7f54f00ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59268/0 (socket says 172.21.15.29:59268) 2024-04-17T19:52:15.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.751+0000 7f54f4c00640 1 -- 172.21.15.29:0/3291323051 learned_addr learned my addr 172.21.15.29:0/3291323051 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:52:15.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.755+0000 7f54f5600640 1 --2- 172.21.15.29:0/3291323051 >> v2:172.21.15.37:3300/0 conn(0x7f54f00cc8c0 0x7f54f00cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:15.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.755+0000 7f54f6000640 1 -- 172.21.15.29:0/3291323051 >> v2:172.21.15.37:3300/0 conn(0x7f54f00cc8c0 msgr2=0x7f54f00cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:15.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.755+0000 7f54f6000640 1 --2- 172.21.15.29:0/3291323051 >> v2:172.21.15.37:3300/0 conn(0x7f54f00cc8c0 0x7f54f00cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.755+0000 7f54f6000640 1 -- 172.21.15.29:0/3291323051 >> v1:172.21.15.29:6789/0 conn(0x7f54f0159390 legacy=0x7f54f00ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:15.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.755+0000 7f54f6000640 1 -- 172.21.15.29:0/3291323051 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54f00c72a0 con 0x7f54f01597a0 2024-04-17T19:52:15.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.755+0000 7f54f5600640 1 --2- 172.21.15.29:0/3291323051 >> v2:172.21.15.37:3300/0 conn(0x7f54f00cc8c0 0x7f54f00cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:52:15.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.755+0000 7f54f6000640 1 --2- 172.21.15.29:0/3291323051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f01597a0 0x7f54f01635e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f54f00721b0 tx=0x7f54e801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b66b3a879a08c8f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:15.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.755+0000 7f54efe00640 1 -- 172.21.15.29:0/3291323051 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e80b2070 con 0x7f54f01597a0 2024-04-17T19:52:15.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.755+0000 7f54efe00640 1 -- 172.21.15.29:0/3291323051 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54e80a57f0 con 0x7f54f01597a0 2024-04-17T19:52:15.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.755+0000 7f54efe00640 1 -- 172.21.15.29:0/3291323051 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e80a5bc0 con 0x7f54f01597a0 2024-04-17T19:52:15.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.759+0000 7f54f8000640 1 -- 172.21.15.29:0/3291323051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f01597a0 msgr2=0x7f54f01635e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:15.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.759+0000 7f54f8000640 1 --2- 172.21.15.29:0/3291323051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f01597a0 0x7f54f01635e0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f54f00721b0 tx=0x7f54e801c930 comp rx=0 tx=0).stop 2024-04-17T19:52:15.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.759+0000 7f54f8000640 1 -- 172.21.15.29:0/3291323051 shutdown_connections 2024-04-17T19:52:15.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.759+0000 7f54f8000640 1 --2- 172.21.15.29:0/3291323051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f01597a0 0x7f54f01635e0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.759+0000 7f54f8000640 1 --2- 172.21.15.29:0/3291323051 >> v2:172.21.15.37:3300/0 conn(0x7f54f00cc8c0 0x7f54f00cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.759+0000 7f54f8000640 1 -- 172.21.15.29:0/3291323051 >> 172.21.15.29:0/3291323051 conn(0x7f54f006fde0 msgr2=0x7f54f00701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:15.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.759+0000 7f54f8000640 1 -- 172.21.15.29:0/3291323051 shutdown_connections 2024-04-17T19:52:15.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.759+0000 7f54f8000640 1 -- 172.21.15.29:0/3291323051 wait complete. 2024-04-17T19:52:15.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.759+0000 7f54f8000640 1 Processor -- start 2024-04-17T19:52:15.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.759+0000 7f54f8000640 1 -- start start 2024-04-17T19:52:15.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.759+0000 7f54f8000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f54f00cc8c0 0x7f54f015ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:15.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.759+0000 7f54f8000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f0159390 0x7f54f0072590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:15.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.759+0000 7f54f8000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f01597a0 0x7f54f0073750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:15.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.759+0000 7f54f8000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f54f00cabc0 con 0x7f54f01597a0 2024-04-17T19:52:15.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.759+0000 7f54f8000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f54f00caa40 con 0x7f54f0159390 2024-04-17T19:52:15.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.759+0000 7f54f8000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f54f00c6880 con 0x7f54f00cc8c0 2024-04-17T19:52:15.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54f6000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f01597a0 0x7f54f0073750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:15.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54f6000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f01597a0 0x7f54f0073750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47484/0 (socket says 172.21.15.29:47484) 2024-04-17T19:52:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54f6000640 1 -- 172.21.15.29:0/3909108010 learned_addr learned my addr 172.21.15.29:0/3909108010 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54f5600640 1 --2- 172.21.15.29:0/3909108010 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f54f00cc8c0 0x7f54f015ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54f4c00640 1 --2- 172.21.15.29:0/3909108010 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f0159390 0x7f54f0072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54f6000640 1 -- 172.21.15.29:0/3909108010 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f54f00cc8c0 msgr2=0x7f54f015ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54f6000640 1 --2- 172.21.15.29:0/3909108010 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f54f00cc8c0 0x7f54f015ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54f6000640 1 -- 172.21.15.29:0/3909108010 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f0159390 msgr2=0x7f54f0072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54f6000640 1 --2- 172.21.15.29:0/3909108010 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f0159390 0x7f54f0072590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54f6000640 1 -- 172.21.15.29:0/3909108010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54f0072ad0 con 0x7f54f01597a0 2024-04-17T19:52:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54f4c00640 1 --2- 172.21.15.29:0/3909108010 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f0159390 0x7f54f0072590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:52:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54f6000640 1 --2- 172.21.15.29:0/3909108010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f01597a0 0x7f54f0073750 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f54e8019540 tx=0x7f54e80196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54ed600640 1 -- 172.21.15.29:0/3909108010 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e80b2050 con 0x7f54f01597a0 2024-04-17T19:52:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54ed600640 1 -- 172.21.15.29:0/3909108010 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54e80be070 con 0x7f54f01597a0 2024-04-17T19:52:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54f8000640 1 -- 172.21.15.29:0/3909108010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54f0141710 con 0x7f54f01597a0 2024-04-17T19:52:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54ed600640 1 -- 172.21.15.29:0/3909108010 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e80ad870 con 0x7f54f01597a0 2024-04-17T19:52:15.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54f8000640 1 -- 172.21.15.29:0/3909108010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54f0163170 con 0x7f54f01597a0 2024-04-17T19:52:15.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54f8000640 1 -- 172.21.15.29:0/3909108010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54f0069b90 con 0x7f54f01597a0 2024-04-17T19:52:15.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.763+0000 7f54ed600640 1 -- 172.21.15.29:0/3909108010 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f54e80ada10 con 0x7f54f01597a0 2024-04-17T19:52:15.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.767+0000 7f54ed600640 1 --2- 172.21.15.29:0/3909108010 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54d00683b0 0x7f54d006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:15.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.767+0000 7f54ed600640 1 -- 172.21.15.29:0/3909108010 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6907+0+0 (secure 0 0 0) 0x7f54e8121e90 con 0x7f54f01597a0 2024-04-17T19:52:15.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.767+0000 7f54f5600640 1 --2- 172.21.15.29:0/3909108010 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54d00683b0 0x7f54d006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:15.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.767+0000 7f54f5600640 1 --2- 172.21.15.29:0/3909108010 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54d00683b0 0x7f54d006a8c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f54e40045f0 tx=0x7f54e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:15.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.771+0000 7f54ed600640 1 -- 172.21.15.29:0/3909108010 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f54e8016520 con 0x7f54f01597a0 2024-04-17T19:52:15.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.971+0000 7f54f8000640 1 -- 172.21.15.29:0/3909108010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54f00721b0 con 0x7f54f01597a0 2024-04-17T19:52:15.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.971+0000 7f54ed600640 1 -- 172.21.15.29:0/3909108010 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6077 (secure 0 0 0) 0x7f54e80f20b0 con 0x7f54f01597a0 2024-04-17T19:52:15.965 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:15.965 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:13.837746+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:52:15.965 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 26 2024-04-17T19:52:15.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.971+0000 7f54f8000640 1 -- 172.21.15.29:0/3909108010 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54d00683b0 msgr2=0x7f54d006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:15.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.971+0000 7f54f8000640 1 --2- 172.21.15.29:0/3909108010 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54d00683b0 0x7f54d006a8c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f54e40045f0 tx=0x7f54e4008040 comp rx=0 tx=0).stop 2024-04-17T19:52:15.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.971+0000 7f54f8000640 1 -- 172.21.15.29:0/3909108010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f01597a0 msgr2=0x7f54f0073750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:15.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.971+0000 7f54f8000640 1 --2- 172.21.15.29:0/3909108010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f01597a0 0x7f54f0073750 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f54e8019540 tx=0x7f54e80196d0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.971+0000 7f54f8000640 1 -- 172.21.15.29:0/3909108010 shutdown_connections 2024-04-17T19:52:15.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.971+0000 7f54f8000640 1 --2- 172.21.15.29:0/3909108010 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54d00683b0 0x7f54d006a8c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.971+0000 7f54f8000640 1 --2- 172.21.15.29:0/3909108010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f01597a0 0x7f54f0073750 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.971+0000 7f54f8000640 1 --2- 172.21.15.29:0/3909108010 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f54f0159390 0x7f54f0072590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.971+0000 7f54f8000640 1 --2- 172.21.15.29:0/3909108010 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f54f00cc8c0 0x7f54f015ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:15.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.971+0000 7f54f8000640 1 -- 172.21.15.29:0/3909108010 >> 172.21.15.29:0/3909108010 conn(0x7f54f006fde0 msgr2=0x7f54f00cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:15.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.971+0000 7f54f8000640 1 -- 172.21.15.29:0/3909108010 shutdown_connections 2024-04-17T19:52:15.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:15.975+0000 7f54f8000640 1 -- 172.21.15.29:0/3909108010 wait complete. 2024-04-17T19:52:15.987 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 26, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-17T19:47:49.315515+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 26, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4229": { "addr": "172.21.15.37:6839/1100598562", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6836", "nonce": 1100598562, "type": "v2" }, { "addr": "172.21.15.37:6839", "nonce": 1100598562, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4229, "incarnation": 16, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 13 }, "gid_4259": { "addr": "172.21.15.29:6836/4184160618", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6832", "nonce": 4184160618, "type": "v2" }, { "addr": "172.21.15.29:6836", "nonce": 4184160618, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4259, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 11 }, "gid_4302": { "addr": "172.21.15.29:6835/270402476", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6834", "nonce": 270402476, "type": "v2" }, { "addr": "172.21.15.29:6835", "nonce": 270402476, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 10, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 9 }, "gid_4305": { "addr": "172.21.15.29:6837/2691590158", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6833", "nonce": 2691590158, "type": "v2" }, { "addr": "172.21.15.29:6837", "nonce": 2691590158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4305, "incarnation": 5, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5212": { "addr": "172.21.15.37:6838/767681664", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6835", "nonce": 767681664, "type": "v2" }, { "addr": "172.21.15.37:6838", "nonce": 767681664, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5212, "incarnation": 26, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-17T19:52:13.837746+0000", "qdb_cluster": [ 4229, 4259, 4302, 4305 ], "qdb_leader": 4305, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4305, "mds_1": 4302, "mds_2": 4259, "mds_3": 4229, "mds_4": 5212 } } } ], "standbys": [] } 2024-04-17T19:52:17.988 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:18.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.159+0000 7fd7d4a00640 1 Processor -- start 2024-04-17T19:52:18.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.159+0000 7fd7d4a00640 1 -- start start 2024-04-17T19:52:18.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.159+0000 7fd7d4a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7d00710a0 0x7fd7d0156310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:18.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.159+0000 7fd7d4a00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d0156850 con 0x7fd7d015c780 2024-04-17T19:52:18.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.159+0000 7fd7d4a00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d015cfa0 con 0x7fd7d015cb90 2024-04-17T19:52:18.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.159+0000 7fd7d4a00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fd7d00caee0 con 0x7fd7d00710a0 2024-04-17T19:52:18.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.159+0000 7fd7cde00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7d00710a0 0x7fd7d0156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:18.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.159+0000 7fd7c7e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd7d015c780 0x7fd7d00cf6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59270/0 (socket says 172.21.15.29:59270) 2024-04-17T19:52:18.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.159+0000 7fd7c7e00640 1 -- 172.21.15.29:0/1674321236 learned_addr learned my addr 172.21.15.29:0/1674321236 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:52:18.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.159+0000 7fd7cde00640 1 -- 172.21.15.29:0/1674321236 >> v1:172.21.15.37:6789/0 conn(0x7fd7d015cb90 legacy=0x7fd7d0166910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:18.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.159+0000 7fd7cde00640 1 -- 172.21.15.29:0/1674321236 >> v1:172.21.15.29:6789/0 conn(0x7fd7d015c780 legacy=0x7fd7d00cf6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:18.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.159+0000 7fd7cde00640 1 -- 172.21.15.29:0/1674321236 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7d00cb130 con 0x7fd7d00710a0 2024-04-17T19:52:18.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.159+0000 7fd7cd400640 1 -- 172.21.15.29:0/1674321236 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2961031261 0 0) 0x7fd7d0156850 con 0x7fd7d015c780 2024-04-17T19:52:18.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.163+0000 7fd7cde00640 1 --2- 172.21.15.29:0/1674321236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d00710a0 0x7fd7d0156310 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd7c00097c0 tx=0x7fd7c001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba95900277efb4cd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:18.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.163+0000 7fd7cd400640 1 -- 172.21.15.29:0/1674321236 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c00b0070 con 0x7fd7d00710a0 2024-04-17T19:52:18.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.163+0000 7fd7cd400640 1 -- 172.21.15.29:0/1674321236 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7c00ab600 con 0x7fd7d00710a0 2024-04-17T19:52:18.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.163+0000 7fd7cd400640 1 -- 172.21.15.29:0/1674321236 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c00ab9d0 con 0x7fd7d00710a0 2024-04-17T19:52:18.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.163+0000 7fd7d4a00640 1 -- 172.21.15.29:0/1674321236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d00710a0 msgr2=0x7fd7d0156310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:18.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.163+0000 7fd7d4a00640 1 --2- 172.21.15.29:0/1674321236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d00710a0 0x7fd7d0156310 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd7c00097c0 tx=0x7fd7c001c4a0 comp rx=0 tx=0).stop 2024-04-17T19:52:18.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.163+0000 7fd7d4a00640 1 -- 172.21.15.29:0/1674321236 shutdown_connections 2024-04-17T19:52:18.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.163+0000 7fd7d4a00640 1 --2- 172.21.15.29:0/1674321236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d00710a0 0x7fd7d0156310 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:18.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.163+0000 7fd7d4a00640 1 -- 172.21.15.29:0/1674321236 >> 172.21.15.29:0/1674321236 conn(0x7fd7d006e1a0 msgr2=0x7fd7d006e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:18.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.163+0000 7fd7d4a00640 1 -- 172.21.15.29:0/1674321236 shutdown_connections 2024-04-17T19:52:18.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.163+0000 7fd7d4a00640 1 -- 172.21.15.29:0/1674321236 wait complete. 2024-04-17T19:52:18.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.167+0000 7fd7d4a00640 1 Processor -- start 2024-04-17T19:52:18.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.167+0000 7fd7d4a00640 1 -- start start 2024-04-17T19:52:18.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.167+0000 7fd7d4a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7d00710a0 0x7fd7d00c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:18.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.167+0000 7fd7d4a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d015c780 0x7fd7d0152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:18.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.167+0000 7fd7d4a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7d015cb90 0x7fd7d014c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:18.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.167+0000 7fd7d4a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd7d00714b0 con 0x7fd7d015c780 2024-04-17T19:52:18.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.167+0000 7fd7d4a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd7d015c2f0 con 0x7fd7d015cb90 2024-04-17T19:52:18.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.167+0000 7fd7d4a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd7d006c960 con 0x7fd7d00710a0 2024-04-17T19:52:18.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.167+0000 7fd7c7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d015c780 0x7fd7d0152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:18.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.167+0000 7fd7c7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d015c780 0x7fd7d0152af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47510/0 (socket says 172.21.15.29:47510) 2024-04-17T19:52:18.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.167+0000 7fd7c7e00640 1 -- 172.21.15.29:0/970951093 learned_addr learned my addr 172.21.15.29:0/970951093 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:18.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.167+0000 7fd7ce800640 1 --2- 172.21.15.29:0/970951093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7d015cb90 0x7fd7d014c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:18.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.167+0000 7fd7cde00640 1 --2- 172.21.15.29:0/970951093 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7d00710a0 0x7fd7d00c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:18.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.171+0000 7fd7c7e00640 1 -- 172.21.15.29:0/970951093 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7d00710a0 msgr2=0x7fd7d00c6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:18.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.171+0000 7fd7c7e00640 1 --2- 172.21.15.29:0/970951093 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7d00710a0 0x7fd7d00c6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:18.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.171+0000 7fd7c7e00640 1 -- 172.21.15.29:0/970951093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7d015cb90 msgr2=0x7fd7d014c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:18.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.171+0000 7fd7c7e00640 1 --2- 172.21.15.29:0/970951093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7d015cb90 0x7fd7d014c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:18.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.171+0000 7fd7c7e00640 1 -- 172.21.15.29:0/970951093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7d00c7290 con 0x7fd7d015c780 2024-04-17T19:52:18.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.171+0000 7fd7c7e00640 1 --2- 172.21.15.29:0/970951093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d015c780 0x7fd7d0152af0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800c440 tx=0x7fd7b800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:18.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.171+0000 7fd7c6000640 1 -- 172.21.15.29:0/970951093 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8018070 con 0x7fd7d015c780 2024-04-17T19:52:18.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.171+0000 7fd7c6000640 1 -- 172.21.15.29:0/970951093 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7b80074e0 con 0x7fd7d015c780 2024-04-17T19:52:18.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.171+0000 7fd7d4a00640 1 -- 172.21.15.29:0/970951093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7d00c8920 con 0x7fd7d015c780 2024-04-17T19:52:18.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.171+0000 7fd7c6000640 1 -- 172.21.15.29:0/970951093 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8004a30 con 0x7fd7d015c780 2024-04-17T19:52:18.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.171+0000 7fd7d4a00640 1 -- 172.21.15.29:0/970951093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7d00c9180 con 0x7fd7d015c780 2024-04-17T19:52:18.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.171+0000 7fd7d4a00640 1 -- 172.21.15.29:0/970951093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7d00c8c10 con 0x7fd7d015c780 2024-04-17T19:52:18.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.171+0000 7fd7c6000640 1 -- 172.21.15.29:0/970951093 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd7b801b070 con 0x7fd7d015c780 2024-04-17T19:52:18.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.171+0000 7fd7c6000640 1 --2- 172.21.15.29:0/970951093 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd7ac068380 0x7fd7ac06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:18.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.171+0000 7fd7c6000640 1 -- 172.21.15.29:0/970951093 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6907+0+0 (secure 0 0 0) 0x7fd7b8085fc0 con 0x7fd7d015c780 2024-04-17T19:52:18.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.175+0000 7fd7cde00640 1 --2- 172.21.15.29:0/970951093 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd7ac068380 0x7fd7ac06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:18.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.175+0000 7fd7cde00640 1 --2- 172.21.15.29:0/970951093 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd7ac068380 0x7fd7ac06a890 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd7c00ad000 tx=0x7fd7c00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:18.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.179+0000 7fd7c6000640 1 -- 172.21.15.29:0/970951093 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd7b8055a70 con 0x7fd7d015c780 2024-04-17T19:52:18.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.375+0000 7fd7d4a00640 1 -- 172.21.15.29:0/970951093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7d0069b90 con 0x7fd7d015c780 2024-04-17T19:52:18.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.375+0000 7fd7c6000640 1 -- 172.21.15.29:0/970951093 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+6077 (secure 0 0 0) 0x7fd7b8059d70 con 0x7fd7d015c780 2024-04-17T19:52:18.371 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:18.371 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:13.837746+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:replay","state_seq":1,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:52:18.371 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 26 2024-04-17T19:52:18.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.379+0000 7fd7d4a00640 1 -- 172.21.15.29:0/970951093 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd7ac068380 msgr2=0x7fd7ac06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:18.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.379+0000 7fd7d4a00640 1 --2- 172.21.15.29:0/970951093 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd7ac068380 0x7fd7ac06a890 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd7c00ad000 tx=0x7fd7c00a6000 comp rx=0 tx=0).stop 2024-04-17T19:52:18.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.379+0000 7fd7d4a00640 1 -- 172.21.15.29:0/970951093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d015c780 msgr2=0x7fd7d0152af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:18.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.379+0000 7fd7d4a00640 1 --2- 172.21.15.29:0/970951093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d015c780 0x7fd7d0152af0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800c440 tx=0x7fd7b800c900 comp rx=0 tx=0).stop 2024-04-17T19:52:18.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.379+0000 7fd7d4a00640 1 -- 172.21.15.29:0/970951093 shutdown_connections 2024-04-17T19:52:18.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.379+0000 7fd7d4a00640 1 --2- 172.21.15.29:0/970951093 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd7ac068380 0x7fd7ac06a890 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:18.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.379+0000 7fd7d4a00640 1 --2- 172.21.15.29:0/970951093 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd7d015cb90 0x7fd7d014c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.379+0000 7fd7d4a00640 1 --2- 172.21.15.29:0/970951093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7d015c780 0x7fd7d0152af0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.379+0000 7fd7d4a00640 1 --2- 172.21.15.29:0/970951093 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd7d00710a0 0x7fd7d00c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.379+0000 7fd7d4a00640 1 -- 172.21.15.29:0/970951093 >> 172.21.15.29:0/970951093 conn(0x7fd7d006e1a0 msgr2=0x7fd7d006bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.379+0000 7fd7d4a00640 1 -- 172.21.15.29:0/970951093 shutdown_connections 2024-04-17T19:52:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:18.379+0000 7fd7d4a00640 1 -- 172.21.15.29:0/970951093 wait complete. 2024-04-17T19:52:18.393 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:52:20.394 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:20.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.563+0000 7f617c800640 1 Processor -- start 2024-04-17T19:52:20.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.563+0000 7f617c800640 1 -- start start 2024-04-17T19:52:20.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.563+0000 7f617c800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f61740cccd0 0x7f61740ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:20.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.563+0000 7f617c800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f61740ca870 0x7f61741635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:20.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.563+0000 7f617c800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61740cd0e0 con 0x7f61740cc8c0 2024-04-17T19:52:20.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.563+0000 7f617c800640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f61740c88b0 con 0x7f61740ca870 2024-04-17T19:52:20.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.563+0000 7f617c800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f61740c6ce0 con 0x7f61740cccd0 2024-04-17T19:52:20.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.563+0000 7f617a800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f61740ca870 0x7f61741635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:20.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.563+0000 7f617a800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f61740ca870 0x7f61741635e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56994/0 (socket says 172.21.15.29:56994) 2024-04-17T19:52:20.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.563+0000 7f617a800640 1 -- 172.21.15.29:0/1195762782 learned_addr learned my addr 172.21.15.29:0/1195762782 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:20.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.563+0000 7f6179400640 1 --2- 172.21.15.29:0/1195762782 >> v2:172.21.15.37:3300/0 conn(0x7f61740cccd0 0x7f61740ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:20.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.563+0000 7f617a800640 1 -- 172.21.15.29:0/1195762782 >> v2:172.21.15.37:3300/0 conn(0x7f61740cccd0 msgr2=0x7f61740ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:20.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.563+0000 7f617a800640 1 --2- 172.21.15.29:0/1195762782 >> v2:172.21.15.37:3300/0 conn(0x7f61740cccd0 0x7f61740ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:20.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.563+0000 7f617a800640 1 -- 172.21.15.29:0/1195762782 >> v1:172.21.15.37:6789/0 conn(0x7f61740cc8c0 legacy=0x7f6174159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:20.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.563+0000 7f617a800640 1 -- 172.21.15.29:0/1195762782 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61740c6fc0 con 0x7f61740ca870 2024-04-17T19:52:20.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.567+0000 7f617a800640 1 --2- 172.21.15.29:0/1195762782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61740ca870 0x7f61741635e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f61740cc7e0 tx=0x7f617001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3ae891ba09f6989 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:20.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.567+0000 7f6178a00640 1 -- 172.21.15.29:0/1195762782 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61700a55c0 con 0x7f61740ca870 2024-04-17T19:52:20.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.567+0000 7f6178a00640 1 -- 172.21.15.29:0/1195762782 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61700a5760 con 0x7f61740ca870 2024-04-17T19:52:20.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.567+0000 7f6178a00640 1 -- 172.21.15.29:0/1195762782 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61700a5b30 con 0x7f61740ca870 2024-04-17T19:52:20.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.567+0000 7f617c800640 1 -- 172.21.15.29:0/1195762782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61740ca870 msgr2=0x7f61741635e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:20.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.567+0000 7f617c800640 1 --2- 172.21.15.29:0/1195762782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61740ca870 0x7f61741635e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f61740cc7e0 tx=0x7f617001c9f0 comp rx=0 tx=0).stop 2024-04-17T19:52:20.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.567+0000 7f617c800640 1 -- 172.21.15.29:0/1195762782 shutdown_connections 2024-04-17T19:52:20.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.567+0000 7f617c800640 1 --2- 172.21.15.29:0/1195762782 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61740ca870 0x7f61741635e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:20.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.571+0000 7f617c800640 1 --2- 172.21.15.29:0/1195762782 >> v2:172.21.15.37:3300/0 conn(0x7f61740cccd0 0x7f61740ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:20.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.571+0000 7f617c800640 1 -- 172.21.15.29:0/1195762782 >> 172.21.15.29:0/1195762782 conn(0x7f617406fde0 msgr2=0x7f61740701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:20.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.571+0000 7f617c800640 1 -- 172.21.15.29:0/1195762782 shutdown_connections 2024-04-17T19:52:20.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.571+0000 7f617c800640 1 -- 172.21.15.29:0/1195762782 wait complete. 2024-04-17T19:52:20.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.571+0000 7f617c800640 1 Processor -- start 2024-04-17T19:52:20.565 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.571+0000 7f617c800640 1 -- start start 2024-04-17T19:52:20.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.571+0000 7f617c800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61740ca870 0x7f61740738e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:20.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.571+0000 7f617c800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61740cc8c0 0x7f6174161960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:20.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.571+0000 7f617c800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61740cccd0 0x7f6174151a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:20.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.571+0000 7f617c800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6174159c20 con 0x7f61740cccd0 2024-04-17T19:52:20.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.571+0000 7f617c800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6174159aa0 con 0x7f61740ca870 2024-04-17T19:52:20.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.571+0000 7f617c800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f61740c6880 con 0x7f61740cc8c0 2024-04-17T19:52:20.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f617a800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61740cccd0 0x7f6174151a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:20.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f617a800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61740cccd0 0x7f6174151a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57000/0 (socket says 172.21.15.29:57000) 2024-04-17T19:52:20.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f617a800640 1 -- 172.21.15.29:0/1874113326 learned_addr learned my addr 172.21.15.29:0/1874113326 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:20.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f6179e00640 1 --2- 172.21.15.29:0/1874113326 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61740ca870 0x7f61740738e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:20.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f6179400640 1 --2- 172.21.15.29:0/1874113326 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61740cc8c0 0x7f6174161960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:20.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f617a800640 1 -- 172.21.15.29:0/1874113326 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61740cc8c0 msgr2=0x7f6174161960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:20.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f617a800640 1 --2- 172.21.15.29:0/1874113326 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61740cc8c0 0x7f6174161960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:20.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f617a800640 1 -- 172.21.15.29:0/1874113326 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61740ca870 msgr2=0x7f61740738e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:20.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f617a800640 1 --2- 172.21.15.29:0/1874113326 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61740ca870 0x7f61740738e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:20.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f617a800640 1 -- 172.21.15.29:0/1874113326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6174138ab0 con 0x7f61740cccd0 2024-04-17T19:52:20.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f6179400640 1 --2- 172.21.15.29:0/1874113326 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61740cc8c0 0x7f6174161960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:52:20.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f617a800640 1 --2- 172.21.15.29:0/1874113326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61740cccd0 0x7f6174151a40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f617001a620 tx=0x7f6170019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:20.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f616e000640 1 -- 172.21.15.29:0/1874113326 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170009510 con 0x7f61740cccd0 2024-04-17T19:52:20.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f616e000640 1 -- 172.21.15.29:0/1874113326 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61700be070 con 0x7f61740cccd0 2024-04-17T19:52:20.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f617c800640 1 -- 172.21.15.29:0/1874113326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6174073e20 con 0x7f61740cccd0 2024-04-17T19:52:20.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f616e000640 1 -- 172.21.15.29:0/1874113326 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61700b88d0 con 0x7f61740cccd0 2024-04-17T19:52:20.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f617c800640 1 -- 172.21.15.29:0/1874113326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61741387d0 con 0x7f61740cccd0 2024-04-17T19:52:20.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f617c800640 1 -- 172.21.15.29:0/1874113326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6174069b90 con 0x7f61740cccd0 2024-04-17T19:52:20.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f616e000640 1 -- 172.21.15.29:0/1874113326 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f61700b8a70 con 0x7f61740cccd0 2024-04-17T19:52:20.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.575+0000 7f616e000640 1 --2- 172.21.15.29:0/1874113326 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6158068380 0x7f615806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:20.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.579+0000 7f616e000640 1 -- 172.21.15.29:0/1874113326 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6907+0+0 (secure 0 0 0) 0x7f6170121d90 con 0x7f61740cccd0 2024-04-17T19:52:20.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.579+0000 7f6179e00640 1 --2- 172.21.15.29:0/1874113326 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6158068380 0x7f615806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:20.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.579+0000 7f6179e00640 1 --2- 172.21.15.29:0/1874113326 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6158068380 0x7f615806a890 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6168007fd0 tx=0x7f6168009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:20.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.583+0000 7f616e000640 1 -- 172.21.15.29:0/1874113326 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6174138ab0 con 0x7f61740cccd0 2024-04-17T19:52:20.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.783+0000 7f617c800640 1 -- 172.21.15.29:0/1874113326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f617415d060 con 0x7f61740cccd0 2024-04-17T19:52:20.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.783+0000 7f616e000640 1 -- 172.21.15.29:0/1874113326 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6079 (secure 0 0 0) 0x7f61700f6e20 con 0x7f61740cccd0 2024-04-17T19:52:20.777 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:20.777 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:18.854391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:resolve","state_seq":25,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229]},"id":1}]} 2024-04-17T19:52:20.778 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-04-17T19:52:20.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.787+0000 7f617c800640 1 -- 172.21.15.29:0/1874113326 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6158068380 msgr2=0x7f615806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:20.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.787+0000 7f617c800640 1 --2- 172.21.15.29:0/1874113326 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6158068380 0x7f615806a890 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6168007fd0 tx=0x7f6168009040 comp rx=0 tx=0).stop 2024-04-17T19:52:20.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.787+0000 7f617c800640 1 -- 172.21.15.29:0/1874113326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61740cccd0 msgr2=0x7f6174151a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:20.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.787+0000 7f617c800640 1 --2- 172.21.15.29:0/1874113326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61740cccd0 0x7f6174151a40 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f617001a620 tx=0x7f6170019790 comp rx=0 tx=0).stop 2024-04-17T19:52:20.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.787+0000 7f617c800640 1 -- 172.21.15.29:0/1874113326 shutdown_connections 2024-04-17T19:52:20.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.787+0000 7f617c800640 1 --2- 172.21.15.29:0/1874113326 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6158068380 0x7f615806a890 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:20.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.787+0000 7f617c800640 1 --2- 172.21.15.29:0/1874113326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61740cccd0 0x7f6174151a40 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:20.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.787+0000 7f617c800640 1 --2- 172.21.15.29:0/1874113326 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61740cc8c0 0x7f6174161960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:20.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.787+0000 7f617c800640 1 --2- 172.21.15.29:0/1874113326 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61740ca870 0x7f61740738e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:20.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.787+0000 7f617c800640 1 -- 172.21.15.29:0/1874113326 >> 172.21.15.29:0/1874113326 conn(0x7f617406fde0 msgr2=0x7f61741528a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:20.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.787+0000 7f617c800640 1 -- 172.21.15.29:0/1874113326 shutdown_connections 2024-04-17T19:52:20.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:20.787+0000 7f617c800640 1 -- 172.21.15.29:0/1874113326 wait complete. 2024-04-17T19:52:20.800 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:52:21.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:52:21.247 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:52:22.801 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:22.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.975+0000 7fab24e00640 1 Processor -- start 2024-04-17T19:52:22.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.975+0000 7fab24e00640 1 -- start start 2024-04-17T19:52:22.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.975+0000 7fab24e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab20156370 0x7fab200ca110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:22.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.975+0000 7fab24e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab2015c970 0x7fab200caff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:22.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.975+0000 7fab24e00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab2006c730 con 0x7fab200cabe0 2024-04-17T19:52:22.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.975+0000 7fab24e00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fab2006c9f0 con 0x7fab2015c970 2024-04-17T19:52:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.975+0000 7fab24e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fab2006ccd0 con 0x7fab20156370 2024-04-17T19:52:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.975+0000 7fab1fe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab2015c970 0x7fab200caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.975+0000 7fab1fe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab2015c970 0x7fab200caff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57016/0 (socket says 172.21.15.29:57016) 2024-04-17T19:52:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.975+0000 7fab1fe00640 1 -- 172.21.15.29:0/2693215937 learned_addr learned my addr 172.21.15.29:0/2693215937 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.975+0000 7fab1f400640 1 --2- 172.21.15.29:0/2693215937 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab20156370 0x7fab200ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.975+0000 7fab1fe00640 1 -- 172.21.15.29:0/2693215937 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab20156370 msgr2=0x7fab200ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.975+0000 7fab1fe00640 1 --2- 172.21.15.29:0/2693215937 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab20156370 0x7fab200ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:22.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.975+0000 7fab1fe00640 1 -- 172.21.15.29:0/2693215937 >> v1:172.21.15.37:6789/0 conn(0x7fab200cabe0 legacy=0x7fab2015c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:22.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.975+0000 7fab1fe00640 1 -- 172.21.15.29:0/2693215937 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab2006cfb0 con 0x7fab2015c970 2024-04-17T19:52:22.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.979+0000 7fab1fe00640 1 --2- 172.21.15.29:0/2693215937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab2015c970 0x7fab200caff0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fab18009e50 tx=0x7fab1801f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a93fed26a2fca73 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:22.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.979+0000 7fab1e000640 1 -- 172.21.15.29:0/2693215937 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab180b4070 con 0x7fab2015c970 2024-04-17T19:52:22.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.979+0000 7fab1e000640 1 -- 172.21.15.29:0/2693215937 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab180af400 con 0x7fab2015c970 2024-04-17T19:52:22.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.979+0000 7fab1e000640 1 -- 172.21.15.29:0/2693215937 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab180af7d0 con 0x7fab2015c970 2024-04-17T19:52:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.979+0000 7fab24e00640 1 -- 172.21.15.29:0/2693215937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab2015c970 msgr2=0x7fab200caff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.979+0000 7fab24e00640 1 --2- 172.21.15.29:0/2693215937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab2015c970 0x7fab200caff0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fab18009e50 tx=0x7fab1801f210 comp rx=0 tx=0).stop 2024-04-17T19:52:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.979+0000 7fab24e00640 1 -- 172.21.15.29:0/2693215937 shutdown_connections 2024-04-17T19:52:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.979+0000 7fab24e00640 1 --2- 172.21.15.29:0/2693215937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab2015c970 0x7fab200caff0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.979+0000 7fab24e00640 1 --2- 172.21.15.29:0/2693215937 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab20156370 0x7fab200ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.979+0000 7fab24e00640 1 -- 172.21.15.29:0/2693215937 >> 172.21.15.29:0/2693215937 conn(0x7fab200680e0 msgr2=0x7fab2006dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.979+0000 7fab24e00640 1 -- 172.21.15.29:0/2693215937 shutdown_connections 2024-04-17T19:52:22.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.979+0000 7fab24e00640 1 -- 172.21.15.29:0/2693215937 wait complete. 2024-04-17T19:52:22.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab24e00640 1 Processor -- start 2024-04-17T19:52:22.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab24e00640 1 -- start start 2024-04-17T19:52:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab24e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab200cabe0 0x7fab201f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab24e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab20156370 0x7fab20157de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab24e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab2015c970 0x7fab200b4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab24e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fab200ca650 con 0x7fab200cabe0 2024-04-17T19:52:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab24e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fab2006d570 con 0x7fab20156370 2024-04-17T19:52:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab24e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fab2006e7c0 con 0x7fab2015c970 2024-04-17T19:52:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab1f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab200cabe0 0x7fab201f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab1f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab200cabe0 0x7fab201f34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57018/0 (socket says 172.21.15.29:57018) 2024-04-17T19:52:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab1f400640 1 -- 172.21.15.29:0/1839609555 learned_addr learned my addr 172.21.15.29:0/1839609555 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab1fe00640 1 --2- 172.21.15.29:0/1839609555 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab2015c970 0x7fab200b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab1ea00640 1 --2- 172.21.15.29:0/1839609555 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab20156370 0x7fab20157de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab1f400640 1 -- 172.21.15.29:0/1839609555 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab2015c970 msgr2=0x7fab200b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab1f400640 1 --2- 172.21.15.29:0/1839609555 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab2015c970 0x7fab200b4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab1f400640 1 -- 172.21.15.29:0/1839609555 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab20156370 msgr2=0x7fab20157de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab1f400640 1 --2- 172.21.15.29:0/1839609555 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab20156370 0x7fab20157de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab1f400640 1 -- 172.21.15.29:0/1839609555 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab20157800 con 0x7fab200cabe0 2024-04-17T19:52:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab1f400640 1 --2- 172.21.15.29:0/1839609555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab200cabe0 0x7fab201f34e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fab1000e8a0 tx=0x7fab1000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.983+0000 7fab17400640 1 -- 172.21.15.29:0/1839609555 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab10017070 con 0x7fab200cabe0 2024-04-17T19:52:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.987+0000 7fab17400640 1 -- 172.21.15.29:0/1839609555 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab1000c940 con 0x7fab200cabe0 2024-04-17T19:52:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.987+0000 7fab24e00640 1 -- 172.21.15.29:0/1839609555 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab20158320 con 0x7fab200cabe0 2024-04-17T19:52:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.987+0000 7fab17400640 1 -- 172.21.15.29:0/1839609555 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1000ccf0 con 0x7fab200cabe0 2024-04-17T19:52:22.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.987+0000 7fab24e00640 1 -- 172.21.15.29:0/1839609555 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab20156bf0 con 0x7fab200cabe0 2024-04-17T19:52:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.987+0000 7fab24e00640 1 -- 172.21.15.29:0/1839609555 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab2004ff10 con 0x7fab200cabe0 2024-04-17T19:52:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.987+0000 7fab17400640 1 -- 172.21.15.29:0/1839609555 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fab1001a070 con 0x7fab200cabe0 2024-04-17T19:52:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.987+0000 7fab17400640 1 --2- 172.21.15.29:0/1839609555 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fab00068380 0x7fab0006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.987+0000 7fab17400640 1 -- 172.21.15.29:0/1839609555 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6907+0+0 (secure 0 0 0) 0x7fab10087220 con 0x7fab200cabe0 2024-04-17T19:52:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.987+0000 7fab1ea00640 1 --2- 172.21.15.29:0/1839609555 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fab00068380 0x7fab0006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:22.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.991+0000 7fab1ea00640 1 --2- 172.21.15.29:0/1839609555 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fab00068380 0x7fab0006a890 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fab20154970 tx=0x7fab0c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:22.991+0000 7fab17400640 1 -- 172.21.15.29:0/1839609555 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fab20157800 con 0x7fab200cabe0 2024-04-17T19:52:23.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:23.191+0000 7fab24e00640 1 -- 172.21.15.29:0/1839609555 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab2006dd80 con 0x7fab200cabe0 2024-04-17T19:52:23.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:23.191+0000 7fab17400640 1 -- 172.21.15.29:0/1839609555 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6079 (secure 0 0 0) 0x7fab10057200 con 0x7fab200cabe0 2024-04-17T19:52:23.186 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:23.186 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:18.854391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:resolve","state_seq":25,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229]},"id":1}]} 2024-04-17T19:52:23.186 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 27 2024-04-17T19:52:23.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:23.195+0000 7fab24e00640 1 -- 172.21.15.29:0/1839609555 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fab00068380 msgr2=0x7fab0006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:23.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:23.195+0000 7fab24e00640 1 --2- 172.21.15.29:0/1839609555 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fab00068380 0x7fab0006a890 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fab20154970 tx=0x7fab0c009040 comp rx=0 tx=0).stop 2024-04-17T19:52:23.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:23.195+0000 7fab24e00640 1 -- 172.21.15.29:0/1839609555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab200cabe0 msgr2=0x7fab201f34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:23.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:23.195+0000 7fab24e00640 1 --2- 172.21.15.29:0/1839609555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab200cabe0 0x7fab201f34e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fab1000e8a0 tx=0x7fab1000eca0 comp rx=0 tx=0).stop 2024-04-17T19:52:23.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:23.195+0000 7fab24e00640 1 -- 172.21.15.29:0/1839609555 shutdown_connections 2024-04-17T19:52:23.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:23.195+0000 7fab24e00640 1 --2- 172.21.15.29:0/1839609555 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fab00068380 0x7fab0006a890 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:23.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:23.195+0000 7fab24e00640 1 --2- 172.21.15.29:0/1839609555 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab2015c970 0x7fab200b4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:23.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:23.195+0000 7fab24e00640 1 --2- 172.21.15.29:0/1839609555 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab20156370 0x7fab20157de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:23.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:23.195+0000 7fab24e00640 1 --2- 172.21.15.29:0/1839609555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab200cabe0 0x7fab201f34e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:23.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:23.195+0000 7fab24e00640 1 -- 172.21.15.29:0/1839609555 >> 172.21.15.29:0/1839609555 conn(0x7fab200680e0 msgr2=0x7fab20075420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:23.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:23.195+0000 7fab24e00640 1 -- 172.21.15.29:0/1839609555 shutdown_connections 2024-04-17T19:52:23.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:23.195+0000 7fab24e00640 1 -- 172.21.15.29:0/1839609555 wait complete. 2024-04-17T19:52:23.208 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:52:25.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:25.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.367+0000 7f339e600640 1 Processor -- start 2024-04-17T19:52:25.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.367+0000 7f339e600640 1 -- start start 2024-04-17T19:52:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f339e600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33980cccd0 0x7f33980ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f339e600640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f33980ca870 0x7f33981635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f339e600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33980cd0e0 con 0x7f33980cc8c0 2024-04-17T19:52:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f339e600640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f33980c88b0 con 0x7f33980ca870 2024-04-17T19:52:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f339e600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f33980c6ce0 con 0x7f33980cccd0 2024-04-17T19:52:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3397400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f33980cc8c0 0x7f3398159390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48286/0 (socket says 172.21.15.29:48286) 2024-04-17T19:52:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3397400640 1 -- 172.21.15.29:0/2560811191 learned_addr learned my addr 172.21.15.29:0/2560811191 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:52:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f338ea00640 1 --2- 172.21.15.29:0/2560811191 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33980cccd0 0x7f33980ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3397e00640 1 --2- 172.21.15.29:0/2560811191 >> v2:172.21.15.37:3300/0 conn(0x7f33980ca870 0x7f33981635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3396a00640 1 -- 172.21.15.29:0/2560811191 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3488588534 0 0) 0x7f33980cd0e0 con 0x7f33980cc8c0 2024-04-17T19:52:25.365 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3396a00640 1 -- 172.21.15.29:0/2560811191 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f337c005740 con 0x7f33980cc8c0 2024-04-17T19:52:25.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3396a00640 1 -- 172.21.15.29:0/2560811191 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2105383377 0 0) 0x7f337c005740 con 0x7f33980cc8c0 2024-04-17T19:52:25.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3396a00640 1 -- 172.21.15.29:0/2560811191 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33980cccd0 msgr2=0x7f33980ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:25.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3396a00640 1 --2- 172.21.15.29:0/2560811191 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33980cccd0 0x7f33980ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:25.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3396a00640 1 -- 172.21.15.29:0/2560811191 >> v2:172.21.15.37:3300/0 conn(0x7f33980ca870 msgr2=0x7f33981635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:25.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3396a00640 1 --2- 172.21.15.29:0/2560811191 >> v2:172.21.15.37:3300/0 conn(0x7f33980ca870 0x7f33981635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:25.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3396a00640 1 -- 172.21.15.29:0/2560811191 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33980c6fc0 con 0x7f33980cc8c0 2024-04-17T19:52:25.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3396a00640 1 -- 172.21.15.29:0/2560811191 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f3388001680 con 0x7f33980cc8c0 2024-04-17T19:52:25.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3396a00640 1 -- 172.21.15.29:0/2560811191 >> v1:172.21.15.29:6789/0 conn(0x7f33980cc8c0 legacy=0x7f3398159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:25.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3396a00640 1 --2- 172.21.15.29:0/2560811191 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f337c005c40 0x7f337c0060b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:25.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3396a00640 1 --2- 172.21.15.29:0/2560811191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f337c006a20 0x7f337c006e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:25.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3396a00640 1 --2- 172.21.15.29:0/2560811191 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33980cccd0 0x7f337c007370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:25.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3396a00640 1 -- 172.21.15.29:0/2560811191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f33980c88b0 con 0x7f337c006a20 2024-04-17T19:52:25.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3396a00640 1 -- 172.21.15.29:0/2560811191 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f33980c6ce0 con 0x7f337c005c40 2024-04-17T19:52:25.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3396a00640 1 -- 172.21.15.29:0/2560811191 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f337c00dcc0 con 0x7f33980cccd0 2024-04-17T19:52:25.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f338ea00640 1 --2- 172.21.15.29:0/2560811191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f337c006a20 0x7f337c006e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:25.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3397400640 1 --2- 172.21.15.29:0/2560811191 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f337c005c40 0x7f337c0060b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:25.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3397e00640 1 --2- 172.21.15.29:0/2560811191 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33980cccd0 0x7f337c007370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:25.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f338ea00640 1 -- 172.21.15.29:0/2560811191 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33980cccd0 msgr2=0x7f337c007370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:25.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f338ea00640 1 --2- 172.21.15.29:0/2560811191 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f33980cccd0 0x7f337c007370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:25.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f338ea00640 1 -- 172.21.15.29:0/2560811191 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f337c005c40 msgr2=0x7f337c0060b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:25.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f338ea00640 1 --2- 172.21.15.29:0/2560811191 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f337c005c40 0x7f337c0060b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:25.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f338ea00640 1 -- 172.21.15.29:0/2560811191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f337c005740 con 0x7f337c006a20 2024-04-17T19:52:25.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.371+0000 7f3397400640 1 --2- 172.21.15.29:0/2560811191 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f337c005c40 0x7f337c0060b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:52:25.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.375+0000 7f338ea00640 1 --2- 172.21.15.29:0/2560811191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f337c006a20 0x7f337c006e30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3380009ab0 tx=0x7f338001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a586ea0de2ec767 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:25.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.375+0000 7f3396a00640 1 -- 172.21.15.29:0/2560811191 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33800b2070 con 0x7f337c006a20 2024-04-17T19:52:25.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.375+0000 7f3396a00640 1 -- 172.21.15.29:0/2560811191 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33800a5550 con 0x7f337c006a20 2024-04-17T19:52:25.369 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.375+0000 7f3396a00640 1 -- 172.21.15.29:0/2560811191 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33800096b0 con 0x7f337c006a20 2024-04-17T19:52:25.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.375+0000 7f339e600640 1 -- 172.21.15.29:0/2560811191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f337c006a20 msgr2=0x7f337c006e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:25.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.375+0000 7f339e600640 1 --2- 172.21.15.29:0/2560811191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f337c006a20 0x7f337c006e30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3380009ab0 tx=0x7f338001ca70 comp rx=0 tx=0).stop 2024-04-17T19:52:25.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.375+0000 7f3397e00640 1 -- 172.21.15.29:0/2560811191 reap_dead start 2024-04-17T19:52:25.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.375+0000 7f339e600640 1 -- 172.21.15.29:0/2560811191 shutdown_connections 2024-04-17T19:52:25.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.375+0000 7f339e600640 1 -- 172.21.15.29:0/2560811191 >> 172.21.15.29:0/2560811191 conn(0x7f339806fde0 msgr2=0x7f33980701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:25.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.375+0000 7f339e600640 1 -- 172.21.15.29:0/2560811191 shutdown_connections 2024-04-17T19:52:25.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.375+0000 7f339e600640 1 -- 172.21.15.29:0/2560811191 wait complete. 2024-04-17T19:52:25.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.379+0000 7f339e600640 1 Processor -- start 2024-04-17T19:52:25.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.379+0000 7f339e600640 1 -- start start 2024-04-17T19:52:25.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.379+0000 7f339e600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f337c006a20 0x7f3398153ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:25.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.379+0000 7f339e600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3398140ed0 0x7f3398157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:25.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.379+0000 7f339e600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3398073aa0 0x7f3398139820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:25.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.379+0000 7f339e600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3398158e70 con 0x7f337c006a20 2024-04-17T19:52:25.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.379+0000 7f339e600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f33980c7580 con 0x7f3398073aa0 2024-04-17T19:52:25.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.379+0000 7f339e600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3398159aa0 con 0x7f3398140ed0 2024-04-17T19:52:25.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.379+0000 7f3397400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f337c006a20 0x7f3398153ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:25.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.379+0000 7f3397400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f337c006a20 0x7f3398153ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57044/0 (socket says 172.21.15.29:57044) 2024-04-17T19:52:25.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.379+0000 7f3397400640 1 -- 172.21.15.29:0/1025911494 learned_addr learned my addr 172.21.15.29:0/1025911494 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:25.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.379+0000 7f3397e00640 1 --2- 172.21.15.29:0/1025911494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3398073aa0 0x7f3398139820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:25.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.379+0000 7f338ea00640 1 --2- 172.21.15.29:0/1025911494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3398140ed0 0x7f3398157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:25.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.379+0000 7f3397400640 1 -- 172.21.15.29:0/1025911494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3398140ed0 msgr2=0x7f3398157c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:25.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.379+0000 7f3397400640 1 --2- 172.21.15.29:0/1025911494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3398140ed0 0x7f3398157c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:25.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.379+0000 7f3397400640 1 -- 172.21.15.29:0/1025911494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3398073aa0 msgr2=0x7f3398139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:25.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.379+0000 7f3397400640 1 --2- 172.21.15.29:0/1025911494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3398073aa0 0x7f3398139820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:25.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.379+0000 7f3397400640 1 -- 172.21.15.29:0/1025911494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33980ca670 con 0x7f337c006a20 2024-04-17T19:52:25.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.383+0000 7f3397400640 1 --2- 172.21.15.29:0/1025911494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f337c006a20 0x7f3398153ef0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3388011870 tx=0x7f3388011d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:25.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.383+0000 7f338fe00640 1 -- 172.21.15.29:0/1025911494 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3388010040 con 0x7f337c006a20 2024-04-17T19:52:25.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.383+0000 7f338fe00640 1 -- 172.21.15.29:0/1025911494 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f338801a070 con 0x7f337c006a20 2024-04-17T19:52:25.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.383+0000 7f339e600640 1 -- 172.21.15.29:0/1025911494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33981565c0 con 0x7f337c006a20 2024-04-17T19:52:25.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.383+0000 7f338fe00640 1 -- 172.21.15.29:0/1025911494 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338800fd60 con 0x7f337c006a20 2024-04-17T19:52:25.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.383+0000 7f339e600640 1 -- 172.21.15.29:0/1025911494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3398140400 con 0x7f337c006a20 2024-04-17T19:52:25.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.383+0000 7f339e600640 1 -- 172.21.15.29:0/1025911494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3398069b90 con 0x7f337c006a20 2024-04-17T19:52:25.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.383+0000 7f338fe00640 1 -- 172.21.15.29:0/1025911494 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f338801f460 con 0x7f337c006a20 2024-04-17T19:52:25.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.383+0000 7f338fe00640 1 --2- 172.21.15.29:0/1025911494 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f33740683b0 0x7f337406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:25.378 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.383+0000 7f338fe00640 1 -- 172.21.15.29:0/1025911494 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6907+0+0 (secure 0 0 0) 0x7f338808a060 con 0x7f337c006a20 2024-04-17T19:52:25.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.387+0000 7f338ea00640 1 --2- 172.21.15.29:0/1025911494 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f33740683b0 0x7f337406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:25.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.387+0000 7f338ea00640 1 --2- 172.21.15.29:0/1025911494 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f33740683b0 0x7f337406a8c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f33800af000 tx=0x7f33800046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:25.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.391+0000 7f338fe00640 1 -- 172.21.15.29:0/1025911494 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3388011e60 con 0x7f337c006a20 2024-04-17T19:52:25.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.591+0000 7f339e600640 1 -- 172.21.15.29:0/1025911494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f339806ed20 con 0x7f337c006a20 2024-04-17T19:52:25.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.591+0000 7f338fe00640 1 -- 172.21.15.29:0/1025911494 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6081 (secure 0 0 0) 0x7f338805a0d0 con 0x7f337c006a20 2024-04-17T19:52:25.584 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:25.584 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:23.869988+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:52:25.585 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 28 2024-04-17T19:52:25.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.591+0000 7f339e600640 1 -- 172.21.15.29:0/1025911494 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f33740683b0 msgr2=0x7f337406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.591+0000 7f339e600640 1 --2- 172.21.15.29:0/1025911494 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f33740683b0 0x7f337406a8c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f33800af000 tx=0x7f33800046d0 comp rx=0 tx=0).stop 2024-04-17T19:52:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.591+0000 7f339e600640 1 -- 172.21.15.29:0/1025911494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f337c006a20 msgr2=0x7f3398153ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.591+0000 7f339e600640 1 --2- 172.21.15.29:0/1025911494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f337c006a20 0x7f3398153ef0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3388011870 tx=0x7f3388011d30 comp rx=0 tx=0).stop 2024-04-17T19:52:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.591+0000 7f339e600640 1 -- 172.21.15.29:0/1025911494 shutdown_connections 2024-04-17T19:52:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.591+0000 7f339e600640 1 --2- 172.21.15.29:0/1025911494 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f33740683b0 0x7f337406a8c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.591+0000 7f339e600640 1 --2- 172.21.15.29:0/1025911494 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3398073aa0 0x7f3398139820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.591+0000 7f339e600640 1 --2- 172.21.15.29:0/1025911494 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3398140ed0 0x7f3398157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.591+0000 7f339e600640 1 --2- 172.21.15.29:0/1025911494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f337c006a20 0x7f3398153ef0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.591+0000 7f339e600640 1 -- 172.21.15.29:0/1025911494 >> 172.21.15.29:0/1025911494 conn(0x7f339806fde0 msgr2=0x7f339806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.591+0000 7f339e600640 1 -- 172.21.15.29:0/1025911494 shutdown_connections 2024-04-17T19:52:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:25.591+0000 7f339e600640 1 -- 172.21.15.29:0/1025911494 wait complete. 2024-04-17T19:52:25.606 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:52:27.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:27.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.779+0000 7f7023800640 1 Processor -- start 2024-04-17T19:52:27.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.779+0000 7f7023800640 1 -- start start 2024-04-17T19:52:27.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.779+0000 7f7023800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f701c0c9f80 0x7f701c0ca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:27.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.779+0000 7f7023800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f701c0ca390 con 0x7f701c0cae90 2024-04-17T19:52:27.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.779+0000 7f7023800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f701c0cb2a0 con 0x7f701c072e60 2024-04-17T19:52:27.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.779+0000 7f7023800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f701c073c90 con 0x7f701c0c9f80 2024-04-17T19:52:27.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.779+0000 7f7021800640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f701c0cae90 0x7f701c15f960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48296/0 (socket says 172.21.15.29:48296) 2024-04-17T19:52:27.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.779+0000 7f7021800640 1 -- 172.21.15.29:0/3071531253 learned_addr learned my addr 172.21.15.29:0/3071531253 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:52:27.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.779+0000 7f701be00640 1 --2- 172.21.15.29:0/3071531253 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f701c0c9f80 0x7f701c0ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:27.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.779+0000 7f701b400640 1 -- 172.21.15.29:0/3071531253 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1707490601 0 0) 0x7f70140014a0 con 0x7f701c0cae90 2024-04-17T19:52:27.775 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.779+0000 7f701b400640 1 -- 172.21.15.29:0/3071531253 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7004003530 con 0x7f701c0cae90 2024-04-17T19:52:27.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f701b400640 1 -- 172.21.15.29:0/3071531253 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 280408708 0 0) 0x7f7004003530 con 0x7f701c0cae90 2024-04-17T19:52:27.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f701b400640 1 -- 172.21.15.29:0/3071531253 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f701c0c9f80 msgr2=0x7f701c0ca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:27.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f701b400640 1 --2- 172.21.15.29:0/3071531253 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f701c0c9f80 0x7f701c0ca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:27.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f701b400640 1 -- 172.21.15.29:0/3071531253 >> v1:172.21.15.37:6789/0 conn(0x7f701c072e60 legacy=0x7f701c1560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:27.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f701b400640 1 -- 172.21.15.29:0/3071531253 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f701c073ee0 con 0x7f701c0cae90 2024-04-17T19:52:27.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f701b400640 1 -- 172.21.15.29:0/3071531253 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f7014001bf0 con 0x7f701c0cae90 2024-04-17T19:52:27.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f701b400640 1 -- 172.21.15.29:0/3071531253 >> v1:172.21.15.29:6789/0 conn(0x7f701c0cae90 legacy=0x7f701c15f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:27.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f701b400640 1 --2- 172.21.15.29:0/3071531253 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7004003a60 0x7f7004003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:27.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f701b400640 1 --2- 172.21.15.29:0/3071531253 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f701c0c9f80 0x7f7004004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:27.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f701b400640 1 --2- 172.21.15.29:0/3071531253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7004004d50 0x7f70040051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:27.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f701b400640 1 -- 172.21.15.29:0/3071531253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f701c073c90 con 0x7f7004004d50 2024-04-17T19:52:27.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f701b400640 1 -- 172.21.15.29:0/3071531253 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f700400bc30 con 0x7f7004003a60 2024-04-17T19:52:27.777 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f701b400640 1 -- 172.21.15.29:0/3071531253 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f700400be30 con 0x7f701c0c9f80 2024-04-17T19:52:27.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f7021800640 1 --2- 172.21.15.29:0/3071531253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7004004d50 0x7f70040051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:27.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f7020e00640 1 --2- 172.21.15.29:0/3071531253 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7004003a60 0x7f7004003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:27.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f701be00640 1 --2- 172.21.15.29:0/3071531253 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f701c0c9f80 0x7f7004004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:27.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f7021800640 1 -- 172.21.15.29:0/3071531253 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f701c0c9f80 msgr2=0x7f7004004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:27.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f7021800640 1 --2- 172.21.15.29:0/3071531253 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f701c0c9f80 0x7f7004004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:27.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f7021800640 1 -- 172.21.15.29:0/3071531253 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7004003a60 msgr2=0x7f7004003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:27.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f7021800640 1 --2- 172.21.15.29:0/3071531253 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7004003a60 0x7f7004003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:27.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f7021800640 1 -- 172.21.15.29:0/3071531253 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7004003530 con 0x7f7004004d50 2024-04-17T19:52:27.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.783+0000 7f7021800640 1 --2- 172.21.15.29:0/3071531253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7004004d50 0x7f70040051e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f701c06e5a0 tx=0x7f701401d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef8b44d2827d6432 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:27.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.787+0000 7f701b400640 1 -- 172.21.15.29:0/3071531253 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f701400a690 con 0x7f7004004d50 2024-04-17T19:52:27.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.787+0000 7f701b400640 1 -- 172.21.15.29:0/3071531253 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70140072f0 con 0x7f7004004d50 2024-04-17T19:52:27.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.787+0000 7f701b400640 1 -- 172.21.15.29:0/3071531253 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70140076e0 con 0x7f7004004d50 2024-04-17T19:52:27.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.787+0000 7f7023800640 1 -- 172.21.15.29:0/3071531253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7004004d50 msgr2=0x7f70040051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:27.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.787+0000 7f7023800640 1 --2- 172.21.15.29:0/3071531253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7004004d50 0x7f70040051e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f701c06e5a0 tx=0x7f701401d2e0 comp rx=0 tx=0).stop 2024-04-17T19:52:27.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.787+0000 7f7021800640 1 -- 172.21.15.29:0/3071531253 reap_dead start 2024-04-17T19:52:27.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.787+0000 7f7023800640 1 -- 172.21.15.29:0/3071531253 shutdown_connections 2024-04-17T19:52:27.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.787+0000 7f7023800640 1 -- 172.21.15.29:0/3071531253 >> 172.21.15.29:0/3071531253 conn(0x7f701c06ced0 msgr2=0x7f701c06d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:27.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.787+0000 7f7023800640 1 -- 172.21.15.29:0/3071531253 shutdown_connections 2024-04-17T19:52:27.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.787+0000 7f7023800640 1 -- 172.21.15.29:0/3071531253 wait complete. 2024-04-17T19:52:27.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.787+0000 7f7023800640 1 Processor -- start 2024-04-17T19:52:27.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.787+0000 7f7023800640 1 -- start start 2024-04-17T19:52:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7023800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7004004d50 0x7f701c1e6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7023800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f701c149700 0x7f701c14bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7023800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f701c1e79b0 0x7f701c0ce110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7023800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f701c071630 con 0x7f7004004d50 2024-04-17T19:52:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7023800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f701c0741c0 con 0x7f701c1e79b0 2024-04-17T19:52:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7023800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f701c160070 con 0x7f701c149700 2024-04-17T19:52:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7020e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7004004d50 0x7f701c1e6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7020e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7004004d50 0x7f701c1e6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57066/0 (socket says 172.21.15.29:57066) 2024-04-17T19:52:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7020e00640 1 -- 172.21.15.29:0/3696032587 learned_addr learned my addr 172.21.15.29:0/3696032587 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f701be00640 1 --2- 172.21.15.29:0/3696032587 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f701c149700 0x7f701c14bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7021800640 1 --2- 172.21.15.29:0/3696032587 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f701c1e79b0 0x7f701c0ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7020e00640 1 -- 172.21.15.29:0/3696032587 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f701c149700 msgr2=0x7f701c14bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7020e00640 1 --2- 172.21.15.29:0/3696032587 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f701c149700 0x7f701c14bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7020e00640 1 -- 172.21.15.29:0/3696032587 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f701c1e79b0 msgr2=0x7f701c0ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7020e00640 1 --2- 172.21.15.29:0/3696032587 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f701c1e79b0 0x7f701c0ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7020e00640 1 -- 172.21.15.29:0/3696032587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f701c1e7060 con 0x7f7004004d50 2024-04-17T19:52:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7021800640 1 --2- 172.21.15.29:0/3696032587 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f701c1e79b0 0x7f701c0ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:52:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7020e00640 1 --2- 172.21.15.29:0/3696032587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7004004d50 0x7f701c1e6b20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7010011be0 tx=0x7f701000f5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7018c00640 1 -- 172.21.15.29:0/3696032587 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7010019020 con 0x7f7004004d50 2024-04-17T19:52:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7018c00640 1 -- 172.21.15.29:0/3696032587 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f701000fcb0 con 0x7f7004004d50 2024-04-17T19:52:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7018c00640 1 -- 172.21.15.29:0/3696032587 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70100052f0 con 0x7f7004004d50 2024-04-17T19:52:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7023800640 1 -- 172.21.15.29:0/3696032587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f701c1e68b0 con 0x7f7004004d50 2024-04-17T19:52:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7023800640 1 -- 172.21.15.29:0/3696032587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f701c1f5d50 con 0x7f7004004d50 2024-04-17T19:52:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.791+0000 7f7023800640 1 -- 172.21.15.29:0/3696032587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f701c1e7230 con 0x7f7004004d50 2024-04-17T19:52:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.795+0000 7f7018c00640 1 -- 172.21.15.29:0/3696032587 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f701001c020 con 0x7f7004004d50 2024-04-17T19:52:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.795+0000 7f7018c00640 1 --2- 172.21.15.29:0/3696032587 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6ffc068380 0x7f6ffc06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.795+0000 7f7018c00640 1 -- 172.21.15.29:0/3696032587 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6907+0+0 (secure 0 0 0) 0x7f701008a4e0 con 0x7f7004004d50 2024-04-17T19:52:27.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.795+0000 7f701be00640 1 --2- 172.21.15.29:0/3696032587 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6ffc068380 0x7f6ffc06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:27.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.795+0000 7f701be00640 1 --2- 172.21.15.29:0/3696032587 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6ffc068380 0x7f6ffc06a890 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f701c0cf150 tx=0x7f7008008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:27.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.799+0000 7f7018c00640 1 -- 172.21.15.29:0/3696032587 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7010023020 con 0x7f7004004d50 2024-04-17T19:52:27.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.999+0000 7f7023800640 1 -- 172.21.15.29:0/3696032587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f701c069b90 con 0x7f7004004d50 2024-04-17T19:52:27.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.999+0000 7f7018c00640 1 -- 172.21.15.29:0/3696032587 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+6081 (secure 0 0 0) 0x7f701005a5e0 con 0x7f7004004d50 2024-04-17T19:52:27.992 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:27.992 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:23.869988+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:reconnect","state_seq":27,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4229,4259,4302,4305]},"id":1}]} 2024-04-17T19:52:27.993 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 28 2024-04-17T19:52:27.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.999+0000 7f7023800640 1 -- 172.21.15.29:0/3696032587 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6ffc068380 msgr2=0x7f6ffc06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:27.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.999+0000 7f7023800640 1 --2- 172.21.15.29:0/3696032587 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6ffc068380 0x7f6ffc06a890 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f701c0cf150 tx=0x7f7008008040 comp rx=0 tx=0).stop 2024-04-17T19:52:27.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.999+0000 7f7023800640 1 -- 172.21.15.29:0/3696032587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7004004d50 msgr2=0x7f701c1e6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:27.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.999+0000 7f7023800640 1 --2- 172.21.15.29:0/3696032587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7004004d50 0x7f701c1e6b20 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7010011be0 tx=0x7f701000f5a0 comp rx=0 tx=0).stop 2024-04-17T19:52:27.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.999+0000 7f7023800640 1 -- 172.21.15.29:0/3696032587 shutdown_connections 2024-04-17T19:52:27.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.999+0000 7f7023800640 1 --2- 172.21.15.29:0/3696032587 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6ffc068380 0x7f6ffc06a890 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:27.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.999+0000 7f7023800640 1 --2- 172.21.15.29:0/3696032587 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f701c1e79b0 0x7f701c0ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:27.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.999+0000 7f7023800640 1 --2- 172.21.15.29:0/3696032587 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f701c149700 0x7f701c14bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:27.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.999+0000 7f7023800640 1 --2- 172.21.15.29:0/3696032587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7004004d50 0x7f701c1e6b20 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:27.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.999+0000 7f7023800640 1 -- 172.21.15.29:0/3696032587 >> 172.21.15.29:0/3696032587 conn(0x7f701c06ced0 msgr2=0x7f701c06d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:27.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.999+0000 7f7023800640 1 -- 172.21.15.29:0/3696032587 shutdown_connections 2024-04-17T19:52:27.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:27.999+0000 7f7023800640 1 -- 172.21.15.29:0/3696032587 wait complete. 2024-04-17T19:52:28.015 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 28, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-17T19:47:49.315515+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 28, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4229": { "addr": "172.21.15.37:6839/1100598562", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6836", "nonce": 1100598562, "type": "v2" }, { "addr": "172.21.15.37:6839", "nonce": 1100598562, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4229, "incarnation": 16, "join_fscid": -1, "name": "f", "rank": 3, "state": "up:active", "state_seq": 13 }, "gid_4259": { "addr": "172.21.15.29:6836/4184160618", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6832", "nonce": 4184160618, "type": "v2" }, { "addr": "172.21.15.29:6836", "nonce": 4184160618, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4259, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 11 }, "gid_4302": { "addr": "172.21.15.29:6835/270402476", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6834", "nonce": 270402476, "type": "v2" }, { "addr": "172.21.15.29:6835", "nonce": 270402476, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 10, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 9 }, "gid_4305": { "addr": "172.21.15.29:6837/2691590158", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6833", "nonce": 2691590158, "type": "v2" }, { "addr": "172.21.15.29:6837", "nonce": 2691590158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4305, "incarnation": 5, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5212": { "addr": "172.21.15.37:6838/767681664", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6835", "nonce": 767681664, "type": "v2" }, { "addr": "172.21.15.37:6838", "nonce": 767681664, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5212, "incarnation": 26, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:reconnect", "state_seq": 27 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-17T19:52:23.869988+0000", "qdb_cluster": [ 4229, 4259, 4302, 4305 ], "qdb_leader": 4305, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4305, "mds_1": 4302, "mds_2": 4259, "mds_3": 4229, "mds_4": 5212 } } } ], "standbys": [] } 2024-04-17T19:52:30.017 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:30.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.175+0000 7f56cd200640 1 Processor -- start 2024-04-17T19:52:30.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.175+0000 7f56cd200640 1 -- start start 2024-04-17T19:52:30.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.175+0000 7f56cd200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f56c80cccd0 0x7f56c80ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:30.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56cd200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56c80cd0e0 con 0x7f56c80cc8c0 2024-04-17T19:52:30.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56cd200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56c80c88b0 con 0x7f56c80ca870 2024-04-17T19:52:30.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56cd200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f56c80c6ce0 con 0x7f56c80cccd0 2024-04-17T19:52:30.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f56c80cc8c0 0x7f56c8159390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54512/0 (socket says 172.21.15.29:54512) 2024-04-17T19:52:30.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5e00640 1 -- 172.21.15.29:0/3521874059 learned_addr learned my addr 172.21.15.29:0/3521874059 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:52:30.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56bfe00640 1 --2- 172.21.15.29:0/3521874059 >> v2:172.21.15.37:3300/0 conn(0x7f56c80cccd0 0x7f56c80ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:30.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5400640 1 -- 172.21.15.29:0/3521874059 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3849621979 0 0) 0x7f56c80cd0e0 con 0x7f56c80cc8c0 2024-04-17T19:52:30.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5400640 1 -- 172.21.15.29:0/3521874059 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56ac003530 con 0x7f56c80cc8c0 2024-04-17T19:52:30.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5400640 1 -- 172.21.15.29:0/3521874059 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2309324182 0 0) 0x7f56ac003530 con 0x7f56c80cc8c0 2024-04-17T19:52:30.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5400640 1 -- 172.21.15.29:0/3521874059 >> v2:172.21.15.37:3300/0 conn(0x7f56c80cccd0 msgr2=0x7f56c80ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:30.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5400640 1 --2- 172.21.15.29:0/3521874059 >> v2:172.21.15.37:3300/0 conn(0x7f56c80cccd0 0x7f56c80ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:30.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5400640 1 -- 172.21.15.29:0/3521874059 >> v1:172.21.15.37:6789/0 conn(0x7f56c80ca870 legacy=0x7f56c81635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:30.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5400640 1 -- 172.21.15.29:0/3521874059 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c80c6fc0 con 0x7f56c80cc8c0 2024-04-17T19:52:30.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5400640 1 -- 172.21.15.29:0/3521874059 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f56b8001680 con 0x7f56c80cc8c0 2024-04-17T19:52:30.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5400640 1 -- 172.21.15.29:0/3521874059 >> v1:172.21.15.29:6789/0 conn(0x7f56c80cc8c0 legacy=0x7f56c8159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:30.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5400640 1 --2- 172.21.15.29:0/3521874059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56ac003a30 0x7f56ac003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:30.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5400640 1 --2- 172.21.15.29:0/3521874059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56ac004810 0x7f56ac004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:30.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5400640 1 --2- 172.21.15.29:0/3521874059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56ac005230 0x7f56ac00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:30.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5400640 1 -- 172.21.15.29:0/3521874059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f56c80c6ce0 con 0x7f56ac003a30 2024-04-17T19:52:30.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5400640 1 -- 172.21.15.29:0/3521874059 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f56ac00c1c0 con 0x7f56ac005230 2024-04-17T19:52:30.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5400640 1 -- 172.21.15.29:0/3521874059 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f56ac00c3c0 con 0x7f56ac004810 2024-04-17T19:52:30.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5e00640 1 --2- 172.21.15.29:0/3521874059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56ac003a30 0x7f56ac003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:30.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c6800640 1 --2- 172.21.15.29:0/3521874059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56ac005230 0x7f56ac00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:30.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56bfe00640 1 --2- 172.21.15.29:0/3521874059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56ac004810 0x7f56ac004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:30.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5e00640 1 -- 172.21.15.29:0/3521874059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56ac004810 msgr2=0x7f56ac004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:30.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5e00640 1 --2- 172.21.15.29:0/3521874059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56ac004810 0x7f56ac004c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:30.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5e00640 1 -- 172.21.15.29:0/3521874059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56ac005230 msgr2=0x7f56ac00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:30.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5e00640 1 --2- 172.21.15.29:0/3521874059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56ac005230 0x7f56ac00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:30.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c5e00640 1 -- 172.21.15.29:0/3521874059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56ac003530 con 0x7f56ac003a30 2024-04-17T19:52:30.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.179+0000 7f56c6800640 1 -- 172.21.15.29:0/3521874059 reap_dead start 2024-04-17T19:52:30.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.183+0000 7f56c5e00640 1 --2- 172.21.15.29:0/3521874059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56ac003a30 0x7f56ac003ea0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f56b800c1c0 tx=0x7f56b801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ad01fb1c0619c9c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:30.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.183+0000 7f56c5400640 1 -- 172.21.15.29:0/3521874059 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8007d00 con 0x7f56ac003a30 2024-04-17T19:52:30.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.183+0000 7f56c5400640 1 -- 172.21.15.29:0/3521874059 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56b80a9020 con 0x7f56ac003a30 2024-04-17T19:52:30.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.183+0000 7f56c5400640 1 -- 172.21.15.29:0/3521874059 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8005b00 con 0x7f56ac003a30 2024-04-17T19:52:30.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.183+0000 7f56cd200640 1 -- 172.21.15.29:0/3521874059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56ac003a30 msgr2=0x7f56ac003ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:30.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.183+0000 7f56cd200640 1 --2- 172.21.15.29:0/3521874059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56ac003a30 0x7f56ac003ea0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f56b800c1c0 tx=0x7f56b801cc60 comp rx=0 tx=0).stop 2024-04-17T19:52:30.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.183+0000 7f56cd200640 1 -- 172.21.15.29:0/3521874059 shutdown_connections 2024-04-17T19:52:30.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.183+0000 7f56cd200640 1 --2- 172.21.15.29:0/3521874059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56ac003a30 0x7f56ac003ea0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:30.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.183+0000 7f56cd200640 1 -- 172.21.15.29:0/3521874059 >> 172.21.15.29:0/3521874059 conn(0x7f56c806fde0 msgr2=0x7f56c80701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:30.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.183+0000 7f56cd200640 1 -- 172.21.15.29:0/3521874059 shutdown_connections 2024-04-17T19:52:30.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.183+0000 7f56cd200640 1 -- 172.21.15.29:0/3521874059 wait complete. 2024-04-17T19:52:30.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56cd200640 1 Processor -- start 2024-04-17T19:52:30.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56cd200640 1 -- start start 2024-04-17T19:52:30.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56cd200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56ac003a30 0x7f56c8072320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:30.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56cd200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56c8073060 0x7f56c81581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:30.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56cd200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56c8157510 0x7f56c80c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:30.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56cd200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f56c80c6a00 con 0x7f56c8157510 2024-04-17T19:52:30.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56cd200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f56c80c6880 con 0x7f56c8073060 2024-04-17T19:52:30.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56cd200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f56c815c250 con 0x7f56ac003a30 2024-04-17T19:52:30.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56c6800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56c8157510 0x7f56c80c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:30.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56c6800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56c8157510 0x7f56c80c8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39636/0 (socket says 172.21.15.29:39636) 2024-04-17T19:52:30.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56c6800640 1 -- 172.21.15.29:0/2177034314 learned_addr learned my addr 172.21.15.29:0/2177034314 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:30.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56bfe00640 1 --2- 172.21.15.29:0/2177034314 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56c8073060 0x7f56c81581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:30.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56c5e00640 1 --2- 172.21.15.29:0/2177034314 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56ac003a30 0x7f56c8072320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:30.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56c6800640 1 -- 172.21.15.29:0/2177034314 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56ac003a30 msgr2=0x7f56c8072320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:30.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56c6800640 1 --2- 172.21.15.29:0/2177034314 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56ac003a30 0x7f56c8072320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:30.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56c6800640 1 -- 172.21.15.29:0/2177034314 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56c8073060 msgr2=0x7f56c81581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:30.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56c6800640 1 --2- 172.21.15.29:0/2177034314 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56c8073060 0x7f56c81581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:30.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56c6800640 1 -- 172.21.15.29:0/2177034314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c80c8430 con 0x7f56c8157510 2024-04-17T19:52:30.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56c6800640 1 --2- 172.21.15.29:0/2177034314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56c8157510 0x7f56c80c8d40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f56c80c67a0 tx=0x7f56c000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:30.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56be000640 1 -- 172.21.15.29:0/2177034314 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c0019070 con 0x7f56c8157510 2024-04-17T19:52:30.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56be000640 1 -- 172.21.15.29:0/2177034314 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56c0006b60 con 0x7f56c8157510 2024-04-17T19:52:30.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56be000640 1 -- 172.21.15.29:0/2177034314 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56c0003980 con 0x7f56c8157510 2024-04-17T19:52:30.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56cd200640 1 -- 172.21.15.29:0/2177034314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56c813a9c0 con 0x7f56c8157510 2024-04-17T19:52:30.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56cd200640 1 -- 172.21.15.29:0/2177034314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56c8139ee0 con 0x7f56c8157510 2024-04-17T19:52:30.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.187+0000 7f56cd200640 1 -- 172.21.15.29:0/2177034314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56c8072990 con 0x7f56c8157510 2024-04-17T19:52:30.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.191+0000 7f56be000640 1 -- 172.21.15.29:0/2177034314 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f56c001c070 con 0x7f56c8157510 2024-04-17T19:52:30.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.191+0000 7f56be000640 1 --2- 172.21.15.29:0/2177034314 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f56a4068380 0x7f56a406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:30.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.191+0000 7f56be000640 1 -- 172.21.15.29:0/2177034314 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6907+0+0 (secure 0 0 0) 0x7f56c0086540 con 0x7f56c8157510 2024-04-17T19:52:30.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.191+0000 7f56c5e00640 1 --2- 172.21.15.29:0/2177034314 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f56a4068380 0x7f56a406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:30.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.191+0000 7f56c5e00640 1 --2- 172.21.15.29:0/2177034314 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f56a4068380 0x7f56a406a890 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f56b80aa000 tx=0x7f56c80c61f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:30.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.195+0000 7f56be000640 1 -- 172.21.15.29:0/2177034314 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f56c0056640 con 0x7f56c8157510 2024-04-17T19:52:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.395+0000 7f56cd200640 1 -- 172.21.15.29:0/2177034314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56c8069b90 con 0x7f56c8157510 2024-04-17T19:52:30.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.395+0000 7f56be000640 1 -- 172.21.15.29:0/2177034314 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6078 (secure 0 0 0) 0x7f56c0055f40 con 0x7f56c8157510 2024-04-17T19:52:30.389 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:30.389 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:28.887005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229]},"id":1}]} 2024-04-17T19:52:30.390 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 29 2024-04-17T19:52:30.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.399+0000 7f56cd200640 1 -- 172.21.15.29:0/2177034314 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f56a4068380 msgr2=0x7f56a406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:30.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.399+0000 7f56cd200640 1 --2- 172.21.15.29:0/2177034314 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f56a4068380 0x7f56a406a890 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f56b80aa000 tx=0x7f56c80c61f0 comp rx=0 tx=0).stop 2024-04-17T19:52:30.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.399+0000 7f56cd200640 1 -- 172.21.15.29:0/2177034314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56c8157510 msgr2=0x7f56c80c8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:30.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.399+0000 7f56cd200640 1 --2- 172.21.15.29:0/2177034314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56c8157510 0x7f56c80c8d40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f56c80c67a0 tx=0x7f56c000ec10 comp rx=0 tx=0).stop 2024-04-17T19:52:30.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.399+0000 7f56cd200640 1 -- 172.21.15.29:0/2177034314 shutdown_connections 2024-04-17T19:52:30.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.399+0000 7f56cd200640 1 --2- 172.21.15.29:0/2177034314 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f56a4068380 0x7f56a406a890 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:30.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.399+0000 7f56cd200640 1 --2- 172.21.15.29:0/2177034314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56c8157510 0x7f56c80c8d40 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:30.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.399+0000 7f56cd200640 1 --2- 172.21.15.29:0/2177034314 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f56c8073060 0x7f56c81581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:30.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.399+0000 7f56cd200640 1 --2- 172.21.15.29:0/2177034314 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f56ac003a30 0x7f56c8072320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:30.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.399+0000 7f56cd200640 1 -- 172.21.15.29:0/2177034314 >> 172.21.15.29:0/2177034314 conn(0x7f56c806fde0 msgr2=0x7f56c813e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:30.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.399+0000 7f56cd200640 1 -- 172.21.15.29:0/2177034314 shutdown_connections 2024-04-17T19:52:30.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:30.399+0000 7f56cd200640 1 -- 172.21.15.29:0/2177034314 wait complete. 2024-04-17T19:52:30.411 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:52:32.412 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:32.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.582+0000 7fb417e00640 1 Processor -- start 2024-04-17T19:52:32.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.582+0000 7fb417e00640 1 -- start start 2024-04-17T19:52:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.582+0000 7fb417e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb4100cc8c0 0x7fb4100cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.586+0000 7fb417e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb410159390 0x7fb4101597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.586+0000 7fb417e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb4100ca330 0x7fb4100ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.586+0000 7fb417e00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fb4100c88b0 con 0x7fb4100cc8c0 2024-04-17T19:52:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.586+0000 7fb417e00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb4100c6ce0 con 0x7fb4100ca330 2024-04-17T19:52:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.586+0000 7fb417e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb4100c6fc0 con 0x7fb410159390 2024-04-17T19:52:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.586+0000 7fb415400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb4100cc8c0 0x7fb4100cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.586+0000 7fb415400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb4100cc8c0 0x7fb4100cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39652/0 (socket says 172.21.15.29:39652) 2024-04-17T19:52:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.586+0000 7fb415400640 1 -- 172.21.15.29:0/3978129107 learned_addr learned my addr 172.21.15.29:0/3978129107 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.586+0000 7fb414a00640 1 --2- 172.21.15.29:0/3978129107 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb410159390 0x7fb4101597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.586+0000 7fb415e00640 1 --2- 172.21.15.29:0/3978129107 >> v2:172.21.15.37:3300/0 conn(0x7fb4100ca330 0x7fb4100ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.586+0000 7fb415400640 1 -- 172.21.15.29:0/3978129107 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb410159390 msgr2=0x7fb4101597a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:32.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.586+0000 7fb415400640 1 --2- 172.21.15.29:0/3978129107 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb410159390 0x7fb4101597a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:32.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.586+0000 7fb415400640 1 -- 172.21.15.29:0/3978129107 >> v2:172.21.15.37:3300/0 conn(0x7fb4100ca330 msgr2=0x7fb4100ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:32.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.586+0000 7fb415400640 1 --2- 172.21.15.29:0/3978129107 >> v2:172.21.15.37:3300/0 conn(0x7fb4100ca330 0x7fb4100ca740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:32.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.586+0000 7fb415400640 1 -- 172.21.15.29:0/3978129107 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4100c72a0 con 0x7fb4100cc8c0 2024-04-17T19:52:32.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.586+0000 7fb415400640 1 --2- 172.21.15.29:0/3978129107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4100cc8c0 0x7fb4100cccd0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb410071e60 tx=0x7fb40401ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c43287de923833f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:32.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.590+0000 7fb40fe00640 1 -- 172.21.15.29:0/3978129107 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4040b4070 con 0x7fb4100cc8c0 2024-04-17T19:52:32.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.590+0000 7fb40fe00640 1 -- 172.21.15.29:0/3978129107 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4040a7a80 con 0x7fb4100cc8c0 2024-04-17T19:52:32.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.590+0000 7fb40fe00640 1 -- 172.21.15.29:0/3978129107 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4040a7e50 con 0x7fb4100cc8c0 2024-04-17T19:52:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.590+0000 7fb417e00640 1 -- 172.21.15.29:0/3978129107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4100cc8c0 msgr2=0x7fb4100cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.590+0000 7fb417e00640 1 --2- 172.21.15.29:0/3978129107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4100cc8c0 0x7fb4100cccd0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fb410071e60 tx=0x7fb40401ebf0 comp rx=0 tx=0).stop 2024-04-17T19:52:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.590+0000 7fb417e00640 1 -- 172.21.15.29:0/3978129107 shutdown_connections 2024-04-17T19:52:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.590+0000 7fb417e00640 1 --2- 172.21.15.29:0/3978129107 >> v2:172.21.15.37:3300/0 conn(0x7fb4100ca330 0x7fb4100ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.590+0000 7fb417e00640 1 --2- 172.21.15.29:0/3978129107 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb410159390 0x7fb4101597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.590+0000 7fb417e00640 1 --2- 172.21.15.29:0/3978129107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4100cc8c0 0x7fb4100cccd0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.590+0000 7fb417e00640 1 -- 172.21.15.29:0/3978129107 >> 172.21.15.29:0/3978129107 conn(0x7fb41006fde0 msgr2=0x7fb4100701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.590+0000 7fb417e00640 1 -- 172.21.15.29:0/3978129107 shutdown_connections 2024-04-17T19:52:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.590+0000 7fb417e00640 1 -- 172.21.15.29:0/3978129107 wait complete. 2024-04-17T19:52:32.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.590+0000 7fb417e00640 1 Processor -- start 2024-04-17T19:52:32.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.590+0000 7fb417e00640 1 -- start start 2024-04-17T19:52:32.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb417e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4100ca330 0x7fb410141290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:32.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb417e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4100cc8c0 0x7fb410139ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:32.588 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb417e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb410159390 0x7fb410139ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb417e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb41015ca60 con 0x7fb4100ca330 2024-04-17T19:52:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb417e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb4100c6a90 con 0x7fb410159390 2024-04-17T19:52:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb417e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb41015cbe0 con 0x7fb4100cc8c0 2024-04-17T19:52:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb415400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4100ca330 0x7fb410141290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb415400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4100ca330 0x7fb410141290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39656/0 (socket says 172.21.15.29:39656) 2024-04-17T19:52:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb415400640 1 -- 172.21.15.29:0/1204547097 learned_addr learned my addr 172.21.15.29:0/1204547097 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb414a00640 1 --2- 172.21.15.29:0/1204547097 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4100cc8c0 0x7fb410139ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb415e00640 1 --2- 172.21.15.29:0/1204547097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb410159390 0x7fb410139ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb415400640 1 -- 172.21.15.29:0/1204547097 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4100cc8c0 msgr2=0x7fb410139ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb415400640 1 --2- 172.21.15.29:0/1204547097 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4100cc8c0 0x7fb410139ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb415400640 1 -- 172.21.15.29:0/1204547097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb410159390 msgr2=0x7fb410139ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb415400640 1 --2- 172.21.15.29:0/1204547097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb410159390 0x7fb410139ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb415400640 1 -- 172.21.15.29:0/1204547097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4101417d0 con 0x7fb4100ca330 2024-04-17T19:52:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb414a00640 1 --2- 172.21.15.29:0/1204547097 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4100cc8c0 0x7fb410139ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:52:32.590 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb415e00640 1 --2- 172.21.15.29:0/1204547097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb410159390 0x7fb410139ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:52:32.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb415400640 1 --2- 172.21.15.29:0/1204547097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4100ca330 0x7fb410141290 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb40400b2c0 tx=0x7fb40401b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:32.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb40d600640 1 -- 172.21.15.29:0/1204547097 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4040bf070 con 0x7fb4100ca330 2024-04-17T19:52:32.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb40d600640 1 -- 172.21.15.29:0/1204547097 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb4040b6050 con 0x7fb4100ca330 2024-04-17T19:52:32.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb417e00640 1 -- 172.21.15.29:0/1204547097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4101f5a40 con 0x7fb4100ca330 2024-04-17T19:52:32.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb40d600640 1 -- 172.21.15.29:0/1204547097 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4040b4030 con 0x7fb4100ca330 2024-04-17T19:52:32.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb417e00640 1 -- 172.21.15.29:0/1204547097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb410140a90 con 0x7fb4100ca330 2024-04-17T19:52:32.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.594+0000 7fb417e00640 1 -- 172.21.15.29:0/1204547097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4101f5f20 con 0x7fb4100ca330 2024-04-17T19:52:32.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.598+0000 7fb40d600640 1 -- 172.21.15.29:0/1204547097 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb4040c2070 con 0x7fb4100ca330 2024-04-17T19:52:32.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.598+0000 7fb40d600640 1 --2- 172.21.15.29:0/1204547097 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb3f00683b0 0x7fb3f006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:32.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.598+0000 7fb40d600640 1 -- 172.21.15.29:0/1204547097 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6907+0+0 (secure 0 0 0) 0x7fb404120f70 con 0x7fb4100ca330 2024-04-17T19:52:32.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.598+0000 7fb414a00640 1 --2- 172.21.15.29:0/1204547097 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb3f00683b0 0x7fb3f006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:32.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.598+0000 7fb414a00640 1 --2- 172.21.15.29:0/1204547097 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb3f00683b0 0x7fb3f006a8c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb4100c9b40 tx=0x7fb3fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:32.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.602+0000 7fb40d600640 1 -- 172.21.15.29:0/1204547097 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb4040187e0 con 0x7fb4100ca330 2024-04-17T19:52:32.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.802+0000 7fb417e00640 1 -- 172.21.15.29:0/1204547097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb410069b90 con 0x7fb4100ca330 2024-04-17T19:52:32.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.802+0000 7fb40d600640 1 -- 172.21.15.29:0/1204547097 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6078 (secure 0 0 0) 0x7fb4040f5e20 con 0x7fb4100ca330 2024-04-17T19:52:32.796 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:32.796 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:28.887005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:rejoin","state_seq":29,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,4302,4259,4229]},"id":1}]} 2024-04-17T19:52:32.796 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 29 2024-04-17T19:52:32.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.802+0000 7fb417e00640 1 -- 172.21.15.29:0/1204547097 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb3f00683b0 msgr2=0x7fb3f006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:32.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.802+0000 7fb417e00640 1 --2- 172.21.15.29:0/1204547097 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb3f00683b0 0x7fb3f006a8c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb4100c9b40 tx=0x7fb3fc008040 comp rx=0 tx=0).stop 2024-04-17T19:52:32.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.802+0000 7fb417e00640 1 -- 172.21.15.29:0/1204547097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4100ca330 msgr2=0x7fb410141290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:32.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.802+0000 7fb417e00640 1 --2- 172.21.15.29:0/1204547097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4100ca330 0x7fb410141290 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb40400b2c0 tx=0x7fb40401b990 comp rx=0 tx=0).stop 2024-04-17T19:52:32.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.802+0000 7fb417e00640 1 -- 172.21.15.29:0/1204547097 shutdown_connections 2024-04-17T19:52:32.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.802+0000 7fb417e00640 1 --2- 172.21.15.29:0/1204547097 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb3f00683b0 0x7fb3f006a8c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:32.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.802+0000 7fb417e00640 1 --2- 172.21.15.29:0/1204547097 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb410159390 0x7fb410139ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:32.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.802+0000 7fb417e00640 1 --2- 172.21.15.29:0/1204547097 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb4100cc8c0 0x7fb410139ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:32.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.802+0000 7fb417e00640 1 --2- 172.21.15.29:0/1204547097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4100ca330 0x7fb410141290 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:32.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.802+0000 7fb417e00640 1 -- 172.21.15.29:0/1204547097 >> 172.21.15.29:0/1204547097 conn(0x7fb41006fde0 msgr2=0x7fb41006bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:32.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.802+0000 7fb417e00640 1 -- 172.21.15.29:0/1204547097 shutdown_connections 2024-04-17T19:52:32.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:32.802+0000 7fb417e00640 1 -- 172.21.15.29:0/1204547097 wait complete. 2024-04-17T19:52:32.819 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:52:34.819 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:34.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:34.990+0000 7fafe7800640 1 Processor -- start 2024-04-17T19:52:34.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:34.990+0000 7fafe7800640 1 -- start start 2024-04-17T19:52:34.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:34.994+0000 7fafe7800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fafe0156210 0x7fafe0070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:34.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:34.994+0000 7fafe7800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafe015c960 con 0x7fafe0155e00 2024-04-17T19:52:34.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:34.994+0000 7fafe7800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafe015cc40 con 0x7fafe0160160 2024-04-17T19:52:34.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:34.994+0000 7fafe7800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fafe00c9210 con 0x7fafe0156210 2024-04-17T19:52:34.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:34.994+0000 7fafdfe00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fafe0155e00 0x7fafe015bff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54526/0 (socket says 172.21.15.29:54526) 2024-04-17T19:52:34.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:34.994+0000 7fafdfe00640 1 -- 172.21.15.29:0/3214318300 learned_addr learned my addr 172.21.15.29:0/3214318300 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:52:34.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:34.994+0000 7fafe5800640 1 --2- 172.21.15.29:0/3214318300 >> v2:172.21.15.37:3300/0 conn(0x7fafe0156210 0x7fafe0070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:34.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:34.994+0000 7fafdf400640 1 -- 172.21.15.29:0/3214318300 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1908817853 0 0) 0x7fafe015c960 con 0x7fafe0155e00 2024-04-17T19:52:34.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:34.994+0000 7fafdf400640 1 -- 172.21.15.29:0/3214318300 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafc8003530 con 0x7fafe0155e00 2024-04-17T19:52:34.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:34.994+0000 7fafe5800640 1 -- 172.21.15.29:0/3214318300 >> v1:172.21.15.37:6789/0 conn(0x7fafe0160160 legacy=0x7fafe0073c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:34.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:34.994+0000 7fafe5800640 1 -- 172.21.15.29:0/3214318300 >> v1:172.21.15.29:6789/0 conn(0x7fafe0155e00 legacy=0x7fafe015bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:34.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:34.994+0000 7fafe5800640 1 -- 172.21.15.29:0/3214318300 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafe015cf20 con 0x7fafe0156210 2024-04-17T19:52:34.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:34.998+0000 7fafe5800640 1 --2- 172.21.15.29:0/3214318300 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe0156210 0x7fafe0070bc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fafd8009d90 tx=0x7fafd801c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78611231f8609c2d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:34.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:34.998+0000 7fafdf400640 1 -- 172.21.15.29:0/3214318300 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd80b2070 con 0x7fafe0156210 2024-04-17T19:52:34.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:34.998+0000 7fafdf400640 1 -- 172.21.15.29:0/3214318300 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafd80a58b0 con 0x7fafe0156210 2024-04-17T19:52:34.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:34.998+0000 7fafdf400640 1 -- 172.21.15.29:0/3214318300 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd80a5c60 con 0x7fafe0156210 2024-04-17T19:52:34.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.002+0000 7fafe7800640 1 -- 172.21.15.29:0/3214318300 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe0156210 msgr2=0x7fafe0070bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:34.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.002+0000 7fafe7800640 1 --2- 172.21.15.29:0/3214318300 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe0156210 0x7fafe0070bc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fafd8009d90 tx=0x7fafd801c8e0 comp rx=0 tx=0).stop 2024-04-17T19:52:34.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.002+0000 7fafe7800640 1 -- 172.21.15.29:0/3214318300 shutdown_connections 2024-04-17T19:52:34.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.002+0000 7fafe7800640 1 --2- 172.21.15.29:0/3214318300 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe0156210 0x7fafe0070bc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:34.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.002+0000 7fafe7800640 1 -- 172.21.15.29:0/3214318300 >> 172.21.15.29:0/3214318300 conn(0x7fafe00ce5e0 msgr2=0x7fafe00ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:34.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.002+0000 7fafe7800640 1 -- 172.21.15.29:0/3214318300 shutdown_connections 2024-04-17T19:52:34.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.002+0000 7fafe7800640 1 -- 172.21.15.29:0/3214318300 wait complete. 2024-04-17T19:52:34.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.002+0000 7fafe7800640 1 Processor -- start 2024-04-17T19:52:34.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.002+0000 7fafe7800640 1 -- start start 2024-04-17T19:52:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe7800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe0155e00 0x7fafe014aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe7800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe0156210 0x7fafe014b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe7800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafe0160160 0x7fafe00cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe7800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fafe015c700 con 0x7fafe0155e00 2024-04-17T19:52:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe7800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fafe0160570 con 0x7fafe0156210 2024-04-17T19:52:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe7800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fafe016a7c0 con 0x7fafe0160160 2024-04-17T19:52:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe4e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe0155e00 0x7fafe014aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe4e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe0155e00 0x7fafe014aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39668/0 (socket says 172.21.15.29:39668) 2024-04-17T19:52:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe4e00640 1 -- 172.21.15.29:0/3621667148 learned_addr learned my addr 172.21.15.29:0/3621667148 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafdfe00640 1 --2- 172.21.15.29:0/3621667148 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe0156210 0x7fafe014b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe5800640 1 --2- 172.21.15.29:0/3621667148 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafe0160160 0x7fafe00cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:35.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe4e00640 1 -- 172.21.15.29:0/3621667148 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafe0160160 msgr2=0x7fafe00cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe4e00640 1 --2- 172.21.15.29:0/3621667148 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafe0160160 0x7fafe00cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe4e00640 1 -- 172.21.15.29:0/3621667148 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe0156210 msgr2=0x7fafe014b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe4e00640 1 --2- 172.21.15.29:0/3621667148 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe0156210 0x7fafe014b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:35.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe4e00640 1 -- 172.21.15.29:0/3621667148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafe0145fd0 con 0x7fafe0155e00 2024-04-17T19:52:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe4e00640 1 --2- 172.21.15.29:0/3621667148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe0155e00 0x7fafe014aac0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fafd400c3a0 tx=0x7fafd400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafdcc00640 1 -- 172.21.15.29:0/3621667148 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd40052d0 con 0x7fafe0155e00 2024-04-17T19:52:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafdcc00640 1 -- 172.21.15.29:0/3621667148 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafd4005470 con 0x7fafe0155e00 2024-04-17T19:52:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafdcc00640 1 -- 172.21.15.29:0/3621667148 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd4007260 con 0x7fafe0155e00 2024-04-17T19:52:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe7800640 1 -- 172.21.15.29:0/3621667148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafe0145a10 con 0x7fafe0155e00 2024-04-17T19:52:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe7800640 1 -- 172.21.15.29:0/3621667148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafe014a2c0 con 0x7fafe0155e00 2024-04-17T19:52:35.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.006+0000 7fafe7800640 1 -- 172.21.15.29:0/3621667148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafe0069b90 con 0x7fafe0155e00 2024-04-17T19:52:35.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.010+0000 7fafdcc00640 1 -- 172.21.15.29:0/3621667148 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fafd4007400 con 0x7fafe0155e00 2024-04-17T19:52:35.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.010+0000 7fafdcc00640 1 --2- 172.21.15.29:0/3621667148 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fafc00683b0 0x7fafc006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:35.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.010+0000 7fafdcc00640 1 -- 172.21.15.29:0/3621667148 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6907+0+0 (secure 0 0 0) 0x7fafd4086740 con 0x7fafe0155e00 2024-04-17T19:52:35.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.010+0000 7fafdfe00640 1 --2- 172.21.15.29:0/3621667148 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fafc00683b0 0x7fafc006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:35.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.010+0000 7fafdfe00640 1 --2- 172.21.15.29:0/3621667148 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fafc00683b0 0x7fafc006a8c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fafe006f690 tx=0x7fafcc007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:35.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.014+0000 7fafdcc00640 1 -- 172.21.15.29:0/3621667148 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fafd4056250 con 0x7fafe0155e00 2024-04-17T19:52:35.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.210+0000 7fafe7800640 1 -- 172.21.15.29:0/3621667148 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafe01534a0 con 0x7fafe0155e00 2024-04-17T19:52:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.210+0000 7fafdcc00640 1 -- 172.21.15.29:0/3621667148 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6083 (secure 0 0 0) 0x7fafd405a550 con 0x7fafe0155e00 2024-04-17T19:52:35.206 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:35.206 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:34.862105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4229,4259,4302,5212,4305]},"id":1}]} 2024-04-17T19:52:35.207 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 30 2024-04-17T19:52:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.214+0000 7fafe7800640 1 -- 172.21.15.29:0/3621667148 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fafc00683b0 msgr2=0x7fafc006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.214+0000 7fafe7800640 1 --2- 172.21.15.29:0/3621667148 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fafc00683b0 0x7fafc006a8c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fafe006f690 tx=0x7fafcc007d40 comp rx=0 tx=0).stop 2024-04-17T19:52:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.214+0000 7fafe7800640 1 -- 172.21.15.29:0/3621667148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe0155e00 msgr2=0x7fafe014aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.214+0000 7fafe7800640 1 --2- 172.21.15.29:0/3621667148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe0155e00 0x7fafe014aac0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fafd400c3a0 tx=0x7fafd400c860 comp rx=0 tx=0).stop 2024-04-17T19:52:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.214+0000 7fafe7800640 1 -- 172.21.15.29:0/3621667148 shutdown_connections 2024-04-17T19:52:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.214+0000 7fafe7800640 1 --2- 172.21.15.29:0/3621667148 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fafc00683b0 0x7fafc006a8c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.214+0000 7fafe7800640 1 --2- 172.21.15.29:0/3621667148 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafe0160160 0x7fafe00cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.214+0000 7fafe7800640 1 --2- 172.21.15.29:0/3621667148 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafe0156210 0x7fafe014b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.214+0000 7fafe7800640 1 --2- 172.21.15.29:0/3621667148 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafe0155e00 0x7fafe014aac0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.214+0000 7fafe7800640 1 -- 172.21.15.29:0/3621667148 >> 172.21.15.29:0/3621667148 conn(0x7fafe00ce5e0 msgr2=0x7fafe00c6600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.214+0000 7fafe7800640 1 -- 172.21.15.29:0/3621667148 shutdown_connections 2024-04-17T19:52:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:35.214+0000 7fafe7800640 1 -- 172.21.15.29:0/3621667148 wait complete. 2024-04-17T19:52:35.229 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=4, replacing gid=4223 2024-04-17T19:52:35.229 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.d 2024-04-17T19:52:37.230 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2024-04-17T19:52:37.230 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T19:52:37.230 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T19:52:37.233 INFO:tasks.ceph.mds.d:Started 2024-04-17T19:52:37.233 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:37.345 INFO:tasks.ceph.mds.d.smithi037.stdout:starting mds.d at 2024-04-17T19:52:37.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.394+0000 7f5f27600640 1 Processor -- start 2024-04-17T19:52:37.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.394+0000 7f5f27600640 1 -- start start 2024-04-17T19:52:37.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.394+0000 7f5f27600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5f2006c3b0 0x7f5f2006d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:37.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.394+0000 7f5f27600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f200caf80 0x7f5f2015f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:37.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.394+0000 7f5f27600640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5f2015ca30 0x7f5f200cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:37.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.394+0000 7f5f27600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f5f2015bfe0 con 0x7f5f2006c3b0 2024-04-17T19:52:37.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.394+0000 7f5f27600640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f5f2015fc90 con 0x7f5f2015ca30 2024-04-17T19:52:37.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.394+0000 7f5f27600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5f2015ff70 con 0x7f5f200caf80 2024-04-17T19:52:37.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.394+0000 7f5f24c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5f2006c3b0 0x7f5f2006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:37.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.394+0000 7f5f1fe00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f200caf80 0x7f5f2015f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:37.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.394+0000 7f5f24c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5f2006c3b0 0x7f5f2006d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39670/0 (socket says 172.21.15.29:39670) 2024-04-17T19:52:37.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.394+0000 7f5f24c00640 1 -- 172.21.15.29:0/3517820647 learned_addr learned my addr 172.21.15.29:0/3517820647 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:37.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.394+0000 7f5f25600640 1 --2- 172.21.15.29:0/3517820647 >> v2:172.21.15.37:3300/0 conn(0x7f5f2015ca30 0x7f5f200cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:37.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.394+0000 7f5f24c00640 1 -- 172.21.15.29:0/3517820647 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f200caf80 msgr2=0x7f5f2015f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:37.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.394+0000 7f5f24c00640 1 --2- 172.21.15.29:0/3517820647 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f200caf80 0x7f5f2015f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:37.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.394+0000 7f5f24c00640 1 -- 172.21.15.29:0/3517820647 >> v2:172.21.15.37:3300/0 conn(0x7f5f2015ca30 msgr2=0x7f5f200cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:37.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.394+0000 7f5f24c00640 1 --2- 172.21.15.29:0/3517820647 >> v2:172.21.15.37:3300/0 conn(0x7f5f2015ca30 0x7f5f200cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:37.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.394+0000 7f5f24c00640 1 -- 172.21.15.29:0/3517820647 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f20160250 con 0x7f5f2006c3b0 2024-04-17T19:52:37.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.398+0000 7f5f24c00640 1 --2- 172.21.15.29:0/3517820647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f2006c3b0 0x7f5f2006d260 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5f20073730 tx=0x7f5f1401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45694148ecd687e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:37.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.398+0000 7f5f1f400640 1 -- 172.21.15.29:0/3517820647 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f140b0070 con 0x7f5f2006c3b0 2024-04-17T19:52:37.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.398+0000 7f5f1f400640 1 -- 172.21.15.29:0/3517820647 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f140ab600 con 0x7f5f2006c3b0 2024-04-17T19:52:37.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.398+0000 7f5f1f400640 1 -- 172.21.15.29:0/3517820647 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f140ab9d0 con 0x7f5f2006c3b0 2024-04-17T19:52:37.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.398+0000 7f5f27600640 1 -- 172.21.15.29:0/3517820647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f2006c3b0 msgr2=0x7f5f2006d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:37.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.398+0000 7f5f27600640 1 --2- 172.21.15.29:0/3517820647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f2006c3b0 0x7f5f2006d260 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5f20073730 tx=0x7f5f1401c520 comp rx=0 tx=0).stop 2024-04-17T19:52:37.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.398+0000 7f5f27600640 1 -- 172.21.15.29:0/3517820647 shutdown_connections 2024-04-17T19:52:37.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.398+0000 7f5f27600640 1 --2- 172.21.15.29:0/3517820647 >> v2:172.21.15.37:3300/0 conn(0x7f5f2015ca30 0x7f5f200cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:37.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.398+0000 7f5f27600640 1 --2- 172.21.15.29:0/3517820647 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f200caf80 0x7f5f2015f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:37.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.398+0000 7f5f27600640 1 --2- 172.21.15.29:0/3517820647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f2006c3b0 0x7f5f2006d260 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:37.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.398+0000 7f5f27600640 1 -- 172.21.15.29:0/3517820647 >> 172.21.15.29:0/3517820647 conn(0x7f5f2006e200 msgr2=0x7f5f2006e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:37.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.398+0000 7f5f27600640 1 -- 172.21.15.29:0/3517820647 shutdown_connections 2024-04-17T19:52:37.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.398+0000 7f5f27600640 1 -- 172.21.15.29:0/3517820647 wait complete. 2024-04-17T19:52:37.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.402+0000 7f5f27600640 1 Processor -- start 2024-04-17T19:52:37.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.402+0000 7f5f27600640 1 -- start start 2024-04-17T19:52:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.402+0000 7f5f27600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f2006c3b0 0x7f5f200cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.402+0000 7f5f27600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f200caf80 0x7f5f20152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.402+0000 7f5f27600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f2015ca30 0x7f5f2014ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.402+0000 7f5f27600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5f200704e0 con 0x7f5f2015ca30 2024-04-17T19:52:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.402+0000 7f5f27600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5f20070360 con 0x7f5f200caf80 2024-04-17T19:52:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.402+0000 7f5f27600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5f20070660 con 0x7f5f2006c3b0 2024-04-17T19:52:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.402+0000 7f5f25600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f2015ca30 0x7f5f2014ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.402+0000 7f5f24c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f2006c3b0 0x7f5f200cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.402+0000 7f5f1fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f200caf80 0x7f5f20152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.402+0000 7f5f25600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f2015ca30 0x7f5f2014ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39684/0 (socket says 172.21.15.29:39684) 2024-04-17T19:52:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.402+0000 7f5f25600640 1 -- 172.21.15.29:0/287675370 learned_addr learned my addr 172.21.15.29:0/287675370 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.402+0000 7f5f25600640 1 -- 172.21.15.29:0/287675370 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f2006c3b0 msgr2=0x7f5f200cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:37.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.402+0000 7f5f25600640 1 --2- 172.21.15.29:0/287675370 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f2006c3b0 0x7f5f200cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.402+0000 7f5f25600640 1 -- 172.21.15.29:0/287675370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f200caf80 msgr2=0x7f5f20152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.402+0000 7f5f25600640 1 --2- 172.21.15.29:0/287675370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f200caf80 0x7f5f20152290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.402+0000 7f5f25600640 1 -- 172.21.15.29:0/287675370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f200cf6e0 con 0x7f5f2015ca30 2024-04-17T19:52:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.406+0000 7f5f25600640 1 --2- 172.21.15.29:0/287675370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f2015ca30 0x7f5f2014ca20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800ef90 tx=0x7f5f1800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:37.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.406+0000 7f5f1cc00640 1 -- 172.21.15.29:0/287675370 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f18018020 con 0x7f5f2015ca30 2024-04-17T19:52:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.406+0000 7f5f1cc00640 1 -- 172.21.15.29:0/287675370 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f18014480 con 0x7f5f2015ca30 2024-04-17T19:52:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.406+0000 7f5f27600640 1 -- 172.21.15.29:0/287675370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f20158a20 con 0x7f5f2015ca30 2024-04-17T19:52:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.406+0000 7f5f1cc00640 1 -- 172.21.15.29:0/287675370 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f18009e60 con 0x7f5f2015ca30 2024-04-17T19:52:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.406+0000 7f5f27600640 1 -- 172.21.15.29:0/287675370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f200ce9a0 con 0x7f5f2015ca30 2024-04-17T19:52:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.406+0000 7f5f27600640 1 -- 172.21.15.29:0/287675370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f20069b90 con 0x7f5f2015ca30 2024-04-17T19:52:37.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.406+0000 7f5f1cc00640 1 -- 172.21.15.29:0/287675370 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5f1801b020 con 0x7f5f2015ca30 2024-04-17T19:52:37.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.410+0000 7f5f1cc00640 1 --2- 172.21.15.29:0/287675370 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5f00068380 0x7f5f0006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:37.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.410+0000 7f5f1cc00640 1 -- 172.21.15.29:0/287675370 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6907+0+0 (secure 0 0 0) 0x7f5f18087ab0 con 0x7f5f2015ca30 2024-04-17T19:52:37.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.410+0000 7f5f24c00640 1 --2- 172.21.15.29:0/287675370 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5f00068380 0x7f5f0006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:37.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.410+0000 7f5f24c00640 1 --2- 172.21.15.29:0/287675370 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5f00068380 0x7f5f0006a890 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5f140ad000 tx=0x7f5f14008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:37.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.410+0000 7f5f1cc00640 1 -- 172.21.15.29:0/287675370 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5f18022030 con 0x7f5f2015ca30 2024-04-17T19:52:37.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.610+0000 7f5f27600640 1 -- 172.21.15.29:0/287675370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f20159d30 con 0x7f5f2015ca30 2024-04-17T19:52:37.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.610+0000 7f5f1cc00640 1 -- 172.21.15.29:0/287675370 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6083 (secure 0 0 0) 0x7f5f18010040 con 0x7f5f2015ca30 2024-04-17T19:52:37.607 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:37.607 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:34.862105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4229,4259,4302,5212,4305]},"id":1}]} 2024-04-17T19:52:37.607 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 30 2024-04-17T19:52:37.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.614+0000 7f5f27600640 1 -- 172.21.15.29:0/287675370 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5f00068380 msgr2=0x7f5f0006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:37.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.614+0000 7f5f27600640 1 --2- 172.21.15.29:0/287675370 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5f00068380 0x7f5f0006a890 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5f140ad000 tx=0x7f5f14008040 comp rx=0 tx=0).stop 2024-04-17T19:52:37.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.614+0000 7f5f27600640 1 -- 172.21.15.29:0/287675370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f2015ca30 msgr2=0x7f5f2014ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:37.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.614+0000 7f5f27600640 1 --2- 172.21.15.29:0/287675370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f2015ca30 0x7f5f2014ca20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5f1800ef90 tx=0x7f5f1800c490 comp rx=0 tx=0).stop 2024-04-17T19:52:37.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.614+0000 7f5f27600640 1 -- 172.21.15.29:0/287675370 shutdown_connections 2024-04-17T19:52:37.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.614+0000 7f5f27600640 1 --2- 172.21.15.29:0/287675370 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5f00068380 0x7f5f0006a890 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:37.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.614+0000 7f5f27600640 1 --2- 172.21.15.29:0/287675370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f2015ca30 0x7f5f2014ca20 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:37.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.614+0000 7f5f27600640 1 --2- 172.21.15.29:0/287675370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5f200caf80 0x7f5f20152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:37.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.614+0000 7f5f27600640 1 --2- 172.21.15.29:0/287675370 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5f2006c3b0 0x7f5f200cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:37.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.614+0000 7f5f27600640 1 -- 172.21.15.29:0/287675370 >> 172.21.15.29:0/287675370 conn(0x7f5f2006e200 msgr2=0x7f5f20157a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:37.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.614+0000 7f5f27600640 1 -- 172.21.15.29:0/287675370 shutdown_connections 2024-04-17T19:52:37.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:37.614+0000 7f5f27600640 1 -- 172.21.15.29:0/287675370 wait complete. 2024-04-17T19:52:37.629 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-04-17T19:52:39.630 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:39.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2d0800640 1 Processor -- start 2024-04-17T19:52:39.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2d0800640 1 -- start start 2024-04-17T19:52:39.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2d0800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd2c81625e0 0x7fd2c8164a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:39.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2d0800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2c80cd0e0 con 0x7fd2c80cccd0 2024-04-17T19:52:39.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2d0800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2c80c88b0 con 0x7fd2c80cc8c0 2024-04-17T19:52:39.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2d0800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd2c80c6ce0 con 0x7fd2c81625e0 2024-04-17T19:52:39.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cd400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd2c80cccd0 0x7fd2c80ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45204/0 (socket says 172.21.15.29:45204) 2024-04-17T19:52:39.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cd400640 1 -- 172.21.15.29:0/240948981 learned_addr learned my addr 172.21.15.29:0/240948981 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:52:39.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2ce800640 1 --2- 172.21.15.29:0/240948981 >> v2:172.21.15.37:3300/0 conn(0x7fd2c81625e0 0x7fd2c8164a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:39.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cca00640 1 -- 172.21.15.29:0/240948981 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3676177067 0 0) 0x7fd2c80cd0e0 con 0x7fd2c80cccd0 2024-04-17T19:52:39.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cca00640 1 -- 172.21.15.29:0/240948981 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2b0003530 con 0x7fd2c80cccd0 2024-04-17T19:52:39.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cca00640 1 -- 172.21.15.29:0/240948981 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2480990560 0 0) 0x7fd2c80c88b0 con 0x7fd2c80cc8c0 2024-04-17T19:52:39.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cca00640 1 -- 172.21.15.29:0/240948981 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2c80cd0e0 con 0x7fd2c80cc8c0 2024-04-17T19:52:39.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cca00640 1 -- 172.21.15.29:0/240948981 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 164203039 0 0) 0x7fd2b0003530 con 0x7fd2c80cccd0 2024-04-17T19:52:39.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cca00640 1 -- 172.21.15.29:0/240948981 >> v2:172.21.15.37:3300/0 conn(0x7fd2c81625e0 msgr2=0x7fd2c8164a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:39.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cca00640 1 --2- 172.21.15.29:0/240948981 >> v2:172.21.15.37:3300/0 conn(0x7fd2c81625e0 0x7fd2c8164a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:39.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cca00640 1 -- 172.21.15.29:0/240948981 >> v1:172.21.15.37:6789/0 conn(0x7fd2c80cc8c0 legacy=0x7fd2c8159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:39.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cca00640 1 -- 172.21.15.29:0/240948981 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c80c6fc0 con 0x7fd2c80cccd0 2024-04-17T19:52:39.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cca00640 1 -- 172.21.15.29:0/240948981 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fd2b8001680 con 0x7fd2c80cccd0 2024-04-17T19:52:39.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cca00640 1 -- 172.21.15.29:0/240948981 >> v1:172.21.15.29:6789/0 conn(0x7fd2c80cccd0 legacy=0x7fd2c80ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:39.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cca00640 1 --2- 172.21.15.29:0/240948981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b0003bc0 0x7fd2b0004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:39.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cca00640 1 --2- 172.21.15.29:0/240948981 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2b0004810 0x7fd2b0004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:39.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cca00640 1 --2- 172.21.15.29:0/240948981 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2b0005220 0x7fd2b000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:39.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cca00640 1 -- 172.21.15.29:0/240948981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c80c6ce0 con 0x7fd2b0003bc0 2024-04-17T19:52:39.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cca00640 1 -- 172.21.15.29:0/240948981 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd2b000c1c0 con 0x7fd2b0005220 2024-04-17T19:52:39.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cca00640 1 -- 172.21.15.29:0/240948981 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd2b000c3c0 con 0x7fd2b0004810 2024-04-17T19:52:39.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cde00640 1 --2- 172.21.15.29:0/240948981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b0003bc0 0x7fd2b0004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:39.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2ce800640 1 --2- 172.21.15.29:0/240948981 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2b0005220 0x7fd2b000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:39.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.790+0000 7fd2cd400640 1 --2- 172.21.15.29:0/240948981 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2b0004810 0x7fd2b0004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:39.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.794+0000 7fd2cde00640 1 -- 172.21.15.29:0/240948981 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2b0004810 msgr2=0x7fd2b0004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:39.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.794+0000 7fd2cde00640 1 --2- 172.21.15.29:0/240948981 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2b0004810 0x7fd2b0004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:39.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.794+0000 7fd2cde00640 1 -- 172.21.15.29:0/240948981 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2b0005220 msgr2=0x7fd2b000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:39.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.794+0000 7fd2cde00640 1 --2- 172.21.15.29:0/240948981 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2b0005220 0x7fd2b000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:39.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.794+0000 7fd2cde00640 1 -- 172.21.15.29:0/240948981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2b0003530 con 0x7fd2b0003bc0 2024-04-17T19:52:39.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.794+0000 7fd2ce800640 1 -- 172.21.15.29:0/240948981 reap_dead start 2024-04-17T19:52:39.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.794+0000 7fd2cde00640 1 --2- 172.21.15.29:0/240948981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b0003bc0 0x7fd2b0004010 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0018d0 tx=0x7fd2bc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4239cbfa38172ecb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:39.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.794+0000 7fd2cca00640 1 -- 172.21.15.29:0/240948981 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc001520 con 0x7fd2b0003bc0 2024-04-17T19:52:39.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.794+0000 7fd2cca00640 1 -- 172.21.15.29:0/240948981 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2bc0052f0 con 0x7fd2b0003bc0 2024-04-17T19:52:39.790 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.794+0000 7fd2cca00640 1 -- 172.21.15.29:0/240948981 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc003840 con 0x7fd2b0003bc0 2024-04-17T19:52:39.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.794+0000 7fd2d0800640 1 -- 172.21.15.29:0/240948981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b0003bc0 msgr2=0x7fd2b0004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:39.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.794+0000 7fd2d0800640 1 --2- 172.21.15.29:0/240948981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b0003bc0 0x7fd2b0004010 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0018d0 tx=0x7fd2bc01c520 comp rx=0 tx=0).stop 2024-04-17T19:52:39.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.798+0000 7fd2d0800640 1 -- 172.21.15.29:0/240948981 shutdown_connections 2024-04-17T19:52:39.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.798+0000 7fd2d0800640 1 --2- 172.21.15.29:0/240948981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b0003bc0 0x7fd2b0004010 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:39.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.798+0000 7fd2d0800640 1 -- 172.21.15.29:0/240948981 >> 172.21.15.29:0/240948981 conn(0x7fd2c806fde0 msgr2=0x7fd2c80701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:39.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.798+0000 7fd2d0800640 1 -- 172.21.15.29:0/240948981 shutdown_connections 2024-04-17T19:52:39.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.798+0000 7fd2d0800640 1 -- 172.21.15.29:0/240948981 wait complete. 2024-04-17T19:52:39.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.798+0000 7fd2d0800640 1 Processor -- start 2024-04-17T19:52:39.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.798+0000 7fd2d0800640 1 -- start start 2024-04-17T19:52:39.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.798+0000 7fd2d0800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b0003bc0 0x7fd2c81f5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:39.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.798+0000 7fd2d0800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2c81e9b00 0x7fd2c813ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:39.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.798+0000 7fd2d0800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2c81e7700 0x7fd2c8140a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:39.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.798+0000 7fd2d0800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c80c6a00 con 0x7fd2b0003bc0 2024-04-17T19:52:39.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.798+0000 7fd2d0800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd2c80c6880 con 0x7fd2c81e9b00 2024-04-17T19:52:39.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.798+0000 7fd2d0800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd2c80ca300 con 0x7fd2c81e7700 2024-04-17T19:52:39.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.802+0000 7fd2cde00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b0003bc0 0x7fd2c81f5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:39.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.802+0000 7fd2cde00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b0003bc0 0x7fd2c81f5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39404/0 (socket says 172.21.15.29:39404) 2024-04-17T19:52:39.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.802+0000 7fd2cde00640 1 -- 172.21.15.29:0/2652736288 learned_addr learned my addr 172.21.15.29:0/2652736288 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:39.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.802+0000 7fd2cd400640 1 --2- 172.21.15.29:0/2652736288 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2c81e9b00 0x7fd2c813ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:39.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.802+0000 7fd2ce800640 1 --2- 172.21.15.29:0/2652736288 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2c81e7700 0x7fd2c8140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:39.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.802+0000 7fd2cde00640 1 -- 172.21.15.29:0/2652736288 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2c81e7700 msgr2=0x7fd2c8140a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:39.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.802+0000 7fd2cde00640 1 --2- 172.21.15.29:0/2652736288 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2c81e7700 0x7fd2c8140a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:39.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.802+0000 7fd2cde00640 1 -- 172.21.15.29:0/2652736288 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2c81e9b00 msgr2=0x7fd2c813ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:39.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.802+0000 7fd2cde00640 1 --2- 172.21.15.29:0/2652736288 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2c81e9b00 0x7fd2c813ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:39.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.802+0000 7fd2cde00640 1 -- 172.21.15.29:0/2652736288 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2c81f9230 con 0x7fd2b0003bc0 2024-04-17T19:52:39.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.802+0000 7fd2cde00640 1 --2- 172.21.15.29:0/2652736288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b0003bc0 0x7fd2c81f5710 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0ad040 tx=0x7fd2b0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:39.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.802+0000 7fd2c2000640 1 -- 172.21.15.29:0/2652736288 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc0054a0 con 0x7fd2b0003bc0 2024-04-17T19:52:39.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.802+0000 7fd2c2000640 1 -- 172.21.15.29:0/2652736288 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2bc001500 con 0x7fd2b0003bc0 2024-04-17T19:52:39.797 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.802+0000 7fd2c2000640 1 -- 172.21.15.29:0/2652736288 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2bc0067e0 con 0x7fd2b0003bc0 2024-04-17T19:52:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.802+0000 7fd2d0800640 1 -- 172.21.15.29:0/2652736288 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2c81f5c50 con 0x7fd2b0003bc0 2024-04-17T19:52:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.802+0000 7fd2d0800640 1 -- 172.21.15.29:0/2652736288 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2c815cb80 con 0x7fd2b0003bc0 2024-04-17T19:52:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.802+0000 7fd2d0800640 1 -- 172.21.15.29:0/2652736288 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2c8069b90 con 0x7fd2b0003bc0 2024-04-17T19:52:39.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.802+0000 7fd2c2000640 1 -- 172.21.15.29:0/2652736288 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd2bc006980 con 0x7fd2b0003bc0 2024-04-17T19:52:39.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.806+0000 7fd2c2000640 1 --2- 172.21.15.29:0/2652736288 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd2ac068380 0x7fd2ac06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:39.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.806+0000 7fd2c2000640 1 -- 172.21.15.29:0/2652736288 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6907+0+0 (secure 0 0 0) 0x7fd2bc120f80 con 0x7fd2b0003bc0 2024-04-17T19:52:39.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.806+0000 7fd2cd400640 1 --2- 172.21.15.29:0/2652736288 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd2ac068380 0x7fd2ac06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:39.801 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.806+0000 7fd2cd400640 1 --2- 172.21.15.29:0/2652736288 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd2ac068380 0x7fd2ac06a890 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd2c81f6340 tx=0x7fd2b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:39.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:39.810+0000 7fd2c2000640 1 -- 172.21.15.29:0/2652736288 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd2bc0b8020 con 0x7fd2b0003bc0 2024-04-17T19:52:39.883 INFO:tasks.workunit.client.0.smithi029.stdout:.//.snap/test1 2024-04-17T19:52:39.883 INFO:tasks.workunit.client.0.smithi029.stdout:.//.snap/test2 2024-04-17T19:52:39.884 INFO:tasks.workunit.client.0.smithi029.stdout:.//.snap/test3 2024-04-17T19:52:39.889 INFO:tasks.workunit.client.0.smithi029.stdout:./274/.snap/snap-subdir-test 2024-04-17T19:52:39.895 INFO:tasks.workunit.client.0.smithi029.stdout:./166/.snap/snap-subdir-test 2024-04-17T19:52:39.900 INFO:tasks.workunit.client.0.smithi029.stdout:./256/.snap/snap-subdir-test 2024-04-17T19:52:39.905 INFO:tasks.workunit.client.0.smithi029.stdout:./247/.snap/snap-subdir-test 2024-04-17T19:52:40.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:40.026+0000 7fd2d0800640 1 -- 172.21.15.29:0/2652736288 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2c80c7e10 con 0x7fd2b0003bc0 2024-04-17T19:52:40.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:40.026+0000 7fd2c2000640 1 -- 172.21.15.29:0/2652736288 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6907 (secure 0 0 0) 0x7fd2bc016230 con 0x7fd2b0003bc0 2024-04-17T19:52:40.020 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:40.020 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:34.862105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5212,4302,4259,4229]},"id":1}]} 2024-04-17T19:52:40.020 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-04-17T19:52:40.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:40.026+0000 7fd2d0800640 1 -- 172.21.15.29:0/2652736288 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd2ac068380 msgr2=0x7fd2ac06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:40.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:40.026+0000 7fd2d0800640 1 --2- 172.21.15.29:0/2652736288 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd2ac068380 0x7fd2ac06a890 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd2c81f6340 tx=0x7fd2b8009040 comp rx=0 tx=0).stop 2024-04-17T19:52:40.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:40.026+0000 7fd2d0800640 1 -- 172.21.15.29:0/2652736288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b0003bc0 msgr2=0x7fd2c81f5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:40.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:40.026+0000 7fd2d0800640 1 --2- 172.21.15.29:0/2652736288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b0003bc0 0x7fd2c81f5710 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0ad040 tx=0x7fd2b0005840 comp rx=0 tx=0).stop 2024-04-17T19:52:40.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:40.026+0000 7fd2d0800640 1 -- 172.21.15.29:0/2652736288 shutdown_connections 2024-04-17T19:52:40.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:40.026+0000 7fd2d0800640 1 --2- 172.21.15.29:0/2652736288 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd2ac068380 0x7fd2ac06a890 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:40.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:40.026+0000 7fd2d0800640 1 --2- 172.21.15.29:0/2652736288 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd2c81e7700 0x7fd2c8140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:40.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:40.026+0000 7fd2d0800640 1 --2- 172.21.15.29:0/2652736288 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd2c81e9b00 0x7fd2c813ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:40.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:40.026+0000 7fd2d0800640 1 --2- 172.21.15.29:0/2652736288 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b0003bc0 0x7fd2c81f5710 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:40.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:40.026+0000 7fd2d0800640 1 -- 172.21.15.29:0/2652736288 >> 172.21.15.29:0/2652736288 conn(0x7fd2c806fde0 msgr2=0x7fd2c806c760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:40.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:40.026+0000 7fd2d0800640 1 -- 172.21.15.29:0/2652736288 shutdown_connections 2024-04-17T19:52:40.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:40.026+0000 7fd2d0800640 1 -- 172.21.15.29:0/2652736288 wait complete. 2024-04-17T19:52:40.042 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2024-04-17T19:52:40.043 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 16 secs before thrashing 2024-04-17T19:52:40.656 INFO:tasks.workunit.client.0.smithi029.stdout:./248/.snap/snap-subdir-test 2024-04-17T19:52:40.662 INFO:tasks.workunit.client.0.smithi029.stdout:./147/.snap/snap-subdir-test 2024-04-17T19:52:40.668 INFO:tasks.workunit.client.0.smithi029.stdout:./151/.snap/snap-subdir-test 2024-04-17T19:52:40.674 INFO:tasks.workunit.client.0.smithi029.stdout:./170/.snap/snap-subdir-test 2024-04-17T19:52:40.681 INFO:tasks.workunit.client.0.smithi029.stdout:./167/.snap/snap-subdir-test 2024-04-17T19:52:40.687 INFO:tasks.workunit.client.0.smithi029.stdout:./140/.snap/snap-subdir-test 2024-04-17T19:52:40.694 INFO:tasks.workunit.client.0.smithi029.stdout:./142/.snap/snap-subdir-test 2024-04-17T19:52:40.700 INFO:tasks.workunit.client.0.smithi029.stdout:./150/.snap/snap-subdir-test 2024-04-17T19:52:40.705 INFO:tasks.workunit.client.0.smithi029.stdout:./273/.snap/snap-subdir-test 2024-04-17T19:52:40.710 INFO:tasks.workunit.client.0.smithi029.stdout:./269/.snap/snap-subdir-test 2024-04-17T19:52:40.717 INFO:tasks.workunit.client.0.smithi029.stdout:./105/.snap/snap-subdir-test 2024-04-17T19:52:40.722 INFO:tasks.workunit.client.0.smithi029.stdout:./210/.snap/snap-subdir-test 2024-04-17T19:52:40.727 INFO:tasks.workunit.client.0.smithi029.stdout:./265/.snap/snap-subdir-test 2024-04-17T19:52:40.732 INFO:tasks.workunit.client.0.smithi029.stdout:./218/.snap/snap-subdir-test 2024-04-17T19:52:40.738 INFO:tasks.workunit.client.0.smithi029.stdout:./290/.snap/snap-subdir-test 2024-04-17T19:52:40.744 INFO:tasks.workunit.client.0.smithi029.stdout:./149/.snap/snap-subdir-test 2024-04-17T19:52:40.750 INFO:tasks.workunit.client.0.smithi029.stdout:./109/.snap/snap-subdir-test 2024-04-17T19:52:40.754 INFO:tasks.workunit.client.0.smithi029.stdout:./292/.snap/snap-subdir-test 2024-04-17T19:52:40.760 INFO:tasks.workunit.client.0.smithi029.stdout:./242/.snap/snap-subdir-test 2024-04-17T19:52:40.765 INFO:tasks.workunit.client.0.smithi029.stdout:./158/.snap/snap-subdir-test 2024-04-17T19:52:40.769 INFO:tasks.workunit.client.0.smithi029.stdout:./243/.snap/snap-subdir-test 2024-04-17T19:52:40.774 INFO:tasks.workunit.client.0.smithi029.stdout:./106/.snap/snap-subdir-test 2024-04-17T19:52:40.779 INFO:tasks.workunit.client.0.smithi029.stdout:./250/.snap/snap-subdir-test 2024-04-17T19:52:40.784 INFO:tasks.workunit.client.0.smithi029.stdout:./221/.snap/snap-subdir-test 2024-04-17T19:52:40.790 INFO:tasks.workunit.client.0.smithi029.stdout:./244/.snap/snap-subdir-test 2024-04-17T19:52:40.795 INFO:tasks.workunit.client.0.smithi029.stdout:./258/.snap/snap-subdir-test 2024-04-17T19:52:40.800 INFO:tasks.workunit.client.0.smithi029.stdout:./268/.snap/snap-subdir-test 2024-04-17T19:52:40.806 INFO:tasks.workunit.client.0.smithi029.stdout:./189/.snap/snap-subdir-test 2024-04-17T19:52:40.810 INFO:tasks.workunit.client.0.smithi029.stdout:./285/.snap/snap-subdir-test 2024-04-17T19:52:40.814 INFO:tasks.workunit.client.0.smithi029.stdout:./208/.snap/snap-subdir-test 2024-04-17T19:52:40.819 INFO:tasks.workunit.client.0.smithi029.stdout:./288/.snap/snap-subdir-test 2024-04-17T19:52:40.825 INFO:tasks.workunit.client.0.smithi029.stdout:./209/.snap/snap-subdir-test 2024-04-17T19:52:40.830 INFO:tasks.workunit.client.0.smithi029.stdout:./253/.snap/snap-subdir-test 2024-04-17T19:52:40.836 INFO:tasks.workunit.client.0.smithi029.stdout:./249/.snap/snap-subdir-test 2024-04-17T19:52:40.842 INFO:tasks.workunit.client.0.smithi029.stdout:./133/.snap/snap-subdir-test 2024-04-17T19:52:40.847 INFO:tasks.workunit.client.0.smithi029.stdout:./291/.snap/snap-subdir-test 2024-04-17T19:52:40.853 INFO:tasks.workunit.client.0.smithi029.stdout:./226/.snap/snap-subdir-test 2024-04-17T19:52:40.858 INFO:tasks.workunit.client.0.smithi029.stdout:./173/.snap/snap-subdir-test 2024-04-17T19:52:40.864 INFO:tasks.workunit.client.0.smithi029.stdout:./174/.snap/snap-subdir-test 2024-04-17T19:52:40.868 INFO:tasks.workunit.client.0.smithi029.stdout:./299/.snap/snap-subdir-test 2024-04-17T19:52:40.873 INFO:tasks.workunit.client.0.smithi029.stdout:./262/.snap/snap-subdir-test 2024-04-17T19:52:40.878 INFO:tasks.workunit.client.0.smithi029.stdout:./231/.snap/snap-subdir-test 2024-04-17T19:52:40.883 INFO:tasks.workunit.client.0.smithi029.stdout:./214/.snap/snap-subdir-test 2024-04-17T19:52:40.897 INFO:tasks.workunit.client.0.smithi029.stdout:./281/.snap/snap-subdir-test 2024-04-17T19:52:40.904 INFO:tasks.workunit.client.0.smithi029.stdout:./289/.snap/snap-subdir-test 2024-04-17T19:52:40.909 INFO:tasks.workunit.client.0.smithi029.stdout:./283/.snap/snap-subdir-test 2024-04-17T19:52:40.915 INFO:tasks.workunit.client.0.smithi029.stdout:./232/.snap/snap-subdir-test 2024-04-17T19:52:40.919 INFO:tasks.workunit.client.0.smithi029.stdout:./298/.snap/snap-subdir-test 2024-04-17T19:52:40.925 INFO:tasks.workunit.client.0.smithi029.stdout:./233/.snap/snap-subdir-test 2024-04-17T19:52:40.931 INFO:tasks.workunit.client.0.smithi029.stdout:./155/.snap/snap-subdir-test 2024-04-17T19:52:40.936 INFO:tasks.workunit.client.0.smithi029.stdout:./114/.snap/snap-subdir-test 2024-04-17T19:52:40.942 INFO:tasks.workunit.client.0.smithi029.stdout:./159/.snap/snap-subdir-test 2024-04-17T19:52:40.947 INFO:tasks.workunit.client.0.smithi029.stdout:./187/.snap/snap-subdir-test 2024-04-17T19:52:40.952 INFO:tasks.workunit.client.0.smithi029.stdout:./185/.snap/snap-subdir-test 2024-04-17T19:52:40.958 INFO:tasks.workunit.client.0.smithi029.stdout:./293/.snap/snap-subdir-test 2024-04-17T19:52:40.963 INFO:tasks.workunit.client.0.smithi029.stdout:./228/.snap/snap-subdir-test 2024-04-17T19:52:40.969 INFO:tasks.workunit.client.0.smithi029.stdout:./120/.snap/snap-subdir-test 2024-04-17T19:52:40.973 INFO:tasks.workunit.client.0.smithi029.stdout:./110/.snap/snap-subdir-test 2024-04-17T19:52:40.978 INFO:tasks.workunit.client.0.smithi029.stdout:./267/.snap/snap-subdir-test 2024-04-17T19:52:41.234 INFO:tasks.workunit.client.0.smithi029.stdout:./202/.snap/snap-subdir-test 2024-04-17T19:52:41.240 INFO:tasks.workunit.client.0.smithi029.stdout:./137/.snap/snap-subdir-test 2024-04-17T19:52:41.245 INFO:tasks.workunit.client.0.smithi029.stdout:./246/.snap/snap-subdir-test 2024-04-17T19:52:41.250 INFO:tasks.workunit.client.0.smithi029.stdout:./219/.snap/snap-subdir-test 2024-04-17T19:52:41.255 INFO:tasks.workunit.client.0.smithi029.stdout:./198/.snap/snap-subdir-test 2024-04-17T19:52:41.261 INFO:tasks.workunit.client.0.smithi029.stdout:./136/.snap/snap-subdir-test 2024-04-17T19:52:41.265 INFO:tasks.workunit.client.0.smithi029.stdout:./270/.snap/snap-subdir-test 2024-04-17T19:52:41.271 INFO:tasks.workunit.client.0.smithi029.stdout:./125/.snap/snap-subdir-test 2024-04-17T19:52:41.277 INFO:tasks.workunit.client.0.smithi029.stdout:./194/.snap/snap-subdir-test 2024-04-17T19:52:41.282 INFO:tasks.workunit.client.0.smithi029.stdout:./119/.snap/snap-subdir-test 2024-04-17T19:52:41.288 INFO:tasks.workunit.client.0.smithi029.stdout:./297/.snap/snap-subdir-test 2024-04-17T19:52:41.293 INFO:tasks.workunit.client.0.smithi029.stdout:./230/.snap/snap-subdir-test 2024-04-17T19:52:41.299 INFO:tasks.workunit.client.0.smithi029.stdout:./260/.snap/snap-subdir-test 2024-04-17T19:52:41.305 INFO:tasks.workunit.client.0.smithi029.stdout:./157/.snap/snap-subdir-test 2024-04-17T19:52:41.310 INFO:tasks.workunit.client.0.smithi029.stdout:./172/.snap/snap-subdir-test 2024-04-17T19:52:41.316 INFO:tasks.workunit.client.0.smithi029.stdout:./277/.snap/snap-subdir-test 2024-04-17T19:52:41.321 INFO:tasks.workunit.client.0.smithi029.stdout:./186/.snap/snap-subdir-test 2024-04-17T19:52:41.327 INFO:tasks.workunit.client.0.smithi029.stdout:./237/.snap/snap-subdir-test 2024-04-17T19:52:41.333 INFO:tasks.workunit.client.0.smithi029.stdout:./190/.snap/snap-subdir-test 2024-04-17T19:52:41.338 INFO:tasks.workunit.client.0.smithi029.stdout:./213/.snap/snap-subdir-test 2024-04-17T19:52:41.344 INFO:tasks.workunit.client.0.smithi029.stdout:./104/.snap/snap-subdir-test 2024-04-17T19:52:41.350 INFO:tasks.workunit.client.0.smithi029.stdout:./164/.snap/snap-subdir-test 2024-04-17T19:52:41.354 INFO:tasks.workunit.client.0.smithi029.stdout:./112/.snap/snap-subdir-test 2024-04-17T19:52:41.359 INFO:tasks.workunit.client.0.smithi029.stdout:./295/.snap/snap-subdir-test 2024-04-17T19:52:41.364 INFO:tasks.workunit.client.0.smithi029.stdout:./272/.snap/snap-subdir-test 2024-04-17T19:52:41.369 INFO:tasks.workunit.client.0.smithi029.stdout:./100/.snap/snap-subdir-test 2024-04-17T19:52:41.375 INFO:tasks.workunit.client.0.smithi029.stdout:./134/.snap/snap-subdir-test 2024-04-17T19:52:41.381 INFO:tasks.workunit.client.0.smithi029.stdout:./200/.snap/snap-subdir-test 2024-04-17T19:52:41.641 INFO:tasks.workunit.client.0.smithi029.stdout:./115/.snap/snap-subdir-test 2024-04-17T19:52:41.648 INFO:tasks.workunit.client.0.smithi029.stdout:./284/.snap/snap-subdir-test 2024-04-17T19:52:41.654 INFO:tasks.workunit.client.0.smithi029.stdout:./280/.snap/snap-subdir-test 2024-04-17T19:52:42.168 INFO:tasks.workunit.client.0.smithi029.stdout:./116/.snap/snap-subdir-test 2024-04-17T19:52:42.190 INFO:tasks.workunit.client.0.smithi029.stdout:./245/.snap/snap-subdir-test 2024-04-17T19:52:42.197 INFO:tasks.workunit.client.0.smithi029.stdout:./195/.snap/snap-subdir-test 2024-04-17T19:52:42.204 INFO:tasks.workunit.client.0.smithi029.stdout:./138/.snap/snap-subdir-test 2024-04-17T19:52:42.211 INFO:tasks.workunit.client.0.smithi029.stdout:./135/.snap/snap-subdir-test 2024-04-17T19:52:42.218 INFO:tasks.workunit.client.0.smithi029.stdout:./117/.snap/snap-subdir-test 2024-04-17T19:52:42.224 INFO:tasks.workunit.client.0.smithi029.stdout:./241/.snap/snap-subdir-test 2024-04-17T19:52:42.230 INFO:tasks.workunit.client.0.smithi029.stdout:./143/.snap/snap-subdir-test 2024-04-17T19:52:42.237 INFO:tasks.workunit.client.0.smithi029.stdout:./180/.snap/snap-subdir-test 2024-04-17T19:52:42.244 INFO:tasks.workunit.client.0.smithi029.stdout:./126/.snap/snap-subdir-test 2024-04-17T19:52:42.250 INFO:tasks.workunit.client.0.smithi029.stdout:./161/.snap/snap-subdir-test 2024-04-17T19:52:42.257 INFO:tasks.workunit.client.0.smithi029.stdout:./204/.snap/snap-subdir-test 2024-04-17T19:52:42.264 INFO:tasks.workunit.client.0.smithi029.stdout:./176/.snap/snap-subdir-test 2024-04-17T19:52:42.270 INFO:tasks.workunit.client.0.smithi029.stdout:./255/.snap/snap-subdir-test 2024-04-17T19:52:42.276 INFO:tasks.workunit.client.0.smithi029.stdout:./215/.snap/snap-subdir-test 2024-04-17T19:52:42.282 INFO:tasks.workunit.client.0.smithi029.stdout:./111/.snap/snap-subdir-test 2024-04-17T19:52:42.288 INFO:tasks.workunit.client.0.smithi029.stdout:./238/.snap/snap-subdir-test 2024-04-17T19:52:42.294 INFO:tasks.workunit.client.0.smithi029.stdout:./191/.snap/snap-subdir-test 2024-04-17T19:52:42.300 INFO:tasks.workunit.client.0.smithi029.stdout:./184/.snap/snap-subdir-test 2024-04-17T19:52:42.306 INFO:tasks.workunit.client.0.smithi029.stdout:./192/.snap/snap-subdir-test 2024-04-17T19:52:42.312 INFO:tasks.workunit.client.0.smithi029.stdout:./103/.snap/snap-subdir-test 2024-04-17T19:52:42.318 INFO:tasks.workunit.client.0.smithi029.stdout:./128/.snap/snap-subdir-test 2024-04-17T19:52:43.032 INFO:tasks.workunit.client.0.smithi029.stdout:./229/.snap/snap-subdir-test 2024-04-17T19:52:43.040 INFO:tasks.workunit.client.0.smithi029.stdout:./144/.snap/snap-subdir-test 2024-04-17T19:52:43.048 INFO:tasks.workunit.client.0.smithi029.stdout:./146/.snap/snap-subdir-test 2024-04-17T19:52:43.055 INFO:tasks.workunit.client.0.smithi029.stdout:./227/.snap/snap-subdir-test 2024-04-17T19:52:43.061 INFO:tasks.workunit.client.0.smithi029.stdout:./278/.snap/snap-subdir-test 2024-04-17T19:52:43.069 INFO:tasks.workunit.client.0.smithi029.stdout:./141/.snap/snap-subdir-test 2024-04-17T19:52:43.076 INFO:tasks.workunit.client.0.smithi029.stdout:./234/.snap/snap-subdir-test 2024-04-17T19:52:43.083 INFO:tasks.workunit.client.0.smithi029.stdout:./225/.snap/snap-subdir-test 2024-04-17T19:52:43.090 INFO:tasks.workunit.client.0.smithi029.stdout:./182/.snap/snap-subdir-test 2024-04-17T19:52:43.097 INFO:tasks.workunit.client.0.smithi029.stdout:./107/.snap/snap-subdir-test 2024-04-17T19:52:43.105 INFO:tasks.workunit.client.0.smithi029.stdout:./178/.snap/snap-subdir-test 2024-04-17T19:52:43.111 INFO:tasks.workunit.client.0.smithi029.stdout:./266/.snap/snap-subdir-test 2024-04-17T19:52:43.118 INFO:tasks.workunit.client.0.smithi029.stdout:./240/.snap/snap-subdir-test 2024-04-17T19:52:43.124 INFO:tasks.workunit.client.0.smithi029.stdout:./156/.snap/snap-subdir-test 2024-04-17T19:52:43.130 INFO:tasks.workunit.client.0.smithi029.stdout:./131/.snap/snap-subdir-test 2024-04-17T19:52:43.136 INFO:tasks.workunit.client.0.smithi029.stdout:./123/.snap/snap-subdir-test 2024-04-17T19:52:43.142 INFO:tasks.workunit.client.0.smithi029.stdout:./160/.snap/snap-subdir-test 2024-04-17T19:52:43.148 INFO:tasks.workunit.client.0.smithi029.stdout:./165/.snap/snap-subdir-test 2024-04-17T19:52:43.153 INFO:tasks.workunit.client.0.smithi029.stdout:./101/.snap/snap-subdir-test 2024-04-17T19:52:43.409 INFO:tasks.workunit.client.0.smithi029.stdout:./148/.snap/snap-subdir-test 2024-04-17T19:52:43.415 INFO:tasks.workunit.client.0.smithi029.stdout:./168/.snap/snap-subdir-test 2024-04-17T19:52:43.421 INFO:tasks.workunit.client.0.smithi029.stdout:./179/.snap/snap-subdir-test 2024-04-17T19:52:43.427 INFO:tasks.workunit.client.0.smithi029.stdout:./257/.snap/snap-subdir-test 2024-04-17T19:52:43.432 INFO:tasks.workunit.client.0.smithi029.stdout:./252/.snap/snap-subdir-test 2024-04-17T19:52:43.437 INFO:tasks.workunit.client.0.smithi029.stdout:./132/.snap/snap-subdir-test 2024-04-17T19:52:43.443 INFO:tasks.workunit.client.0.smithi029.stdout:./236/.snap/snap-subdir-test 2024-04-17T19:52:43.449 INFO:tasks.workunit.client.0.smithi029.stdout:./211/.snap/snap-subdir-test 2024-04-17T19:52:43.455 INFO:tasks.workunit.client.0.smithi029.stdout:./199/.snap/snap-subdir-test 2024-04-17T19:52:43.460 INFO:tasks.workunit.client.0.smithi029.stdout:./205/.snap/snap-subdir-test 2024-04-17T19:52:43.466 INFO:tasks.workunit.client.0.smithi029.stdout:./163/.snap/snap-subdir-test 2024-04-17T19:52:43.471 INFO:tasks.workunit.client.0.smithi029.stdout:./261/.snap/snap-subdir-test 2024-04-17T19:52:43.477 INFO:tasks.workunit.client.0.smithi029.stdout:./201/.snap/snap-subdir-test 2024-04-17T19:52:43.482 INFO:tasks.workunit.client.0.smithi029.stdout:./282/.snap/snap-subdir-test 2024-04-17T19:52:43.488 INFO:tasks.workunit.client.0.smithi029.stdout:./296/.snap/snap-subdir-test 2024-04-17T19:52:46.745 INFO:tasks.workunit.client.0.smithi029.stdout:./217/.snap/snap-subdir-test 2024-04-17T19:52:46.753 INFO:tasks.workunit.client.0.smithi029.stdout:./153/.snap/snap-subdir-test 2024-04-17T19:52:46.759 INFO:tasks.workunit.client.0.smithi029.stdout:./206/.snap/snap-subdir-test 2024-04-17T19:52:46.766 INFO:tasks.workunit.client.0.smithi029.stdout:./122/.snap/snap-subdir-test 2024-04-17T19:52:46.773 INFO:tasks.workunit.client.0.smithi029.stdout:./162/.snap/snap-subdir-test 2024-04-17T19:52:46.780 INFO:tasks.workunit.client.0.smithi029.stdout:./197/.snap/snap-subdir-test 2024-04-17T19:52:46.785 INFO:tasks.workunit.client.0.smithi029.stdout:./216/.snap/snap-subdir-test 2024-04-17T19:52:48.001 INFO:tasks.workunit.client.0.smithi029.stdout:./254/.snap/snap-subdir-test 2024-04-17T19:52:48.006 INFO:tasks.workunit.client.0.smithi029.stdout:./113/.snap/snap-subdir-test 2024-04-17T19:52:48.011 INFO:tasks.workunit.client.0.smithi029.stdout:./152/.snap/snap-subdir-test 2024-04-17T19:52:48.017 INFO:tasks.workunit.client.0.smithi029.stdout:./121/.snap/snap-subdir-test 2024-04-17T19:52:48.022 INFO:tasks.workunit.client.0.smithi029.stdout:./118/.snap/snap-subdir-test 2024-04-17T19:52:48.029 INFO:tasks.workunit.client.0.smithi029.stdout:./196/.snap/snap-subdir-test 2024-04-17T19:52:48.034 INFO:tasks.workunit.client.0.smithi029.stdout:./124/.snap/snap-subdir-test 2024-04-17T19:52:48.039 INFO:tasks.workunit.client.0.smithi029.stdout:./239/.snap/snap-subdir-test 2024-04-17T19:52:48.045 INFO:tasks.workunit.client.0.smithi029.stdout:./188/.snap/snap-subdir-test 2024-04-17T19:52:48.051 INFO:tasks.workunit.client.0.smithi029.stdout:./287/.snap/snap-subdir-test 2024-04-17T19:52:48.056 INFO:tasks.workunit.client.0.smithi029.stdout:./275/.snap/snap-subdir-test 2024-04-17T19:52:48.061 INFO:tasks.workunit.client.0.smithi029.stdout:./259/.snap/snap-subdir-test 2024-04-17T19:52:48.066 INFO:tasks.workunit.client.0.smithi029.stdout:./224/.snap/snap-subdir-test 2024-04-17T19:52:48.071 INFO:tasks.workunit.client.0.smithi029.stdout:./264/.snap/snap-subdir-test 2024-04-17T19:52:48.076 INFO:tasks.workunit.client.0.smithi029.stdout:./145/.snap/snap-subdir-test 2024-04-17T19:52:48.082 INFO:tasks.workunit.client.0.smithi029.stdout:./279/.snap/snap-subdir-test 2024-04-17T19:52:48.086 INFO:tasks.workunit.client.0.smithi029.stdout:./263/.snap/snap-subdir-test 2024-04-17T19:52:48.092 INFO:tasks.workunit.client.0.smithi029.stdout:./130/.snap/snap-subdir-test 2024-04-17T19:52:48.255 INFO:tasks.workunit.client.0.smithi029.stdout:./193/.snap/snap-subdir-test 2024-04-17T19:52:48.261 INFO:tasks.workunit.client.0.smithi029.stdout:./223/.snap/snap-subdir-test 2024-04-17T19:52:48.266 INFO:tasks.workunit.client.0.smithi029.stdout:./207/.snap/snap-subdir-test 2024-04-17T19:52:48.271 INFO:tasks.workunit.client.0.smithi029.stdout:./276/.snap/snap-subdir-test 2024-04-17T19:52:48.276 INFO:tasks.workunit.client.0.smithi029.stdout:./271/.snap/snap-subdir-test 2024-04-17T19:52:48.282 INFO:tasks.workunit.client.0.smithi029.stdout:./251/.snap/snap-subdir-test 2024-04-17T19:52:48.287 INFO:tasks.workunit.client.0.smithi029.stdout:./175/.snap/snap-subdir-test 2024-04-17T19:52:48.293 INFO:tasks.workunit.client.0.smithi029.stdout:./222/.snap/snap-subdir-test 2024-04-17T19:52:48.297 INFO:tasks.workunit.client.0.smithi029.stdout:./220/.snap/snap-subdir-test 2024-04-17T19:52:48.302 INFO:tasks.workunit.client.0.smithi029.stdout:./203/.snap/snap-subdir-test 2024-04-17T19:52:48.307 INFO:tasks.workunit.client.0.smithi029.stdout:./181/.snap/snap-subdir-test 2024-04-17T19:52:48.312 INFO:tasks.workunit.client.0.smithi029.stdout:./286/.snap/snap-subdir-test 2024-04-17T19:52:48.317 INFO:tasks.workunit.client.0.smithi029.stdout:./127/.snap/snap-subdir-test 2024-04-17T19:52:48.322 INFO:tasks.workunit.client.0.smithi029.stdout:./108/.snap/snap-subdir-test 2024-04-17T19:52:48.327 INFO:tasks.workunit.client.0.smithi029.stdout:./183/.snap/snap-subdir-test 2024-04-17T19:52:48.332 INFO:tasks.workunit.client.0.smithi029.stdout:./294/.snap/snap-subdir-test 2024-04-17T19:52:48.337 INFO:tasks.workunit.client.0.smithi029.stdout:./235/.snap/snap-subdir-test 2024-04-17T19:52:48.342 INFO:tasks.workunit.client.0.smithi029.stdout:./177/.snap/snap-subdir-test 2024-04-17T19:52:48.349 INFO:tasks.workunit.client.0.smithi029.stdout:./169/.snap/snap-subdir-test 2024-04-17T19:52:48.354 INFO:tasks.workunit.client.0.smithi029.stdout:./212/.snap/snap-subdir-test 2024-04-17T19:52:48.360 INFO:tasks.workunit.client.0.smithi029.stdout:./154/.snap/snap-subdir-test 2024-04-17T19:52:48.365 INFO:tasks.workunit.client.0.smithi029.stdout:./171/.snap/snap-subdir-test 2024-04-17T19:52:48.370 INFO:tasks.workunit.client.0.smithi029.stdout:./139/.snap/snap-subdir-test 2024-04-17T19:52:48.375 INFO:tasks.workunit.client.0.smithi029.stdout:./102/.snap/snap-subdir-test 2024-04-17T19:52:48.380 INFO:tasks.workunit.client.0.smithi029.stdout:./129/.snap/snap-subdir-test 2024-04-17T19:52:51.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:52:51.495 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:52:56.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:56.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.362+0000 7fa076000640 1 Processor -- start 2024-04-17T19:52:56.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.362+0000 7fa076000640 1 -- start start 2024-04-17T19:52:56.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.370+0000 7fa076000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0700cb9a0 0x7fa07015a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:56.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.370+0000 7fa076000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa0700c5dd0 0x7fa0700cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.374+0000 7fa076000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa07006cf70 con 0x7fa0700cc6f0 2024-04-17T19:52:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.374+0000 7fa06f400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0700cb9a0 0x7fa07015a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.374+0000 7fa06f400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0700cb9a0 0x7fa07015a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:52454/0 (socket says 172.21.15.29:52454) 2024-04-17T19:52:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.374+0000 7fa06f400640 1 -- 172.21.15.29:0/32649993 learned_addr learned my addr 172.21.15.29:0/32649993 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.374+0000 7fa06ea00640 1 --2- 172.21.15.29:0/32649993 >> v2:172.21.15.37:3300/0 conn(0x7fa0700c5dd0 0x7fa0700cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.374+0000 7fa076000640 1 -- 172.21.15.29:0/32649993 --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa0700cb770 con 0x7fa0700c5dd0 2024-04-17T19:52:56.368 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.374+0000 7fa076000640 1 -- 172.21.15.29:0/32649993 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa07006d250 con 0x7fa0700cb9a0 2024-04-17T19:52:56.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.374+0000 7fa06e000640 1 -- 172.21.15.29:0/32649993 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4246062281 0 0) 0x7fa07006cf70 con 0x7fa0700cc6f0 2024-04-17T19:52:56.372 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.374+0000 7fa06e000640 1 -- 172.21.15.29:0/32649993 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa054003530 con 0x7fa0700cc6f0 2024-04-17T19:52:56.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06e000640 1 -- 172.21.15.29:0/32649993 <== mon.1 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4219361868 0 0) 0x7fa054003530 con 0x7fa0700cc6f0 2024-04-17T19:52:56.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06e000640 1 -- 172.21.15.29:0/32649993 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0700cb9a0 msgr2=0x7fa07015a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:56.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06e000640 1 --2- 172.21.15.29:0/32649993 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0700cb9a0 0x7fa07015a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:56.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06e000640 1 -- 172.21.15.29:0/32649993 >> v2:172.21.15.37:3300/0 conn(0x7fa0700c5dd0 msgr2=0x7fa0700cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:56.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06e000640 1 --2- 172.21.15.29:0/32649993 >> v2:172.21.15.37:3300/0 conn(0x7fa0700c5dd0 0x7fa0700cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:56.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06e000640 1 -- 172.21.15.29:0/32649993 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0700c5200 con 0x7fa0700cc6f0 2024-04-17T19:52:56.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06e000640 1 -- 172.21.15.29:0/32649993 <== mon.1 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fa068001a00 con 0x7fa0700cc6f0 2024-04-17T19:52:56.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06e000640 1 -- 172.21.15.29:0/32649993 >> v1:172.21.15.37:6789/0 conn(0x7fa0700cc6f0 legacy=0x7fa07006d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:56.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06e000640 1 --2- 172.21.15.29:0/32649993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa054003a30 0x7fa054003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:56.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06e000640 1 --2- 172.21.15.29:0/32649993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa054004810 0x7fa054004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:56.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06e000640 1 --2- 172.21.15.29:0/32649993 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa054005400 0x7fa05400ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:56.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06e000640 1 -- 172.21.15.29:0/32649993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa07006d250 con 0x7fa054004810 2024-04-17T19:52:56.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06e000640 1 -- 172.21.15.29:0/32649993 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa05400c2e0 con 0x7fa054003a30 2024-04-17T19:52:56.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06e000640 1 -- 172.21.15.29:0/32649993 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa05400c4e0 con 0x7fa054005400 2024-04-17T19:52:56.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06fe00640 1 --2- 172.21.15.29:0/32649993 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa054005400 0x7fa05400ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:56.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06ea00640 1 --2- 172.21.15.29:0/32649993 >> v2:172.21.15.37:3300/0 conn(0x7fa0700c5dd0 0x7fa0700cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T19:52:56.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06fe00640 1 -- 172.21.15.29:0/32649993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa054003a30 msgr2=0x7fa054003ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-17T19:52:56.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06fe00640 1 --2- 172.21.15.29:0/32649993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa054003a30 0x7fa054003ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:56.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06fe00640 1 -- 172.21.15.29:0/32649993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa054004810 msgr2=0x7fa054004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:56.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06fe00640 1 --2- 172.21.15.29:0/32649993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa054004810 0x7fa054004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:56.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06fe00640 1 -- 172.21.15.29:0/32649993 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa054003530 con 0x7fa054005400 2024-04-17T19:52:56.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.378+0000 7fa06f400640 1 --2- 172.21.15.29:0/32649993 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa0700cb9a0 0x7fa07015a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T19:52:56.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.386+0000 7fa06fe00640 1 --2- 172.21.15.29:0/32649993 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa054005400 0x7fa05400ba50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa0700c4a10 tx=0x7fa06801f5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd235b1c03db1d6a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:56.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.386+0000 7fa06e000640 1 -- 172.21.15.29:0/32649993 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0680b4070 con 0x7fa054005400 2024-04-17T19:52:56.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.386+0000 7fa06e000640 1 -- 172.21.15.29:0/32649993 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0680af620 con 0x7fa054005400 2024-04-17T19:52:56.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.386+0000 7fa06e000640 1 -- 172.21.15.29:0/32649993 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa06800c0e0 con 0x7fa054005400 2024-04-17T19:52:56.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.394+0000 7fa076000640 1 -- 172.21.15.29:0/32649993 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa054005400 msgr2=0x7fa05400ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:56.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.394+0000 7fa076000640 1 --2- 172.21.15.29:0/32649993 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa054005400 0x7fa05400ba50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa0700c4a10 tx=0x7fa06801f5e0 comp rx=0 tx=0).stop 2024-04-17T19:52:56.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.398+0000 7fa06fe00640 1 -- 172.21.15.29:0/32649993 reap_dead start 2024-04-17T19:52:56.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.406+0000 7fa076000640 1 -- 172.21.15.29:0/32649993 shutdown_connections 2024-04-17T19:52:56.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.406+0000 7fa076000640 1 -- 172.21.15.29:0/32649993 >> 172.21.15.29:0/32649993 conn(0x7fa070071ca0 msgr2=0x7fa0700720e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:56.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.406+0000 7fa076000640 1 -- 172.21.15.29:0/32649993 shutdown_connections 2024-04-17T19:52:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.406+0000 7fa076000640 1 -- 172.21.15.29:0/32649993 wait complete. 2024-04-17T19:52:56.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.422+0000 7fa076000640 1 Processor -- start 2024-04-17T19:52:56.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.422+0000 7fa076000640 1 -- start start 2024-04-17T19:52:56.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.430+0000 7fa076000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa07014f630 0x7fa0701eea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:56.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.430+0000 7fa076000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa070144da0 0x7fa0701464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:56.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.430+0000 7fa076000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa070145380 0x7fa0701458d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:56.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.430+0000 7fa076000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa07006e060 con 0x7fa070144da0 2024-04-17T19:52:56.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.430+0000 7fa076000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa0700c57c0 con 0x7fa070145380 2024-04-17T19:52:56.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.430+0000 7fa076000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa07006e1e0 con 0x7fa07014f630 2024-04-17T19:52:56.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.430+0000 7fa06f400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa07014f630 0x7fa0701eea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:56.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.430+0000 7fa06f400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa07014f630 0x7fa0701eea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:52470/0 (socket says 172.21.15.29:52470) 2024-04-17T19:52:56.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.430+0000 7fa06f400640 1 -- 172.21.15.29:0/1936935922 learned_addr learned my addr 172.21.15.29:0/1936935922 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:56.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.430+0000 7fa06f400640 1 -- 172.21.15.29:0/1936935922 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa070145380 msgr2=0x7fa0701458d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-17T19:52:56.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.430+0000 7fa06f400640 1 --2- 172.21.15.29:0/1936935922 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa070145380 0x7fa0701458d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:56.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.430+0000 7fa06f400640 1 -- 172.21.15.29:0/1936935922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa070144da0 msgr2=0x7fa0701464c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:56.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.430+0000 7fa06f400640 1 --2- 172.21.15.29:0/1936935922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa070144da0 0x7fa0701464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:56.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.430+0000 7fa06f400640 1 -- 172.21.15.29:0/1936935922 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0701f1d50 con 0x7fa07014f630 2024-04-17T19:52:56.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.438+0000 7fa06f400640 1 --2- 172.21.15.29:0/1936935922 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa07014f630 0x7fa0701eea20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa06000c4a0 tx=0x7fa06000c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:56.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.442+0000 7fa067400640 1 -- 172.21.15.29:0/1936935922 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa060015070 con 0x7fa07014f630 2024-04-17T19:52:56.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.442+0000 7fa067400640 1 -- 172.21.15.29:0/1936935922 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa060011800 con 0x7fa07014f630 2024-04-17T19:52:56.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.442+0000 7fa067400640 1 -- 172.21.15.29:0/1936935922 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa060011bf0 con 0x7fa07014f630 2024-04-17T19:52:56.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.442+0000 7fa076000640 1 -- 172.21.15.29:0/1936935922 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0701f22d0 con 0x7fa07014f630 2024-04-17T19:52:56.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.442+0000 7fa076000640 1 -- 172.21.15.29:0/1936935922 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0701edf40 con 0x7fa07014f630 2024-04-17T19:52:56.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.446+0000 7fa076000640 1 -- 172.21.15.29:0/1936935922 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa070069b90 con 0x7fa07014f630 2024-04-17T19:52:56.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.450+0000 7fa067400640 1 -- 172.21.15.29:0/1936935922 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa060018070 con 0x7fa07014f630 2024-04-17T19:52:56.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.454+0000 7fa067400640 1 --2- 172.21.15.29:0/1936935922 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa050068480 0x7fa05006a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:56.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.454+0000 7fa067400640 1 -- 172.21.15.29:0/1936935922 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(31..31 src has 1..31) ==== 9771+0+0 (secure 0 0 0) 0x7fa0600875a0 con 0x7fa07014f630 2024-04-17T19:52:56.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.454+0000 7fa06ea00640 1 --2- 172.21.15.29:0/1936935922 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa050068480 0x7fa05006a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:56.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.454+0000 7fa06ea00640 1 --2- 172.21.15.29:0/1936935922 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa050068480 0x7fa05006a990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa070148030 tx=0x7fa05c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:56.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.462+0000 7fa067400640 1 -- 172.21.15.29:0/1936935922 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa060056ca0 con 0x7fa07014f630 2024-04-17T19:52:56.933 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.938+0000 7fa076000640 1 -- 172.21.15.29:0/1936935922 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa07014fc60 con 0x7fa07014f630 2024-04-17T19:52:56.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.942+0000 7fa067400640 1 -- 172.21.15.29:0/1936935922 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6907 (secure 0 0 0) 0x7fa0600565a0 con 0x7fa07014f630 2024-04-17T19:52:56.939 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:56.939 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:34.862105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5212,4302,4259,4229]},"id":1}]} 2024-04-17T19:52:56.939 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-04-17T19:52:56.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.950+0000 7fa076000640 1 -- 172.21.15.29:0/1936935922 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa050068480 msgr2=0x7fa05006a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:56.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.950+0000 7fa076000640 1 --2- 172.21.15.29:0/1936935922 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa050068480 0x7fa05006a990 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fa070148030 tx=0x7fa05c008040 comp rx=0 tx=0).stop 2024-04-17T19:52:56.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.950+0000 7fa076000640 1 -- 172.21.15.29:0/1936935922 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa07014f630 msgr2=0x7fa0701eea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:56.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.950+0000 7fa076000640 1 --2- 172.21.15.29:0/1936935922 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa07014f630 0x7fa0701eea20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa06000c4a0 tx=0x7fa06000c960 comp rx=0 tx=0).stop 2024-04-17T19:52:56.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.950+0000 7fa076000640 1 -- 172.21.15.29:0/1936935922 shutdown_connections 2024-04-17T19:52:56.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.950+0000 7fa076000640 1 --2- 172.21.15.29:0/1936935922 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa050068480 0x7fa05006a990 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:56.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.950+0000 7fa076000640 1 --2- 172.21.15.29:0/1936935922 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa070145380 0x7fa0701458d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:56.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.950+0000 7fa076000640 1 --2- 172.21.15.29:0/1936935922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa070144da0 0x7fa0701464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:56.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.950+0000 7fa076000640 1 --2- 172.21.15.29:0/1936935922 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa07014f630 0x7fa0701eea20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:56.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.950+0000 7fa076000640 1 -- 172.21.15.29:0/1936935922 >> 172.21.15.29:0/1936935922 conn(0x7fa070071ca0 msgr2=0x7fa0700720e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:56.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.950+0000 7fa076000640 1 -- 172.21.15.29:0/1936935922 shutdown_connections 2024-04-17T19:52:56.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:56.950+0000 7fa076000640 1 -- 172.21.15.29:0/1936935922 wait complete. 2024-04-17T19:52:56.986 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=3) 2024-04-17T19:52:56.987 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-04-17T19:52:56.987 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T19:52:57.007 INFO:tasks.ceph.mds.f:Stopped 2024-04-17T19:52:57.007 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:57.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.310+0000 7f36f9600640 1 Processor -- start 2024-04-17T19:52:57.308 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.314+0000 7f36f9600640 1 -- start start 2024-04-17T19:52:57.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.322+0000 7f36f9600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36f40cc8c0 0x7f36f40cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:57.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.322+0000 7f36f9600640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f36f41597a0 0x7f36f41635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:57.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.322+0000 7f36f9600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36f40c88b0 con 0x7f36f4159390 2024-04-17T19:52:57.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.322+0000 7f36f9600640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f36f40c6ce0 con 0x7f36f41597a0 2024-04-17T19:52:57.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.322+0000 7f36f9600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f36f40c6fc0 con 0x7f36f40cc8c0 2024-04-17T19:52:57.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.326+0000 7f36f3400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f36f41597a0 0x7f36f41635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:57.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.326+0000 7f36f2a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36f40cc8c0 0x7f36f40cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:57.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.326+0000 7f36f2000640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f36f4159390 0x7f36f40ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46748/0 (socket says 172.21.15.29:46748) 2024-04-17T19:52:57.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.326+0000 7f36f2000640 1 -- 172.21.15.29:0/734753576 learned_addr learned my addr 172.21.15.29:0/734753576 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:52:57.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.326+0000 7f36f1600640 1 -- 172.21.15.29:0/734753576 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1639039512 0 0) 0x7f36f40c88b0 con 0x7f36f4159390 2024-04-17T19:52:57.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.326+0000 7f36f1600640 1 -- 172.21.15.29:0/734753576 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f36d8003530 con 0x7f36f4159390 2024-04-17T19:52:57.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.326+0000 7f36f2a00640 1 -- 172.21.15.29:0/734753576 >> v2:172.21.15.37:3300/0 conn(0x7f36f41597a0 msgr2=0x7f36f41635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:52:57.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.326+0000 7f36f2a00640 1 --2- 172.21.15.29:0/734753576 >> v2:172.21.15.37:3300/0 conn(0x7f36f41597a0 0x7f36f41635e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:57.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.326+0000 7f36f2a00640 1 -- 172.21.15.29:0/734753576 >> v1:172.21.15.29:6789/0 conn(0x7f36f4159390 legacy=0x7f36f40ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:57.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.326+0000 7f36f2a00640 1 -- 172.21.15.29:0/734753576 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f40c72a0 con 0x7f36f40cc8c0 2024-04-17T19:52:57.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.330+0000 7f36f2a00640 1 --2- 172.21.15.29:0/734753576 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36f40cc8c0 0x7f36f40cccd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f36ec009d90 tx=0x7f36ec01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c349169d59b6392f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:57.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.330+0000 7f36f1600640 1 -- 172.21.15.29:0/734753576 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec0b2070 con 0x7f36f40cc8c0 2024-04-17T19:52:57.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.330+0000 7f36f1600640 1 -- 172.21.15.29:0/734753576 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36ec0a5730 con 0x7f36f40cc8c0 2024-04-17T19:52:57.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.330+0000 7f36f1600640 1 -- 172.21.15.29:0/734753576 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec0a5b00 con 0x7f36f40cc8c0 2024-04-17T19:52:57.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.334+0000 7f36f9600640 1 -- 172.21.15.29:0/734753576 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36f40cc8c0 msgr2=0x7f36f40cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:57.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.334+0000 7f36f9600640 1 --2- 172.21.15.29:0/734753576 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36f40cc8c0 0x7f36f40cccd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f36ec009d90 tx=0x7f36ec01c970 comp rx=0 tx=0).stop 2024-04-17T19:52:57.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.334+0000 7f36f9600640 1 -- 172.21.15.29:0/734753576 shutdown_connections 2024-04-17T19:52:57.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.334+0000 7f36f9600640 1 --2- 172.21.15.29:0/734753576 >> v2:172.21.15.37:3300/0 conn(0x7f36f41597a0 0x7f36f41635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:57.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.334+0000 7f36f9600640 1 --2- 172.21.15.29:0/734753576 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36f40cc8c0 0x7f36f40cccd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:57.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.334+0000 7f36f9600640 1 -- 172.21.15.29:0/734753576 >> 172.21.15.29:0/734753576 conn(0x7f36f406fde0 msgr2=0x7f36f40701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:57.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.334+0000 7f36f9600640 1 -- 172.21.15.29:0/734753576 shutdown_connections 2024-04-17T19:52:57.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.334+0000 7f36f9600640 1 -- 172.21.15.29:0/734753576 wait complete. 2024-04-17T19:52:57.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.338+0000 7f36f9600640 1 Processor -- start 2024-04-17T19:52:57.334 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.338+0000 7f36f9600640 1 -- start start 2024-04-17T19:52:57.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36f9600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36f40cc8c0 0x7f36f4168e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:57.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36f9600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36f4159390 0x7f36f4169e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:57.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36f9600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36f41597a0 0x7f36f415c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:57.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36f9600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f36f40cabc0 con 0x7f36f40cc8c0 2024-04-17T19:52:57.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36f9600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f36f40caa40 con 0x7f36f41597a0 2024-04-17T19:52:57.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36f9600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f36f40c6880 con 0x7f36f4159390 2024-04-17T19:52:57.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36f2a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36f40cc8c0 0x7f36f4168e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:57.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36f2a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36f40cc8c0 0x7f36f4168e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41468/0 (socket says 172.21.15.29:41468) 2024-04-17T19:52:57.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36f2a00640 1 -- 172.21.15.29:0/3756821265 learned_addr learned my addr 172.21.15.29:0/3756821265 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:57.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36f2000640 1 --2- 172.21.15.29:0/3756821265 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36f4159390 0x7f36f4169e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:57.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36f2a00640 1 -- 172.21.15.29:0/3756821265 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36f4159390 msgr2=0x7f36f4169e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:57.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36f2a00640 1 --2- 172.21.15.29:0/3756821265 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36f4159390 0x7f36f4169e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:57.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36f2a00640 1 -- 172.21.15.29:0/3756821265 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36f41597a0 msgr2=0x7f36f415c3d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-17T19:52:57.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36f2a00640 1 --2- 172.21.15.29:0/3756821265 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36f41597a0 0x7f36f415c3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:57.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36f2a00640 1 -- 172.21.15.29:0/3756821265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f4167510 con 0x7f36f40cc8c0 2024-04-17T19:52:57.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36f2a00640 1 --2- 172.21.15.29:0/3756821265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36f40cc8c0 0x7f36f4168e20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f36ec009d90 tx=0x7f36ec019730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:57.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36eaa00640 1 -- 172.21.15.29:0/3756821265 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec0b2040 con 0x7f36f40cc8c0 2024-04-17T19:52:57.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36f9600640 1 -- 172.21.15.29:0/3756821265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36f41677f0 con 0x7f36f40cc8c0 2024-04-17T19:52:57.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36eaa00640 1 -- 172.21.15.29:0/3756821265 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36ec0bd070 con 0x7f36f40cc8c0 2024-04-17T19:52:57.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36f9600640 1 -- 172.21.15.29:0/3756821265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36f4154280 con 0x7f36f40cc8c0 2024-04-17T19:52:57.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.342+0000 7f36eaa00640 1 -- 172.21.15.29:0/3756821265 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec0ad680 con 0x7f36f40cc8c0 2024-04-17T19:52:57.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.346+0000 7f36f9600640 1 -- 172.21.15.29:0/3756821265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36f41544e0 con 0x7f36f40cc8c0 2024-04-17T19:52:57.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.346+0000 7f36eaa00640 1 -- 172.21.15.29:0/3756821265 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f36ec0ad820 con 0x7f36f40cc8c0 2024-04-17T19:52:57.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.346+0000 7f36eaa00640 1 --2- 172.21.15.29:0/3756821265 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f36d40793c0 0x7f36d407b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:57.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.346+0000 7f36eaa00640 1 -- 172.21.15.29:0/3756821265 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 8587+0+0 (secure 0 0 0) 0x7f36ec002020 con 0x7f36f40cc8c0 2024-04-17T19:52:57.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.350+0000 7f36f2000640 1 --2- 172.21.15.29:0/3756821265 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f36d40793c0 0x7f36d407b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:57.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.350+0000 7f36f2000640 1 --2- 172.21.15.29:0/3756821265 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f36d40793c0 0x7f36d407b8d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f36f415ce90 tx=0x7f36e4007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:57.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.358+0000 7f36eaa00640 1 -- 172.21.15.29:0/3756821265 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f36ec016580 con 0x7f36f40cc8c0 2024-04-17T19:52:57.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.762+0000 7f36f9600640 1 -- 172.21.15.29:0/3756821265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36f4069b90 con 0x7f36f40cc8c0 2024-04-17T19:52:57.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.766+0000 7f36eaa00640 1 -- 172.21.15.29:0/3756821265 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6907 (secure 0 0 0) 0x7f36ec0f13d0 con 0x7f36f40cc8c0 2024-04-17T19:52:57.762 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:52:57.762 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:34.862105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5212,4302,4259,4229]},"id":1}]} 2024-04-17T19:52:57.762 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-04-17T19:52:57.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.770+0000 7f36f9600640 1 -- 172.21.15.29:0/3756821265 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f36d40793c0 msgr2=0x7f36d407b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.770+0000 7f36f9600640 1 --2- 172.21.15.29:0/3756821265 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f36d40793c0 0x7f36d407b8d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f36f415ce90 tx=0x7f36e4007d40 comp rx=0 tx=0).stop 2024-04-17T19:52:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.770+0000 7f36f9600640 1 -- 172.21.15.29:0/3756821265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36f40cc8c0 msgr2=0x7f36f4168e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.770+0000 7f36f9600640 1 --2- 172.21.15.29:0/3756821265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36f40cc8c0 0x7f36f4168e20 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f36ec009d90 tx=0x7f36ec019730 comp rx=0 tx=0).stop 2024-04-17T19:52:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.770+0000 7f36f9600640 1 -- 172.21.15.29:0/3756821265 shutdown_connections 2024-04-17T19:52:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.770+0000 7f36f9600640 1 --2- 172.21.15.29:0/3756821265 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f36d40793c0 0x7f36d407b8d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.770+0000 7f36f9600640 1 --2- 172.21.15.29:0/3756821265 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f36f41597a0 0x7f36f415c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.770+0000 7f36f9600640 1 --2- 172.21.15.29:0/3756821265 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f36f4159390 0x7f36f4169e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.770+0000 7f36f9600640 1 --2- 172.21.15.29:0/3756821265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36f40cc8c0 0x7f36f4168e20 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:57.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.770+0000 7f36f9600640 1 -- 172.21.15.29:0/3756821265 >> 172.21.15.29:0/3756821265 conn(0x7f36f406fde0 msgr2=0x7f36f40c5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:57.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.770+0000 7f36f9600640 1 -- 172.21.15.29:0/3756821265 shutdown_connections 2024-04-17T19:52:57.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:57.770+0000 7f36f9600640 1 -- 172.21.15.29:0/3756821265 wait complete. 2024-04-17T19:52:57.803 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-17T19:52:59.804 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:52:59.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.970+0000 7f9460200640 1 Processor -- start 2024-04-17T19:52:59.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.970+0000 7f9460200640 1 -- start start 2024-04-17T19:52:59.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f9460200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f94580c6f70 0x7f94580c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:59.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f9460200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f945815a0a0 con 0x7f945806c660 2024-04-17T19:52:59.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f9460200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f945815abe0 con 0x7f9458159c90 2024-04-17T19:52:59.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f9460200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f945815a340 con 0x7f94580c6f70 2024-04-17T19:52:59.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f945e200640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f945806c660 0x7f945806ca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46292/0 (socket says 172.21.15.29:46292) 2024-04-17T19:52:59.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f945ce00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f94580c6f70 0x7f94580c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:59.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f945e200640 1 -- 172.21.15.29:0/360377116 learned_addr learned my addr 172.21.15.29:0/360377116 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:52:59.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f945ce00640 1 -- 172.21.15.29:0/360377116 >> v1:172.21.15.37:6789/0 conn(0x7f9458159c90 legacy=0x7f94580ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:59.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f945ce00640 1 -- 172.21.15.29:0/360377116 >> v1:172.21.15.29:6789/0 conn(0x7f945806c660 legacy=0x7f945806ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:59.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f945ce00640 1 -- 172.21.15.29:0/360377116 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f945815a620 con 0x7f94580c6f70 2024-04-17T19:52:59.965 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f9457e00640 1 -- 172.21.15.29:0/360377116 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1369961563 0 0) 0x7f945815a0a0 con 0x7f945806c660 2024-04-17T19:52:59.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f945ce00640 1 --2- 172.21.15.29:0/360377116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94580c6f70 0x7f94580c7380 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f94440097c0 tx=0x7f944401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d2f7f84fdfcad81 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:59.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f9457e00640 1 -- 172.21.15.29:0/360377116 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94440b0070 con 0x7f94580c6f70 2024-04-17T19:52:59.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f9457e00640 1 -- 172.21.15.29:0/360377116 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94440ab600 con 0x7f94580c6f70 2024-04-17T19:52:59.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f9457e00640 1 -- 172.21.15.29:0/360377116 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94440ab9d0 con 0x7f94580c6f70 2024-04-17T19:52:59.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f9460200640 1 -- 172.21.15.29:0/360377116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94580c6f70 msgr2=0x7f94580c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:59.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f9460200640 1 --2- 172.21.15.29:0/360377116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94580c6f70 0x7f94580c7380 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f94440097c0 tx=0x7f944401c4a0 comp rx=0 tx=0).stop 2024-04-17T19:52:59.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f9460200640 1 -- 172.21.15.29:0/360377116 shutdown_connections 2024-04-17T19:52:59.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f9460200640 1 --2- 172.21.15.29:0/360377116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94580c6f70 0x7f94580c7380 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:59.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f9460200640 1 -- 172.21.15.29:0/360377116 >> 172.21.15.29:0/360377116 conn(0x7f945806fde0 msgr2=0x7f94580701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:52:59.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f9460200640 1 -- 172.21.15.29:0/360377116 shutdown_connections 2024-04-17T19:52:59.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f9460200640 1 -- 172.21.15.29:0/360377116 wait complete. 2024-04-17T19:52:59.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f9460200640 1 Processor -- start 2024-04-17T19:52:59.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.974+0000 7f9460200640 1 -- start start 2024-04-17T19:52:59.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f9460200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f945806c660 0x7f9458150c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:59.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f9460200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94580c6f70 0x7f94581f5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:59.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f9460200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9458159c90 0x7f945814ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:59.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f9460200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f94580cab40 con 0x7f945806c660 2024-04-17T19:52:59.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f9460200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f94580ca9c0 con 0x7f94580c6f70 2024-04-17T19:52:59.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f9460200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9458167e40 con 0x7f9458159c90 2024-04-17T19:52:59.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f945d800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f945806c660 0x7f9458150c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:59.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f945ce00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94580c6f70 0x7f94581f5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:59.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f945d800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f945806c660 0x7f9458150c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49258/0 (socket says 172.21.15.29:49258) 2024-04-17T19:52:59.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f945d800640 1 -- 172.21.15.29:0/624581979 learned_addr learned my addr 172.21.15.29:0/624581979 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:52:59.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f945e200640 1 --2- 172.21.15.29:0/624581979 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9458159c90 0x7f945814ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:59.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f945d800640 1 -- 172.21.15.29:0/624581979 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9458159c90 msgr2=0x7f945814ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:59.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f945d800640 1 --2- 172.21.15.29:0/624581979 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9458159c90 0x7f945814ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:59.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f945d800640 1 -- 172.21.15.29:0/624581979 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94580c6f70 msgr2=0x7f94581f5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:52:59.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f945d800640 1 --2- 172.21.15.29:0/624581979 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94580c6f70 0x7f94581f5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:52:59.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f945d800640 1 -- 172.21.15.29:0/624581979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9458162de0 con 0x7f945806c660 2024-04-17T19:52:59.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f945d800640 1 --2- 172.21.15.29:0/624581979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f945806c660 0x7f9458150c80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f944c00c390 tx=0x7f944c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:59.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f9455600640 1 -- 172.21.15.29:0/624581979 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f944c017070 con 0x7f945806c660 2024-04-17T19:52:59.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f9455600640 1 -- 172.21.15.29:0/624581979 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f944c0052d0 con 0x7f945806c660 2024-04-17T19:52:59.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f9460200640 1 -- 172.21.15.29:0/624581979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94581f6310 con 0x7f945806c660 2024-04-17T19:52:59.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f9455600640 1 -- 172.21.15.29:0/624581979 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f944c003820 con 0x7f945806c660 2024-04-17T19:52:59.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f9460200640 1 -- 172.21.15.29:0/624581979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94581644e0 con 0x7f945806c660 2024-04-17T19:52:59.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.978+0000 7f9460200640 1 -- 172.21.15.29:0/624581979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9458069b90 con 0x7f945806c660 2024-04-17T19:52:59.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.982+0000 7f9455600640 1 -- 172.21.15.29:0/624581979 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f944c01a070 con 0x7f945806c660 2024-04-17T19:52:59.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.982+0000 7f9455600640 1 --2- 172.21.15.29:0/624581979 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9438068380 0x7f943806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:52:59.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.982+0000 7f9455600640 1 -- 172.21.15.29:0/624581979 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 8651+0+0 (secure 0 0 0) 0x7f944c086960 con 0x7f945806c660 2024-04-17T19:52:59.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.982+0000 7f945ce00640 1 --2- 172.21.15.29:0/624581979 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9438068380 0x7f943806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:52:59.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.982+0000 7f945ce00640 1 --2- 172.21.15.29:0/624581979 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9438068380 0x7f943806a890 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f94440ad000 tx=0x7f94440a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:52:59.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:52:59.990+0000 7f9455600640 1 -- 172.21.15.29:0/624581979 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9458162de0 con 0x7f945806c660 2024-04-17T19:53:00.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:00.238+0000 7f9460200640 1 -- 172.21.15.29:0/624581979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9458075a90 con 0x7f945806c660 2024-04-17T19:53:00.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:00.242+0000 7f9455600640 1 -- 172.21.15.29:0/624581979 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6907 (secure 0 0 0) 0x7f944c005480 con 0x7f945806c660 2024-04-17T19:53:00.236 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:00.236 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:34.862105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5212,4302,4259,4229]},"id":1}]} 2024-04-17T19:53:00.236 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-04-17T19:53:00.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:00.242+0000 7f9460200640 1 -- 172.21.15.29:0/624581979 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9438068380 msgr2=0x7f943806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:00.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:00.242+0000 7f9460200640 1 --2- 172.21.15.29:0/624581979 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9438068380 0x7f943806a890 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f94440ad000 tx=0x7f94440a6000 comp rx=0 tx=0).stop 2024-04-17T19:53:00.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:00.242+0000 7f9460200640 1 -- 172.21.15.29:0/624581979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f945806c660 msgr2=0x7f9458150c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:00.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:00.242+0000 7f9460200640 1 --2- 172.21.15.29:0/624581979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f945806c660 0x7f9458150c80 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f944c00c390 tx=0x7f944c00c850 comp rx=0 tx=0).stop 2024-04-17T19:53:00.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:00.242+0000 7f9460200640 1 -- 172.21.15.29:0/624581979 shutdown_connections 2024-04-17T19:53:00.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:00.242+0000 7f9460200640 1 --2- 172.21.15.29:0/624581979 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9438068380 0x7f943806a890 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:00.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:00.242+0000 7f9460200640 1 --2- 172.21.15.29:0/624581979 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9458159c90 0x7f945814ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:00.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:00.242+0000 7f9460200640 1 --2- 172.21.15.29:0/624581979 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f94580c6f70 0x7f94581f5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:00.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:00.242+0000 7f9460200640 1 --2- 172.21.15.29:0/624581979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f945806c660 0x7f9458150c80 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:00.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:00.242+0000 7f9460200640 1 -- 172.21.15.29:0/624581979 >> 172.21.15.29:0/624581979 conn(0x7f945806fde0 msgr2=0x7f945806ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:00.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:00.242+0000 7f9460200640 1 -- 172.21.15.29:0/624581979 shutdown_connections 2024-04-17T19:53:00.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:00.242+0000 7f9460200640 1 -- 172.21.15.29:0/624581979 wait complete. 2024-04-17T19:53:00.258 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-17T19:53:02.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:53:02.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.434+0000 7f16fbe00640 1 Processor -- start 2024-04-17T19:53:02.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.434+0000 7f16fbe00640 1 -- start start 2024-04-17T19:53:02.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.438+0000 7f16fbe00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f16f40c6f70 0x7f16f40c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:02.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.438+0000 7f16fbe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f16f406c660 0x7f16f406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:02.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.438+0000 7f16fbe00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16f415a0a0 con 0x7f16f4159c90 2024-04-17T19:53:02.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.438+0000 7f16fbe00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f16f415abe0 con 0x7f16f406c660 2024-04-17T19:53:02.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.438+0000 7f16fbe00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f16f415a340 con 0x7f16f40c6f70 2024-04-17T19:53:02.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.438+0000 7f16f9e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f16f406c660 0x7f16f406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:02.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.438+0000 7f16f9e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f16f406c660 0x7f16f406ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49270/0 (socket says 172.21.15.29:49270) 2024-04-17T19:53:02.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.438+0000 7f16f9e00640 1 -- 172.21.15.29:0/1853228417 learned_addr learned my addr 172.21.15.29:0/1853228417 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:02.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.438+0000 7f16f8a00640 1 --2- 172.21.15.29:0/1853228417 >> v2:172.21.15.37:3300/0 conn(0x7f16f40c6f70 0x7f16f40c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:02.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.438+0000 7f16f9e00640 1 -- 172.21.15.29:0/1853228417 >> v2:172.21.15.37:3300/0 conn(0x7f16f40c6f70 msgr2=0x7f16f40c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:02.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.438+0000 7f16f9e00640 1 --2- 172.21.15.29:0/1853228417 >> v2:172.21.15.37:3300/0 conn(0x7f16f40c6f70 0x7f16f40c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:02.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.438+0000 7f16f9e00640 1 -- 172.21.15.29:0/1853228417 >> v1:172.21.15.37:6789/0 conn(0x7f16f4159c90 legacy=0x7f16f40ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:02.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.438+0000 7f16f9e00640 1 -- 172.21.15.29:0/1853228417 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f415a620 con 0x7f16f406c660 2024-04-17T19:53:02.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.442+0000 7f16f9e00640 1 --2- 172.21.15.29:0/1853228417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16f406c660 0x7f16f406ca70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f16f40cd310 tx=0x7f16ec01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5617b3c003d69468 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:02.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.442+0000 7f16f3e00640 1 -- 172.21.15.29:0/1853228417 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec0a56c0 con 0x7f16f406c660 2024-04-17T19:53:02.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.442+0000 7f16f3e00640 1 -- 172.21.15.29:0/1853228417 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16ec0a5860 con 0x7f16f406c660 2024-04-17T19:53:02.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.442+0000 7f16f3e00640 1 -- 172.21.15.29:0/1853228417 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec0a5c30 con 0x7f16f406c660 2024-04-17T19:53:02.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.442+0000 7f16fbe00640 1 -- 172.21.15.29:0/1853228417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16f406c660 msgr2=0x7f16f406ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:02.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.442+0000 7f16fbe00640 1 --2- 172.21.15.29:0/1853228417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16f406c660 0x7f16f406ca70 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f16f40cd310 tx=0x7f16ec01c9f0 comp rx=0 tx=0).stop 2024-04-17T19:53:02.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.442+0000 7f16fbe00640 1 -- 172.21.15.29:0/1853228417 shutdown_connections 2024-04-17T19:53:02.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.442+0000 7f16fbe00640 1 --2- 172.21.15.29:0/1853228417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16f406c660 0x7f16f406ca70 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:02.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.442+0000 7f16fbe00640 1 --2- 172.21.15.29:0/1853228417 >> v2:172.21.15.37:3300/0 conn(0x7f16f40c6f70 0x7f16f40c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:02.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.442+0000 7f16fbe00640 1 -- 172.21.15.29:0/1853228417 >> 172.21.15.29:0/1853228417 conn(0x7f16f406fde0 msgr2=0x7f16f40701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:02.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.442+0000 7f16fbe00640 1 -- 172.21.15.29:0/1853228417 shutdown_connections 2024-04-17T19:53:02.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.442+0000 7f16fbe00640 1 -- 172.21.15.29:0/1853228417 wait complete. 2024-04-17T19:53:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.442+0000 7f16fbe00640 1 Processor -- start 2024-04-17T19:53:02.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.442+0000 7f16fbe00640 1 -- start start 2024-04-17T19:53:02.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16fbe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16f406c660 0x7f16f41407e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16fbe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16f40c6f70 0x7f16f4141b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16fbe00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16f4159c90 0x7f16f41446a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16fbe00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f16f40ca9c0 con 0x7f16f40c6f70 2024-04-17T19:53:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16fbe00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f16f406cfb0 con 0x7f16f406c660 2024-04-17T19:53:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16fbe00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f16f40cab40 con 0x7f16f4159c90 2024-04-17T19:53:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16f8a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16f40c6f70 0x7f16f4141b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16f8a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16f40c6f70 0x7f16f4141b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49274/0 (socket says 172.21.15.29:49274) 2024-04-17T19:53:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16f8a00640 1 -- 172.21.15.29:0/1276989260 learned_addr learned my addr 172.21.15.29:0/1276989260 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16f9400640 1 --2- 172.21.15.29:0/1276989260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16f406c660 0x7f16f41407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16f9e00640 1 --2- 172.21.15.29:0/1276989260 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16f4159c90 0x7f16f41446a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16f8a00640 1 -- 172.21.15.29:0/1276989260 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16f4159c90 msgr2=0x7f16f41446a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16f8a00640 1 --2- 172.21.15.29:0/1276989260 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16f4159c90 0x7f16f41446a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16f8a00640 1 -- 172.21.15.29:0/1276989260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16f406c660 msgr2=0x7f16f41407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16f8a00640 1 --2- 172.21.15.29:0/1276989260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16f406c660 0x7f16f41407e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:02.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16f8a00640 1 -- 172.21.15.29:0/1276989260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f41f51d0 con 0x7f16f40c6f70 2024-04-17T19:53:02.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16f8a00640 1 --2- 172.21.15.29:0/1276989260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16f40c6f70 0x7f16f4141b90 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f16e000d7d0 tx=0x7f16e000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:02.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16f1600640 1 -- 172.21.15.29:0/1276989260 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e000b6c0 con 0x7f16f40c6f70 2024-04-17T19:53:02.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16f1600640 1 -- 172.21.15.29:0/1276989260 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16e0019070 con 0x7f16f40c6f70 2024-04-17T19:53:02.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16f1600640 1 -- 172.21.15.29:0/1276989260 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e000ba80 con 0x7f16f40c6f70 2024-04-17T19:53:02.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16fbe00640 1 -- 172.21.15.29:0/1276989260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f41f54b0 con 0x7f16f40c6f70 2024-04-17T19:53:02.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16fbe00640 1 -- 172.21.15.29:0/1276989260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f406cb00 con 0x7f16f40c6f70 2024-04-17T19:53:02.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.446+0000 7f16fbe00640 1 -- 172.21.15.29:0/1276989260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16f4069b90 con 0x7f16f40c6f70 2024-04-17T19:53:02.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.450+0000 7f16f1600640 1 -- 172.21.15.29:0/1276989260 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f16e000bc20 con 0x7f16f40c6f70 2024-04-17T19:53:02.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.450+0000 7f16f1600640 1 --2- 172.21.15.29:0/1276989260 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f16d40683b0 0x7f16d406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:02.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.450+0000 7f16f1600640 1 -- 172.21.15.29:0/1276989260 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 8651+0+0 (secure 0 0 0) 0x7f16e0087960 con 0x7f16f40c6f70 2024-04-17T19:53:02.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.450+0000 7f16f9400640 1 --2- 172.21.15.29:0/1276989260 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f16d40683b0 0x7f16d406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:02.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.450+0000 7f16f9400640 1 --2- 172.21.15.29:0/1276989260 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f16d40683b0 0x7f16d406a8c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f16e8007fd0 tx=0x7f16e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:02.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.454+0000 7f16f1600640 1 -- 172.21.15.29:0/1276989260 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f16e00572f0 con 0x7f16f40c6f70 2024-04-17T19:53:02.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.666+0000 7f16fbe00640 1 -- 172.21.15.29:0/1276989260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16f4075550 con 0x7f16f40c6f70 2024-04-17T19:53:02.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.666+0000 7f16f1600640 1 -- 172.21.15.29:0/1276989260 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6907 (secure 0 0 0) 0x7f16e0056bf0 con 0x7f16f40c6f70 2024-04-17T19:53:02.657 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:02.657 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:34.862105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5212,4302,4259,4229]},"id":1}]} 2024-04-17T19:53:02.657 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-04-17T19:53:02.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.666+0000 7f16fbe00640 1 -- 172.21.15.29:0/1276989260 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f16d40683b0 msgr2=0x7f16d406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:02.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.666+0000 7f16fbe00640 1 --2- 172.21.15.29:0/1276989260 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f16d40683b0 0x7f16d406a8c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f16e8007fd0 tx=0x7f16e8009040 comp rx=0 tx=0).stop 2024-04-17T19:53:02.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.666+0000 7f16fbe00640 1 -- 172.21.15.29:0/1276989260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16f40c6f70 msgr2=0x7f16f4141b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:02.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.666+0000 7f16fbe00640 1 --2- 172.21.15.29:0/1276989260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16f40c6f70 0x7f16f4141b90 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f16e000d7d0 tx=0x7f16e000dc90 comp rx=0 tx=0).stop 2024-04-17T19:53:02.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.666+0000 7f16fbe00640 1 -- 172.21.15.29:0/1276989260 shutdown_connections 2024-04-17T19:53:02.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.666+0000 7f16fbe00640 1 --2- 172.21.15.29:0/1276989260 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f16d40683b0 0x7f16d406a8c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:02.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.666+0000 7f16fbe00640 1 --2- 172.21.15.29:0/1276989260 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f16f4159c90 0x7f16f41446a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:02.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.666+0000 7f16fbe00640 1 --2- 172.21.15.29:0/1276989260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16f40c6f70 0x7f16f4141b90 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:02.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.666+0000 7f16fbe00640 1 --2- 172.21.15.29:0/1276989260 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16f406c660 0x7f16f41407e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:02.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.666+0000 7f16fbe00640 1 -- 172.21.15.29:0/1276989260 >> 172.21.15.29:0/1276989260 conn(0x7f16f406fde0 msgr2=0x7f16f40701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:02.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.666+0000 7f16fbe00640 1 -- 172.21.15.29:0/1276989260 shutdown_connections 2024-04-17T19:53:02.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:02.666+0000 7f16fbe00640 1 -- 172.21.15.29:0/1276989260 wait complete. 2024-04-17T19:53:02.679 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-17T19:53:04.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:53:04.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.854+0000 7f695d200640 1 Processor -- start 2024-04-17T19:53:04.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.854+0000 7f695d200640 1 -- start start 2024-04-17T19:53:04.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.858+0000 7f695d200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f69580cc8c0 0x7f69580cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:04.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.858+0000 7f695d200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6958159390 0x7f69581597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:04.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.858+0000 7f695d200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69580ca330 0x7f69580ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:04.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.858+0000 7f695d200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f69580c88b0 con 0x7f6958159390 2024-04-17T19:53:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.858+0000 7f695d200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f69580c6ce0 con 0x7f69580cc8c0 2024-04-17T19:53:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.858+0000 7f695d200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f69580c6fc0 con 0x7f69580ca330 2024-04-17T19:53:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.858+0000 7f694d400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6958159390 0x7f69581597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.858+0000 7f694d400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6958159390 0x7f69581597a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49280/0 (socket says 172.21.15.29:49280) 2024-04-17T19:53:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.858+0000 7f694d400640 1 -- 172.21.15.29:0/2694829662 learned_addr learned my addr 172.21.15.29:0/2694829662 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.858+0000 7f6955e00640 1 --2- 172.21.15.29:0/2694829662 >> v2:172.21.15.37:3300/0 conn(0x7f69580cc8c0 0x7f69580cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.858+0000 7f6956800640 1 --2- 172.21.15.29:0/2694829662 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69580ca330 0x7f69580ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.858+0000 7f694d400640 1 -- 172.21.15.29:0/2694829662 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69580ca330 msgr2=0x7f69580ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.858+0000 7f694d400640 1 --2- 172.21.15.29:0/2694829662 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69580ca330 0x7f69580ca740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.858+0000 7f694d400640 1 -- 172.21.15.29:0/2694829662 >> v2:172.21.15.37:3300/0 conn(0x7f69580cc8c0 msgr2=0x7f69580cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.858+0000 7f694d400640 1 --2- 172.21.15.29:0/2694829662 >> v2:172.21.15.37:3300/0 conn(0x7f69580cc8c0 0x7f69580cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:04.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.858+0000 7f694d400640 1 -- 172.21.15.29:0/2694829662 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69580c72a0 con 0x7f6958159390 2024-04-17T19:53:04.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.858+0000 7f694d400640 1 --2- 172.21.15.29:0/2694829662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6958159390 0x7f69581597a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f69580721b0 tx=0x7f694001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f71ea50e201289 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:04.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.862+0000 7f6955400640 1 -- 172.21.15.29:0/2694829662 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69400b0070 con 0x7f6958159390 2024-04-17T19:53:04.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.862+0000 7f6955400640 1 -- 172.21.15.29:0/2694829662 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69400ab600 con 0x7f6958159390 2024-04-17T19:53:04.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.862+0000 7f6955400640 1 -- 172.21.15.29:0/2694829662 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69400ab9d0 con 0x7f6958159390 2024-04-17T19:53:04.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.862+0000 7f695d200640 1 -- 172.21.15.29:0/2694829662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6958159390 msgr2=0x7f69581597a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:04.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.862+0000 7f695d200640 1 --2- 172.21.15.29:0/2694829662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6958159390 0x7f69581597a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f69580721b0 tx=0x7f694001c520 comp rx=0 tx=0).stop 2024-04-17T19:53:04.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.862+0000 7f695d200640 1 -- 172.21.15.29:0/2694829662 shutdown_connections 2024-04-17T19:53:04.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.862+0000 7f695d200640 1 --2- 172.21.15.29:0/2694829662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6958159390 0x7f69581597a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:04.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.862+0000 7f695d200640 1 --2- 172.21.15.29:0/2694829662 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69580ca330 0x7f69580ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:04.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.862+0000 7f695d200640 1 --2- 172.21.15.29:0/2694829662 >> v2:172.21.15.37:3300/0 conn(0x7f69580cc8c0 0x7f69580cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:04.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.862+0000 7f695d200640 1 -- 172.21.15.29:0/2694829662 >> 172.21.15.29:0/2694829662 conn(0x7f695806fde0 msgr2=0x7f69580701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:04.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.862+0000 7f695d200640 1 -- 172.21.15.29:0/2694829662 shutdown_connections 2024-04-17T19:53:04.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.862+0000 7f695d200640 1 -- 172.21.15.29:0/2694829662 wait complete. 2024-04-17T19:53:04.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.862+0000 7f695d200640 1 Processor -- start 2024-04-17T19:53:04.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.862+0000 7f695d200640 1 -- start start 2024-04-17T19:53:04.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f695d200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69580ca330 0x7f695815a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:04.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f695d200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69580cc8c0 0x7f695813a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:04.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f695d200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6958159390 0x7f6958151bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:04.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f695d200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f695815ca60 con 0x7f69580ca330 2024-04-17T19:53:04.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f695d200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f69580c6a90 con 0x7f6958159390 2024-04-17T19:53:04.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f695d200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f695815cbe0 con 0x7f69580cc8c0 2024-04-17T19:53:04.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f6955e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69580ca330 0x7f695815a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:04.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f6955e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69580ca330 0x7f695815a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49282/0 (socket says 172.21.15.29:49282) 2024-04-17T19:53:04.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f6955e00640 1 -- 172.21.15.29:0/3756864865 learned_addr learned my addr 172.21.15.29:0/3756864865 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:04.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f694d400640 1 --2- 172.21.15.29:0/3756864865 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69580cc8c0 0x7f695813a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:04.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f6956800640 1 --2- 172.21.15.29:0/3756864865 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6958159390 0x7f6958151bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:04.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f6955e00640 1 -- 172.21.15.29:0/3756864865 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69580cc8c0 msgr2=0x7f695813a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:04.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f6955e00640 1 --2- 172.21.15.29:0/3756864865 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69580cc8c0 0x7f695813a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:04.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f6955e00640 1 -- 172.21.15.29:0/3756864865 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6958159390 msgr2=0x7f6958151bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:04.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f6955e00640 1 --2- 172.21.15.29:0/3756864865 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6958159390 0x7f6958151bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:04.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f6955e00640 1 -- 172.21.15.29:0/3756864865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f695815ba80 con 0x7f69580ca330 2024-04-17T19:53:04.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f6955e00640 1 --2- 172.21.15.29:0/3756864865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69580ca330 0x7f695815a750 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f694800d620 tx=0x7f694800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:04.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f694ea00640 1 -- 172.21.15.29:0/3756864865 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694800b620 con 0x7f69580ca330 2024-04-17T19:53:04.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f694ea00640 1 -- 172.21.15.29:0/3756864865 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6948019070 con 0x7f69580ca330 2024-04-17T19:53:04.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f695d200640 1 -- 172.21.15.29:0/3756864865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f695815bd60 con 0x7f69580ca330 2024-04-17T19:53:04.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f694ea00640 1 -- 172.21.15.29:0/3756864865 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694800b9e0 con 0x7f69580ca330 2024-04-17T19:53:04.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f695d200640 1 -- 172.21.15.29:0/3756864865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69580724c0 con 0x7f69580ca330 2024-04-17T19:53:04.859 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.866+0000 7f695d200640 1 -- 172.21.15.29:0/3756864865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6958069b90 con 0x7f69580ca330 2024-04-17T19:53:04.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.870+0000 7f694ea00640 1 -- 172.21.15.29:0/3756864865 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f694800bb80 con 0x7f69580ca330 2024-04-17T19:53:04.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.870+0000 7f694ea00640 1 --2- 172.21.15.29:0/3756864865 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6934068380 0x7f693406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:04.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.870+0000 7f694ea00640 1 -- 172.21.15.29:0/3756864865 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 8499+0+0 (secure 0 0 0) 0x7f6948087970 con 0x7f69580ca330 2024-04-17T19:53:04.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.870+0000 7f694d400640 1 --2- 172.21.15.29:0/3756864865 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6934068380 0x7f693406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:04.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.870+0000 7f694d400640 1 --2- 172.21.15.29:0/3756864865 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6934068380 0x7f693406a890 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f69400ad000 tx=0x7f6940008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:04.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:04.874+0000 7f694ea00640 1 -- 172.21.15.29:0/3756864865 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6948057390 con 0x7f69580ca330 2024-04-17T19:53:05.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:05.082+0000 7f695d200640 1 -- 172.21.15.29:0/3756864865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69580c44d0 con 0x7f69580ca330 2024-04-17T19:53:05.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:05.082+0000 7f694ea00640 1 -- 172.21.15.29:0/3756864865 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6907 (secure 0 0 0) 0x7f6948056c90 con 0x7f69580ca330 2024-04-17T19:53:05.073 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:05.073 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:34.862105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5212,4302,4259,4229]},"id":1}]} 2024-04-17T19:53:05.073 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-04-17T19:53:05.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:05.082+0000 7f695d200640 1 -- 172.21.15.29:0/3756864865 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6934068380 msgr2=0x7f693406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:05.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:05.082+0000 7f695d200640 1 --2- 172.21.15.29:0/3756864865 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6934068380 0x7f693406a890 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f69400ad000 tx=0x7f6940008040 comp rx=0 tx=0).stop 2024-04-17T19:53:05.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:05.082+0000 7f695d200640 1 -- 172.21.15.29:0/3756864865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69580ca330 msgr2=0x7f695815a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:05.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:05.082+0000 7f695d200640 1 --2- 172.21.15.29:0/3756864865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69580ca330 0x7f695815a750 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f694800d620 tx=0x7f694800dae0 comp rx=0 tx=0).stop 2024-04-17T19:53:05.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:05.082+0000 7f695d200640 1 -- 172.21.15.29:0/3756864865 shutdown_connections 2024-04-17T19:53:05.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:05.082+0000 7f695d200640 1 --2- 172.21.15.29:0/3756864865 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6934068380 0x7f693406a890 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:05.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:05.082+0000 7f695d200640 1 --2- 172.21.15.29:0/3756864865 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6958159390 0x7f6958151bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:05.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:05.082+0000 7f695d200640 1 --2- 172.21.15.29:0/3756864865 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69580cc8c0 0x7f695813a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:05.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:05.082+0000 7f695d200640 1 --2- 172.21.15.29:0/3756864865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69580ca330 0x7f695815a750 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:05.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:05.082+0000 7f695d200640 1 -- 172.21.15.29:0/3756864865 >> 172.21.15.29:0/3756864865 conn(0x7f695806fde0 msgr2=0x7f69580ccd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:05.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:05.082+0000 7f695d200640 1 -- 172.21.15.29:0/3756864865 shutdown_connections 2024-04-17T19:53:05.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:05.082+0000 7f695d200640 1 -- 172.21.15.29:0/3756864865 wait complete. 2024-04-17T19:53:05.095 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-17T19:53:07.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:53:07.370 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.270+0000 7f7f84800640 1 Processor -- start 2024-04-17T19:53:07.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.270+0000 7f7f84800640 1 -- start start 2024-04-17T19:53:07.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.270+0000 7f7f84800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7f7c08d670 0x7f7f7c1662a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:07.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.270+0000 7f7f84800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f7c168e20 con 0x7f7f7c071ed0 2024-04-17T19:53:07.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.270+0000 7f7f84800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f7c16c890 con 0x7f7f7c0ca240 2024-04-17T19:53:07.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.270+0000 7f7f84800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f7f7c16d0f0 con 0x7f7f7c08d670 2024-04-17T19:53:07.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.270+0000 7f7f81400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7f7c071ed0 0x7f7f7c16c180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46308/0 (socket says 172.21.15.29:46308) 2024-04-17T19:53:07.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.270+0000 7f7f81400640 1 -- 172.21.15.29:0/1697732422 learned_addr learned my addr 172.21.15.29:0/1697732422 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:53:07.371 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.270+0000 7f7f81e00640 1 --2- 172.21.15.29:0/1697732422 >> v2:172.21.15.37:3300/0 conn(0x7f7f7c08d670 0x7f7f7c1662a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:07.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.382+0000 7f7f81e00640 1 -- 172.21.15.29:0/1697732422 >> v1:172.21.15.37:6789/0 conn(0x7f7f7c0ca240 legacy=0x7f7f7c1734b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:07.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.382+0000 7f7f81e00640 1 -- 172.21.15.29:0/1697732422 >> v1:172.21.15.29:6789/0 conn(0x7f7f7c071ed0 legacy=0x7f7f7c16c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:07.373 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.382+0000 7f7f81e00640 1 -- 172.21.15.29:0/1697732422 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7c0c9c30 con 0x7f7f7c08d670 2024-04-17T19:53:07.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.382+0000 7f7f81e00640 1 --2- 172.21.15.29:0/1697732422 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f7c08d670 0x7f7f7c1662a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7f70009c20 tx=0x7f7f7001c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ca8b8e01f60b172 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.382+0000 7f7f80a00640 1 -- 172.21.15.29:0/1697732422 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f700ac070 con 0x7f7f7c08d670 2024-04-17T19:53:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.382+0000 7f7f80a00640 1 -- 172.21.15.29:0/1697732422 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f700092d0 con 0x7f7f7c08d670 2024-04-17T19:53:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.382+0000 7f7f80a00640 1 -- 172.21.15.29:0/1697732422 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f700a6650 con 0x7f7f7c08d670 2024-04-17T19:53:07.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.386+0000 7f7f84800640 1 -- 172.21.15.29:0/1697732422 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f7c08d670 msgr2=0x7f7f7c1662a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:07.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.386+0000 7f7f84800640 1 --2- 172.21.15.29:0/1697732422 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f7c08d670 0x7f7f7c1662a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7f70009c20 tx=0x7f7f7001c760 comp rx=0 tx=0).stop 2024-04-17T19:53:07.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.386+0000 7f7f84800640 1 -- 172.21.15.29:0/1697732422 shutdown_connections 2024-04-17T19:53:07.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.386+0000 7f7f84800640 1 --2- 172.21.15.29:0/1697732422 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f7c08d670 0x7f7f7c1662a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:07.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.386+0000 7f7f84800640 1 -- 172.21.15.29:0/1697732422 >> 172.21.15.29:0/1697732422 conn(0x7f7f7c06fde0 msgr2=0x7f7f7c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:07.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.386+0000 7f7f84800640 1 -- 172.21.15.29:0/1697732422 shutdown_connections 2024-04-17T19:53:07.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.386+0000 7f7f84800640 1 -- 172.21.15.29:0/1697732422 wait complete. 2024-04-17T19:53:07.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.386+0000 7f7f84800640 1 Processor -- start 2024-04-17T19:53:07.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.386+0000 7f7f84800640 1 -- start start 2024-04-17T19:53:07.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f84800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f7c071ed0 0x7f7f7c169d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:07.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f84800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f7c08d670 0x7f7f7c167410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:07.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f84800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f7c0ca240 0x7f7f7c168210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:07.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f84800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c0caa60 con 0x7f7f7c0ca240 2024-04-17T19:53:07.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f84800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c1693a0 con 0x7f7f7c071ed0 2024-04-17T19:53:07.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f84800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7f7c0722e0 con 0x7f7f7c08d670 2024-04-17T19:53:07.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f81e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f7c071ed0 0x7f7f7c169d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:07.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f82800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f7c0ca240 0x7f7f7c168210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:07.382 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f81e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f7c071ed0 0x7f7f7c169d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:36472/0 (socket says 172.21.15.29:36472) 2024-04-17T19:53:07.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f81e00640 1 -- 172.21.15.29:0/2989692995 learned_addr learned my addr 172.21.15.29:0/2989692995 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:07.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f81400640 1 --2- 172.21.15.29:0/2989692995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f7c08d670 0x7f7f7c167410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:07.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f82800640 1 -- 172.21.15.29:0/2989692995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f7c08d670 msgr2=0x7f7f7c167410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:07.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f82800640 1 --2- 172.21.15.29:0/2989692995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f7c08d670 0x7f7f7c167410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:07.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f82800640 1 -- 172.21.15.29:0/2989692995 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f7c071ed0 msgr2=0x7f7f7c169d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:07.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f82800640 1 --2- 172.21.15.29:0/2989692995 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f7c071ed0 0x7f7f7c169d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:07.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f82800640 1 -- 172.21.15.29:0/2989692995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7c16bcf0 con 0x7f7f7c0ca240 2024-04-17T19:53:07.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f81e00640 1 --2- 172.21.15.29:0/2989692995 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f7c071ed0 0x7f7f7c169d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:53:07.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f82800640 1 --2- 172.21.15.29:0/2989692995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f7c0ca240 0x7f7f7c168210 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7f7800c700 tx=0x7f7f7800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:07.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f76000640 1 -- 172.21.15.29:0/2989692995 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f78003b80 con 0x7f7f7c0ca240 2024-04-17T19:53:07.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f76000640 1 -- 172.21.15.29:0/2989692995 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f78003d20 con 0x7f7f7c0ca240 2024-04-17T19:53:07.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f76000640 1 -- 172.21.15.29:0/2989692995 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f78005310 con 0x7f7f7c0ca240 2024-04-17T19:53:07.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f84800640 1 -- 172.21.15.29:0/2989692995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f7c166c90 con 0x7f7f7c0ca240 2024-04-17T19:53:07.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.390+0000 7f7f84800640 1 -- 172.21.15.29:0/2989692995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f7c16b640 con 0x7f7f7c0ca240 2024-04-17T19:53:07.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.394+0000 7f7f84800640 1 -- 172.21.15.29:0/2989692995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f7c166a80 con 0x7f7f7c0ca240 2024-04-17T19:53:07.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.394+0000 7f7f76000640 1 -- 172.21.15.29:0/2989692995 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7f780054b0 con 0x7f7f7c0ca240 2024-04-17T19:53:07.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.394+0000 7f7f76000640 1 --2- 172.21.15.29:0/2989692995 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7f600683b0 0x7f7f6006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:07.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.394+0000 7f7f76000640 1 -- 172.21.15.29:0/2989692995 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 8499+0+0 (secure 0 0 0) 0x7f7f78087230 con 0x7f7f7c0ca240 2024-04-17T19:53:07.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.394+0000 7f7f81e00640 1 --2- 172.21.15.29:0/2989692995 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7f600683b0 0x7f7f6006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:07.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.394+0000 7f7f81e00640 1 --2- 172.21.15.29:0/2989692995 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7f600683b0 0x7f7f6006a8c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7f70019370 tx=0x7f7f700b3040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:07.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.398+0000 7f7f76000640 1 -- 172.21.15.29:0/2989692995 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7f78056600 con 0x7f7f7c0ca240 2024-04-17T19:53:07.592 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.602+0000 7f7f84800640 1 -- 172.21.15.29:0/2989692995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f7c069b90 con 0x7f7f7c0ca240 2024-04-17T19:53:07.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.602+0000 7f7f76000640 1 -- 172.21.15.29:0/2989692995 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6907 (secure 0 0 0) 0x7f7f7805a900 con 0x7f7f7c0ca240 2024-04-17T19:53:07.593 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:07.593 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:34.862105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5212,4302,4259,4229]},"id":1}]} 2024-04-17T19:53:07.594 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-04-17T19:53:07.595 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.602+0000 7f7f84800640 1 -- 172.21.15.29:0/2989692995 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7f600683b0 msgr2=0x7f7f6006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:07.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.602+0000 7f7f84800640 1 --2- 172.21.15.29:0/2989692995 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7f600683b0 0x7f7f6006a8c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7f70019370 tx=0x7f7f700b3040 comp rx=0 tx=0).stop 2024-04-17T19:53:07.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.602+0000 7f7f84800640 1 -- 172.21.15.29:0/2989692995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f7c0ca240 msgr2=0x7f7f7c168210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:07.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.602+0000 7f7f84800640 1 --2- 172.21.15.29:0/2989692995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f7c0ca240 0x7f7f7c168210 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7f7800c700 tx=0x7f7f7800cbc0 comp rx=0 tx=0).stop 2024-04-17T19:53:07.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.602+0000 7f7f84800640 1 -- 172.21.15.29:0/2989692995 shutdown_connections 2024-04-17T19:53:07.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.602+0000 7f7f84800640 1 --2- 172.21.15.29:0/2989692995 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7f600683b0 0x7f7f6006a8c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:07.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.602+0000 7f7f84800640 1 --2- 172.21.15.29:0/2989692995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f7c0ca240 0x7f7f7c168210 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:07.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.602+0000 7f7f84800640 1 --2- 172.21.15.29:0/2989692995 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f7c08d670 0x7f7f7c167410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:07.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.602+0000 7f7f84800640 1 --2- 172.21.15.29:0/2989692995 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f7c071ed0 0x7f7f7c169d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:07.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.602+0000 7f7f84800640 1 -- 172.21.15.29:0/2989692995 >> 172.21.15.29:0/2989692995 conn(0x7f7f7c06fde0 msgr2=0x7f7f7c15fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:07.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.602+0000 7f7f84800640 1 -- 172.21.15.29:0/2989692995 shutdown_connections 2024-04-17T19:53:07.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:07.606+0000 7f7f84800640 1 -- 172.21.15.29:0/2989692995 wait complete. 2024-04-17T19:53:07.615 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-17T19:53:09.615 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:53:09.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.790+0000 7f10f6600640 1 Processor -- start 2024-04-17T19:53:09.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.790+0000 7f10f6600640 1 -- start start 2024-04-17T19:53:09.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.790+0000 7f10f6600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f10f00c6f70 0x7f10f00c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:09.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.790+0000 7f10f6600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10f006c660 0x7f10f006ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:09.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.790+0000 7f10f6600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10f015a0a0 con 0x7f10f0159c90 2024-04-17T19:53:09.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.790+0000 7f10f6600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f10f015abe0 con 0x7f10f00c6f70 2024-04-17T19:53:09.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.790+0000 7f10f6600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f10f015a340 con 0x7f10f006c660 2024-04-17T19:53:09.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.790+0000 7f10e7e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f10f00c6f70 0x7f10f00c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:09.783 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.790+0000 7f10e7e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f10f00c6f70 0x7f10f00c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59348/0 (socket says 172.21.15.29:59348) 2024-04-17T19:53:09.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.790+0000 7f10e7e00640 1 -- 172.21.15.29:0/3338499258 learned_addr learned my addr 172.21.15.29:0/3338499258 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:09.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.790+0000 7f10efe00640 1 --2- 172.21.15.29:0/3338499258 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10f006c660 0x7f10f006ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:09.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.790+0000 7f10e7e00640 1 -- 172.21.15.29:0/3338499258 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10f006c660 msgr2=0x7f10f006ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:09.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.790+0000 7f10e7e00640 1 --2- 172.21.15.29:0/3338499258 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10f006c660 0x7f10f006ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:09.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.790+0000 7f10e7e00640 1 -- 172.21.15.29:0/3338499258 >> v1:172.21.15.37:6789/0 conn(0x7f10f0159c90 legacy=0x7f10f00ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:09.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.790+0000 7f10e7e00640 1 -- 172.21.15.29:0/3338499258 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f015a620 con 0x7f10f00c6f70 2024-04-17T19:53:09.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.794+0000 7f10e7e00640 1 --2- 172.21.15.29:0/3338499258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f00c6f70 0x7f10f00c7380 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f10d8009870 tx=0x7f10d801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5885e9f31d6e5aa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:09.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.794+0000 7f10eea00640 1 -- 172.21.15.29:0/3338499258 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d80b2070 con 0x7f10f00c6f70 2024-04-17T19:53:09.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.794+0000 7f10eea00640 1 -- 172.21.15.29:0/3338499258 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10d80a9070 con 0x7f10f00c6f70 2024-04-17T19:53:09.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.794+0000 7f10eea00640 1 -- 172.21.15.29:0/3338499258 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d80ad880 con 0x7f10f00c6f70 2024-04-17T19:53:09.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.794+0000 7f10f6600640 1 -- 172.21.15.29:0/3338499258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f00c6f70 msgr2=0x7f10f00c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:09.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.794+0000 7f10f6600640 1 --2- 172.21.15.29:0/3338499258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f00c6f70 0x7f10f00c7380 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f10d8009870 tx=0x7f10d801c560 comp rx=0 tx=0).stop 2024-04-17T19:53:09.787 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.794+0000 7f10f6600640 1 -- 172.21.15.29:0/3338499258 shutdown_connections 2024-04-17T19:53:09.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.794+0000 7f10f6600640 1 --2- 172.21.15.29:0/3338499258 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10f006c660 0x7f10f006ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:09.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.794+0000 7f10f6600640 1 --2- 172.21.15.29:0/3338499258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f00c6f70 0x7f10f00c7380 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:09.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.794+0000 7f10f6600640 1 -- 172.21.15.29:0/3338499258 >> 172.21.15.29:0/3338499258 conn(0x7f10f006fde0 msgr2=0x7f10f00701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:09.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.794+0000 7f10f6600640 1 -- 172.21.15.29:0/3338499258 shutdown_connections 2024-04-17T19:53:09.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.794+0000 7f10f6600640 1 -- 172.21.15.29:0/3338499258 wait complete. 2024-04-17T19:53:09.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.798+0000 7f10f6600640 1 Processor -- start 2024-04-17T19:53:09.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.798+0000 7f10f6600640 1 -- start start 2024-04-17T19:53:09.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.798+0000 7f10f6600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f006c660 0x7f10f014f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.798+0000 7f10f6600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10f00c6f70 0x7f10f0146da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.798+0000 7f10f6600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10f0159c90 0x7f10f013afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.798+0000 7f10f6600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f10f00ca9c0 con 0x7f10f006c660 2024-04-17T19:53:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.798+0000 7f10f6600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f10f006cfb0 con 0x7f10f0159c90 2024-04-17T19:53:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.798+0000 7f10f6600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f10f00cab40 con 0x7f10f00c6f70 2024-04-17T19:53:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.798+0000 7f10ef400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f006c660 0x7f10f014f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.798+0000 7f10ef400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f006c660 0x7f10f014f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59356/0 (socket says 172.21.15.29:59356) 2024-04-17T19:53:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.798+0000 7f10ef400640 1 -- 172.21.15.29:0/3004939963 learned_addr learned my addr 172.21.15.29:0/3004939963 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.798+0000 7f10e7e00640 1 --2- 172.21.15.29:0/3004939963 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10f00c6f70 0x7f10f0146da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.798+0000 7f10efe00640 1 --2- 172.21.15.29:0/3004939963 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10f0159c90 0x7f10f013afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.802+0000 7f10ef400640 1 -- 172.21.15.29:0/3004939963 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10f00c6f70 msgr2=0x7f10f0146da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.802+0000 7f10ef400640 1 --2- 172.21.15.29:0/3004939963 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10f00c6f70 0x7f10f0146da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.802+0000 7f10ef400640 1 -- 172.21.15.29:0/3004939963 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10f0159c90 msgr2=0x7f10f013afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.802+0000 7f10ef400640 1 --2- 172.21.15.29:0/3004939963 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10f0159c90 0x7f10f013afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:09.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.802+0000 7f10ef400640 1 -- 172.21.15.29:0/3004939963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f014fe70 con 0x7f10f006c660 2024-04-17T19:53:09.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.802+0000 7f10ef400640 1 --2- 172.21.15.29:0/3004939963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f006c660 0x7f10f014f930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f10e000c370 tx=0x7f10e000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:09.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.802+0000 7f10e7400640 1 -- 172.21.15.29:0/3004939963 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e0017070 con 0x7f10f006c660 2024-04-17T19:53:09.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.802+0000 7f10e7400640 1 -- 172.21.15.29:0/3004939963 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10e00052d0 con 0x7f10f006c660 2024-04-17T19:53:09.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.802+0000 7f10f6600640 1 -- 172.21.15.29:0/3004939963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f01f7c90 con 0x7f10f006c660 2024-04-17T19:53:09.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.802+0000 7f10e7400640 1 -- 172.21.15.29:0/3004939963 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e00054c0 con 0x7f10f006c660 2024-04-17T19:53:09.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.802+0000 7f10f6600640 1 -- 172.21.15.29:0/3004939963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f01f76d0 con 0x7f10f006c660 2024-04-17T19:53:09.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.802+0000 7f10f6600640 1 -- 172.21.15.29:0/3004939963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10f0069b90 con 0x7f10f006c660 2024-04-17T19:53:09.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.802+0000 7f10e7400640 1 -- 172.21.15.29:0/3004939963 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f10e001a070 con 0x7f10f006c660 2024-04-17T19:53:09.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.802+0000 7f10e7400640 1 --2- 172.21.15.29:0/3004939963 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f10cc068380 0x7f10cc06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:09.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.802+0000 7f10e7400640 1 -- 172.21.15.29:0/3004939963 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7703+0+0 (secure 0 0 0) 0x7f10e0086560 con 0x7f10f006c660 2024-04-17T19:53:09.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.806+0000 7f10e7e00640 1 --2- 172.21.15.29:0/3004939963 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f10cc068380 0x7f10cc06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:09.796 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.806+0000 7f10e7e00640 1 --2- 172.21.15.29:0/3004939963 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f10cc068380 0x7f10cc06a890 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f10d80af000 tx=0x7f10d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:09.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:09.810+0000 7f10e7400640 1 -- 172.21.15.29:0/3004939963 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f10e00562b0 con 0x7f10f006c660 2024-04-17T19:53:10.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:10.014+0000 7f10f6600640 1 -- 172.21.15.29:0/3004939963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10f01598e0 con 0x7f10f006c660 2024-04-17T19:53:10.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:10.014+0000 7f10e7400640 1 -- 172.21.15.29:0/3004939963 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6907 (secure 0 0 0) 0x7f10e0055bb0 con 0x7f10f006c660 2024-04-17T19:53:10.006 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:10.006 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:34.862105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5212,4302,4259,4229]},"id":1}]} 2024-04-17T19:53:10.007 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-04-17T19:53:10.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:10.018+0000 7f10f6600640 1 -- 172.21.15.29:0/3004939963 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f10cc068380 msgr2=0x7f10cc06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:10.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:10.018+0000 7f10f6600640 1 --2- 172.21.15.29:0/3004939963 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f10cc068380 0x7f10cc06a890 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f10d80af000 tx=0x7f10d8008040 comp rx=0 tx=0).stop 2024-04-17T19:53:10.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:10.018+0000 7f10f6600640 1 -- 172.21.15.29:0/3004939963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f006c660 msgr2=0x7f10f014f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:10.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:10.018+0000 7f10f6600640 1 --2- 172.21.15.29:0/3004939963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f006c660 0x7f10f014f930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f10e000c370 tx=0x7f10e000c830 comp rx=0 tx=0).stop 2024-04-17T19:53:10.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:10.018+0000 7f10f6600640 1 -- 172.21.15.29:0/3004939963 shutdown_connections 2024-04-17T19:53:10.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:10.018+0000 7f10f6600640 1 --2- 172.21.15.29:0/3004939963 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f10cc068380 0x7f10cc06a890 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:10.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:10.018+0000 7f10f6600640 1 --2- 172.21.15.29:0/3004939963 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10f0159c90 0x7f10f013afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:10.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:10.018+0000 7f10f6600640 1 --2- 172.21.15.29:0/3004939963 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10f00c6f70 0x7f10f0146da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:10.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:10.018+0000 7f10f6600640 1 --2- 172.21.15.29:0/3004939963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f006c660 0x7f10f014f930 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:10.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:10.018+0000 7f10f6600640 1 -- 172.21.15.29:0/3004939963 >> 172.21.15.29:0/3004939963 conn(0x7f10f006fde0 msgr2=0x7f10f00cb530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:10.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:10.018+0000 7f10f6600640 1 -- 172.21.15.29:0/3004939963 shutdown_connections 2024-04-17T19:53:10.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:10.018+0000 7f10f6600640 1 -- 172.21.15.29:0/3004939963 wait complete. 2024-04-17T19:53:10.030 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-17T19:53:12.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:53:12.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.206+0000 7fab8f200640 1 Processor -- start 2024-04-17T19:53:12.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.206+0000 7fab8f200640 1 -- start start 2024-04-17T19:53:12.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.206+0000 7fab8f200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab90157250 0x7fab900cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:12.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.206+0000 7fab8f200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fab9015a700 0x7fab900ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:12.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.206+0000 7fab8f200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab900c4c10 con 0x7fab900c9ea0 2024-04-17T19:53:12.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.206+0000 7fab8f200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fab9006ef50 con 0x7fab90157250 2024-04-17T19:53:12.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.206+0000 7fab8f200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fab900c5a70 con 0x7fab9015a700 2024-04-17T19:53:12.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.206+0000 7fab8de00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab90157250 0x7fab900cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:12.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.206+0000 7fab8de00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab90157250 0x7fab900cb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59360/0 (socket says 172.21.15.29:59360) 2024-04-17T19:53:12.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.206+0000 7fab8de00640 1 -- 172.21.15.29:0/1254938583 learned_addr learned my addr 172.21.15.29:0/1254938583 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:12.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.206+0000 7fab8e800640 1 --2- 172.21.15.29:0/1254938583 >> v2:172.21.15.37:3300/0 conn(0x7fab9015a700 0x7fab900ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:12.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.206+0000 7fab8de00640 1 -- 172.21.15.29:0/1254938583 >> v2:172.21.15.37:3300/0 conn(0x7fab9015a700 msgr2=0x7fab900ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:12.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.206+0000 7fab8de00640 1 --2- 172.21.15.29:0/1254938583 >> v2:172.21.15.37:3300/0 conn(0x7fab9015a700 0x7fab900ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:12.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.206+0000 7fab8de00640 1 -- 172.21.15.29:0/1254938583 >> v1:172.21.15.37:6789/0 conn(0x7fab900c9ea0 legacy=0x7fab90159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:12.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.206+0000 7fab8de00640 1 -- 172.21.15.29:0/1254938583 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab900c4ef0 con 0x7fab90157250 2024-04-17T19:53:12.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.210+0000 7fab8de00640 1 --2- 172.21.15.29:0/1254938583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab90157250 0x7fab900cb2f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fab9006e8f0 tx=0x7fab8001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d9a9ecc0dfb872 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:12.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.210+0000 7fab8ca00640 1 -- 172.21.15.29:0/1254938583 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab800b5070 con 0x7fab90157250 2024-04-17T19:53:12.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.210+0000 7fab8ca00640 1 -- 172.21.15.29:0/1254938583 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab800a8a10 con 0x7fab90157250 2024-04-17T19:53:12.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.210+0000 7fab8ca00640 1 -- 172.21.15.29:0/1254938583 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab800a8de0 con 0x7fab90157250 2024-04-17T19:53:12.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.210+0000 7fab8f200640 1 -- 172.21.15.29:0/1254938583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab90157250 msgr2=0x7fab900cb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:12.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.210+0000 7fab8f200640 1 --2- 172.21.15.29:0/1254938583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab90157250 0x7fab900cb2f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fab9006e8f0 tx=0x7fab8001ed10 comp rx=0 tx=0).stop 2024-04-17T19:53:12.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.210+0000 7fab8f200640 1 -- 172.21.15.29:0/1254938583 shutdown_connections 2024-04-17T19:53:12.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.210+0000 7fab8f200640 1 --2- 172.21.15.29:0/1254938583 >> v2:172.21.15.37:3300/0 conn(0x7fab9015a700 0x7fab900ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:12.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.210+0000 7fab8f200640 1 --2- 172.21.15.29:0/1254938583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab90157250 0x7fab900cb2f0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:12.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.210+0000 7fab8f200640 1 -- 172.21.15.29:0/1254938583 >> 172.21.15.29:0/1254938583 conn(0x7fab900680e0 msgr2=0x7fab9006dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:12.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.210+0000 7fab8f200640 1 -- 172.21.15.29:0/1254938583 shutdown_connections 2024-04-17T19:53:12.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.210+0000 7fab8f200640 1 -- 172.21.15.29:0/1254938583 wait complete. 2024-04-17T19:53:12.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8f200640 1 Processor -- start 2024-04-17T19:53:12.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8f200640 1 -- start start 2024-04-17T19:53:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8f200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab900c9ea0 0x7fab901f61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8f200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab90157250 0x7fab901e5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8f200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab9015a700 0x7fab9006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8f200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fab900c54b0 con 0x7fab9015a700 2024-04-17T19:53:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8f200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fab90157660 con 0x7fab90157250 2024-04-17T19:53:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8f200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fab900c5790 con 0x7fab900c9ea0 2024-04-17T19:53:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8e800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab9015a700 0x7fab9006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:12.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8e800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab9015a700 0x7fab9006d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59370/0 (socket says 172.21.15.29:59370) 2024-04-17T19:53:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8e800640 1 -- 172.21.15.29:0/1737446382 learned_addr learned my addr 172.21.15.29:0/1737446382 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8de00640 1 --2- 172.21.15.29:0/1737446382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab900c9ea0 0x7fab901f61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8d400640 1 --2- 172.21.15.29:0/1737446382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab90157250 0x7fab901e5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8e800640 1 -- 172.21.15.29:0/1737446382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab900c9ea0 msgr2=0x7fab901f61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8e800640 1 --2- 172.21.15.29:0/1737446382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab900c9ea0 0x7fab901f61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8e800640 1 -- 172.21.15.29:0/1737446382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab90157250 msgr2=0x7fab901e5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8e800640 1 --2- 172.21.15.29:0/1737446382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab90157250 0x7fab901e5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8e800640 1 -- 172.21.15.29:0/1737446382 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab901e3c90 con 0x7fab9015a700 2024-04-17T19:53:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.214+0000 7fab8e800640 1 --2- 172.21.15.29:0/1737446382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab9015a700 0x7fab9006d4e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fab88007db0 tx=0x7fab8800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.218+0000 7fab86000640 1 -- 172.21.15.29:0/1737446382 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab88014420 con 0x7fab9015a700 2024-04-17T19:53:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.218+0000 7fab86000640 1 -- 172.21.15.29:0/1737446382 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab88019070 con 0x7fab9015a700 2024-04-17T19:53:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.218+0000 7fab86000640 1 -- 172.21.15.29:0/1737446382 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab88014800 con 0x7fab9015a700 2024-04-17T19:53:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.218+0000 7fab8f200640 1 -- 172.21.15.29:0/1737446382 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab901f2440 con 0x7fab9015a700 2024-04-17T19:53:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.218+0000 7fab8f200640 1 -- 172.21.15.29:0/1737446382 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab901f2a00 con 0x7fab9015a700 2024-04-17T19:53:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.218+0000 7fab8f200640 1 -- 172.21.15.29:0/1737446382 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab901e3a00 con 0x7fab9015a700 2024-04-17T19:53:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.218+0000 7fab86000640 1 -- 172.21.15.29:0/1737446382 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fab880149a0 con 0x7fab9015a700 2024-04-17T19:53:12.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.218+0000 7fab86000640 1 --2- 172.21.15.29:0/1737446382 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fab700683b0 0x7fab7006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:12.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.218+0000 7fab86000640 1 -- 172.21.15.29:0/1737446382 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7703+0+0 (secure 0 0 0) 0x7fab88087970 con 0x7fab9015a700 2024-04-17T19:53:12.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.218+0000 7fab8de00640 1 --2- 172.21.15.29:0/1737446382 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fab700683b0 0x7fab7006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:12.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.222+0000 7fab8de00640 1 --2- 172.21.15.29:0/1737446382 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fab700683b0 0x7fab7006a8c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fab800b2000 tx=0x7fab80009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:12.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.226+0000 7fab86000640 1 -- 172.21.15.29:0/1737446382 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fab88056fe0 con 0x7fab9015a700 2024-04-17T19:53:12.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.426+0000 7fab8f200640 1 -- 172.21.15.29:0/1737446382 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab901f2230 con 0x7fab9015a700 2024-04-17T19:53:12.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.426+0000 7fab86000640 1 -- 172.21.15.29:0/1737446382 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6907 (secure 0 0 0) 0x7fab8805b2e0 con 0x7fab9015a700 2024-04-17T19:53:12.420 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:12.420 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5350,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:52:34.862105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":4229,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"f","rank":3,"incarnation":16,"state":"up:active","state_seq":13,"addr":"172.21.15.37:6839/1100598562","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":1100598562},{"type":"v1","addr":"172.21.15.37:6839","nonce":1100598562}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5212,4302,4259,4229]},"id":1}]} 2024-04-17T19:53:12.420 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 31 2024-04-17T19:53:12.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.430+0000 7fab8f200640 1 -- 172.21.15.29:0/1737446382 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fab700683b0 msgr2=0x7fab7006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:12.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.430+0000 7fab8f200640 1 --2- 172.21.15.29:0/1737446382 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fab700683b0 0x7fab7006a8c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fab800b2000 tx=0x7fab80009040 comp rx=0 tx=0).stop 2024-04-17T19:53:12.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.430+0000 7fab8f200640 1 -- 172.21.15.29:0/1737446382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab9015a700 msgr2=0x7fab9006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:12.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.430+0000 7fab8f200640 1 --2- 172.21.15.29:0/1737446382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab9015a700 0x7fab9006d4e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fab88007db0 tx=0x7fab8800e4b0 comp rx=0 tx=0).stop 2024-04-17T19:53:12.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.430+0000 7fab8f200640 1 -- 172.21.15.29:0/1737446382 shutdown_connections 2024-04-17T19:53:12.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.430+0000 7fab8f200640 1 --2- 172.21.15.29:0/1737446382 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fab700683b0 0x7fab7006a8c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:12.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.430+0000 7fab8f200640 1 --2- 172.21.15.29:0/1737446382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab9015a700 0x7fab9006d4e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:12.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.430+0000 7fab8f200640 1 --2- 172.21.15.29:0/1737446382 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fab90157250 0x7fab901e5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:12.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.430+0000 7fab8f200640 1 --2- 172.21.15.29:0/1737446382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fab900c9ea0 0x7fab901f61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:12.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.430+0000 7fab8f200640 1 -- 172.21.15.29:0/1737446382 >> 172.21.15.29:0/1737446382 conn(0x7fab900680e0 msgr2=0x7fab90155940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:12.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.430+0000 7fab8f200640 1 -- 172.21.15.29:0/1737446382 shutdown_connections 2024-04-17T19:53:12.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:12.430+0000 7fab8f200640 1 -- 172.21.15.29:0/1737446382 wait complete. 2024-04-17T19:53:12.441 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-17T19:53:14.442 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:53:14.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.618+0000 7fe900e00640 1 Processor -- start 2024-04-17T19:53:14.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.618+0000 7fe900e00640 1 -- start start 2024-04-17T19:53:14.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.618+0000 7fe900e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe8fc0c9370 0x7fe8fc167940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:14.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.618+0000 7fe900e00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8fc06c1e0 con 0x7fe8fc15ca30 2024-04-17T19:53:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.618+0000 7fe900e00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8fc15f720 con 0x7fe8fc0c8f60 2024-04-17T19:53:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.618+0000 7fe900e00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fe8fc1604c0 con 0x7fe8fc0c9370 2024-04-17T19:53:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.618+0000 7fe8f9e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe8fc15ca30 0x7fe8fc156170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54796/0 (socket says 172.21.15.29:54796) 2024-04-17T19:53:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.618+0000 7fe8fa800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe8fc0c9370 0x7fe8fc167940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.618+0000 7fe8f9e00640 1 -- 172.21.15.29:0/1880262516 learned_addr learned my addr 172.21.15.29:0/1880262516 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:53:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.618+0000 7fe8fa800640 1 -- 172.21.15.29:0/1880262516 >> v1:172.21.15.37:6789/0 conn(0x7fe8fc0c8f60 legacy=0x7fe8fc067ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.618+0000 7fe8fa800640 1 -- 172.21.15.29:0/1880262516 >> v1:172.21.15.29:6789/0 conn(0x7fe8fc15ca30 legacy=0x7fe8fc156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:14.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.618+0000 7fe8fa800640 1 -- 172.21.15.29:0/1880262516 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8fc06c450 con 0x7fe8fc0c9370 2024-04-17T19:53:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.622+0000 7fe8fa800640 1 --2- 172.21.15.29:0/1880262516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0c9370 0x7fe8fc167940 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4009dc0 tx=0x7fe8f401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=547350ffebc9c739 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:14.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.622+0000 7fe8f8a00640 1 -- 172.21.15.29:0/1880262516 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f40b2070 con 0x7fe8fc0c9370 2024-04-17T19:53:14.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.622+0000 7fe8f8a00640 1 -- 172.21.15.29:0/1880262516 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8f40a5810 con 0x7fe8fc0c9370 2024-04-17T19:53:14.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.622+0000 7fe8f8a00640 1 -- 172.21.15.29:0/1880262516 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8f40a5be0 con 0x7fe8fc0c9370 2024-04-17T19:53:14.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.622+0000 7fe900e00640 1 -- 172.21.15.29:0/1880262516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0c9370 msgr2=0x7fe8fc167940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.622+0000 7fe900e00640 1 --2- 172.21.15.29:0/1880262516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0c9370 0x7fe8fc167940 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe8f4009dc0 tx=0x7fe8f401c950 comp rx=0 tx=0).stop 2024-04-17T19:53:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.622+0000 7fe900e00640 1 -- 172.21.15.29:0/1880262516 shutdown_connections 2024-04-17T19:53:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.622+0000 7fe900e00640 1 --2- 172.21.15.29:0/1880262516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0c9370 0x7fe8fc167940 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.622+0000 7fe900e00640 1 -- 172.21.15.29:0/1880262516 >> 172.21.15.29:0/1880262516 conn(0x7fe8fc0713d0 msgr2=0x7fe8fc0ce350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.626+0000 7fe900e00640 1 -- 172.21.15.29:0/1880262516 shutdown_connections 2024-04-17T19:53:14.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.626+0000 7fe900e00640 1 -- 172.21.15.29:0/1880262516 wait complete. 2024-04-17T19:53:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.626+0000 7fe900e00640 1 Processor -- start 2024-04-17T19:53:14.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.626+0000 7fe900e00640 1 -- start start 2024-04-17T19:53:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe900e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0c8f60 0x7fe8fc1f5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe900e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8fc0c9370 0x7fe8fc1465e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe900e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8fc15ca30 0x7fe8fc1f61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe900e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe8fc06d680 con 0x7fe8fc0c8f60 2024-04-17T19:53:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe900e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe8fc15ce40 con 0x7fe8fc15ca30 2024-04-17T19:53:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe900e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe8fc0683f0 con 0x7fe8fc0c9370 2024-04-17T19:53:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe8f9e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0c8f60 0x7fe8fc1f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe8f9e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0c8f60 0x7fe8fc1f5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59394/0 (socket says 172.21.15.29:59394) 2024-04-17T19:53:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe8f9e00640 1 -- 172.21.15.29:0/1459841960 learned_addr learned my addr 172.21.15.29:0/1459841960 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe8fa800640 1 --2- 172.21.15.29:0/1459841960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8fc15ca30 0x7fe8fc1f61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe8f9400640 1 --2- 172.21.15.29:0/1459841960 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8fc0c9370 0x7fe8fc1465e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe8f9e00640 1 -- 172.21.15.29:0/1459841960 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8fc0c9370 msgr2=0x7fe8fc1465e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe8f9e00640 1 --2- 172.21.15.29:0/1459841960 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8fc0c9370 0x7fe8fc1465e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:14.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe8f9e00640 1 -- 172.21.15.29:0/1459841960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8fc15ca30 msgr2=0x7fe8fc1f61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe8f9e00640 1 --2- 172.21.15.29:0/1459841960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8fc15ca30 0x7fe8fc1f61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe8f9e00640 1 -- 172.21.15.29:0/1459841960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8fc146b20 con 0x7fe8fc0c8f60 2024-04-17T19:53:14.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe8fa800640 1 --2- 172.21.15.29:0/1459841960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8fc15ca30 0x7fe8fc1f61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:53:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe8f9e00640 1 --2- 172.21.15.29:0/1459841960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0c8f60 0x7fe8fc1f5870 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00c430 tx=0x7fe8ec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe8f2000640 1 -- 172.21.15.29:0/1459841960 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec017070 con 0x7fe8fc0c8f60 2024-04-17T19:53:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe8f2000640 1 -- 172.21.15.29:0/1459841960 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8ec005cf0 con 0x7fe8fc0c8f60 2024-04-17T19:53:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe900e00640 1 -- 172.21.15.29:0/1459841960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8fc152580 con 0x7fe8fc0c8f60 2024-04-17T19:53:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe8f2000640 1 -- 172.21.15.29:0/1459841960 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec0072d0 con 0x7fe8fc0c8f60 2024-04-17T19:53:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe900e00640 1 -- 172.21.15.29:0/1459841960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8fc152de0 con 0x7fe8fc0c8f60 2024-04-17T19:53:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.630+0000 7fe900e00640 1 -- 172.21.15.29:0/1459841960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8fc069b90 con 0x7fe8fc0c8f60 2024-04-17T19:53:14.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.634+0000 7fe8f2000640 1 -- 172.21.15.29:0/1459841960 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe8ec01a070 con 0x7fe8fc0c8f60 2024-04-17T19:53:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.634+0000 7fe8f2000640 1 --2- 172.21.15.29:0/1459841960 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe8dc0683b0 0x7fe8dc06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.634+0000 7fe8f2000640 1 -- 172.21.15.29:0/1459841960 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6993+0+0 (secure 0 0 0) 0x7fe8ec0861e0 con 0x7fe8fc0c8f60 2024-04-17T19:53:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.634+0000 7fe8f9400640 1 --2- 172.21.15.29:0/1459841960 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe8dc0683b0 0x7fe8dc06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.634+0000 7fe8f9400640 1 --2- 172.21.15.29:0/1459841960 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe8dc0683b0 0x7fe8dc06a8c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc146050 tx=0x7fe8e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:14.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.638+0000 7fe8f2000640 1 -- 172.21.15.29:0/1459841960 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe8ec055b20 con 0x7fe8fc0c8f60 2024-04-17T19:53:14.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.838+0000 7fe900e00640 1 -- 172.21.15.29:0/1459841960 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8fc0c8d40 con 0x7fe8fc0c8f60 2024-04-17T19:53:14.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.838+0000 7fe8f2000640 1 -- 172.21.15.29:0/1459841960 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6077 (secure 0 0 0) 0x7fe8ec059e20 con 0x7fe8fc0c8f60 2024-04-17T19:53:14.829 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:14.829 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:13.852365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4259,4302,5212,4305]},"id":1}]} 2024-04-17T19:53:14.829 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 32 2024-04-17T19:53:14.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.838+0000 7fe900e00640 1 -- 172.21.15.29:0/1459841960 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe8dc0683b0 msgr2=0x7fe8dc06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:14.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.838+0000 7fe900e00640 1 --2- 172.21.15.29:0/1459841960 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe8dc0683b0 0x7fe8dc06a8c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe8fc146050 tx=0x7fe8e8009040 comp rx=0 tx=0).stop 2024-04-17T19:53:14.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.838+0000 7fe900e00640 1 -- 172.21.15.29:0/1459841960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0c8f60 msgr2=0x7fe8fc1f5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:14.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.838+0000 7fe900e00640 1 --2- 172.21.15.29:0/1459841960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0c8f60 0x7fe8fc1f5870 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00c430 tx=0x7fe8ec00c8f0 comp rx=0 tx=0).stop 2024-04-17T19:53:14.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.838+0000 7fe900e00640 1 -- 172.21.15.29:0/1459841960 shutdown_connections 2024-04-17T19:53:14.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.838+0000 7fe900e00640 1 --2- 172.21.15.29:0/1459841960 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe8dc0683b0 0x7fe8dc06a8c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:14.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.838+0000 7fe900e00640 1 --2- 172.21.15.29:0/1459841960 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe8fc15ca30 0x7fe8fc1f61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:14.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.838+0000 7fe900e00640 1 --2- 172.21.15.29:0/1459841960 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8fc0c9370 0x7fe8fc1465e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:14.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.838+0000 7fe900e00640 1 --2- 172.21.15.29:0/1459841960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8fc0c8f60 0x7fe8fc1f5870 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:14.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.838+0000 7fe900e00640 1 -- 172.21.15.29:0/1459841960 >> 172.21.15.29:0/1459841960 conn(0x7fe8fc0713d0 msgr2=0x7fe8fc1543b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:14.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.838+0000 7fe900e00640 1 -- 172.21.15.29:0/1459841960 shutdown_connections 2024-04-17T19:53:14.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:14.838+0000 7fe900e00640 1 -- 172.21.15.29:0/1459841960 wait complete. 2024-04-17T19:53:14.851 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2024-04-17T19:53:14.851 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-17T19:53:14.851 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:53:15.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.014+0000 7f86a5400640 1 Processor -- start 2024-04-17T19:53:15.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.014+0000 7f86a5400640 1 -- start start 2024-04-17T19:53:15.006 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.014+0000 7f86a5400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f86a00c4e10 0x7f86a0153d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:15.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.014+0000 7f86a5400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f86a015a460 0x7f86a015a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:15.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.014+0000 7f86a5400640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86a0159cc0 con 0x7f86a00cf730 2024-04-17T19:53:15.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.014+0000 7f86a5400640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f86a00cbad0 con 0x7f86a00c4e10 2024-04-17T19:53:15.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.014+0000 7f86a5400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f86a00c9e80 con 0x7f86a015a460 2024-04-17T19:53:15.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.014+0000 7f869e800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f86a00c4e10 0x7f86a0153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:15.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.014+0000 7f869e800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f86a00c4e10 0x7f86a0153d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59402/0 (socket says 172.21.15.29:59402) 2024-04-17T19:53:15.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.014+0000 7f869e800640 1 -- 172.21.15.29:0/3841740532 learned_addr learned my addr 172.21.15.29:0/3841740532 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:15.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.014+0000 7f8697e00640 1 --2- 172.21.15.29:0/3841740532 >> v2:172.21.15.37:3300/0 conn(0x7f86a015a460 0x7f86a015a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:15.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.014+0000 7f869e800640 1 -- 172.21.15.29:0/3841740532 >> v2:172.21.15.37:3300/0 conn(0x7f86a015a460 msgr2=0x7f86a015a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:15.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.014+0000 7f869e800640 1 --2- 172.21.15.29:0/3841740532 >> v2:172.21.15.37:3300/0 conn(0x7f86a015a460 0x7f86a015a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:15.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.014+0000 7f869e800640 1 -- 172.21.15.29:0/3841740532 >> v1:172.21.15.37:6789/0 conn(0x7f86a00cf730 legacy=0x7f86a015d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:15.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.014+0000 7f869e800640 1 -- 172.21.15.29:0/3841740532 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a00ca140 con 0x7f86a00c4e10 2024-04-17T19:53:15.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.018+0000 7f869e800640 1 --2- 172.21.15.29:0/3841740532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86a00c4e10 0x7f86a0153d80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f86a0072e20 tx=0x7f869001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86ebe60cb6eb3b0d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:15.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.018+0000 7f869de00640 1 -- 172.21.15.29:0/3841740532 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86900aa650 con 0x7f86a00c4e10 2024-04-17T19:53:15.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.018+0000 7f869de00640 1 -- 172.21.15.29:0/3841740532 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86900af070 con 0x7f86a00c4e10 2024-04-17T19:53:15.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.018+0000 7f869de00640 1 -- 172.21.15.29:0/3841740532 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86900aaa10 con 0x7f86a00c4e10 2024-04-17T19:53:15.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.018+0000 7f86a5400640 1 -- 172.21.15.29:0/3841740532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86a00c4e10 msgr2=0x7f86a0153d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:15.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.018+0000 7f86a5400640 1 --2- 172.21.15.29:0/3841740532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86a00c4e10 0x7f86a0153d80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f86a0072e20 tx=0x7f869001c4f0 comp rx=0 tx=0).stop 2024-04-17T19:53:15.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.018+0000 7f86a5400640 1 -- 172.21.15.29:0/3841740532 shutdown_connections 2024-04-17T19:53:15.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.018+0000 7f86a5400640 1 --2- 172.21.15.29:0/3841740532 >> v2:172.21.15.37:3300/0 conn(0x7f86a015a460 0x7f86a015a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:15.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.018+0000 7f86a5400640 1 --2- 172.21.15.29:0/3841740532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86a00c4e10 0x7f86a0153d80 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:15.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.018+0000 7f86a5400640 1 -- 172.21.15.29:0/3841740532 >> 172.21.15.29:0/3841740532 conn(0x7f86a006e1a0 msgr2=0x7f86a006e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:15.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.018+0000 7f86a5400640 1 -- 172.21.15.29:0/3841740532 shutdown_connections 2024-04-17T19:53:15.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.018+0000 7f86a5400640 1 -- 172.21.15.29:0/3841740532 wait complete. 2024-04-17T19:53:15.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f86a5400640 1 Processor -- start 2024-04-17T19:53:15.013 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f86a5400640 1 -- start start 2024-04-17T19:53:15.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f86a5400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86a00c4e10 0x7f86a01642f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:15.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f86a5400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a00cf730 0x7f86a01f77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:15.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f86a5400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a015a460 0x7f86a0143af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:15.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f86a5400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f86a00cff50 con 0x7f86a00c4e10 2024-04-17T19:53:15.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f86a5400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f86a0159f40 con 0x7f86a00cf730 2024-04-17T19:53:15.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f86a5400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f86a01542c0 con 0x7f86a015a460 2024-04-17T19:53:15.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f869e800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86a00c4e10 0x7f86a01642f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:15.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f869e800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86a00c4e10 0x7f86a01642f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59404/0 (socket says 172.21.15.29:59404) 2024-04-17T19:53:15.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f869e800640 1 -- 172.21.15.29:0/964557467 learned_addr learned my addr 172.21.15.29:0/964557467 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:15.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f8697e00640 1 --2- 172.21.15.29:0/964557467 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a00cf730 0x7f86a01f77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:15.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f869f200640 1 --2- 172.21.15.29:0/964557467 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a015a460 0x7f86a0143af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:15.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f869e800640 1 -- 172.21.15.29:0/964557467 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a015a460 msgr2=0x7f86a0143af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:15.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f869e800640 1 --2- 172.21.15.29:0/964557467 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a015a460 0x7f86a0143af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:15.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f869e800640 1 -- 172.21.15.29:0/964557467 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a00cf730 msgr2=0x7f86a01f77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:15.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f869e800640 1 --2- 172.21.15.29:0/964557467 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a00cf730 0x7f86a01f77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:15.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f869e800640 1 -- 172.21.15.29:0/964557467 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a01f7cf0 con 0x7f86a00c4e10 2024-04-17T19:53:15.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.022+0000 7f8697e00640 1 --2- 172.21.15.29:0/964557467 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a00cf730 0x7f86a01f77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:53:15.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.026+0000 7f869e800640 1 --2- 172.21.15.29:0/964557467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86a00c4e10 0x7f86a01642f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f86900ac040 tx=0x7f869001ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:15.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.026+0000 7f8696a00640 1 -- 172.21.15.29:0/964557467 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86900bd5f0 con 0x7f86a00c4e10 2024-04-17T19:53:15.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.026+0000 7f8696a00640 1 -- 172.21.15.29:0/964557467 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86900a6040 con 0x7f86a00c4e10 2024-04-17T19:53:15.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.026+0000 7f86a5400640 1 -- 172.21.15.29:0/964557467 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a0163ea0 con 0x7f86a00c4e10 2024-04-17T19:53:15.017 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.026+0000 7f8696a00640 1 -- 172.21.15.29:0/964557467 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86900bd9d0 con 0x7f86a00c4e10 2024-04-17T19:53:15.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.026+0000 7f86a5400640 1 -- 172.21.15.29:0/964557467 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a01f6fb0 con 0x7f86a00c4e10 2024-04-17T19:53:15.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.026+0000 7f86a5400640 1 -- 172.21.15.29:0/964557467 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a0069b90 con 0x7f86a00c4e10 2024-04-17T19:53:15.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.026+0000 7f8696a00640 1 -- 172.21.15.29:0/964557467 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f86900b33f0 con 0x7f86a00c4e10 2024-04-17T19:53:15.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.026+0000 7f8696a00640 1 --2- 172.21.15.29:0/964557467 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f867c068380 0x7f867c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:15.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.026+0000 7f8696a00640 1 -- 172.21.15.29:0/964557467 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6993+0+0 (secure 0 0 0) 0x7f86901215b0 con 0x7f86a00c4e10 2024-04-17T19:53:15.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.026+0000 7f8697e00640 1 --2- 172.21.15.29:0/964557467 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f867c068380 0x7f867c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:15.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.030+0000 7f8697e00640 1 --2- 172.21.15.29:0/964557467 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f867c068380 0x7f867c06a890 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f86a0158be0 tx=0x7f8688008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:15.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.030+0000 7f8696a00640 1 -- 172.21.15.29:0/964557467 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f86900c3060 con 0x7f86a00c4e10 2024-04-17T19:53:15.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.234+0000 7f86a5400640 1 -- 172.21.15.29:0/964557467 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86a00ca9d0 con 0x7f86a00c4e10 2024-04-17T19:53:15.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.234+0000 7f8696a00640 1 -- 172.21.15.29:0/964557467 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6077 (secure 0 0 0) 0x7f8690016110 con 0x7f86a00c4e10 2024-04-17T19:53:15.226 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:15.226 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:13.852365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4259,4302,5212,4305]},"id":1}]} 2024-04-17T19:53:15.226 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 32 2024-04-17T19:53:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.238+0000 7f86a5400640 1 -- 172.21.15.29:0/964557467 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f867c068380 msgr2=0x7f867c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.238+0000 7f86a5400640 1 --2- 172.21.15.29:0/964557467 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f867c068380 0x7f867c06a890 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f86a0158be0 tx=0x7f8688008040 comp rx=0 tx=0).stop 2024-04-17T19:53:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.238+0000 7f86a5400640 1 -- 172.21.15.29:0/964557467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86a00c4e10 msgr2=0x7f86a01642f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.238+0000 7f86a5400640 1 --2- 172.21.15.29:0/964557467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86a00c4e10 0x7f86a01642f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f86900ac040 tx=0x7f869001ad30 comp rx=0 tx=0).stop 2024-04-17T19:53:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.238+0000 7f86a5400640 1 -- 172.21.15.29:0/964557467 shutdown_connections 2024-04-17T19:53:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.238+0000 7f86a5400640 1 --2- 172.21.15.29:0/964557467 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f867c068380 0x7f867c06a890 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.238+0000 7f86a5400640 1 --2- 172.21.15.29:0/964557467 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f86a015a460 0x7f86a0143af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:15.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.238+0000 7f86a5400640 1 --2- 172.21.15.29:0/964557467 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f86a00cf730 0x7f86a01f77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:15.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.238+0000 7f86a5400640 1 --2- 172.21.15.29:0/964557467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86a00c4e10 0x7f86a01642f0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:15.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.238+0000 7f86a5400640 1 -- 172.21.15.29:0/964557467 >> 172.21.15.29:0/964557467 conn(0x7f86a006e1a0 msgr2=0x7f86a0159520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:15.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.238+0000 7f86a5400640 1 -- 172.21.15.29:0/964557467 shutdown_connections 2024-04-17T19:53:15.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:15.238+0000 7f86a5400640 1 -- 172.21.15.29:0/964557467 wait complete. 2024-04-17T19:53:15.248 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 32, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-17T19:47:49.315515+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 32, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4259": { "addr": "172.21.15.29:6836/4184160618", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6832", "nonce": 4184160618, "type": "v2" }, { "addr": "172.21.15.29:6836", "nonce": 4184160618, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4259, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 11 }, "gid_4302": { "addr": "172.21.15.29:6835/270402476", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6834", "nonce": 270402476, "type": "v2" }, { "addr": "172.21.15.29:6835", "nonce": 270402476, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 10, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 9 }, "gid_4305": { "addr": "172.21.15.29:6837/2691590158", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6833", "nonce": 2691590158, "type": "v2" }, { "addr": "172.21.15.29:6837", "nonce": 2691590158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4305, "incarnation": 5, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5212": { "addr": "172.21.15.37:6838/767681664", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6835", "nonce": 767681664, "type": "v2" }, { "addr": "172.21.15.37:6838", "nonce": 767681664, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5212, "incarnation": 26, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 31 }, "gid_5350": { "addr": "172.21.15.37:6837/2495953425", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6834", "nonce": 2495953425, "type": "v2" }, { "addr": "172.21.15.37:6837", "nonce": 2495953425, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5350, "incarnation": 32, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-17T19:53:13.852365+0000", "qdb_cluster": [ 4259, 4302, 5212, 4305 ], "qdb_leader": 4305, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4305, "mds_1": 4302, "mds_2": 4259, "mds_3": 5350, "mds_4": 5212 } } } ], "standbys": [] } 2024-04-17T19:53:17.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:53:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.422+0000 7fbe03c00640 1 Processor -- start 2024-04-17T19:53:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.422+0000 7fbe03c00640 1 -- start start 2024-04-17T19:53:17.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbe03c00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fbdfc1523d0 0x7fbdfc1527e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:17.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbe03c00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdfc0c60e0 con 0x7fbdfc0c71b0 2024-04-17T19:53:17.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbe03c00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdfc0c6350 con 0x7fbdfc0c75c0 2024-04-17T19:53:17.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbe03c00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fbdfc15ca30 con 0x7fbdfc1523d0 2024-04-17T19:53:17.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbe01200640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbdfc0c71b0 0x7fbdfc15c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54804/0 (socket says 172.21.15.29:54804) 2024-04-17T19:53:17.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbe01200640 1 -- 172.21.15.29:0/1190032243 learned_addr learned my addr 172.21.15.29:0/1190032243 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:53:17.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbe01c00640 1 --2- 172.21.15.29:0/1190032243 >> v2:172.21.15.37:3300/0 conn(0x7fbdfc1523d0 0x7fbdfc1527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:17.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbdfbe00640 1 -- 172.21.15.29:0/1190032243 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2973158569 0 0) 0x7fbdfc0c60e0 con 0x7fbdfc0c71b0 2024-04-17T19:53:17.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbdfbe00640 1 -- 172.21.15.29:0/1190032243 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbde0003530 con 0x7fbdfc0c71b0 2024-04-17T19:53:17.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbdfbe00640 1 -- 172.21.15.29:0/1190032243 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4101818101 0 0) 0x7fbde0003530 con 0x7fbdfc0c71b0 2024-04-17T19:53:17.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbdfbe00640 1 -- 172.21.15.29:0/1190032243 >> v2:172.21.15.37:3300/0 conn(0x7fbdfc1523d0 msgr2=0x7fbdfc1527e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:17.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbdfbe00640 1 --2- 172.21.15.29:0/1190032243 >> v2:172.21.15.37:3300/0 conn(0x7fbdfc1523d0 0x7fbdfc1527e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:17.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbdfbe00640 1 -- 172.21.15.29:0/1190032243 >> v1:172.21.15.37:6789/0 conn(0x7fbdfc0c75c0 legacy=0x7fbdfc06bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:17.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbdfbe00640 1 -- 172.21.15.29:0/1190032243 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdfc0c6630 con 0x7fbdfc0c71b0 2024-04-17T19:53:17.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbdfbe00640 1 -- 172.21.15.29:0/1190032243 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fbdec001680 con 0x7fbdfc0c71b0 2024-04-17T19:53:17.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbdfbe00640 1 -- 172.21.15.29:0/1190032243 >> v1:172.21.15.29:6789/0 conn(0x7fbdfc0c71b0 legacy=0x7fbdfc15c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:17.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbdfbe00640 1 --2- 172.21.15.29:0/1190032243 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbde0003a30 0x7fbde0003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:17.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbdfbe00640 1 --2- 172.21.15.29:0/1190032243 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbde0004810 0x7fbde0004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:17.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbdfbe00640 1 --2- 172.21.15.29:0/1190032243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbde0005230 0x7fbde000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:17.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbdfbe00640 1 -- 172.21.15.29:0/1190032243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbdfc15ca30 con 0x7fbde0005230 2024-04-17T19:53:17.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbdfbe00640 1 -- 172.21.15.29:0/1190032243 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbde000c1b0 con 0x7fbde0004810 2024-04-17T19:53:17.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbdfbe00640 1 -- 172.21.15.29:0/1190032243 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbde000c3b0 con 0x7fbde0003a30 2024-04-17T19:53:17.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbe01c00640 1 --2- 172.21.15.29:0/1190032243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbde0005230 0x7fbde000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:17.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbe01200640 1 --2- 172.21.15.29:0/1190032243 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbde0003a30 0x7fbde0003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:17.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbdf3e00640 1 --2- 172.21.15.29:0/1190032243 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbde0004810 0x7fbde0004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:17.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbe01c00640 1 -- 172.21.15.29:0/1190032243 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbde0003a30 msgr2=0x7fbde0003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:17.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbe01c00640 1 --2- 172.21.15.29:0/1190032243 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbde0003a30 0x7fbde0003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:17.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbe01c00640 1 -- 172.21.15.29:0/1190032243 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbde0004810 msgr2=0x7fbde0004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:17.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbe01c00640 1 --2- 172.21.15.29:0/1190032243 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbde0004810 0x7fbde0004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:17.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.426+0000 7fbe01c00640 1 -- 172.21.15.29:0/1190032243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde0003530 con 0x7fbde0005230 2024-04-17T19:53:17.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.430+0000 7fbe01c00640 1 -- 172.21.15.29:0/1190032243 reap_dead start 2024-04-17T19:53:17.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.430+0000 7fbe01c00640 1 --2- 172.21.15.29:0/1190032243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbde0005230 0x7fbde000b9d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbdf400f270 tx=0x7fbdf401ddc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b1a90936ddd6fa server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:17.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.430+0000 7fbdfbe00640 1 -- 172.21.15.29:0/1190032243 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf400ad20 con 0x7fbde0005230 2024-04-17T19:53:17.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.430+0000 7fbdfbe00640 1 -- 172.21.15.29:0/1190032243 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdf40a6bc0 con 0x7fbde0005230 2024-04-17T19:53:17.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.430+0000 7fbdfbe00640 1 -- 172.21.15.29:0/1190032243 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf40039a0 con 0x7fbde0005230 2024-04-17T19:53:17.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.430+0000 7fbe03c00640 1 -- 172.21.15.29:0/1190032243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbde0005230 msgr2=0x7fbde000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:17.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.430+0000 7fbe03c00640 1 --2- 172.21.15.29:0/1190032243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbde0005230 0x7fbde000b9d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbdf400f270 tx=0x7fbdf401ddc0 comp rx=0 tx=0).stop 2024-04-17T19:53:17.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.430+0000 7fbe03c00640 1 -- 172.21.15.29:0/1190032243 shutdown_connections 2024-04-17T19:53:17.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.430+0000 7fbe03c00640 1 --2- 172.21.15.29:0/1190032243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbde0005230 0x7fbde000b9d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:17.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.430+0000 7fbe03c00640 1 -- 172.21.15.29:0/1190032243 >> 172.21.15.29:0/1190032243 conn(0x7fbdfc06fde0 msgr2=0x7fbdfc0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:17.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.430+0000 7fbe03c00640 1 -- 172.21.15.29:0/1190032243 shutdown_connections 2024-04-17T19:53:17.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.430+0000 7fbe03c00640 1 -- 172.21.15.29:0/1190032243 wait complete. 2024-04-17T19:53:17.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe03c00640 1 Processor -- start 2024-04-17T19:53:17.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe03c00640 1 -- start start 2024-04-17T19:53:17.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe03c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbde0005230 0x7fbdfc1e2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:17.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe03c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbdfc1e18c0 0x7fbdfc150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:17.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe03c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbdfc151220 0x7fbdfc1f5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:17.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe03c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbdfc0cb140 con 0x7fbde0005230 2024-04-17T19:53:17.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe03c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbdfc15be90 con 0x7fbdfc1e18c0 2024-04-17T19:53:17.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe03c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbdfc0c6910 con 0x7fbdfc151220 2024-04-17T19:53:17.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe01200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbde0005230 0x7fbdfc1e2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:17.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe01200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbde0005230 0x7fbdfc1e2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59412/0 (socket says 172.21.15.29:59412) 2024-04-17T19:53:17.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe01200640 1 -- 172.21.15.29:0/1896834581 learned_addr learned my addr 172.21.15.29:0/1896834581 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:17.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbdf3e00640 1 --2- 172.21.15.29:0/1896834581 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbdfc1e18c0 0x7fbdfc150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:17.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe01c00640 1 --2- 172.21.15.29:0/1896834581 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbdfc151220 0x7fbdfc1f5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:17.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe01200640 1 -- 172.21.15.29:0/1896834581 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbdfc151220 msgr2=0x7fbdfc1f5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:17.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe01200640 1 --2- 172.21.15.29:0/1896834581 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbdfc151220 0x7fbdfc1f5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:17.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe01200640 1 -- 172.21.15.29:0/1896834581 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbdfc1e18c0 msgr2=0x7fbdfc150ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:17.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe01200640 1 --2- 172.21.15.29:0/1896834581 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbdfc1e18c0 0x7fbdfc150ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:17.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe01200640 1 -- 172.21.15.29:0/1896834581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdfc1e2dd0 con 0x7fbde0005230 2024-04-17T19:53:17.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbdf3e00640 1 --2- 172.21.15.29:0/1896834581 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbdfc1e18c0 0x7fbdfc150ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:53:17.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe01c00640 1 --2- 172.21.15.29:0/1896834581 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbdfc151220 0x7fbdfc1f5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:53:17.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe01200640 1 --2- 172.21.15.29:0/1896834581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbde0005230 0x7fbdfc1e2890 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbdec007290 tx=0x7fbdec007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:17.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbdf9600640 1 -- 172.21.15.29:0/1896834581 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec007e20 con 0x7fbde0005230 2024-04-17T19:53:17.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbdf9600640 1 -- 172.21.15.29:0/1896834581 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdec018070 con 0x7fbde0005230 2024-04-17T19:53:17.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbdf9600640 1 -- 172.21.15.29:0/1896834581 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdec00fae0 con 0x7fbde0005230 2024-04-17T19:53:17.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe03c00640 1 -- 172.21.15.29:0/1896834581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdfc1e46f0 con 0x7fbde0005230 2024-04-17T19:53:17.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.434+0000 7fbe03c00640 1 -- 172.21.15.29:0/1896834581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdfc151f50 con 0x7fbde0005230 2024-04-17T19:53:17.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.438+0000 7fbe03c00640 1 -- 172.21.15.29:0/1896834581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdfc069b90 con 0x7fbde0005230 2024-04-17T19:53:17.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.438+0000 7fbdf9600640 1 -- 172.21.15.29:0/1896834581 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbdec00fc80 con 0x7fbde0005230 2024-04-17T19:53:17.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.438+0000 7fbdf9600640 1 --2- 172.21.15.29:0/1896834581 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbdd8068380 0x7fbdd806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:17.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.438+0000 7fbdf9600640 1 -- 172.21.15.29:0/1896834581 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7585+0+0 (secure 0 0 0) 0x7fbdec087010 con 0x7fbde0005230 2024-04-17T19:53:17.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.438+0000 7fbdf3e00640 1 --2- 172.21.15.29:0/1896834581 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbdd8068380 0x7fbdd806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:17.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.438+0000 7fbdf3e00640 1 --2- 172.21.15.29:0/1896834581 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbdd8068380 0x7fbdd806a890 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc14ea40 tx=0x7fbde4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:17.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.442+0000 7fbdf9600640 1 -- 172.21.15.29:0/1896834581 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbdec056e70 con 0x7fbde0005230 2024-04-17T19:53:17.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.642+0000 7fbe03c00640 1 -- 172.21.15.29:0/1896834581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbdfc075960 con 0x7fbde0005230 2024-04-17T19:53:17.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.646+0000 7fbdf9600640 1 -- 172.21.15.29:0/1896834581 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+6077 (secure 0 0 0) 0x7fbdec056770 con 0x7fbde0005230 2024-04-17T19:53:17.636 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:17.636 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:13.852365+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4259,4302,5212,4305]},"id":1}]} 2024-04-17T19:53:17.637 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 32 2024-04-17T19:53:17.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.646+0000 7fbe03c00640 1 -- 172.21.15.29:0/1896834581 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbdd8068380 msgr2=0x7fbdd806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:17.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.646+0000 7fbe03c00640 1 --2- 172.21.15.29:0/1896834581 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbdd8068380 0x7fbdd806a890 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc14ea40 tx=0x7fbde4009040 comp rx=0 tx=0).stop 2024-04-17T19:53:17.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.646+0000 7fbe03c00640 1 -- 172.21.15.29:0/1896834581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbde0005230 msgr2=0x7fbdfc1e2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:17.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.646+0000 7fbe03c00640 1 --2- 172.21.15.29:0/1896834581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbde0005230 0x7fbdfc1e2890 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbdec007290 tx=0x7fbdec007750 comp rx=0 tx=0).stop 2024-04-17T19:53:17.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.646+0000 7fbe03c00640 1 -- 172.21.15.29:0/1896834581 shutdown_connections 2024-04-17T19:53:17.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.646+0000 7fbe03c00640 1 --2- 172.21.15.29:0/1896834581 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbdd8068380 0x7fbdd806a890 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:17.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.646+0000 7fbe03c00640 1 --2- 172.21.15.29:0/1896834581 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbdfc151220 0x7fbdfc1f5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:17.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.646+0000 7fbe03c00640 1 --2- 172.21.15.29:0/1896834581 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbdfc1e18c0 0x7fbdfc150ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:17.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.646+0000 7fbe03c00640 1 --2- 172.21.15.29:0/1896834581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbde0005230 0x7fbdfc1e2890 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:17.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.646+0000 7fbe03c00640 1 -- 172.21.15.29:0/1896834581 >> 172.21.15.29:0/1896834581 conn(0x7fbdfc06fde0 msgr2=0x7fbdfc0c8310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:17.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.646+0000 7fbe03c00640 1 -- 172.21.15.29:0/1896834581 shutdown_connections 2024-04-17T19:53:17.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:17.646+0000 7fbe03c00640 1 -- 172.21.15.29:0/1896834581 wait complete. 2024-04-17T19:53:17.658 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:53:19.659 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:53:19.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.818+0000 7f9331800640 1 Processor -- start 2024-04-17T19:53:19.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.818+0000 7f9331800640 1 -- start start 2024-04-17T19:53:19.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.822+0000 7f9331800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f932c0c7080 0x7f932c0cca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:19.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.822+0000 7f9331800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f932c0c75d0 con 0x7f932c0c6310 2024-04-17T19:53:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.822+0000 7f9331800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f932c0c7870 con 0x7f932c15fdb0 2024-04-17T19:53:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.822+0000 7f9331800640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f932c0c7b50 con 0x7f932c0c7080 2024-04-17T19:53:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.822+0000 7f9323e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f932c0c6310 0x7f932c15f6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51274/0 (socket says 172.21.15.29:51274) 2024-04-17T19:53:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.822+0000 7f9323e00640 1 -- 172.21.15.29:0/461073096 learned_addr learned my addr 172.21.15.29:0/461073096 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:53:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.822+0000 7f932aa00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f932c0c7080 0x7f932c0cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.822+0000 7f932a000640 1 -- 172.21.15.29:0/461073096 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 213471627 0 0) 0x7f932c0c75d0 con 0x7f932c0c6310 2024-04-17T19:53:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.822+0000 7f932a000640 1 -- 172.21.15.29:0/461073096 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9310003530 con 0x7f932c0c6310 2024-04-17T19:53:19.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.822+0000 7f932aa00640 1 -- 172.21.15.29:0/461073096 >> v1:172.21.15.37:6789/0 conn(0x7f932c15fdb0 legacy=0x7f932c166810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:19.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.822+0000 7f932aa00640 1 -- 172.21.15.29:0/461073096 >> v1:172.21.15.29:6789/0 conn(0x7f932c0c6310 legacy=0x7f932c15f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:19.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.822+0000 7f932aa00640 1 -- 172.21.15.29:0/461073096 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f932c0c7e30 con 0x7f932c0c7080 2024-04-17T19:53:19.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.826+0000 7f932aa00640 1 --2- 172.21.15.29:0/461073096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f932c0c7080 0x7f932c0cca80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f931c0097c0 tx=0x7f931c01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce73e859fd6a9c46 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:19.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.826+0000 7f932a000640 1 -- 172.21.15.29:0/461073096 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c0b5070 con 0x7f932c0c7080 2024-04-17T19:53:19.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.826+0000 7f932a000640 1 -- 172.21.15.29:0/461073096 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f931c0a8990 con 0x7f932c0c7080 2024-04-17T19:53:19.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.826+0000 7f932a000640 1 -- 172.21.15.29:0/461073096 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c0a8d40 con 0x7f932c0c7080 2024-04-17T19:53:19.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.830+0000 7f9331800640 1 -- 172.21.15.29:0/461073096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f932c0c7080 msgr2=0x7f932c0cca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:19.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.830+0000 7f9331800640 1 --2- 172.21.15.29:0/461073096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f932c0c7080 0x7f932c0cca80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f931c0097c0 tx=0x7f931c01ec90 comp rx=0 tx=0).stop 2024-04-17T19:53:19.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.830+0000 7f9331800640 1 -- 172.21.15.29:0/461073096 shutdown_connections 2024-04-17T19:53:19.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.830+0000 7f9331800640 1 --2- 172.21.15.29:0/461073096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f932c0c7080 0x7f932c0cca80 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:19.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.830+0000 7f9331800640 1 -- 172.21.15.29:0/461073096 >> 172.21.15.29:0/461073096 conn(0x7f932c0723e0 msgr2=0x7f932c0727f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:19.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.830+0000 7f9331800640 1 -- 172.21.15.29:0/461073096 shutdown_connections 2024-04-17T19:53:19.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.830+0000 7f9331800640 1 -- 172.21.15.29:0/461073096 wait complete. 2024-04-17T19:53:19.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.830+0000 7f9331800640 1 Processor -- start 2024-04-17T19:53:19.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.830+0000 7f9331800640 1 -- start start 2024-04-17T19:53:19.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f9331800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f932c0c6310 0x7f932c14fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:19.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f9331800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f932c0c7080 0x7f932c0cf640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:19.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f9331800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f932c15fdb0 0x7f932c0cfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f9331800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f932c0c6b30 con 0x7f932c0c6310 2024-04-17T19:53:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f9331800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f932c0c83f0 con 0x7f932c0c7080 2024-04-17T19:53:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f9331800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f932c0c6cb0 con 0x7f932c15fdb0 2024-04-17T19:53:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f932aa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f932c0c6310 0x7f932c14fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f932aa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f932c0c6310 0x7f932c14fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60620/0 (socket says 172.21.15.29:60620) 2024-04-17T19:53:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f932aa00640 1 -- 172.21.15.29:0/2153269358 learned_addr learned my addr 172.21.15.29:0/2153269358 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f9323e00640 1 --2- 172.21.15.29:0/2153269358 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f932c0c7080 0x7f932c0cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f932b400640 1 --2- 172.21.15.29:0/2153269358 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f932c15fdb0 0x7f932c0cfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f932aa00640 1 -- 172.21.15.29:0/2153269358 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f932c15fdb0 msgr2=0x7f932c0cfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f932aa00640 1 --2- 172.21.15.29:0/2153269358 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f932c15fdb0 0x7f932c0cfdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f932aa00640 1 -- 172.21.15.29:0/2153269358 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f932c0c7080 msgr2=0x7f932c0cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f932aa00640 1 --2- 172.21.15.29:0/2153269358 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f932c0c7080 0x7f932c0cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f932aa00640 1 -- 172.21.15.29:0/2153269358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f932c1511a0 con 0x7f932c0c6310 2024-04-17T19:53:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f932aa00640 1 --2- 172.21.15.29:0/2153269358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f932c0c6310 0x7f932c14fd30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f931c0b3040 tx=0x7f931c01ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:19.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f9322a00640 1 -- 172.21.15.29:0/2153269358 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c0c0070 con 0x7f932c0c6310 2024-04-17T19:53:19.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f9322a00640 1 -- 172.21.15.29:0/2153269358 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f931c004230 con 0x7f932c0c6310 2024-04-17T19:53:19.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f9331800640 1 -- 172.21.15.29:0/2153269358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f932c0cfb80 con 0x7f932c0c6310 2024-04-17T19:53:19.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f9322a00640 1 -- 172.21.15.29:0/2153269358 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f931c0b5040 con 0x7f932c0c6310 2024-04-17T19:53:19.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f9331800640 1 -- 172.21.15.29:0/2153269358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f932c1f5f80 con 0x7f932c0c6310 2024-04-17T19:53:19.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.834+0000 7f9331800640 1 -- 172.21.15.29:0/2153269358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f932c069b90 con 0x7f932c0c6310 2024-04-17T19:53:19.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.838+0000 7f9322a00640 1 -- 172.21.15.29:0/2153269358 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f931c0c3070 con 0x7f932c0c6310 2024-04-17T19:53:19.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.838+0000 7f9322a00640 1 --2- 172.21.15.29:0/2153269358 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f93080683b0 0x7f930806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:19.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.838+0000 7f9322a00640 1 -- 172.21.15.29:0/2153269358 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7585+0+0 (secure 0 0 0) 0x7f931c123950 con 0x7f932c0c6310 2024-04-17T19:53:19.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.838+0000 7f9323e00640 1 --2- 172.21.15.29:0/2153269358 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f93080683b0 0x7f930806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:19.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.838+0000 7f9323e00640 1 --2- 172.21.15.29:0/2153269358 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f93080683b0 0x7f930806a8c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f93140011f0 tx=0x7f9314009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:19.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:19.842+0000 7f9322a00640 1 -- 172.21.15.29:0/2153269358 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f931c0187f0 con 0x7f932c0c6310 2024-04-17T19:53:20.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:20.046+0000 7f9331800640 1 -- 172.21.15.29:0/2153269358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f932c0cd5c0 con 0x7f932c0c6310 2024-04-17T19:53:20.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:20.050+0000 7f9322a00640 1 -- 172.21.15.29:0/2153269358 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6079 (secure 0 0 0) 0x7f931c0f3680 con 0x7f932c0c6310 2024-04-17T19:53:20.040 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:20.040 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:18.868672+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:resolve","state_seq":12,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5212,4302,4259]},"id":1}]} 2024-04-17T19:53:20.040 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 33 2024-04-17T19:53:20.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:20.050+0000 7f9331800640 1 -- 172.21.15.29:0/2153269358 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f93080683b0 msgr2=0x7f930806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:20.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:20.050+0000 7f9331800640 1 --2- 172.21.15.29:0/2153269358 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f93080683b0 0x7f930806a8c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f93140011f0 tx=0x7f9314009040 comp rx=0 tx=0).stop 2024-04-17T19:53:20.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:20.050+0000 7f9331800640 1 -- 172.21.15.29:0/2153269358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f932c0c6310 msgr2=0x7f932c14fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:20.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:20.050+0000 7f9331800640 1 --2- 172.21.15.29:0/2153269358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f932c0c6310 0x7f932c14fd30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f931c0b3040 tx=0x7f931c01ba30 comp rx=0 tx=0).stop 2024-04-17T19:53:20.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:20.050+0000 7f9331800640 1 -- 172.21.15.29:0/2153269358 shutdown_connections 2024-04-17T19:53:20.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:20.050+0000 7f9331800640 1 --2- 172.21.15.29:0/2153269358 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f93080683b0 0x7f930806a8c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:20.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:20.050+0000 7f9331800640 1 --2- 172.21.15.29:0/2153269358 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f932c15fdb0 0x7f932c0cfdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:20.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:20.050+0000 7f9331800640 1 --2- 172.21.15.29:0/2153269358 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f932c0c7080 0x7f932c0cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:20.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:20.050+0000 7f9331800640 1 --2- 172.21.15.29:0/2153269358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f932c0c6310 0x7f932c14fd30 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:20.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:20.050+0000 7f9331800640 1 -- 172.21.15.29:0/2153269358 >> 172.21.15.29:0/2153269358 conn(0x7f932c0723e0 msgr2=0x7f932c152750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:20.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:20.050+0000 7f9331800640 1 -- 172.21.15.29:0/2153269358 shutdown_connections 2024-04-17T19:53:20.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:20.050+0000 7f9331800640 1 -- 172.21.15.29:0/2153269358 wait complete. 2024-04-17T19:53:20.062 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:53:21.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:53:21.549 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:53:22.063 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:53:22.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.234+0000 7f0794600640 1 Processor -- start 2024-04-17T19:53:22.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.234+0000 7f0794600640 1 -- start start 2024-04-17T19:53:22.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f0794600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f078c0cccd0 0x7f078c0ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:22.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f0794600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f078c0cd0e0 con 0x7f078c0ca870 2024-04-17T19:53:22.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f0794600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f078c0c88b0 con 0x7f078c0cc8c0 2024-04-17T19:53:22.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f0794600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f078c0c6ce0 con 0x7f078c0cccd0 2024-04-17T19:53:22.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f0792600640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f078c0ca870 0x7f078c1635e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51282/0 (socket says 172.21.15.29:51282) 2024-04-17T19:53:22.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f0792600640 1 -- 172.21.15.29:0/2621350676 learned_addr learned my addr 172.21.15.29:0/2621350676 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:53:22.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f0791200640 1 --2- 172.21.15.29:0/2621350676 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f078c0cccd0 0x7f078c0ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:22.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f078be00640 1 -- 172.21.15.29:0/2621350676 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3471391905 0 0) 0x7f078c0cd0e0 con 0x7f078c0ca870 2024-04-17T19:53:22.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f078be00640 1 -- 172.21.15.29:0/2621350676 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0774003530 con 0x7f078c0ca870 2024-04-17T19:53:22.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f078be00640 1 -- 172.21.15.29:0/2621350676 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2878676150 0 0) 0x7f078c0c88b0 con 0x7f078c0cc8c0 2024-04-17T19:53:22.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f078be00640 1 -- 172.21.15.29:0/2621350676 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f078c0cd0e0 con 0x7f078c0cc8c0 2024-04-17T19:53:22.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f078be00640 1 -- 172.21.15.29:0/2621350676 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 689246273 0 0) 0x7f0774003530 con 0x7f078c0ca870 2024-04-17T19:53:22.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f078be00640 1 -- 172.21.15.29:0/2621350676 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f078c0cccd0 msgr2=0x7f078c0ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:22.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f078be00640 1 --2- 172.21.15.29:0/2621350676 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f078c0cccd0 0x7f078c0ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:22.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f078be00640 1 -- 172.21.15.29:0/2621350676 >> v1:172.21.15.37:6789/0 conn(0x7f078c0cc8c0 legacy=0x7f078c159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:22.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f078be00640 1 -- 172.21.15.29:0/2621350676 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078c0c6fc0 con 0x7f078c0ca870 2024-04-17T19:53:22.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f078be00640 1 -- 172.21.15.29:0/2621350676 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f0784001a00 con 0x7f078c0ca870 2024-04-17T19:53:22.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f078be00640 1 -- 172.21.15.29:0/2621350676 >> v1:172.21.15.29:6789/0 conn(0x7f078c0ca870 legacy=0x7f078c1635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:22.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f078be00640 1 --2- 172.21.15.29:0/2621350676 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0774003bf0 0x7f0774004040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:22.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f078be00640 1 --2- 172.21.15.29:0/2621350676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0774004810 0x7f0774004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:22.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f078be00640 1 --2- 172.21.15.29:0/2621350676 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07740051b0 0x7f077400dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:22.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f078be00640 1 -- 172.21.15.29:0/2621350676 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f078c0c6ce0 con 0x7f0774004810 2024-04-17T19:53:22.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f078be00640 1 -- 172.21.15.29:0/2621350676 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f077400e330 con 0x7f07740051b0 2024-04-17T19:53:22.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f078be00640 1 -- 172.21.15.29:0/2621350676 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f077400e530 con 0x7f0774003bf0 2024-04-17T19:53:22.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f0791200640 1 --2- 172.21.15.29:0/2621350676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0774004810 0x7f0774004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:22.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f0791c00640 1 --2- 172.21.15.29:0/2621350676 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0774003bf0 0x7f0774004040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:22.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f0792600640 1 --2- 172.21.15.29:0/2621350676 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07740051b0 0x7f077400dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:22.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f0791200640 1 -- 172.21.15.29:0/2621350676 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0774003bf0 msgr2=0x7f0774004040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:22.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f0791200640 1 --2- 172.21.15.29:0/2621350676 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0774003bf0 0x7f0774004040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:22.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f0791200640 1 -- 172.21.15.29:0/2621350676 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07740051b0 msgr2=0x7f077400dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:22.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f0791200640 1 --2- 172.21.15.29:0/2621350676 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f07740051b0 0x7f077400dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:22.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.238+0000 7f0791200640 1 -- 172.21.15.29:0/2621350676 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0774003530 con 0x7f0774004810 2024-04-17T19:53:22.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.242+0000 7f0791200640 1 --2- 172.21.15.29:0/2621350676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0774004810 0x7f0774004c20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f078c06d610 tx=0x7f077801c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e01bc73fff00cede server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:22.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.242+0000 7f078be00640 1 -- 172.21.15.29:0/2621350676 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778009670 con 0x7f0774004810 2024-04-17T19:53:22.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.242+0000 7f078be00640 1 -- 172.21.15.29:0/2621350676 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07780a64f0 con 0x7f0774004810 2024-04-17T19:53:22.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.242+0000 7f078be00640 1 -- 172.21.15.29:0/2621350676 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07780a68e0 con 0x7f0774004810 2024-04-17T19:53:22.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.242+0000 7f0794600640 1 -- 172.21.15.29:0/2621350676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0774004810 msgr2=0x7f0774004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:22.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.242+0000 7f0794600640 1 --2- 172.21.15.29:0/2621350676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0774004810 0x7f0774004c20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f078c06d610 tx=0x7f077801c8f0 comp rx=0 tx=0).stop 2024-04-17T19:53:22.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.242+0000 7f0792600640 1 -- 172.21.15.29:0/2621350676 reap_dead start 2024-04-17T19:53:22.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.242+0000 7f0794600640 1 -- 172.21.15.29:0/2621350676 shutdown_connections 2024-04-17T19:53:22.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.242+0000 7f0794600640 1 -- 172.21.15.29:0/2621350676 >> 172.21.15.29:0/2621350676 conn(0x7f078c06fde0 msgr2=0x7f078c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:22.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.242+0000 7f0794600640 1 -- 172.21.15.29:0/2621350676 shutdown_connections 2024-04-17T19:53:22.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.242+0000 7f0794600640 1 -- 172.21.15.29:0/2621350676 wait complete. 2024-04-17T19:53:22.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0794600640 1 Processor -- start 2024-04-17T19:53:22.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0794600640 1 -- start start 2024-04-17T19:53:22.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0794600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0774004810 0x7f078c1f3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:22.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0794600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f078c1f6140 0x7f078c15c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:22.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0794600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f078c15c5d0 0x7f078c153720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:22.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0794600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f078c158e70 con 0x7f078c1f6140 2024-04-17T19:53:22.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0794600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f078c0c7580 con 0x7f078c15c5d0 2024-04-17T19:53:22.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0794600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f078c159aa0 con 0x7f0774004810 2024-04-17T19:53:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0791200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f078c1f6140 0x7f078c15c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0791200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f078c1f6140 0x7f078c15c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60644/0 (socket says 172.21.15.29:60644) 2024-04-17T19:53:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0791200640 1 -- 172.21.15.29:0/3942588640 learned_addr learned my addr 172.21.15.29:0/3942588640 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0792600640 1 --2- 172.21.15.29:0/3942588640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f078c15c5d0 0x7f078c153720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0791c00640 1 --2- 172.21.15.29:0/3942588640 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0774004810 0x7f078c1f3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0791200640 1 -- 172.21.15.29:0/3942588640 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0774004810 msgr2=0x7f078c1f3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0791200640 1 --2- 172.21.15.29:0/3942588640 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0774004810 0x7f078c1f3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0791200640 1 -- 172.21.15.29:0/3942588640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f078c15c5d0 msgr2=0x7f078c153720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0791200640 1 --2- 172.21.15.29:0/3942588640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f078c15c5d0 0x7f078c153720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0791200640 1 -- 172.21.15.29:0/3942588640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078c1e2af0 con 0x7f078c1f6140 2024-04-17T19:53:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0792600640 1 --2- 172.21.15.29:0/3942588640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f078c15c5d0 0x7f078c153720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:53:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0791c00640 1 --2- 172.21.15.29:0/3942588640 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0774004810 0x7f078c1f3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:53:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0791200640 1 --2- 172.21.15.29:0/3942588640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f078c1f6140 0x7f078c15c090 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f07780b0040 tx=0x7f0774005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0789600640 1 -- 172.21.15.29:0/3942588640 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0778005bb0 con 0x7f078c1f6140 2024-04-17T19:53:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0789600640 1 -- 172.21.15.29:0/3942588640 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07780bd070 con 0x7f078c1f6140 2024-04-17T19:53:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0789600640 1 -- 172.21.15.29:0/3942588640 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07780a66a0 con 0x7f078c1f6140 2024-04-17T19:53:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0794600640 1 -- 172.21.15.29:0/3942588640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078c1e2db0 con 0x7f078c1f6140 2024-04-17T19:53:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.246+0000 7f0794600640 1 -- 172.21.15.29:0/3942588640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f078c1e3370 con 0x7f078c1f6140 2024-04-17T19:53:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.250+0000 7f0794600640 1 -- 172.21.15.29:0/3942588640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f078c069b90 con 0x7f078c1f6140 2024-04-17T19:53:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.250+0000 7f0789600640 1 -- 172.21.15.29:0/3942588640 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f07780a6840 con 0x7f078c1f6140 2024-04-17T19:53:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.250+0000 7f0789600640 1 --2- 172.21.15.29:0/3942588640 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f076c068380 0x7f076c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.250+0000 7f0789600640 1 -- 172.21.15.29:0/3942588640 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7585+0+0 (secure 0 0 0) 0x7f0778121ab0 con 0x7f078c1f6140 2024-04-17T19:53:22.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.250+0000 7f0791c00640 1 --2- 172.21.15.29:0/3942588640 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f076c068380 0x7f076c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:22.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.250+0000 7f0791c00640 1 --2- 172.21.15.29:0/3942588640 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f076c068380 0x7f076c06a890 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0780000c10 tx=0x7f0780009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:22.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.254+0000 7f0789600640 1 -- 172.21.15.29:0/3942588640 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0778016600 con 0x7f078c1f6140 2024-04-17T19:53:22.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.458+0000 7f0794600640 1 -- 172.21.15.29:0/3942588640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f078c0c9660 con 0x7f078c1f6140 2024-04-17T19:53:22.449 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.458+0000 7f0789600640 1 -- 172.21.15.29:0/3942588640 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+6079 (secure 0 0 0) 0x7f07780ad3d0 con 0x7f078c1f6140 2024-04-17T19:53:22.450 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:22.450 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:18.868672+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:resolve","state_seq":12,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5212,4302,4259]},"id":1}]} 2024-04-17T19:53:22.450 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 33 2024-04-17T19:53:22.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.458+0000 7f0794600640 1 -- 172.21.15.29:0/3942588640 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f076c068380 msgr2=0x7f076c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:22.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.458+0000 7f0794600640 1 --2- 172.21.15.29:0/3942588640 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f076c068380 0x7f076c06a890 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f0780000c10 tx=0x7f0780009040 comp rx=0 tx=0).stop 2024-04-17T19:53:22.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.458+0000 7f0794600640 1 -- 172.21.15.29:0/3942588640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f078c1f6140 msgr2=0x7f078c15c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:22.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.458+0000 7f0794600640 1 --2- 172.21.15.29:0/3942588640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f078c1f6140 0x7f078c15c090 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f07780b0040 tx=0x7f0774005840 comp rx=0 tx=0).stop 2024-04-17T19:53:22.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.458+0000 7f0794600640 1 -- 172.21.15.29:0/3942588640 shutdown_connections 2024-04-17T19:53:22.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.458+0000 7f0794600640 1 --2- 172.21.15.29:0/3942588640 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f076c068380 0x7f076c06a890 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:22.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.458+0000 7f0794600640 1 --2- 172.21.15.29:0/3942588640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f078c15c5d0 0x7f078c153720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:22.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.458+0000 7f0794600640 1 --2- 172.21.15.29:0/3942588640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f078c1f6140 0x7f078c15c090 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:22.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.458+0000 7f0794600640 1 --2- 172.21.15.29:0/3942588640 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0774004810 0x7f078c1f3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:22.452 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.458+0000 7f0794600640 1 -- 172.21.15.29:0/3942588640 >> 172.21.15.29:0/3942588640 conn(0x7f078c06fde0 msgr2=0x7f078c0cb140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:22.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.458+0000 7f0794600640 1 -- 172.21.15.29:0/3942588640 shutdown_connections 2024-04-17T19:53:22.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:22.462+0000 7f0794600640 1 -- 172.21.15.29:0/3942588640 wait complete. 2024-04-17T19:53:22.470 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:53:24.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:53:24.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.646+0000 7f0d98e00640 1 Processor -- start 2024-04-17T19:53:24.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.646+0000 7f0d98e00640 1 -- start start 2024-04-17T19:53:24.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.646+0000 7f0d98e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d940c6f70 0x7f0d940c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:24.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.646+0000 7f0d98e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0d9406c660 0x7f0d9406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:24.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.646+0000 7f0d98e00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d9415a0a0 con 0x7f0d94159c90 2024-04-17T19:53:24.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.646+0000 7f0d98e00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f0d9415abe0 con 0x7f0d9406c660 2024-04-17T19:53:24.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.646+0000 7f0d98e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0d9415a340 con 0x7f0d940c6f70 2024-04-17T19:53:24.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.646+0000 7f0d92800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0d9406c660 0x7f0d9406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:24.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.646+0000 7f0d8be00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d940c6f70 0x7f0d940c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:24.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.646+0000 7f0d91e00640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f0d94159c90 0x7f0d940ca2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:52298/0 (socket says 172.21.15.29:52298) 2024-04-17T19:53:24.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.646+0000 7f0d91e00640 1 -- 172.21.15.29:0/2752104446 learned_addr learned my addr 172.21.15.29:0/2752104446 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:53:24.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.646+0000 7f0d91400640 1 -- 172.21.15.29:0/2752104446 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3921598804 0 0) 0x7f0d9415a0a0 con 0x7f0d94159c90 2024-04-17T19:53:24.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.646+0000 7f0d91400640 1 -- 172.21.15.29:0/2752104446 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d78003530 con 0x7f0d94159c90 2024-04-17T19:53:24.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.646+0000 7f0d8be00640 1 -- 172.21.15.29:0/2752104446 >> v2:172.21.15.37:3300/0 conn(0x7f0d9406c660 msgr2=0x7f0d9406ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:24.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.646+0000 7f0d8be00640 1 --2- 172.21.15.29:0/2752104446 >> v2:172.21.15.37:3300/0 conn(0x7f0d9406c660 0x7f0d9406ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:24.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.646+0000 7f0d8be00640 1 -- 172.21.15.29:0/2752104446 >> v1:172.21.15.37:6789/0 conn(0x7f0d94159c90 legacy=0x7f0d940ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:24.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.646+0000 7f0d8be00640 1 -- 172.21.15.29:0/2752104446 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9415a620 con 0x7f0d940c6f70 2024-04-17T19:53:24.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.646+0000 7f0d92800640 1 --2- 172.21.15.29:0/2752104446 >> v2:172.21.15.37:3300/0 conn(0x7f0d9406c660 0x7f0d9406ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:53:24.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.650+0000 7f0d8be00640 1 --2- 172.21.15.29:0/2752104446 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d940c6f70 0x7f0d940c7380 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0097c0 tx=0x7f0d7c01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ae77ab55ab60ede server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:24.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.650+0000 7f0d91400640 1 -- 172.21.15.29:0/2752104446 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c0b0070 con 0x7f0d940c6f70 2024-04-17T19:53:24.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.650+0000 7f0d91400640 1 -- 172.21.15.29:0/2752104446 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d7c0ab600 con 0x7f0d940c6f70 2024-04-17T19:53:24.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.650+0000 7f0d91400640 1 -- 172.21.15.29:0/2752104446 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7c0ab9b0 con 0x7f0d940c6f70 2024-04-17T19:53:24.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.650+0000 7f0d98e00640 1 -- 172.21.15.29:0/2752104446 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d940c6f70 msgr2=0x7f0d940c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:24.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.650+0000 7f0d98e00640 1 --2- 172.21.15.29:0/2752104446 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d940c6f70 0x7f0d940c7380 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0097c0 tx=0x7f0d7c01c4a0 comp rx=0 tx=0).stop 2024-04-17T19:53:24.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.650+0000 7f0d98e00640 1 -- 172.21.15.29:0/2752104446 shutdown_connections 2024-04-17T19:53:24.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.650+0000 7f0d98e00640 1 --2- 172.21.15.29:0/2752104446 >> v2:172.21.15.37:3300/0 conn(0x7f0d9406c660 0x7f0d9406ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:24.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.650+0000 7f0d98e00640 1 --2- 172.21.15.29:0/2752104446 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d940c6f70 0x7f0d940c7380 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:24.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.650+0000 7f0d98e00640 1 -- 172.21.15.29:0/2752104446 >> 172.21.15.29:0/2752104446 conn(0x7f0d9406fde0 msgr2=0x7f0d940701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:24.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.654+0000 7f0d98e00640 1 -- 172.21.15.29:0/2752104446 shutdown_connections 2024-04-17T19:53:24.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.654+0000 7f0d98e00640 1 -- 172.21.15.29:0/2752104446 wait complete. 2024-04-17T19:53:24.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.654+0000 7f0d98e00640 1 Processor -- start 2024-04-17T19:53:24.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.654+0000 7f0d98e00640 1 -- start start 2024-04-17T19:53:24.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.654+0000 7f0d98e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d9406c660 0x7f0d94162f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:24.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.654+0000 7f0d98e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d940c6f70 0x7f0d9413fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:24.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.654+0000 7f0d98e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d94159c90 0x7f0d94145f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:24.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.654+0000 7f0d98e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0d940ca9c0 con 0x7f0d9406c660 2024-04-17T19:53:24.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.654+0000 7f0d98e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0d9406cfb0 con 0x7f0d94159c90 2024-04-17T19:53:24.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.654+0000 7f0d98e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0d940cab40 con 0x7f0d940c6f70 2024-04-17T19:53:24.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.658+0000 7f0d91e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d9406c660 0x7f0d94162f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:24.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.658+0000 7f0d91e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d9406c660 0x7f0d94162f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60660/0 (socket says 172.21.15.29:60660) 2024-04-17T19:53:24.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.658+0000 7f0d91e00640 1 -- 172.21.15.29:0/3461297259 learned_addr learned my addr 172.21.15.29:0/3461297259 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:24.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.658+0000 7f0d92800640 1 --2- 172.21.15.29:0/3461297259 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d94159c90 0x7f0d94145f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:24.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.658+0000 7f0d8be00640 1 --2- 172.21.15.29:0/3461297259 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d940c6f70 0x7f0d9413fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:24.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.658+0000 7f0d91e00640 1 -- 172.21.15.29:0/3461297259 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d940c6f70 msgr2=0x7f0d9413fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:24.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.658+0000 7f0d91e00640 1 --2- 172.21.15.29:0/3461297259 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d940c6f70 0x7f0d9413fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:24.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.658+0000 7f0d91e00640 1 -- 172.21.15.29:0/3461297259 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d94159c90 msgr2=0x7f0d94145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:24.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.658+0000 7f0d91e00640 1 --2- 172.21.15.29:0/3461297259 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d94159c90 0x7f0d94145f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:24.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.658+0000 7f0d91e00640 1 -- 172.21.15.29:0/3461297259 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d94163470 con 0x7f0d9406c660 2024-04-17T19:53:24.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.658+0000 7f0d91e00640 1 --2- 172.21.15.29:0/3461297259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d9406c660 0x7f0d94162f30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0d8400c390 tx=0x7f0d8400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:24.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.658+0000 7f0d8a000640 1 -- 172.21.15.29:0/3461297259 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d84017070 con 0x7f0d9406c660 2024-04-17T19:53:24.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.658+0000 7f0d8a000640 1 -- 172.21.15.29:0/3461297259 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d84005cf0 con 0x7f0d9406c660 2024-04-17T19:53:24.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.658+0000 7f0d98e00640 1 -- 172.21.15.29:0/3461297259 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d941f5980 con 0x7f0d9406c660 2024-04-17T19:53:24.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.658+0000 7f0d8a000640 1 -- 172.21.15.29:0/3461297259 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d840072d0 con 0x7f0d9406c660 2024-04-17T19:53:24.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.658+0000 7f0d98e00640 1 -- 172.21.15.29:0/3461297259 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d941f5f40 con 0x7f0d9406c660 2024-04-17T19:53:24.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.658+0000 7f0d98e00640 1 -- 172.21.15.29:0/3461297259 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d94069b90 con 0x7f0d9406c660 2024-04-17T19:53:24.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.658+0000 7f0d8a000640 1 -- 172.21.15.29:0/3461297259 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0d8401a070 con 0x7f0d9406c660 2024-04-17T19:53:24.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.662+0000 7f0d8a000640 1 --2- 172.21.15.29:0/3461297259 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0d700683b0 0x7f0d7006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:24.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.662+0000 7f0d8a000640 1 -- 172.21.15.29:0/3461297259 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7497+0+0 (secure 0 0 0) 0x7f0d840862d0 con 0x7f0d9406c660 2024-04-17T19:53:24.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.662+0000 7f0d8be00640 1 --2- 172.21.15.29:0/3461297259 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0d700683b0 0x7f0d7006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:24.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.662+0000 7f0d8be00640 1 --2- 172.21.15.29:0/3461297259 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0d700683b0 0x7f0d7006a8c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0ad000 tx=0x7f0d7c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:24.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.666+0000 7f0d8a000640 1 -- 172.21.15.29:0/3461297259 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0d84055aa0 con 0x7f0d9406c660 2024-04-17T19:53:24.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.870+0000 7f0d98e00640 1 -- 172.21.15.29:0/3461297259 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d94144820 con 0x7f0d9406c660 2024-04-17T19:53:24.863 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.870+0000 7f0d8a000640 1 -- 172.21.15.29:0/3461297259 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6081 (secure 0 0 0) 0x7f0d84059da0 con 0x7f0d9406c660 2024-04-17T19:53:24.863 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:24.863 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:23.943300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4259,4302,5212,4305]},"id":1}]} 2024-04-17T19:53:24.864 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 34 2024-04-17T19:53:24.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.874+0000 7f0d98e00640 1 -- 172.21.15.29:0/3461297259 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0d700683b0 msgr2=0x7f0d7006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:24.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.874+0000 7f0d98e00640 1 --2- 172.21.15.29:0/3461297259 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0d700683b0 0x7f0d7006a8c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c0ad000 tx=0x7f0d7c0a6000 comp rx=0 tx=0).stop 2024-04-17T19:53:24.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.874+0000 7f0d98e00640 1 -- 172.21.15.29:0/3461297259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d9406c660 msgr2=0x7f0d94162f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:24.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.874+0000 7f0d98e00640 1 --2- 172.21.15.29:0/3461297259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d9406c660 0x7f0d94162f30 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0d8400c390 tx=0x7f0d8400c790 comp rx=0 tx=0).stop 2024-04-17T19:53:24.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.874+0000 7f0d98e00640 1 -- 172.21.15.29:0/3461297259 shutdown_connections 2024-04-17T19:53:24.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.874+0000 7f0d98e00640 1 --2- 172.21.15.29:0/3461297259 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0d700683b0 0x7f0d7006a8c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:24.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.874+0000 7f0d98e00640 1 --2- 172.21.15.29:0/3461297259 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0d94159c90 0x7f0d94145f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:24.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.874+0000 7f0d98e00640 1 --2- 172.21.15.29:0/3461297259 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0d940c6f70 0x7f0d9413fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:24.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.874+0000 7f0d98e00640 1 --2- 172.21.15.29:0/3461297259 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d9406c660 0x7f0d94162f30 secure :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0d8400c390 tx=0x7f0d8400c790 comp rx=0 tx=0).stop 2024-04-17T19:53:24.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.874+0000 7f0d98e00640 1 -- 172.21.15.29:0/3461297259 >> 172.21.15.29:0/3461297259 conn(0x7f0d9406fde0 msgr2=0x7f0d940701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:24.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.874+0000 7f0d98e00640 1 -- 172.21.15.29:0/3461297259 shutdown_connections 2024-04-17T19:53:24.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:24.874+0000 7f0d98e00640 1 -- 172.21.15.29:0/3461297259 wait complete. 2024-04-17T19:53:24.890 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:53:26.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:53:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.066+0000 7f2c81000640 1 Processor -- start 2024-04-17T19:53:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.066+0000 7f2c81000640 1 -- start start 2024-04-17T19:53:27.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.070+0000 7f2c81000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c7c0cc8c0 0x7f2c7c0cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:27.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.070+0000 7f2c81000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2c7c159390 0x7f2c7c1597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:27.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.070+0000 7f2c81000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c7c0c88b0 con 0x7f2c7c0ca330 2024-04-17T19:53:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.070+0000 7f2c81000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f2c7c0c6ce0 con 0x7f2c7c159390 2024-04-17T19:53:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.070+0000 7f2c81000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2c7c0c6fc0 con 0x7f2c7c0cc8c0 2024-04-17T19:53:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.070+0000 7f2c73e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2c7c159390 0x7f2c7c1597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.070+0000 7f2c73e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2c7c159390 0x7f2c7c1597a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60672/0 (socket says 172.21.15.29:60672) 2024-04-17T19:53:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.070+0000 7f2c73e00640 1 -- 172.21.15.29:0/2593893863 learned_addr learned my addr 172.21.15.29:0/2593893863 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.070+0000 7f2c79e00640 1 --2- 172.21.15.29:0/2593893863 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c7c0cc8c0 0x7f2c7c0cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.070+0000 7f2c73e00640 1 -- 172.21.15.29:0/2593893863 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c7c0cc8c0 msgr2=0x7f2c7c0cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.070+0000 7f2c73e00640 1 --2- 172.21.15.29:0/2593893863 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c7c0cc8c0 0x7f2c7c0cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.070+0000 7f2c73e00640 1 -- 172.21.15.29:0/2593893863 >> v1:172.21.15.37:6789/0 conn(0x7f2c7c0ca330 legacy=0x7f2c7c15c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.070+0000 7f2c73e00640 1 -- 172.21.15.29:0/2593893863 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c7c0c72a0 con 0x7f2c7c159390 2024-04-17T19:53:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.074+0000 7f2c73e00640 1 --2- 172.21.15.29:0/2593893863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7c159390 0x7f2c7c1597a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009870 tx=0x7f2c6401e770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d849e88d57610fc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.074+0000 7f2c79400640 1 -- 172.21.15.29:0/2593893863 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c640ab640 con 0x7f2c7c159390 2024-04-17T19:53:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.074+0000 7f2c79400640 1 -- 172.21.15.29:0/2593893863 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c640ab7e0 con 0x7f2c7c159390 2024-04-17T19:53:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.074+0000 7f2c79400640 1 -- 172.21.15.29:0/2593893863 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c640abbb0 con 0x7f2c7c159390 2024-04-17T19:53:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.074+0000 7f2c81000640 1 -- 172.21.15.29:0/2593893863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7c159390 msgr2=0x7f2c7c1597a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.074+0000 7f2c81000640 1 --2- 172.21.15.29:0/2593893863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7c159390 0x7f2c7c1597a0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009870 tx=0x7f2c6401e770 comp rx=0 tx=0).stop 2024-04-17T19:53:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.074+0000 7f2c81000640 1 -- 172.21.15.29:0/2593893863 shutdown_connections 2024-04-17T19:53:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.074+0000 7f2c81000640 1 --2- 172.21.15.29:0/2593893863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7c159390 0x7f2c7c1597a0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.074+0000 7f2c81000640 1 --2- 172.21.15.29:0/2593893863 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c7c0cc8c0 0x7f2c7c0cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.074+0000 7f2c81000640 1 -- 172.21.15.29:0/2593893863 >> 172.21.15.29:0/2593893863 conn(0x7f2c7c06fde0 msgr2=0x7f2c7c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.074+0000 7f2c81000640 1 -- 172.21.15.29:0/2593893863 shutdown_connections 2024-04-17T19:53:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.074+0000 7f2c81000640 1 -- 172.21.15.29:0/2593893863 wait complete. 2024-04-17T19:53:27.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.074+0000 7f2c81000640 1 Processor -- start 2024-04-17T19:53:27.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.074+0000 7f2c81000640 1 -- start start 2024-04-17T19:53:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c81000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2c7c0ca330 0x7f2c7c153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c81000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c7c0cc8c0 0x7f2c7c0ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c81000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7c159390 0x7f2c7c13d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c81000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2c7c0c6a00 con 0x7f2c7c159390 2024-04-17T19:53:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c81000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2c7c0c6880 con 0x7f2c7c0ca330 2024-04-17T19:53:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c81000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2c7c06bdc0 con 0x7f2c7c0cc8c0 2024-04-17T19:53:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c7a800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7c159390 0x7f2c7c13d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c7a800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7c159390 0x7f2c7c13d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60678/0 (socket says 172.21.15.29:60678) 2024-04-17T19:53:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c7a800640 1 -- 172.21.15.29:0/3774749142 learned_addr learned my addr 172.21.15.29:0/3774749142 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c79e00640 1 --2- 172.21.15.29:0/3774749142 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2c7c0ca330 0x7f2c7c153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c73e00640 1 --2- 172.21.15.29:0/3774749142 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c7c0cc8c0 0x7f2c7c0ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c7a800640 1 -- 172.21.15.29:0/3774749142 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c7c0cc8c0 msgr2=0x7f2c7c0ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c7a800640 1 --2- 172.21.15.29:0/3774749142 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c7c0cc8c0 0x7f2c7c0ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c7a800640 1 -- 172.21.15.29:0/3774749142 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2c7c0ca330 msgr2=0x7f2c7c153550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c7a800640 1 --2- 172.21.15.29:0/3774749142 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2c7c0ca330 0x7f2c7c153550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c7a800640 1 -- 172.21.15.29:0/3774749142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c7c157e00 con 0x7f2c7c159390 2024-04-17T19:53:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c7a800640 1 --2- 172.21.15.29:0/3774749142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7c159390 0x7f2c7c13d610 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2c7400c760 tx=0x7f2c7400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c72000640 1 -- 172.21.15.29:0/3774749142 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c74003b80 con 0x7f2c7c159390 2024-04-17T19:53:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c72000640 1 -- 172.21.15.29:0/3774749142 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c74003d20 con 0x7f2c7c159390 2024-04-17T19:53:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c72000640 1 -- 172.21.15.29:0/3774749142 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c7400aa30 con 0x7f2c7c159390 2024-04-17T19:53:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c81000640 1 -- 172.21.15.29:0/3774749142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c7c1580e0 con 0x7f2c7c159390 2024-04-17T19:53:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c81000640 1 -- 172.21.15.29:0/3774749142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c7c1586a0 con 0x7f2c7c159390 2024-04-17T19:53:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.078+0000 7f2c81000640 1 -- 172.21.15.29:0/3774749142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c7c069b90 con 0x7f2c7c159390 2024-04-17T19:53:27.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.082+0000 7f2c72000640 1 -- 172.21.15.29:0/3774749142 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2c7400abd0 con 0x7f2c7c159390 2024-04-17T19:53:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.082+0000 7f2c72000640 1 --2- 172.21.15.29:0/3774749142 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2c580683b0 0x7f2c5806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.082+0000 7f2c72000640 1 -- 172.21.15.29:0/3774749142 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7497+0+0 (secure 0 0 0) 0x7f2c74086e40 con 0x7f2c7c159390 2024-04-17T19:53:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.082+0000 7f2c79e00640 1 --2- 172.21.15.29:0/3774749142 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2c580683b0 0x7f2c5806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:27.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.082+0000 7f2c79e00640 1 --2- 172.21.15.29:0/3774749142 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2c580683b0 0x7f2c5806a8c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0079c0 tx=0x7f2c6c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:27.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.086+0000 7f2c72000640 1 -- 172.21.15.29:0/3774749142 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2c740566a0 con 0x7f2c7c159390 2024-04-17T19:53:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.286+0000 7f2c81000640 1 -- 172.21.15.29:0/3774749142 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c7c0c7e10 con 0x7f2c7c159390 2024-04-17T19:53:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.286+0000 7f2c72000640 1 -- 172.21.15.29:0/3774749142 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6081 (secure 0 0 0) 0x7f2c7405a9a0 con 0x7f2c7c159390 2024-04-17T19:53:27.278 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:27.279 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:23.943300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4259,4302,5212,4305]},"id":1}]} 2024-04-17T19:53:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 34 2024-04-17T19:53:27.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.290+0000 7f2c81000640 1 -- 172.21.15.29:0/3774749142 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2c580683b0 msgr2=0x7f2c5806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:27.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.290+0000 7f2c81000640 1 --2- 172.21.15.29:0/3774749142 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2c580683b0 0x7f2c5806a8c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0079c0 tx=0x7f2c6c008040 comp rx=0 tx=0).stop 2024-04-17T19:53:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.290+0000 7f2c81000640 1 -- 172.21.15.29:0/3774749142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7c159390 msgr2=0x7f2c7c13d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.290+0000 7f2c81000640 1 --2- 172.21.15.29:0/3774749142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7c159390 0x7f2c7c13d610 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2c7400c760 tx=0x7f2c7400cc20 comp rx=0 tx=0).stop 2024-04-17T19:53:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.290+0000 7f2c81000640 1 -- 172.21.15.29:0/3774749142 shutdown_connections 2024-04-17T19:53:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.290+0000 7f2c81000640 1 --2- 172.21.15.29:0/3774749142 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2c580683b0 0x7f2c5806a8c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.290+0000 7f2c81000640 1 --2- 172.21.15.29:0/3774749142 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c7c159390 0x7f2c7c13d610 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.290+0000 7f2c81000640 1 --2- 172.21.15.29:0/3774749142 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c7c0cc8c0 0x7f2c7c0ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.290+0000 7f2c81000640 1 --2- 172.21.15.29:0/3774749142 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2c7c0ca330 0x7f2c7c153550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.290+0000 7f2c81000640 1 -- 172.21.15.29:0/3774749142 >> 172.21.15.29:0/3774749142 conn(0x7f2c7c06fde0 msgr2=0x7f2c7c15c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.290+0000 7f2c81000640 1 -- 172.21.15.29:0/3774749142 shutdown_connections 2024-04-17T19:53:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:27.290+0000 7f2c81000640 1 -- 172.21.15.29:0/3774749142 wait complete. 2024-04-17T19:53:27.301 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 34, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-17T19:47:49.315515+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 34, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4259": { "addr": "172.21.15.29:6836/4184160618", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6832", "nonce": 4184160618, "type": "v2" }, { "addr": "172.21.15.29:6836", "nonce": 4184160618, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4259, "incarnation": 14, "join_fscid": -1, "name": "a", "rank": 2, "state": "up:active", "state_seq": 11 }, "gid_4302": { "addr": "172.21.15.29:6835/270402476", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6834", "nonce": 270402476, "type": "v2" }, { "addr": "172.21.15.29:6835", "nonce": 270402476, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 10, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 9 }, "gid_4305": { "addr": "172.21.15.29:6837/2691590158", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6833", "nonce": 2691590158, "type": "v2" }, { "addr": "172.21.15.29:6837", "nonce": 2691590158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4305, "incarnation": 5, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5212": { "addr": "172.21.15.37:6838/767681664", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6835", "nonce": 767681664, "type": "v2" }, { "addr": "172.21.15.37:6838", "nonce": 767681664, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5212, "incarnation": 26, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 31 }, "gid_5350": { "addr": "172.21.15.37:6837/2495953425", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6834", "nonce": 2495953425, "type": "v2" }, { "addr": "172.21.15.37:6837", "nonce": 2495953425, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5350, "incarnation": 32, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:reconnect", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-17T19:53:23.943300+0000", "qdb_cluster": [ 4259, 4302, 5212, 4305 ], "qdb_leader": 4305, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4305, "mds_1": 4302, "mds_2": 4259, "mds_3": 5350, "mds_4": 5212 } } } ], "standbys": [] } 2024-04-17T19:53:29.301 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:53:29.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.474+0000 7fe851200640 1 Processor -- start 2024-04-17T19:53:29.467 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.474+0000 7fe851200640 1 -- start start 2024-04-17T19:53:29.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe851200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe84c159c90 0x7fe84c0ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:29.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe851200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe84c06c660 0x7fe84c06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:29.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe851200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe84c15a0a0 con 0x7fe84c0ca7f0 2024-04-17T19:53:29.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe851200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fe84c15abe0 con 0x7fe84c06c660 2024-04-17T19:53:29.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe851200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe84c15a340 con 0x7fe84c159c90 2024-04-17T19:53:29.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe843e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe84c0ca7f0 0x7fe84c0c6f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57340/0 (socket says 172.21.15.29:57340) 2024-04-17T19:53:29.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe843e00640 1 -- 172.21.15.29:0/2168683798 learned_addr learned my addr 172.21.15.29:0/2168683798 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:53:29.469 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe849e00640 1 --2- 172.21.15.29:0/2168683798 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe84c159c90 0x7fe84c0ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:29.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe84a800640 1 --2- 172.21.15.29:0/2168683798 >> v2:172.21.15.37:3300/0 conn(0x7fe84c06c660 0x7fe84c06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:29.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe849400640 1 -- 172.21.15.29:0/2168683798 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1385050735 0 0) 0x7fe84c15a0a0 con 0x7fe84c0ca7f0 2024-04-17T19:53:29.470 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe849400640 1 -- 172.21.15.29:0/2168683798 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe830003530 con 0x7fe84c0ca7f0 2024-04-17T19:53:29.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe849400640 1 -- 172.21.15.29:0/2168683798 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2085614468 0 0) 0x7fe830003530 con 0x7fe84c0ca7f0 2024-04-17T19:53:29.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe849400640 1 -- 172.21.15.29:0/2168683798 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe84c159c90 msgr2=0x7fe84c0ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:29.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe849400640 1 --2- 172.21.15.29:0/2168683798 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe84c159c90 0x7fe84c0ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:29.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe849400640 1 -- 172.21.15.29:0/2168683798 >> v2:172.21.15.37:3300/0 conn(0x7fe84c06c660 msgr2=0x7fe84c06ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:29.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe849400640 1 --2- 172.21.15.29:0/2168683798 >> v2:172.21.15.37:3300/0 conn(0x7fe84c06c660 0x7fe84c06ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:29.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe849400640 1 -- 172.21.15.29:0/2168683798 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84c15a620 con 0x7fe84c0ca7f0 2024-04-17T19:53:29.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe849400640 1 -- 172.21.15.29:0/2168683798 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fe8340016a0 con 0x7fe84c0ca7f0 2024-04-17T19:53:29.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe849400640 1 -- 172.21.15.29:0/2168683798 >> v1:172.21.15.29:6789/0 conn(0x7fe84c0ca7f0 legacy=0x7fe84c0c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:29.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe849400640 1 --2- 172.21.15.29:0/2168683798 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe830003a30 0x7fe830003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:29.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe849400640 1 --2- 172.21.15.29:0/2168683798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe830004810 0x7fe830004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:29.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe849400640 1 --2- 172.21.15.29:0/2168683798 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe84c159c90 0x7fe830005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:29.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe849400640 1 -- 172.21.15.29:0/2168683798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c15abe0 con 0x7fe830004810 2024-04-17T19:53:29.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe849400640 1 -- 172.21.15.29:0/2168683798 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c15a340 con 0x7fe830003a30 2024-04-17T19:53:29.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe849400640 1 -- 172.21.15.29:0/2168683798 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe83000bab0 con 0x7fe84c159c90 2024-04-17T19:53:29.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe843e00640 1 --2- 172.21.15.29:0/2168683798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe830004810 0x7fe830004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:29.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe849e00640 1 --2- 172.21.15.29:0/2168683798 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe830003a30 0x7fe830003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:29.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe84a800640 1 --2- 172.21.15.29:0/2168683798 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe84c159c90 0x7fe830005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:29.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe843e00640 1 -- 172.21.15.29:0/2168683798 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe84c159c90 msgr2=0x7fe830005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:29.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe843e00640 1 --2- 172.21.15.29:0/2168683798 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe84c159c90 0x7fe830005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:29.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe843e00640 1 -- 172.21.15.29:0/2168683798 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe830003a30 msgr2=0x7fe830003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:29.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe843e00640 1 --2- 172.21.15.29:0/2168683798 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe830003a30 0x7fe830003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:29.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe843e00640 1 -- 172.21.15.29:0/2168683798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe830003530 con 0x7fe830004810 2024-04-17T19:53:29.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.478+0000 7fe84a800640 1 --2- 172.21.15.29:0/2168683798 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe84c159c90 0x7fe830005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T19:53:29.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.482+0000 7fe843e00640 1 --2- 172.21.15.29:0/2168683798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe830004810 0x7fe830004c20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe84c0cb260 tx=0x7fe83401f2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=379263e4796f89da server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:29.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.482+0000 7fe849400640 1 -- 172.21.15.29:0/2168683798 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe834007cb0 con 0x7fe830004810 2024-04-17T19:53:29.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.482+0000 7fe849400640 1 -- 172.21.15.29:0/2168683798 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8340ac020 con 0x7fe830004810 2024-04-17T19:53:29.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.482+0000 7fe849400640 1 -- 172.21.15.29:0/2168683798 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8340058e0 con 0x7fe830004810 2024-04-17T19:53:29.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.482+0000 7fe851200640 1 -- 172.21.15.29:0/2168683798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe830004810 msgr2=0x7fe830004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:29.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.482+0000 7fe851200640 1 --2- 172.21.15.29:0/2168683798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe830004810 0x7fe830004c20 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe84c0cb260 tx=0x7fe83401f2b0 comp rx=0 tx=0).stop 2024-04-17T19:53:29.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.482+0000 7fe84a800640 1 -- 172.21.15.29:0/2168683798 reap_dead start 2024-04-17T19:53:29.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.482+0000 7fe851200640 1 -- 172.21.15.29:0/2168683798 shutdown_connections 2024-04-17T19:53:29.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.482+0000 7fe851200640 1 -- 172.21.15.29:0/2168683798 >> 172.21.15.29:0/2168683798 conn(0x7fe84c06fde0 msgr2=0x7fe84c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:29.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.482+0000 7fe851200640 1 -- 172.21.15.29:0/2168683798 shutdown_connections 2024-04-17T19:53:29.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.482+0000 7fe851200640 1 -- 172.21.15.29:0/2168683798 wait complete. 2024-04-17T19:53:29.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe851200640 1 Processor -- start 2024-04-17T19:53:29.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe851200640 1 -- start start 2024-04-17T19:53:29.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe851200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe830004810 0x7fe84c1566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:29.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe851200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe84c1f3c90 0x7fe84c14e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:29.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe851200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe84c1e30c0 0x7fe84c1e34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe851200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c06c140 con 0x7fe84c1f3c90 2024-04-17T19:53:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe851200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c06be60 con 0x7fe830004810 2024-04-17T19:53:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe851200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe84c06cfb0 con 0x7fe84c1e30c0 2024-04-17T19:53:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe843e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe84c1f3c90 0x7fe84c14e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe843e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe84c1f3c90 0x7fe84c14e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55064/0 (socket says 172.21.15.29:55064) 2024-04-17T19:53:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe843e00640 1 -- 172.21.15.29:0/1369452687 learned_addr learned my addr 172.21.15.29:0/1369452687 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe849e00640 1 --2- 172.21.15.29:0/1369452687 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe830004810 0x7fe84c1566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe84a800640 1 --2- 172.21.15.29:0/1369452687 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe84c1e30c0 0x7fe84c1e34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe843e00640 1 -- 172.21.15.29:0/1369452687 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe84c1e30c0 msgr2=0x7fe84c1e34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe843e00640 1 --2- 172.21.15.29:0/1369452687 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe84c1e30c0 0x7fe84c1e34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe843e00640 1 -- 172.21.15.29:0/1369452687 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe830004810 msgr2=0x7fe84c1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe843e00640 1 --2- 172.21.15.29:0/1369452687 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe830004810 0x7fe84c1566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe843e00640 1 -- 172.21.15.29:0/1369452687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84c1f1040 con 0x7fe84c1f3c90 2024-04-17T19:53:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe843e00640 1 --2- 172.21.15.29:0/1369452687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe84c1f3c90 0x7fe84c14e050 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe834007040 tx=0x7fe830005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe842000640 1 -- 172.21.15.29:0/1369452687 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8340049f0 con 0x7fe84c1f3c90 2024-04-17T19:53:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe842000640 1 -- 172.21.15.29:0/1369452687 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8340b6030 con 0x7fe84c1f3c90 2024-04-17T19:53:29.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe851200640 1 -- 172.21.15.29:0/1369452687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe84c0c8b00 con 0x7fe84c1f3c90 2024-04-17T19:53:29.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe842000640 1 -- 172.21.15.29:0/1369452687 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe834004df0 con 0x7fe84c1f3c90 2024-04-17T19:53:29.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.486+0000 7fe851200640 1 -- 172.21.15.29:0/1369452687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe84c157260 con 0x7fe84c1f3c90 2024-04-17T19:53:29.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.490+0000 7fe851200640 1 -- 172.21.15.29:0/1369452687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe84c069b90 con 0x7fe84c1f3c90 2024-04-17T19:53:29.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.490+0000 7fe842000640 1 -- 172.21.15.29:0/1369452687 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe834007070 con 0x7fe84c1f3c90 2024-04-17T19:53:29.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.490+0000 7fe842000640 1 --2- 172.21.15.29:0/1369452687 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe828068380 0x7fe82806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:29.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.490+0000 7fe842000640 1 -- 172.21.15.29:0/1369452687 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7497+0+0 (secure 0 0 0) 0x7fe834123b20 con 0x7fe84c1f3c90 2024-04-17T19:53:29.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.490+0000 7fe849e00640 1 --2- 172.21.15.29:0/1369452687 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe828068380 0x7fe82806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:29.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.494+0000 7fe849e00640 1 --2- 172.21.15.29:0/1369452687 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe828068380 0x7fe82806a890 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe83c0090e0 tx=0x7fe83c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:29.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.494+0000 7fe842000640 1 -- 172.21.15.29:0/1369452687 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe834018fc0 con 0x7fe84c1f3c90 2024-04-17T19:53:29.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.698+0000 7fe851200640 1 -- 172.21.15.29:0/1369452687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe84c15e120 con 0x7fe84c1f3c90 2024-04-17T19:53:29.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.698+0000 7fe842000640 1 -- 172.21.15.29:0/1369452687 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+6081 (secure 0 0 0) 0x7fe8340f3380 con 0x7fe84c1f3c90 2024-04-17T19:53:29.689 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:29.689 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:23.943300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:reconnect","state_seq":14,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4259,4302,5212,4305]},"id":1}]} 2024-04-17T19:53:29.689 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 34 2024-04-17T19:53:29.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.698+0000 7fe851200640 1 -- 172.21.15.29:0/1369452687 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe828068380 msgr2=0x7fe82806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:29.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.698+0000 7fe851200640 1 --2- 172.21.15.29:0/1369452687 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe828068380 0x7fe82806a890 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe83c0090e0 tx=0x7fe83c008040 comp rx=0 tx=0).stop 2024-04-17T19:53:29.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.698+0000 7fe851200640 1 -- 172.21.15.29:0/1369452687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe84c1f3c90 msgr2=0x7fe84c14e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:29.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.698+0000 7fe851200640 1 --2- 172.21.15.29:0/1369452687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe84c1f3c90 0x7fe84c14e050 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe834007040 tx=0x7fe830005840 comp rx=0 tx=0).stop 2024-04-17T19:53:29.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.698+0000 7fe851200640 1 -- 172.21.15.29:0/1369452687 shutdown_connections 2024-04-17T19:53:29.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.698+0000 7fe851200640 1 --2- 172.21.15.29:0/1369452687 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe828068380 0x7fe82806a890 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:29.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.698+0000 7fe851200640 1 --2- 172.21.15.29:0/1369452687 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe84c1e30c0 0x7fe84c1e34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:29.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.698+0000 7fe851200640 1 --2- 172.21.15.29:0/1369452687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe84c1f3c90 0x7fe84c14e050 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:29.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.698+0000 7fe851200640 1 --2- 172.21.15.29:0/1369452687 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe830004810 0x7fe84c1566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:29.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.698+0000 7fe851200640 1 -- 172.21.15.29:0/1369452687 >> 172.21.15.29:0/1369452687 conn(0x7fe84c06fde0 msgr2=0x7fe84c072320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:29.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.698+0000 7fe851200640 1 -- 172.21.15.29:0/1369452687 shutdown_connections 2024-04-17T19:53:29.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:29.698+0000 7fe851200640 1 -- 172.21.15.29:0/1369452687 wait complete. 2024-04-17T19:53:29.710 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:53:31.712 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:53:31.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.886+0000 7f153ca00640 1 Processor -- start 2024-04-17T19:53:31.877 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.886+0000 7f153ca00640 1 -- start start 2024-04-17T19:53:31.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.886+0000 7f153ca00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f15380c7f50 0x7f15380ca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:31.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.886+0000 7f153ca00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15380cbba0 con 0x7f15380cbf90 2024-04-17T19:53:31.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.886+0000 7f153ca00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f153806c010 con 0x7f1538155da0 2024-04-17T19:53:31.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.886+0000 7f153ca00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f153806bdc0 con 0x7f15380c7f50 2024-04-17T19:53:31.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.886+0000 7f152d400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f15380c7f50 0x7f15380ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:31.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.886+0000 7f1535e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f15380cbf90 0x7f153815ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57342/0 (socket says 172.21.15.29:57342) 2024-04-17T19:53:31.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.886+0000 7f1535e00640 1 -- 172.21.15.29:0/2413782215 learned_addr learned my addr 172.21.15.29:0/2413782215 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:53:31.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.886+0000 7f152d400640 1 --2- 172.21.15.29:0/2413782215 >> v2:172.21.15.29:3300/0 conn(0x7f15380c7f50 0x7f15380ca410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55080/0 (socket says 172.21.15.29:55080) 2024-04-17T19:53:31.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.890+0000 7f152d400640 1 -- 172.21.15.29:0/2413782215 >> v1:172.21.15.37:6789/0 conn(0x7f1538155da0 legacy=0x7f15381561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:31.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.890+0000 7f152d400640 1 -- 172.21.15.29:0/2413782215 >> v1:172.21.15.29:6789/0 conn(0x7f15380cbf90 legacy=0x7f153815ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:31.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.890+0000 7f152d400640 1 -- 172.21.15.29:0/2413782215 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153806c2f0 con 0x7f15380c7f50 2024-04-17T19:53:31.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.890+0000 7f152d400640 1 --2- 172.21.15.29:0/2413782215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15380c7f50 0x7f15380ca410 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f15200097c0 tx=0x7f152001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b2f04d9cb140716 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:31.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.890+0000 7f1535400640 1 -- 172.21.15.29:0/2413782215 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15200b1070 con 0x7f15380c7f50 2024-04-17T19:53:31.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.890+0000 7f1535400640 1 -- 172.21.15.29:0/2413782215 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15200ac410 con 0x7f15380c7f50 2024-04-17T19:53:31.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.890+0000 7f1535400640 1 -- 172.21.15.29:0/2413782215 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15200ac7e0 con 0x7f15380c7f50 2024-04-17T19:53:31.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.890+0000 7f153ca00640 1 -- 172.21.15.29:0/2413782215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15380c7f50 msgr2=0x7f15380ca410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:31.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.890+0000 7f153ca00640 1 --2- 172.21.15.29:0/2413782215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15380c7f50 0x7f15380ca410 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f15200097c0 tx=0x7f152001c4a0 comp rx=0 tx=0).stop 2024-04-17T19:53:31.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.890+0000 7f153ca00640 1 -- 172.21.15.29:0/2413782215 shutdown_connections 2024-04-17T19:53:31.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.890+0000 7f153ca00640 1 --2- 172.21.15.29:0/2413782215 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15380c7f50 0x7f15380ca410 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:31.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.890+0000 7f153ca00640 1 -- 172.21.15.29:0/2413782215 >> 172.21.15.29:0/2413782215 conn(0x7f153806fde0 msgr2=0x7f15380701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:31.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.894+0000 7f153ca00640 1 -- 172.21.15.29:0/2413782215 shutdown_connections 2024-04-17T19:53:31.884 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.894+0000 7f153ca00640 1 -- 172.21.15.29:0/2413782215 wait complete. 2024-04-17T19:53:31.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.894+0000 7f153ca00640 1 Processor -- start 2024-04-17T19:53:31.886 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.894+0000 7f153ca00640 1 -- start start 2024-04-17T19:53:31.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.894+0000 7f153ca00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15380c7f50 0x7f15381f6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:31.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.894+0000 7f153ca00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15380cbf90 0x7f15381f96d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:31.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f153ca00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1538155da0 0x7f1538146c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:31.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f153ca00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f15380cb8e0 con 0x7f15380cbf90 2024-04-17T19:53:31.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f153ca00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f15380cb760 con 0x7f15380c7f50 2024-04-17T19:53:31.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f153ca00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f153816a5a0 con 0x7f1538155da0 2024-04-17T19:53:31.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f152d400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15380cbf90 0x7f15381f96d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:31.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f152d400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15380cbf90 0x7f15381f96d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55090/0 (socket says 172.21.15.29:55090) 2024-04-17T19:53:31.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f152d400640 1 -- 172.21.15.29:0/3869497059 learned_addr learned my addr 172.21.15.29:0/3869497059 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:31.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f1535e00640 1 --2- 172.21.15.29:0/3869497059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15380c7f50 0x7f15381f6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:31.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f1536800640 1 --2- 172.21.15.29:0/3869497059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1538155da0 0x7f1538146c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:31.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f152d400640 1 -- 172.21.15.29:0/3869497059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1538155da0 msgr2=0x7f1538146c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:31.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f152d400640 1 --2- 172.21.15.29:0/3869497059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1538155da0 0x7f1538146c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:31.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f152d400640 1 -- 172.21.15.29:0/3869497059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15380c7f50 msgr2=0x7f15381f6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:31.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f152d400640 1 --2- 172.21.15.29:0/3869497059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15380c7f50 0x7f15381f6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:31.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f152d400640 1 -- 172.21.15.29:0/3869497059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15381f9c10 con 0x7f15380cbf90 2024-04-17T19:53:31.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f152d400640 1 --2- 172.21.15.29:0/3869497059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15380cbf90 0x7f15381f96d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f15200190b0 tx=0x7f1520019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:31.890 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f152ea00640 1 -- 172.21.15.29:0/3869497059 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15200b1040 con 0x7f15380cbf90 2024-04-17T19:53:31.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f152ea00640 1 -- 172.21.15.29:0/3869497059 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15200ba070 con 0x7f15380cbf90 2024-04-17T19:53:31.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f152ea00640 1 -- 172.21.15.29:0/3869497059 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15200b5650 con 0x7f15380cbf90 2024-04-17T19:53:31.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f153ca00640 1 -- 172.21.15.29:0/3869497059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15380732f0 con 0x7f15380cbf90 2024-04-17T19:53:31.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f153ca00640 1 -- 172.21.15.29:0/3869497059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15381f58b0 con 0x7f15380cbf90 2024-04-17T19:53:31.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f153ca00640 1 -- 172.21.15.29:0/3869497059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15381497e0 con 0x7f15380cbf90 2024-04-17T19:53:31.891 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.898+0000 7f152ea00640 1 -- 172.21.15.29:0/3869497059 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f15200b57f0 con 0x7f15380cbf90 2024-04-17T19:53:31.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.902+0000 7f152ea00640 1 --2- 172.21.15.29:0/3869497059 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f15140683b0 0x7f151406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:31.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.902+0000 7f152ea00640 1 -- 172.21.15.29:0/3869497059 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7497+0+0 (secure 0 0 0) 0x7f1520120eb0 con 0x7f15380cbf90 2024-04-17T19:53:31.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.902+0000 7f1535e00640 1 --2- 172.21.15.29:0/3869497059 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f15140683b0 0x7f151406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:31.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.902+0000 7f1535e00640 1 --2- 172.21.15.29:0/3869497059 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f15140683b0 0x7f151406a8c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1528005890 tx=0x7f1528009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:31.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:31.906+0000 7f152ea00640 1 -- 172.21.15.29:0/3869497059 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1520016090 con 0x7f15380cbf90 2024-04-17T19:53:32.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:32.110+0000 7f153ca00640 1 -- 172.21.15.29:0/3869497059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1538069b90 con 0x7f15380cbf90 2024-04-17T19:53:32.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:32.110+0000 7f152ea00640 1 -- 172.21.15.29:0/3869497059 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6078 (secure 0 0 0) 0x7f15200f0c40 con 0x7f15380cbf90 2024-04-17T19:53:32.101 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:32.101 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:29.872505+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5212,4302,4259]},"id":1}]} 2024-04-17T19:53:32.102 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 35 2024-04-17T19:53:32.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:32.110+0000 7f153ca00640 1 -- 172.21.15.29:0/3869497059 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f15140683b0 msgr2=0x7f151406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:32.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:32.110+0000 7f153ca00640 1 --2- 172.21.15.29:0/3869497059 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f15140683b0 0x7f151406a8c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1528005890 tx=0x7f1528009040 comp rx=0 tx=0).stop 2024-04-17T19:53:32.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:32.110+0000 7f153ca00640 1 -- 172.21.15.29:0/3869497059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15380cbf90 msgr2=0x7f15381f96d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:32.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:32.110+0000 7f153ca00640 1 --2- 172.21.15.29:0/3869497059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15380cbf90 0x7f15381f96d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f15200190b0 tx=0x7f1520019240 comp rx=0 tx=0).stop 2024-04-17T19:53:32.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:32.110+0000 7f153ca00640 1 -- 172.21.15.29:0/3869497059 shutdown_connections 2024-04-17T19:53:32.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:32.110+0000 7f153ca00640 1 --2- 172.21.15.29:0/3869497059 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f15140683b0 0x7f151406a8c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:32.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:32.110+0000 7f153ca00640 1 --2- 172.21.15.29:0/3869497059 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1538155da0 0x7f1538146c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:32.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:32.110+0000 7f153ca00640 1 --2- 172.21.15.29:0/3869497059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15380cbf90 0x7f15381f96d0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:32.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:32.110+0000 7f153ca00640 1 --2- 172.21.15.29:0/3869497059 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15380c7f50 0x7f15381f6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:32.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:32.110+0000 7f153ca00640 1 -- 172.21.15.29:0/3869497059 >> 172.21.15.29:0/3869497059 conn(0x7f153806fde0 msgr2=0x7f1538154d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:32.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:32.110+0000 7f153ca00640 1 -- 172.21.15.29:0/3869497059 shutdown_connections 2024-04-17T19:53:32.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:32.110+0000 7f153ca00640 1 -- 172.21.15.29:0/3869497059 wait complete. 2024-04-17T19:53:32.123 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:53:34.125 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:53:34.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.298+0000 7fdcb3000640 1 Processor -- start 2024-04-17T19:53:34.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.298+0000 7fdcb3000640 1 -- start start 2024-04-17T19:53:34.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.302+0000 7fdcb3000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcac0cbf90 0x7fdcac15ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:34.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.302+0000 7fdcb3000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdcac0c7f50 0x7fdcac0ca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:34.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.302+0000 7fdcb3000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcac15cf70 con 0x7fdcac155da0 2024-04-17T19:53:34.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.302+0000 7fdcb3000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fdcac06bdc0 con 0x7fdcac0c7f50 2024-04-17T19:53:34.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.302+0000 7fdcb3000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fdcac0cbba0 con 0x7fdcac0cbf90 2024-04-17T19:53:34.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.302+0000 7fdca3e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdcac0c7f50 0x7fdcac0ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:34.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.302+0000 7fdca3e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdcac0c7f50 0x7fdcac0ca410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55096/0 (socket says 172.21.15.29:55096) 2024-04-17T19:53:34.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.302+0000 7fdca3e00640 1 -- 172.21.15.29:0/979471066 learned_addr learned my addr 172.21.15.29:0/979471066 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:34.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.302+0000 7fdcabe00640 1 --2- 172.21.15.29:0/979471066 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcac0cbf90 0x7fdcac15ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:34.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.302+0000 7fdca3e00640 1 -- 172.21.15.29:0/979471066 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcac0cbf90 msgr2=0x7fdcac15ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:34.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.302+0000 7fdca3e00640 1 --2- 172.21.15.29:0/979471066 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcac0cbf90 0x7fdcac15ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:34.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.302+0000 7fdca3e00640 1 -- 172.21.15.29:0/979471066 >> v1:172.21.15.37:6789/0 conn(0x7fdcac155da0 legacy=0x7fdcac1561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:34.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.302+0000 7fdca3e00640 1 -- 172.21.15.29:0/979471066 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcac06c010 con 0x7fdcac0c7f50 2024-04-17T19:53:34.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.306+0000 7fdca3e00640 1 --2- 172.21.15.29:0/979471066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcac0c7f50 0x7fdcac0ca410 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fdc94009870 tx=0x7fdc9401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b915e095b256fc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:34.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.306+0000 7fdcab400640 1 -- 172.21.15.29:0/979471066 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc940ab650 con 0x7fdcac0c7f50 2024-04-17T19:53:34.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.306+0000 7fdcab400640 1 -- 172.21.15.29:0/979471066 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc940b0070 con 0x7fdcac0c7f50 2024-04-17T19:53:34.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.306+0000 7fdcab400640 1 -- 172.21.15.29:0/979471066 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc940aba10 con 0x7fdcac0c7f50 2024-04-17T19:53:34.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.306+0000 7fdcb3000640 1 -- 172.21.15.29:0/979471066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcac0c7f50 msgr2=0x7fdcac0ca410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:34.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.306+0000 7fdcb3000640 1 --2- 172.21.15.29:0/979471066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcac0c7f50 0x7fdcac0ca410 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fdc94009870 tx=0x7fdc9401c560 comp rx=0 tx=0).stop 2024-04-17T19:53:34.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.306+0000 7fdcb3000640 1 -- 172.21.15.29:0/979471066 shutdown_connections 2024-04-17T19:53:34.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.306+0000 7fdcb3000640 1 --2- 172.21.15.29:0/979471066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcac0c7f50 0x7fdcac0ca410 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:34.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.306+0000 7fdcb3000640 1 --2- 172.21.15.29:0/979471066 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcac0cbf90 0x7fdcac15ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:34.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.306+0000 7fdcb3000640 1 -- 172.21.15.29:0/979471066 >> 172.21.15.29:0/979471066 conn(0x7fdcac06fde0 msgr2=0x7fdcac0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:34.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.306+0000 7fdcb3000640 1 -- 172.21.15.29:0/979471066 shutdown_connections 2024-04-17T19:53:34.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.306+0000 7fdcb3000640 1 -- 172.21.15.29:0/979471066 wait complete. 2024-04-17T19:53:34.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdcb3000640 1 Processor -- start 2024-04-17T19:53:34.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdcb3000640 1 -- start start 2024-04-17T19:53:34.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdcb3000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcac0c7f50 0x7fdcac1f8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:34.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdcb3000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcac0cbf90 0x7fdcac1588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:34.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdcb3000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcac155da0 0x7fdcac146040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:34.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdcb3000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fdcac0caad0 con 0x7fdcac0cbf90 2024-04-17T19:53:34.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdcb3000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fdcac0ca950 con 0x7fdcac0c7f50 2024-04-17T19:53:34.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdcb3000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fdcac0cb760 con 0x7fdcac155da0 2024-04-17T19:53:34.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdca3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcac0cbf90 0x7fdcac1588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:34.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdca3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcac0cbf90 0x7fdcac1588b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55104/0 (socket says 172.21.15.29:55104) 2024-04-17T19:53:34.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdca3e00640 1 -- 172.21.15.29:0/249978702 learned_addr learned my addr 172.21.15.29:0/249978702 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:34.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdcabe00640 1 --2- 172.21.15.29:0/249978702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcac0c7f50 0x7fdcac1f8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:34.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdcb1000640 1 --2- 172.21.15.29:0/249978702 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcac155da0 0x7fdcac146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:34.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdca3e00640 1 -- 172.21.15.29:0/249978702 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcac155da0 msgr2=0x7fdcac146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:34.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdca3e00640 1 --2- 172.21.15.29:0/249978702 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcac155da0 0x7fdcac146040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:34.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdca3e00640 1 -- 172.21.15.29:0/249978702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcac0c7f50 msgr2=0x7fdcac1f8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:34.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdca3e00640 1 --2- 172.21.15.29:0/249978702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcac0c7f50 0x7fdcac1f8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:34.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdca3e00640 1 -- 172.21.15.29:0/249978702 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcac158df0 con 0x7fdcac0cbf90 2024-04-17T19:53:34.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdca3e00640 1 --2- 172.21.15.29:0/249978702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcac0cbf90 0x7fdcac1588b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fdc9401ac10 tx=0x7fdc9401ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:34.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdca8c00640 1 -- 172.21.15.29:0/249978702 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc940be5f0 con 0x7fdcac0cbf90 2024-04-17T19:53:34.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdca8c00640 1 -- 172.21.15.29:0/249978702 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc940be7b0 con 0x7fdcac0cbf90 2024-04-17T19:53:34.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdca8c00640 1 -- 172.21.15.29:0/249978702 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc940beba0 con 0x7fdcac0cbf90 2024-04-17T19:53:34.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdcb3000640 1 -- 172.21.15.29:0/249978702 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcac15d480 con 0x7fdcac0cbf90 2024-04-17T19:53:34.305 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdcb3000640 1 -- 172.21.15.29:0/249978702 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcac157dd0 con 0x7fdcac0cbf90 2024-04-17T19:53:34.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.310+0000 7fdcb3000640 1 -- 172.21.15.29:0/249978702 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcac069b90 con 0x7fdcac0cbf90 2024-04-17T19:53:34.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.314+0000 7fdca8c00640 1 -- 172.21.15.29:0/249978702 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fdc940b43f0 con 0x7fdcac0cbf90 2024-04-17T19:53:34.306 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.314+0000 7fdca8c00640 1 --2- 172.21.15.29:0/249978702 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fdc880683b0 0x7fdc8806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:34.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.314+0000 7fdca8c00640 1 -- 172.21.15.29:0/249978702 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7497+0+0 (secure 0 0 0) 0x7fdc941213d0 con 0x7fdcac0cbf90 2024-04-17T19:53:34.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.314+0000 7fdcabe00640 1 --2- 172.21.15.29:0/249978702 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fdc880683b0 0x7fdc8806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:34.307 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.314+0000 7fdcabe00640 1 --2- 172.21.15.29:0/249978702 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fdc880683b0 0x7fdc8806a8c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c009a50 tx=0x7fdc9c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:34.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.318+0000 7fdca8c00640 1 -- 172.21.15.29:0/249978702 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fdc94016150 con 0x7fdcac0cbf90 2024-04-17T19:53:34.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.518+0000 7fdcb3000640 1 -- 172.21.15.29:0/249978702 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdcac0c65a0 con 0x7fdcac0cbf90 2024-04-17T19:53:34.511 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.518+0000 7fdca8c00640 1 -- 172.21.15.29:0/249978702 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+6078 (secure 0 0 0) 0x7fdc940f0b10 con 0x7fdcac0cbf90 2024-04-17T19:53:34.511 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:34.512 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:29.872505+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5212,4302,4259]},"id":1}]} 2024-04-17T19:53:34.512 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 35 2024-04-17T19:53:34.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.522+0000 7fdcb3000640 1 -- 172.21.15.29:0/249978702 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fdc880683b0 msgr2=0x7fdc8806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:34.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.522+0000 7fdcb3000640 1 --2- 172.21.15.29:0/249978702 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fdc880683b0 0x7fdc8806a8c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c009a50 tx=0x7fdc9c008040 comp rx=0 tx=0).stop 2024-04-17T19:53:34.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.522+0000 7fdcb3000640 1 -- 172.21.15.29:0/249978702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcac0cbf90 msgr2=0x7fdcac1588b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:34.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.522+0000 7fdcb3000640 1 --2- 172.21.15.29:0/249978702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcac0cbf90 0x7fdcac1588b0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fdc9401ac10 tx=0x7fdc9401ada0 comp rx=0 tx=0).stop 2024-04-17T19:53:34.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.522+0000 7fdcb3000640 1 -- 172.21.15.29:0/249978702 shutdown_connections 2024-04-17T19:53:34.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.522+0000 7fdcb3000640 1 --2- 172.21.15.29:0/249978702 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fdc880683b0 0x7fdc8806a8c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:34.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.522+0000 7fdcb3000640 1 --2- 172.21.15.29:0/249978702 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fdcac155da0 0x7fdcac146040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:34.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.522+0000 7fdcb3000640 1 --2- 172.21.15.29:0/249978702 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdcac0cbf90 0x7fdcac1588b0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:34.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.522+0000 7fdcb3000640 1 --2- 172.21.15.29:0/249978702 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fdcac0c7f50 0x7fdcac1f8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:34.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.522+0000 7fdcb3000640 1 -- 172.21.15.29:0/249978702 >> 172.21.15.29:0/249978702 conn(0x7fdcac06fde0 msgr2=0x7fdcac0c8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:34.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.522+0000 7fdcb3000640 1 -- 172.21.15.29:0/249978702 shutdown_connections 2024-04-17T19:53:34.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:34.522+0000 7fdcb3000640 1 -- 172.21.15.29:0/249978702 wait complete. 2024-04-17T19:53:34.533 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:53:36.535 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:53:36.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.694+0000 7f04c0600640 1 Processor -- start 2024-04-17T19:53:36.688 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.694+0000 7f04c0600640 1 -- start start 2024-04-17T19:53:36.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.698+0000 7f04c0600640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f04b80cc8c0 0x7f04b80cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:36.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.698+0000 7f04c0600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f04b81597a0 0x7f04b81635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:36.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.698+0000 7f04c0600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04b80c88b0 con 0x7f04b8159390 2024-04-17T19:53:36.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.698+0000 7f04c0600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f04b80c6ce0 con 0x7f04b81597a0 2024-04-17T19:53:36.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.698+0000 7f04c0600640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f04b80c6fc0 con 0x7f04b80cc8c0 2024-04-17T19:53:36.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.698+0000 7f04bd200640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f04b8159390 0x7f04b80ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57358/0 (socket says 172.21.15.29:57358) 2024-04-17T19:53:36.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.698+0000 7f04bd200640 1 -- 172.21.15.29:0/481118430 learned_addr learned my addr 172.21.15.29:0/481118430 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:53:36.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.698+0000 7f04bdc00640 1 --2- 172.21.15.29:0/481118430 >> v2:172.21.15.37:3300/0 conn(0x7f04b80cc8c0 0x7f04b80cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:36.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.698+0000 7f04be600640 1 --2- 172.21.15.29:0/481118430 >> v2:172.21.15.29:3300/0 conn(0x7f04b81597a0 0x7f04b81635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:36.691 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.698+0000 7f04b7e00640 1 -- 172.21.15.29:0/481118430 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1110118700 0 0) 0x7f04b80c88b0 con 0x7f04b8159390 2024-04-17T19:53:36.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.698+0000 7f04b7e00640 1 -- 172.21.15.29:0/481118430 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04a0003530 con 0x7f04b8159390 2024-04-17T19:53:36.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.698+0000 7f04be600640 1 -- 172.21.15.29:0/481118430 >> v2:172.21.15.37:3300/0 conn(0x7f04b80cc8c0 msgr2=0x7f04b80cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:36.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.698+0000 7f04be600640 1 --2- 172.21.15.29:0/481118430 >> v2:172.21.15.37:3300/0 conn(0x7f04b80cc8c0 0x7f04b80cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:36.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.698+0000 7f04be600640 1 -- 172.21.15.29:0/481118430 >> v1:172.21.15.29:6789/0 conn(0x7f04b8159390 legacy=0x7f04b80ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:36.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.698+0000 7f04be600640 1 -- 172.21.15.29:0/481118430 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b80c72a0 con 0x7f04b81597a0 2024-04-17T19:53:36.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.702+0000 7f04be600640 1 --2- 172.21.15.29:0/481118430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b81597a0 0x7f04b81635e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f04b80721b0 tx=0x7f04b001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e77912d5dda9025 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:36.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.702+0000 7f04b7e00640 1 -- 172.21.15.29:0/481118430 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b00b2070 con 0x7f04b81597a0 2024-04-17T19:53:36.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.702+0000 7f04b7e00640 1 -- 172.21.15.29:0/481118430 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b00a57f0 con 0x7f04b81597a0 2024-04-17T19:53:36.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.702+0000 7f04b7e00640 1 -- 172.21.15.29:0/481118430 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b00a5bc0 con 0x7f04b81597a0 2024-04-17T19:53:36.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.702+0000 7f04c0600640 1 -- 172.21.15.29:0/481118430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b81597a0 msgr2=0x7f04b81635e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:36.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.702+0000 7f04c0600640 1 --2- 172.21.15.29:0/481118430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b81597a0 0x7f04b81635e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f04b80721b0 tx=0x7f04b001c930 comp rx=0 tx=0).stop 2024-04-17T19:53:36.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.702+0000 7f04c0600640 1 -- 172.21.15.29:0/481118430 shutdown_connections 2024-04-17T19:53:36.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.702+0000 7f04c0600640 1 --2- 172.21.15.29:0/481118430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b81597a0 0x7f04b81635e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:36.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.702+0000 7f04c0600640 1 --2- 172.21.15.29:0/481118430 >> v2:172.21.15.37:3300/0 conn(0x7f04b80cc8c0 0x7f04b80cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:36.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.702+0000 7f04c0600640 1 -- 172.21.15.29:0/481118430 >> 172.21.15.29:0/481118430 conn(0x7f04b806fde0 msgr2=0x7f04b80701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:36.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.702+0000 7f04c0600640 1 -- 172.21.15.29:0/481118430 shutdown_connections 2024-04-17T19:53:36.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.702+0000 7f04c0600640 1 -- 172.21.15.29:0/481118430 wait complete. 2024-04-17T19:53:36.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04c0600640 1 Processor -- start 2024-04-17T19:53:36.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04c0600640 1 -- start start 2024-04-17T19:53:36.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04c0600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04b80cc8c0 0x7f04b815c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:36.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04c0600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b8159390 0x7f04b813ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:36.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04c0600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04b81597a0 0x7f04b8139220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:36.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04c0600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f04b80cabc0 con 0x7f04b8159390 2024-04-17T19:53:36.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04c0600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f04b80caa40 con 0x7f04b81597a0 2024-04-17T19:53:36.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04c0600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f04b80c6880 con 0x7f04b80cc8c0 2024-04-17T19:53:36.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04bd200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b8159390 0x7f04b813ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:36.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04bd200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b8159390 0x7f04b813ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55130/0 (socket says 172.21.15.29:55130) 2024-04-17T19:53:36.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04bd200640 1 -- 172.21.15.29:0/1103099306 learned_addr learned my addr 172.21.15.29:0/1103099306 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:36.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04be600640 1 --2- 172.21.15.29:0/1103099306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04b81597a0 0x7f04b8139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:36.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04bdc00640 1 --2- 172.21.15.29:0/1103099306 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04b80cc8c0 0x7f04b815c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:36.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04bd200640 1 -- 172.21.15.29:0/1103099306 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04b80cc8c0 msgr2=0x7f04b815c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:36.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04bd200640 1 --2- 172.21.15.29:0/1103099306 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04b80cc8c0 0x7f04b815c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:36.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04bd200640 1 -- 172.21.15.29:0/1103099306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04b81597a0 msgr2=0x7f04b8139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:36.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04bd200640 1 --2- 172.21.15.29:0/1103099306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04b81597a0 0x7f04b8139220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:36.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04bd200640 1 -- 172.21.15.29:0/1103099306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b815c7e0 con 0x7f04b8159390 2024-04-17T19:53:36.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04bd200640 1 --2- 172.21.15.29:0/1103099306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b8159390 0x7f04b813ad50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f04a400c3c0 tx=0x7f04a400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:36.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04b5600640 1 -- 172.21.15.29:0/1103099306 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a4017070 con 0x7f04b8159390 2024-04-17T19:53:36.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04b5600640 1 -- 172.21.15.29:0/1103099306 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a4005cf0 con 0x7f04b8159390 2024-04-17T19:53:36.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04b5600640 1 -- 172.21.15.29:0/1103099306 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a40072d0 con 0x7f04b8159390 2024-04-17T19:53:36.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.706+0000 7f04c0600640 1 -- 172.21.15.29:0/1103099306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04b81f6df0 con 0x7f04b8159390 2024-04-17T19:53:36.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.710+0000 7f04c0600640 1 -- 172.21.15.29:0/1103099306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04b81f73b0 con 0x7f04b8159390 2024-04-17T19:53:36.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.710+0000 7f04c0600640 1 -- 172.21.15.29:0/1103099306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04b81f6b40 con 0x7f04b8159390 2024-04-17T19:53:36.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.710+0000 7f04b5600640 1 -- 172.21.15.29:0/1103099306 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f04a401a070 con 0x7f04b8159390 2024-04-17T19:53:36.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.710+0000 7f04b5600640 1 --2- 172.21.15.29:0/1103099306 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f04980683b0 0x7f049806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:36.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.710+0000 7f04b5600640 1 -- 172.21.15.29:0/1103099306 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7497+0+0 (secure 0 0 0) 0x7f04a4086390 con 0x7f04b8159390 2024-04-17T19:53:36.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.710+0000 7f04bdc00640 1 --2- 172.21.15.29:0/1103099306 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f04980683b0 0x7f049806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:36.704 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.714+0000 7f04bdc00640 1 --2- 172.21.15.29:0/1103099306 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f04980683b0 0x7f049806a8c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009ac0 tx=0x7f04ac009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:36.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.718+0000 7f04b5600640 1 -- 172.21.15.29:0/1103099306 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f04a4055b60 con 0x7f04b8159390 2024-04-17T19:53:36.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.914+0000 7f04c0600640 1 -- 172.21.15.29:0/1103099306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04b8069b90 con 0x7f04b8159390 2024-04-17T19:53:36.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.918+0000 7f04b5600640 1 -- 172.21.15.29:0/1103099306 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6083 (secure 0 0 0) 0x7f04a4059e60 con 0x7f04b8159390 2024-04-17T19:53:36.908 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:36.909 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:34.909002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4259,4302,5212,4305,5350]},"id":1}]} 2024-04-17T19:53:36.909 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 36 2024-04-17T19:53:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.918+0000 7f04c0600640 1 -- 172.21.15.29:0/1103099306 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f04980683b0 msgr2=0x7f049806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.918+0000 7f04c0600640 1 --2- 172.21.15.29:0/1103099306 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f04980683b0 0x7f049806a8c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f04ac009ac0 tx=0x7f04ac009190 comp rx=0 tx=0).stop 2024-04-17T19:53:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.918+0000 7f04c0600640 1 -- 172.21.15.29:0/1103099306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b8159390 msgr2=0x7f04b813ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.918+0000 7f04c0600640 1 --2- 172.21.15.29:0/1103099306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b8159390 0x7f04b813ad50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f04a400c3c0 tx=0x7f04a400c880 comp rx=0 tx=0).stop 2024-04-17T19:53:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.918+0000 7f04c0600640 1 -- 172.21.15.29:0/1103099306 shutdown_connections 2024-04-17T19:53:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.918+0000 7f04c0600640 1 --2- 172.21.15.29:0/1103099306 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f04980683b0 0x7f049806a8c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.918+0000 7f04c0600640 1 --2- 172.21.15.29:0/1103099306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f04b81597a0 0x7f04b8139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.918+0000 7f04c0600640 1 --2- 172.21.15.29:0/1103099306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04b8159390 0x7f04b813ad50 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.918+0000 7f04c0600640 1 --2- 172.21.15.29:0/1103099306 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f04b80cc8c0 0x7f04b815c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.918+0000 7f04c0600640 1 -- 172.21.15.29:0/1103099306 >> 172.21.15.29:0/1103099306 conn(0x7f04b806fde0 msgr2=0x7f04b813ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.918+0000 7f04c0600640 1 -- 172.21.15.29:0/1103099306 shutdown_connections 2024-04-17T19:53:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:36.918+0000 7f04c0600640 1 -- 172.21.15.29:0/1103099306 wait complete. 2024-04-17T19:53:36.931 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=3, replacing gid=4229 2024-04-17T19:53:36.931 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.f 2024-04-17T19:53:45.933 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2024-04-17T19:53:45.933 INFO:tasks.ceph.mds.f:Restarting daemon 2024-04-17T19:53:45.933 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-04-17T19:53:45.941 INFO:tasks.ceph.mds.f:Started 2024-04-17T19:53:45.941 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:53:45.985 INFO:tasks.workunit.client.0.smithi029.stdout:Delete all the files and directories ... 2024-04-17T19:53:46.168 INFO:tasks.ceph.mds.f.smithi037.stdout:starting mds.f at 2024-04-17T19:53:46.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.282+0000 7fe1a2600640 1 Processor -- start 2024-04-17T19:53:46.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.282+0000 7fe1a2600640 1 -- start start 2024-04-17T19:53:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.286+0000 7fe1a2600640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe19c0c6f70 0x7fe19c0c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.286+0000 7fe1a2600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe19c15a0a0 con 0x7fe19c06c660 2024-04-17T19:53:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.286+0000 7fe1a2600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe19c15abe0 con 0x7fe19c159c90 2024-04-17T19:53:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.286+0000 7fe1a2600640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fe19c15a340 con 0x7fe19c0c6f70 2024-04-17T19:53:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.286+0000 7fe19be00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe19c06c660 0x7fe19c06ca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60924/0 (socket says 172.21.15.29:60924) 2024-04-17T19:53:46.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.286+0000 7fe19be00640 1 -- 172.21.15.29:0/1701444927 learned_addr learned my addr 172.21.15.29:0/1701444927 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:53:46.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.286+0000 7fe192a00640 1 --2- 172.21.15.29:0/1701444927 >> v2:172.21.15.37:3300/0 conn(0x7fe19c0c6f70 0x7fe19c0c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:46.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.286+0000 7fe19aa00640 1 -- 172.21.15.29:0/1701444927 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3659394498 0 0) 0x7fe19c15abe0 con 0x7fe19c159c90 2024-04-17T19:53:46.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.286+0000 7fe19aa00640 1 -- 172.21.15.29:0/1701444927 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe180003530 con 0x7fe19c159c90 2024-04-17T19:53:46.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.286+0000 7fe192a00640 1 -- 172.21.15.29:0/1701444927 >> v1:172.21.15.37:6789/0 conn(0x7fe19c159c90 legacy=0x7fe19c0ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:46.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.286+0000 7fe192a00640 1 -- 172.21.15.29:0/1701444927 >> v1:172.21.15.29:6789/0 conn(0x7fe19c06c660 legacy=0x7fe19c06ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:46.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.286+0000 7fe192a00640 1 -- 172.21.15.29:0/1701444927 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe19c15a620 con 0x7fe19c0c6f70 2024-04-17T19:53:46.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.290+0000 7fe192a00640 1 --2- 172.21.15.29:0/1701444927 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe19c0c6f70 0x7fe19c0c7380 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe1840097c0 tx=0x7fe18401c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3515dadd075114a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:46.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.294+0000 7fe19aa00640 1 -- 172.21.15.29:0/1701444927 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1840b0070 con 0x7fe19c0c6f70 2024-04-17T19:53:46.284 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.294+0000 7fe19aa00640 1 -- 172.21.15.29:0/1701444927 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1840ab600 con 0x7fe19c0c6f70 2024-04-17T19:53:46.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.294+0000 7fe19aa00640 1 -- 172.21.15.29:0/1701444927 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1840ab9d0 con 0x7fe19c0c6f70 2024-04-17T19:53:46.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.294+0000 7fe1a2600640 1 -- 172.21.15.29:0/1701444927 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe19c0c6f70 msgr2=0x7fe19c0c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:46.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.294+0000 7fe1a2600640 1 --2- 172.21.15.29:0/1701444927 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe19c0c6f70 0x7fe19c0c7380 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe1840097c0 tx=0x7fe18401c330 comp rx=0 tx=0).stop 2024-04-17T19:53:46.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.294+0000 7fe1a2600640 1 -- 172.21.15.29:0/1701444927 shutdown_connections 2024-04-17T19:53:46.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.294+0000 7fe1a2600640 1 --2- 172.21.15.29:0/1701444927 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe19c0c6f70 0x7fe19c0c7380 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:46.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.294+0000 7fe1a2600640 1 -- 172.21.15.29:0/1701444927 >> 172.21.15.29:0/1701444927 conn(0x7fe19c06fde0 msgr2=0x7fe19c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:46.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.294+0000 7fe1a2600640 1 -- 172.21.15.29:0/1701444927 shutdown_connections 2024-04-17T19:53:46.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.294+0000 7fe1a2600640 1 -- 172.21.15.29:0/1701444927 wait complete. 2024-04-17T19:53:46.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.298+0000 7fe1a2600640 1 Processor -- start 2024-04-17T19:53:46.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.298+0000 7fe1a2600640 1 -- start start 2024-04-17T19:53:46.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.302+0000 7fe1a2600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe19c06c660 0x7fe19c14f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:46.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.302+0000 7fe1a2600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe19c0c6f70 0x7fe19c15d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:46.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.302+0000 7fe1a2600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe19c159c90 0x7fe19c15daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:46.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.302+0000 7fe1a2600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c06c140 con 0x7fe19c06c660 2024-04-17T19:53:46.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.302+0000 7fe1a2600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe19c06be60 con 0x7fe19c159c90 2024-04-17T19:53:46.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.302+0000 7fe1a2600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe19c0ca9c0 con 0x7fe19c0c6f70 2024-04-17T19:53:46.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.302+0000 7fe19b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe19c06c660 0x7fe19c14f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:46.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.302+0000 7fe192a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe19c0c6f70 0x7fe19c15d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:46.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.302+0000 7fe192a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe19c0c6f70 0x7fe19c15d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:35104/0 (socket says 172.21.15.29:35104) 2024-04-17T19:53:46.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.302+0000 7fe192a00640 1 -- 172.21.15.29:0/1518027349 learned_addr learned my addr 172.21.15.29:0/1518027349 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:46.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.302+0000 7fe19be00640 1 --2- 172.21.15.29:0/1518027349 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe19c159c90 0x7fe19c15daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:46.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.302+0000 7fe19be00640 1 -- 172.21.15.29:0/1518027349 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe19c0c6f70 msgr2=0x7fe19c15d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:46.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.302+0000 7fe19be00640 1 --2- 172.21.15.29:0/1518027349 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe19c0c6f70 0x7fe19c15d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:46.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.302+0000 7fe19be00640 1 -- 172.21.15.29:0/1518027349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe19c06c660 msgr2=0x7fe19c14f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:46.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.302+0000 7fe19be00640 1 --2- 172.21.15.29:0/1518027349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe19c06c660 0x7fe19c14f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:46.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.302+0000 7fe19be00640 1 -- 172.21.15.29:0/1518027349 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe19c1518a0 con 0x7fe19c159c90 2024-04-17T19:53:46.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.302+0000 7fe19be00640 1 --2- 172.21.15.29:0/1518027349 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe19c159c90 0x7fe19c15daf0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe19400f8e0 tx=0x7fe19400fda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:46.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.306+0000 7fe193e00640 1 -- 172.21.15.29:0/1518027349 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19401b070 con 0x7fe19c159c90 2024-04-17T19:53:46.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.306+0000 7fe193e00640 1 -- 172.21.15.29:0/1518027349 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe194007de0 con 0x7fe19c159c90 2024-04-17T19:53:46.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.306+0000 7fe193e00640 1 -- 172.21.15.29:0/1518027349 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19400a310 con 0x7fe19c159c90 2024-04-17T19:53:46.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.306+0000 7fe1a2600640 1 -- 172.21.15.29:0/1518027349 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe19c14fc90 con 0x7fe19c159c90 2024-04-17T19:53:46.297 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.306+0000 7fe1a2600640 1 -- 172.21.15.29:0/1518027349 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe19c157060 con 0x7fe19c159c90 2024-04-17T19:53:46.298 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.306+0000 7fe1a2600640 1 -- 172.21.15.29:0/1518027349 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe19c069b90 con 0x7fe19c159c90 2024-04-17T19:53:46.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.310+0000 7fe193e00640 1 -- 172.21.15.29:0/1518027349 <== mon.1 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe19401e070 con 0x7fe19c159c90 2024-04-17T19:53:46.302 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.310+0000 7fe193e00640 1 --2- 172.21.15.29:0/1518027349 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe178068400 0x7fe17806a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:46.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.310+0000 7fe193e00640 1 -- 172.21.15.29:0/1518027349 <== mon.1 v2:172.21.15.37:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 9089+0+0 (secure 0 0 0) 0x7fe194089ea0 con 0x7fe19c159c90 2024-04-17T19:53:46.303 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.310+0000 7fe19b400640 1 --2- 172.21.15.29:0/1518027349 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe178068400 0x7fe17806a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:46.304 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.314+0000 7fe19b400640 1 --2- 172.21.15.29:0/1518027349 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe178068400 0x7fe17806a910 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe18c001570 tx=0x7fe18c012920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:46.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.318+0000 7fe193e00640 1 -- 172.21.15.29:0/1518027349 <== mon.1 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe194059190 con 0x7fe19c159c90 2024-04-17T19:53:46.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.522+0000 7fe1a2600640 1 -- 172.21.15.29:0/1518027349 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe19c0c60d0 con 0x7fe19c159c90 2024-04-17T19:53:46.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.522+0000 7fe193e00640 1 -- 172.21.15.29:0/1518027349 <== mon.1 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6083 (secure 0 0 0) 0x7fe19405d490 con 0x7fe19c159c90 2024-04-17T19:53:46.514 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:46.514 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:34.909002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4259,4302,5212,4305,5350]},"id":1}]} 2024-04-17T19:53:46.514 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 36 2024-04-17T19:53:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.522+0000 7fe1a2600640 1 -- 172.21.15.29:0/1518027349 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe178068400 msgr2=0x7fe17806a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.522+0000 7fe1a2600640 1 --2- 172.21.15.29:0/1518027349 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe178068400 0x7fe17806a910 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe18c001570 tx=0x7fe18c012920 comp rx=0 tx=0).stop 2024-04-17T19:53:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.522+0000 7fe1a2600640 1 -- 172.21.15.29:0/1518027349 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe19c159c90 msgr2=0x7fe19c15daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.522+0000 7fe1a2600640 1 --2- 172.21.15.29:0/1518027349 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe19c159c90 0x7fe19c15daf0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe19400f8e0 tx=0x7fe19400fda0 comp rx=0 tx=0).stop 2024-04-17T19:53:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.522+0000 7fe1a2600640 1 -- 172.21.15.29:0/1518027349 shutdown_connections 2024-04-17T19:53:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.522+0000 7fe1a2600640 1 --2- 172.21.15.29:0/1518027349 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe178068400 0x7fe17806a910 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.522+0000 7fe1a2600640 1 --2- 172.21.15.29:0/1518027349 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe19c159c90 0x7fe19c15daf0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.522+0000 7fe1a2600640 1 --2- 172.21.15.29:0/1518027349 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe19c0c6f70 0x7fe19c15d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.522+0000 7fe1a2600640 1 --2- 172.21.15.29:0/1518027349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe19c06c660 0x7fe19c14f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.522+0000 7fe1a2600640 1 -- 172.21.15.29:0/1518027349 >> 172.21.15.29:0/1518027349 conn(0x7fe19c06fde0 msgr2=0x7fe19c0c24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:46.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.522+0000 7fe1a2600640 1 -- 172.21.15.29:0/1518027349 shutdown_connections 2024-04-17T19:53:46.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:46.522+0000 7fe1a2600640 1 -- 172.21.15.29:0/1518027349 wait complete. 2024-04-17T19:53:46.535 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2024-04-17T19:53:48.536 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:53:48.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.694+0000 7fbd6e600640 1 Processor -- start 2024-04-17T19:53:48.687 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.694+0000 7fbd6e600640 1 -- start start 2024-04-17T19:53:48.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd6e600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd680c75c0 0x7fbd6806bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:48.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd6e600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd680c60e0 con 0x7fbd681523d0 2024-04-17T19:53:48.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd6e600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd680c6350 con 0x7fbd680c71b0 2024-04-17T19:53:48.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd6e600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbd6815ca30 con 0x7fbd680c75c0 2024-04-17T19:53:48.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd67e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbd681523d0 0x7fbd68162e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55070/0 (socket says 172.21.15.29:55070) 2024-04-17T19:53:48.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd67e00640 1 -- 172.21.15.29:0/1930496306 learned_addr learned my addr 172.21.15.29:0/1930496306 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:53:48.690 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd5fe00640 1 --2- 172.21.15.29:0/1930496306 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd680c75c0 0x7fbd6806bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd66a00640 1 -- 172.21.15.29:0/1930496306 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2903461282 0 0) 0x7fbd680c60e0 con 0x7fbd681523d0 2024-04-17T19:53:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd66a00640 1 -- 172.21.15.29:0/1930496306 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd4c003530 con 0x7fbd681523d0 2024-04-17T19:53:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd66a00640 1 -- 172.21.15.29:0/1930496306 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3642006596 0 0) 0x7fbd680c6350 con 0x7fbd680c71b0 2024-04-17T19:53:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd66a00640 1 -- 172.21.15.29:0/1930496306 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd680c60e0 con 0x7fbd680c71b0 2024-04-17T19:53:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd66a00640 1 -- 172.21.15.29:0/1930496306 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3384464541 0 0) 0x7fbd4c003530 con 0x7fbd681523d0 2024-04-17T19:53:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd66a00640 1 -- 172.21.15.29:0/1930496306 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd680c75c0 msgr2=0x7fbd6806bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd66a00640 1 --2- 172.21.15.29:0/1930496306 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd680c75c0 0x7fbd6806bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd66a00640 1 -- 172.21.15.29:0/1930496306 >> v1:172.21.15.37:6789/0 conn(0x7fbd680c71b0 legacy=0x7fbd6815c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd66a00640 1 -- 172.21.15.29:0/1930496306 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd680c6630 con 0x7fbd681523d0 2024-04-17T19:53:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd66a00640 1 -- 172.21.15.29:0/1930496306 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fbd60001a20 con 0x7fbd681523d0 2024-04-17T19:53:48.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd66a00640 1 -- 172.21.15.29:0/1930496306 >> v1:172.21.15.29:6789/0 conn(0x7fbd681523d0 legacy=0x7fbd68162e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:48.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd66a00640 1 --2- 172.21.15.29:0/1930496306 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd680c75c0 0x7fbd4c003bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:48.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd66a00640 1 --2- 172.21.15.29:0/1930496306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd4c004150 0x7fbd4c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:48.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd66a00640 1 --2- 172.21.15.29:0/1930496306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd4c004d50 0x7fbd4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:48.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd66a00640 1 -- 172.21.15.29:0/1930496306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6815ca30 con 0x7fbd4c004d50 2024-04-17T19:53:48.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd66a00640 1 -- 172.21.15.29:0/1930496306 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c00ddd0 con 0x7fbd4c004150 2024-04-17T19:53:48.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd66a00640 1 -- 172.21.15.29:0/1930496306 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbd4c00dfd0 con 0x7fbd680c75c0 2024-04-17T19:53:48.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd67e00640 1 --2- 172.21.15.29:0/1930496306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd4c004d50 0x7fbd4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:48.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd5fe00640 1 --2- 172.21.15.29:0/1930496306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd4c004150 0x7fbd4c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:48.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd67400640 1 --2- 172.21.15.29:0/1930496306 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd680c75c0 0x7fbd4c003bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:48.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd67e00640 1 -- 172.21.15.29:0/1930496306 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd680c75c0 msgr2=0x7fbd4c003bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:48.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd67e00640 1 --2- 172.21.15.29:0/1930496306 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd680c75c0 0x7fbd4c003bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:48.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd67e00640 1 -- 172.21.15.29:0/1930496306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd4c004150 msgr2=0x7fbd4c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:53:48.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd67e00640 1 --2- 172.21.15.29:0/1930496306 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd4c004150 0x7fbd4c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:48.693 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.698+0000 7fbd67e00640 1 -- 172.21.15.29:0/1930496306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd4c003530 con 0x7fbd4c004d50 2024-04-17T19:53:48.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.702+0000 7fbd67e00640 1 --2- 172.21.15.29:0/1930496306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd4c004d50 0x7fbd4c0051c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fbd68071e80 tx=0x7fbd6001d280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1bad30579ff9396 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:48.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.702+0000 7fbd66a00640 1 -- 172.21.15.29:0/1930496306 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6000a450 con 0x7fbd4c004d50 2024-04-17T19:53:48.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.702+0000 7fbd66a00640 1 -- 172.21.15.29:0/1930496306 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd600072f0 con 0x7fbd4c004d50 2024-04-17T19:53:48.694 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.702+0000 7fbd66a00640 1 -- 172.21.15.29:0/1930496306 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd600a5e60 con 0x7fbd4c004d50 2024-04-17T19:53:48.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.702+0000 7fbd6e600640 1 -- 172.21.15.29:0/1930496306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd4c004d50 msgr2=0x7fbd4c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:48.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.702+0000 7fbd6e600640 1 --2- 172.21.15.29:0/1930496306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd4c004d50 0x7fbd4c0051c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fbd68071e80 tx=0x7fbd6001d280 comp rx=0 tx=0).stop 2024-04-17T19:53:48.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.702+0000 7fbd67e00640 1 -- 172.21.15.29:0/1930496306 reap_dead start 2024-04-17T19:53:48.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.702+0000 7fbd6e600640 1 -- 172.21.15.29:0/1930496306 shutdown_connections 2024-04-17T19:53:48.695 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.702+0000 7fbd6e600640 1 -- 172.21.15.29:0/1930496306 >> 172.21.15.29:0/1930496306 conn(0x7fbd6806fde0 msgr2=0x7fbd680701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:48.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.702+0000 7fbd6e600640 1 -- 172.21.15.29:0/1930496306 shutdown_connections 2024-04-17T19:53:48.696 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.702+0000 7fbd6e600640 1 -- 172.21.15.29:0/1930496306 wait complete. 2024-04-17T19:53:48.697 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd6e600640 1 Processor -- start 2024-04-17T19:53:48.698 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd6e600640 1 -- start start 2024-04-17T19:53:48.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd6e600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd4c004d50 0x7fbd681f2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:48.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd6e600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd6815a340 0x7fbd6814e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:48.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd6e600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd68144bf0 0x7fbd68145000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:48.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd6e600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbd680cb140 con 0x7fbd68144bf0 2024-04-17T19:53:48.700 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd6e600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6815be90 con 0x7fbd6815a340 2024-04-17T19:53:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd6e600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbd680c6910 con 0x7fbd4c004d50 2024-04-17T19:53:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd67e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd68144bf0 0x7fbd68145000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd67e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd68144bf0 0x7fbd68145000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59712/0 (socket says 172.21.15.29:59712) 2024-04-17T19:53:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd67e00640 1 -- 172.21.15.29:0/3774888684 learned_addr learned my addr 172.21.15.29:0/3774888684 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:53:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd67400640 1 --2- 172.21.15.29:0/3774888684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd4c004d50 0x7fbd681f2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd5fe00640 1 --2- 172.21.15.29:0/3774888684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd6815a340 0x7fbd6814e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd67e00640 1 -- 172.21.15.29:0/3774888684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd4c004d50 msgr2=0x7fbd681f2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd67e00640 1 --2- 172.21.15.29:0/3774888684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd4c004d50 0x7fbd681f2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd67e00640 1 -- 172.21.15.29:0/3774888684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd6815a340 msgr2=0x7fbd6814e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd67e00640 1 --2- 172.21.15.29:0/3774888684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd6815a340 0x7fbd6814e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd67e00640 1 -- 172.21.15.29:0/3774888684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd68145540 con 0x7fbd68144bf0 2024-04-17T19:53:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.706+0000 7fbd67e00640 1 --2- 172.21.15.29:0/3774888684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd68144bf0 0x7fbd68145000 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fbd60001ca0 tx=0x7fbd4c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.710+0000 7fbd5f400640 1 -- 172.21.15.29:0/3774888684 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6000a450 con 0x7fbd68144bf0 2024-04-17T19:53:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.710+0000 7fbd5f400640 1 -- 172.21.15.29:0/3774888684 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd600a5e60 con 0x7fbd68144bf0 2024-04-17T19:53:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.710+0000 7fbd6e600640 1 -- 172.21.15.29:0/3774888684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd6815af50 con 0x7fbd68144bf0 2024-04-17T19:53:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.710+0000 7fbd5f400640 1 -- 172.21.15.29:0/3774888684 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd60001fb0 con 0x7fbd68144bf0 2024-04-17T19:53:48.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.710+0000 7fbd6e600640 1 -- 172.21.15.29:0/3774888684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd68159860 con 0x7fbd68144bf0 2024-04-17T19:53:48.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.710+0000 7fbd6e600640 1 -- 172.21.15.29:0/3774888684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd68159650 con 0x7fbd68144bf0 2024-04-17T19:53:48.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.710+0000 7fbd5f400640 1 -- 172.21.15.29:0/3774888684 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbd600b5070 con 0x7fbd68144bf0 2024-04-17T19:53:48.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.710+0000 7fbd5f400640 1 --2- 172.21.15.29:0/3774888684 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbd44068380 0x7fbd4406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:53:48.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.710+0000 7fbd5f400640 1 -- 172.21.15.29:0/3774888684 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8769+0+0 (secure 0 0 0) 0x7fbd60120b60 con 0x7fbd68144bf0 2024-04-17T19:53:48.703 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.710+0000 7fbd67400640 1 --2- 172.21.15.29:0/3774888684 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbd44068380 0x7fbd4406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:53:48.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.714+0000 7fbd67400640 1 --2- 172.21.15.29:0/3774888684 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbd44068380 0x7fbd4406a890 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbd58007af0 tx=0x7fbd58009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:53:48.708 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.714+0000 7fbd5f400640 1 -- 172.21.15.29:0/3774888684 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbd68145540 con 0x7fbd68144bf0 2024-04-17T19:53:48.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.922+0000 7fbd6e600640 1 -- 172.21.15.29:0/3774888684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd68069b90 con 0x7fbd68144bf0 2024-04-17T19:53:48.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.922+0000 7fbd5f400640 1 -- 172.21.15.29:0/3774888684 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6904 (secure 0 0 0) 0x7fbd60016f90 con 0x7fbd68144bf0 2024-04-17T19:53:48.914 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:53:48.914 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5144,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:34.909002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,4305,5212,4302,4259]},"id":1}]} 2024-04-17T19:53:48.914 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-04-17T19:53:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.922+0000 7fbd6e600640 1 -- 172.21.15.29:0/3774888684 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbd44068380 msgr2=0x7fbd4406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.922+0000 7fbd6e600640 1 --2- 172.21.15.29:0/3774888684 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbd44068380 0x7fbd4406a890 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbd58007af0 tx=0x7fbd58009040 comp rx=0 tx=0).stop 2024-04-17T19:53:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.922+0000 7fbd6e600640 1 -- 172.21.15.29:0/3774888684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd68144bf0 msgr2=0x7fbd68145000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:53:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.922+0000 7fbd6e600640 1 --2- 172.21.15.29:0/3774888684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd68144bf0 0x7fbd68145000 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fbd60001ca0 tx=0x7fbd4c005840 comp rx=0 tx=0).stop 2024-04-17T19:53:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.922+0000 7fbd6e600640 1 -- 172.21.15.29:0/3774888684 shutdown_connections 2024-04-17T19:53:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.922+0000 7fbd6e600640 1 --2- 172.21.15.29:0/3774888684 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbd44068380 0x7fbd4406a890 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.922+0000 7fbd6e600640 1 --2- 172.21.15.29:0/3774888684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbd68144bf0 0x7fbd68145000 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.922+0000 7fbd6e600640 1 --2- 172.21.15.29:0/3774888684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbd6815a340 0x7fbd6814e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.922+0000 7fbd6e600640 1 --2- 172.21.15.29:0/3774888684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbd4c004d50 0x7fbd681f2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:53:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.922+0000 7fbd6e600640 1 -- 172.21.15.29:0/3774888684 >> 172.21.15.29:0/3774888684 conn(0x7fbd6806fde0 msgr2=0x7fbd680cc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:53:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.922+0000 7fbd6e600640 1 -- 172.21.15.29:0/3774888684 shutdown_connections 2024-04-17T19:53:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:53:48.922+0000 7fbd6e600640 1 -- 172.21.15.29:0/3774888684 wait complete. 2024-04-17T19:53:48.936 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2024-04-17T19:53:48.936 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 73 secs before thrashing 2024-04-17T19:53:49.711 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/19' 2024-04-17T19:53:49.712 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/15' 2024-04-17T19:53:49.712 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/31' 2024-04-17T19:53:49.712 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/36' 2024-04-17T19:53:49.712 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/20' 2024-04-17T19:53:49.712 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/12' 2024-04-17T19:53:49.712 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/39' 2024-04-17T19:53:49.712 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/10' 2024-04-17T19:53:49.712 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/34' 2024-04-17T19:53:49.712 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/38' 2024-04-17T19:53:49.712 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/11' 2024-04-17T19:53:49.712 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/16' 2024-04-17T19:53:49.712 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/32' 2024-04-17T19:53:49.712 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/30' 2024-04-17T19:53:49.712 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/14' 2024-04-17T19:53:49.712 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/18' 2024-04-17T19:53:49.713 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/13' 2024-04-17T19:53:49.713 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/33' 2024-04-17T19:53:49.713 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/17' 2024-04-17T19:53:49.713 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/35' 2024-04-17T19:53:49.713 INFO:tasks.workunit.client.0.smithi029.stdout:removed './100/37' 2024-04-17T19:53:49.713 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './100' 2024-04-17T19:53:49.713 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/19' 2024-04-17T19:53:49.713 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/15' 2024-04-17T19:53:49.713 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/31' 2024-04-17T19:53:49.713 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/36' 2024-04-17T19:53:49.713 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/20' 2024-04-17T19:53:49.713 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/12' 2024-04-17T19:53:49.713 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/39' 2024-04-17T19:53:49.713 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/10' 2024-04-17T19:53:49.713 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/34' 2024-04-17T19:53:49.714 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/38' 2024-04-17T19:53:49.714 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/11' 2024-04-17T19:53:49.714 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/16' 2024-04-17T19:53:49.714 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/32' 2024-04-17T19:53:49.714 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/30' 2024-04-17T19:53:49.714 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/14' 2024-04-17T19:53:49.714 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/18' 2024-04-17T19:53:49.714 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/13' 2024-04-17T19:53:49.714 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/33' 2024-04-17T19:53:49.714 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/17' 2024-04-17T19:53:49.714 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/35' 2024-04-17T19:53:49.714 INFO:tasks.workunit.client.0.smithi029.stdout:removed './101/37' 2024-04-17T19:53:49.714 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './101' 2024-04-17T19:53:49.714 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/19' 2024-04-17T19:53:49.715 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/15' 2024-04-17T19:53:49.715 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/31' 2024-04-17T19:53:49.715 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/36' 2024-04-17T19:53:49.715 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/20' 2024-04-17T19:53:49.715 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/12' 2024-04-17T19:53:49.715 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/39' 2024-04-17T19:53:49.715 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/10' 2024-04-17T19:53:49.715 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/34' 2024-04-17T19:53:49.715 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/38' 2024-04-17T19:53:49.715 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/11' 2024-04-17T19:53:49.715 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/16' 2024-04-17T19:53:49.715 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/32' 2024-04-17T19:53:49.715 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/30' 2024-04-17T19:53:49.715 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/14' 2024-04-17T19:53:49.715 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/18' 2024-04-17T19:53:49.715 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/13' 2024-04-17T19:53:49.716 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/33' 2024-04-17T19:53:49.716 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/17' 2024-04-17T19:53:49.716 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/35' 2024-04-17T19:53:49.716 INFO:tasks.workunit.client.0.smithi029.stdout:removed './102/37' 2024-04-17T19:53:49.716 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './102' 2024-04-17T19:53:49.716 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/19' 2024-04-17T19:53:49.716 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/15' 2024-04-17T19:53:49.716 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/31' 2024-04-17T19:53:49.716 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/36' 2024-04-17T19:53:49.716 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/20' 2024-04-17T19:53:49.716 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/12' 2024-04-17T19:53:49.716 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/39' 2024-04-17T19:53:49.716 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/10' 2024-04-17T19:53:49.716 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/34' 2024-04-17T19:53:49.717 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/38' 2024-04-17T19:53:49.717 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/11' 2024-04-17T19:53:49.717 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/16' 2024-04-17T19:53:49.717 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/32' 2024-04-17T19:53:49.717 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/30' 2024-04-17T19:53:49.717 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/14' 2024-04-17T19:53:49.717 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/18' 2024-04-17T19:53:49.717 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/13' 2024-04-17T19:53:49.717 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/33' 2024-04-17T19:53:49.717 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/17' 2024-04-17T19:53:49.717 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/35' 2024-04-17T19:53:49.717 INFO:tasks.workunit.client.0.smithi029.stdout:removed './103/37' 2024-04-17T19:53:49.717 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './103' 2024-04-17T19:53:49.717 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/19' 2024-04-17T19:53:49.717 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/15' 2024-04-17T19:53:49.717 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/31' 2024-04-17T19:53:49.718 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/36' 2024-04-17T19:53:49.718 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/20' 2024-04-17T19:53:49.718 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/12' 2024-04-17T19:53:49.718 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/39' 2024-04-17T19:53:49.718 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/10' 2024-04-17T19:53:49.718 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/34' 2024-04-17T19:53:49.718 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/38' 2024-04-17T19:53:49.718 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/11' 2024-04-17T19:53:49.718 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/16' 2024-04-17T19:53:49.718 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/32' 2024-04-17T19:53:49.718 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/30' 2024-04-17T19:53:49.718 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/14' 2024-04-17T19:53:49.718 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/18' 2024-04-17T19:53:49.718 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/13' 2024-04-17T19:53:49.718 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/33' 2024-04-17T19:53:49.719 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/17' 2024-04-17T19:53:49.719 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/35' 2024-04-17T19:53:49.719 INFO:tasks.workunit.client.0.smithi029.stdout:removed './104/37' 2024-04-17T19:53:49.719 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './104' 2024-04-17T19:53:49.719 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/19' 2024-04-17T19:53:49.719 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/15' 2024-04-17T19:53:49.719 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/31' 2024-04-17T19:53:49.719 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/36' 2024-04-17T19:53:49.719 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/20' 2024-04-17T19:53:49.719 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/12' 2024-04-17T19:53:49.719 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/39' 2024-04-17T19:53:49.719 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/10' 2024-04-17T19:53:49.719 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/34' 2024-04-17T19:53:49.719 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/38' 2024-04-17T19:53:49.719 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/11' 2024-04-17T19:53:49.720 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/16' 2024-04-17T19:53:49.720 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/32' 2024-04-17T19:53:49.720 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/30' 2024-04-17T19:53:49.720 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/14' 2024-04-17T19:53:49.720 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/18' 2024-04-17T19:53:49.720 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/13' 2024-04-17T19:53:49.720 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/33' 2024-04-17T19:53:49.720 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/17' 2024-04-17T19:53:49.720 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/35' 2024-04-17T19:53:49.720 INFO:tasks.workunit.client.0.smithi029.stdout:removed './105/37' 2024-04-17T19:53:49.720 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './105' 2024-04-17T19:53:49.720 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/19' 2024-04-17T19:53:49.720 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/15' 2024-04-17T19:53:49.720 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/31' 2024-04-17T19:53:49.720 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/36' 2024-04-17T19:53:49.720 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/20' 2024-04-17T19:53:49.721 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/12' 2024-04-17T19:53:49.721 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/39' 2024-04-17T19:53:49.721 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/10' 2024-04-17T19:53:49.721 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/34' 2024-04-17T19:53:49.721 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/38' 2024-04-17T19:53:49.721 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/11' 2024-04-17T19:53:49.721 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/16' 2024-04-17T19:53:49.721 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/32' 2024-04-17T19:53:49.721 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/30' 2024-04-17T19:53:49.721 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/14' 2024-04-17T19:53:49.721 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/18' 2024-04-17T19:53:49.721 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/13' 2024-04-17T19:53:49.721 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/33' 2024-04-17T19:53:49.721 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/17' 2024-04-17T19:53:49.721 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/35' 2024-04-17T19:53:49.722 INFO:tasks.workunit.client.0.smithi029.stdout:removed './106/37' 2024-04-17T19:53:49.722 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './106' 2024-04-17T19:53:49.722 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/19' 2024-04-17T19:53:49.722 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/15' 2024-04-17T19:53:49.722 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/31' 2024-04-17T19:53:49.722 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/36' 2024-04-17T19:53:49.722 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/20' 2024-04-17T19:53:49.722 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/12' 2024-04-17T19:53:49.722 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/39' 2024-04-17T19:53:49.722 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/10' 2024-04-17T19:53:49.722 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/34' 2024-04-17T19:53:49.722 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/38' 2024-04-17T19:53:49.722 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/11' 2024-04-17T19:53:49.722 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/16' 2024-04-17T19:53:49.722 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/32' 2024-04-17T19:53:49.723 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/30' 2024-04-17T19:53:49.723 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/14' 2024-04-17T19:53:49.723 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/18' 2024-04-17T19:53:49.723 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/13' 2024-04-17T19:53:49.723 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/33' 2024-04-17T19:53:49.723 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/17' 2024-04-17T19:53:49.723 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/35' 2024-04-17T19:53:49.723 INFO:tasks.workunit.client.0.smithi029.stdout:removed './107/37' 2024-04-17T19:53:49.723 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './107' 2024-04-17T19:53:49.723 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/19' 2024-04-17T19:53:49.723 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/15' 2024-04-17T19:53:49.723 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/31' 2024-04-17T19:53:49.723 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/36' 2024-04-17T19:53:49.723 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/20' 2024-04-17T19:53:49.723 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/12' 2024-04-17T19:53:49.723 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/39' 2024-04-17T19:53:49.724 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/10' 2024-04-17T19:53:49.724 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/34' 2024-04-17T19:53:49.724 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/38' 2024-04-17T19:53:49.724 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/11' 2024-04-17T19:53:49.724 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/16' 2024-04-17T19:53:49.724 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/32' 2024-04-17T19:53:49.724 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/30' 2024-04-17T19:53:49.724 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/14' 2024-04-17T19:53:49.724 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/18' 2024-04-17T19:53:49.724 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/13' 2024-04-17T19:53:49.724 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/33' 2024-04-17T19:53:49.724 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/17' 2024-04-17T19:53:49.724 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/35' 2024-04-17T19:53:49.724 INFO:tasks.workunit.client.0.smithi029.stdout:removed './108/37' 2024-04-17T19:53:49.724 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './108' 2024-04-17T19:53:49.725 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/19' 2024-04-17T19:53:49.725 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/15' 2024-04-17T19:53:49.725 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/31' 2024-04-17T19:53:49.725 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/36' 2024-04-17T19:53:49.725 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/20' 2024-04-17T19:53:49.725 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/12' 2024-04-17T19:53:49.725 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/39' 2024-04-17T19:53:49.725 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/10' 2024-04-17T19:53:49.725 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/34' 2024-04-17T19:53:49.725 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/38' 2024-04-17T19:53:49.725 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/11' 2024-04-17T19:53:49.725 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/16' 2024-04-17T19:53:49.725 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/32' 2024-04-17T19:53:49.725 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/30' 2024-04-17T19:53:50.618 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/14' 2024-04-17T19:53:50.618 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/18' 2024-04-17T19:53:50.618 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/13' 2024-04-17T19:53:50.618 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/33' 2024-04-17T19:53:50.618 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/17' 2024-04-17T19:53:50.618 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/35' 2024-04-17T19:53:50.618 INFO:tasks.workunit.client.0.smithi029.stdout:removed './109/37' 2024-04-17T19:53:50.618 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './109' 2024-04-17T19:53:50.619 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/19' 2024-04-17T19:53:50.619 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/15' 2024-04-17T19:53:50.619 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/31' 2024-04-17T19:53:50.619 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/36' 2024-04-17T19:53:50.619 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/20' 2024-04-17T19:53:50.619 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/12' 2024-04-17T19:53:50.619 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/39' 2024-04-17T19:53:50.619 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/10' 2024-04-17T19:53:50.619 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/34' 2024-04-17T19:53:50.619 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/38' 2024-04-17T19:53:50.619 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/11' 2024-04-17T19:53:50.619 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/16' 2024-04-17T19:53:50.619 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/32' 2024-04-17T19:53:50.619 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/30' 2024-04-17T19:53:50.620 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/14' 2024-04-17T19:53:50.620 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/18' 2024-04-17T19:53:50.620 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/13' 2024-04-17T19:53:50.620 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/33' 2024-04-17T19:53:50.620 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/17' 2024-04-17T19:53:50.620 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/35' 2024-04-17T19:53:50.620 INFO:tasks.workunit.client.0.smithi029.stdout:removed './110/37' 2024-04-17T19:53:50.620 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './110' 2024-04-17T19:53:50.620 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/19' 2024-04-17T19:53:50.620 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/15' 2024-04-17T19:53:50.620 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/31' 2024-04-17T19:53:50.620 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/36' 2024-04-17T19:53:50.620 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/20' 2024-04-17T19:53:50.620 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/12' 2024-04-17T19:53:50.620 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/39' 2024-04-17T19:53:50.621 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/10' 2024-04-17T19:53:50.621 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/34' 2024-04-17T19:53:50.621 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/38' 2024-04-17T19:53:50.621 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/11' 2024-04-17T19:53:50.621 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/16' 2024-04-17T19:53:50.621 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/32' 2024-04-17T19:53:50.621 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/30' 2024-04-17T19:53:50.621 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/14' 2024-04-17T19:53:50.621 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/18' 2024-04-17T19:53:50.621 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/13' 2024-04-17T19:53:50.621 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/33' 2024-04-17T19:53:50.621 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/17' 2024-04-17T19:53:50.621 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/35' 2024-04-17T19:53:50.622 INFO:tasks.workunit.client.0.smithi029.stdout:removed './111/37' 2024-04-17T19:53:50.622 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './111' 2024-04-17T19:53:50.622 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/19' 2024-04-17T19:53:50.622 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/15' 2024-04-17T19:53:50.622 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/31' 2024-04-17T19:53:50.622 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/36' 2024-04-17T19:53:50.622 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/20' 2024-04-17T19:53:50.622 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/12' 2024-04-17T19:53:50.622 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/39' 2024-04-17T19:53:50.622 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/10' 2024-04-17T19:53:50.622 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/34' 2024-04-17T19:53:50.622 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/38' 2024-04-17T19:53:50.622 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/11' 2024-04-17T19:53:50.623 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/16' 2024-04-17T19:53:50.623 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/32' 2024-04-17T19:53:50.623 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/30' 2024-04-17T19:53:50.623 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/14' 2024-04-17T19:53:50.623 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/18' 2024-04-17T19:53:50.623 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/13' 2024-04-17T19:53:50.623 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/33' 2024-04-17T19:53:50.623 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/17' 2024-04-17T19:53:50.623 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/35' 2024-04-17T19:53:50.623 INFO:tasks.workunit.client.0.smithi029.stdout:removed './112/37' 2024-04-17T19:53:50.623 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './112' 2024-04-17T19:53:50.623 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/19' 2024-04-17T19:53:50.623 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/15' 2024-04-17T19:53:50.623 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/31' 2024-04-17T19:53:50.624 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/36' 2024-04-17T19:53:50.624 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/20' 2024-04-17T19:53:50.624 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/12' 2024-04-17T19:53:50.624 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/39' 2024-04-17T19:53:50.624 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/10' 2024-04-17T19:53:50.624 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/34' 2024-04-17T19:53:50.624 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/38' 2024-04-17T19:53:50.624 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/11' 2024-04-17T19:53:50.624 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/16' 2024-04-17T19:53:50.624 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/32' 2024-04-17T19:53:50.624 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/30' 2024-04-17T19:53:50.624 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/14' 2024-04-17T19:53:50.624 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/18' 2024-04-17T19:53:50.624 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/13' 2024-04-17T19:53:50.624 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/33' 2024-04-17T19:53:50.625 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/17' 2024-04-17T19:53:50.625 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/35' 2024-04-17T19:53:50.625 INFO:tasks.workunit.client.0.smithi029.stdout:removed './113/37' 2024-04-17T19:53:50.625 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './113' 2024-04-17T19:53:50.625 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/19' 2024-04-17T19:53:50.625 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/15' 2024-04-17T19:53:50.625 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/31' 2024-04-17T19:53:50.625 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/36' 2024-04-17T19:53:50.625 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/20' 2024-04-17T19:53:50.625 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/12' 2024-04-17T19:53:50.625 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/39' 2024-04-17T19:53:50.625 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/10' 2024-04-17T19:53:50.625 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/34' 2024-04-17T19:53:50.625 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/38' 2024-04-17T19:53:50.626 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/11' 2024-04-17T19:53:50.626 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/16' 2024-04-17T19:53:50.626 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/32' 2024-04-17T19:53:50.626 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/30' 2024-04-17T19:53:50.626 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/14' 2024-04-17T19:53:50.626 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/18' 2024-04-17T19:53:50.626 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/13' 2024-04-17T19:53:50.626 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/33' 2024-04-17T19:53:50.626 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/17' 2024-04-17T19:53:50.626 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/35' 2024-04-17T19:53:50.626 INFO:tasks.workunit.client.0.smithi029.stdout:removed './114/37' 2024-04-17T19:53:50.626 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './114' 2024-04-17T19:53:50.626 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/19' 2024-04-17T19:53:50.626 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/15' 2024-04-17T19:53:50.626 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/31' 2024-04-17T19:53:50.627 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/36' 2024-04-17T19:53:50.627 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/20' 2024-04-17T19:53:50.627 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/12' 2024-04-17T19:53:50.627 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/39' 2024-04-17T19:53:50.627 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/10' 2024-04-17T19:53:50.627 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/34' 2024-04-17T19:53:50.627 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/38' 2024-04-17T19:53:50.627 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/11' 2024-04-17T19:53:50.627 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/16' 2024-04-17T19:53:50.627 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/32' 2024-04-17T19:53:50.627 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/30' 2024-04-17T19:53:50.627 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/14' 2024-04-17T19:53:50.627 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/18' 2024-04-17T19:53:50.627 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/13' 2024-04-17T19:53:50.627 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/33' 2024-04-17T19:53:50.627 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/17' 2024-04-17T19:53:50.628 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/35' 2024-04-17T19:53:50.628 INFO:tasks.workunit.client.0.smithi029.stdout:removed './115/37' 2024-04-17T19:53:50.628 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './115' 2024-04-17T19:53:50.628 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/19' 2024-04-17T19:53:50.628 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/15' 2024-04-17T19:53:50.628 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/31' 2024-04-17T19:53:50.628 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/36' 2024-04-17T19:53:50.628 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/20' 2024-04-17T19:53:50.628 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/12' 2024-04-17T19:53:50.628 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/39' 2024-04-17T19:53:50.628 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/10' 2024-04-17T19:53:50.628 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/34' 2024-04-17T19:53:50.628 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/38' 2024-04-17T19:53:50.628 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/11' 2024-04-17T19:53:50.628 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/16' 2024-04-17T19:53:50.629 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/32' 2024-04-17T19:53:50.629 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/30' 2024-04-17T19:53:50.629 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/14' 2024-04-17T19:53:50.629 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/18' 2024-04-17T19:53:50.629 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/13' 2024-04-17T19:53:50.629 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/33' 2024-04-17T19:53:50.629 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/17' 2024-04-17T19:53:50.629 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/35' 2024-04-17T19:53:50.629 INFO:tasks.workunit.client.0.smithi029.stdout:removed './116/37' 2024-04-17T19:53:50.629 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './116' 2024-04-17T19:53:50.629 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/19' 2024-04-17T19:53:50.629 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/15' 2024-04-17T19:53:50.629 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/31' 2024-04-17T19:53:50.629 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/36' 2024-04-17T19:53:50.629 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/20' 2024-04-17T19:53:50.630 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/12' 2024-04-17T19:53:50.630 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/39' 2024-04-17T19:53:50.630 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/10' 2024-04-17T19:53:50.630 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/34' 2024-04-17T19:53:50.630 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/38' 2024-04-17T19:53:50.630 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/11' 2024-04-17T19:53:50.630 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/16' 2024-04-17T19:53:50.630 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/32' 2024-04-17T19:53:50.630 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/30' 2024-04-17T19:53:50.630 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/14' 2024-04-17T19:53:50.630 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/18' 2024-04-17T19:53:50.630 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/13' 2024-04-17T19:53:50.630 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/33' 2024-04-17T19:53:50.630 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/17' 2024-04-17T19:53:50.631 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/35' 2024-04-17T19:53:50.631 INFO:tasks.workunit.client.0.smithi029.stdout:removed './117/37' 2024-04-17T19:53:50.631 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './117' 2024-04-17T19:53:50.631 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/19' 2024-04-17T19:53:50.631 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/15' 2024-04-17T19:53:50.631 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/31' 2024-04-17T19:53:50.631 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/36' 2024-04-17T19:53:50.631 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/20' 2024-04-17T19:53:50.631 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/12' 2024-04-17T19:53:50.631 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/39' 2024-04-17T19:53:50.631 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/10' 2024-04-17T19:53:50.631 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/34' 2024-04-17T19:53:50.631 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/38' 2024-04-17T19:53:50.631 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/11' 2024-04-17T19:53:50.632 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/16' 2024-04-17T19:53:50.632 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/32' 2024-04-17T19:53:50.632 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/30' 2024-04-17T19:53:50.632 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/14' 2024-04-17T19:53:50.632 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/18' 2024-04-17T19:53:50.632 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/13' 2024-04-17T19:53:50.632 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/33' 2024-04-17T19:53:50.632 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/17' 2024-04-17T19:53:50.632 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/35' 2024-04-17T19:53:50.632 INFO:tasks.workunit.client.0.smithi029.stdout:removed './118/37' 2024-04-17T19:53:50.632 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './118' 2024-04-17T19:53:50.632 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/19' 2024-04-17T19:53:50.632 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/15' 2024-04-17T19:53:50.632 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/31' 2024-04-17T19:53:50.632 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/36' 2024-04-17T19:53:50.633 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/20' 2024-04-17T19:53:50.633 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/12' 2024-04-17T19:53:51.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:53:51.611 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:53:52.105 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/39' 2024-04-17T19:53:52.106 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/10' 2024-04-17T19:53:52.106 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/34' 2024-04-17T19:53:52.106 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/38' 2024-04-17T19:53:52.106 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/11' 2024-04-17T19:53:52.106 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/16' 2024-04-17T19:53:52.106 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/32' 2024-04-17T19:53:52.106 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/30' 2024-04-17T19:53:52.106 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/14' 2024-04-17T19:53:52.106 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/18' 2024-04-17T19:53:52.106 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/13' 2024-04-17T19:53:52.106 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/33' 2024-04-17T19:53:52.106 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/17' 2024-04-17T19:53:52.106 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/35' 2024-04-17T19:53:52.106 INFO:tasks.workunit.client.0.smithi029.stdout:removed './119/37' 2024-04-17T19:53:52.107 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './119' 2024-04-17T19:53:52.107 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/19' 2024-04-17T19:53:52.107 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/15' 2024-04-17T19:53:52.107 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/31' 2024-04-17T19:53:52.107 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/36' 2024-04-17T19:53:52.107 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/20' 2024-04-17T19:53:52.107 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/12' 2024-04-17T19:53:52.107 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/39' 2024-04-17T19:53:52.107 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/10' 2024-04-17T19:53:52.107 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/34' 2024-04-17T19:53:52.107 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/38' 2024-04-17T19:53:52.107 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/11' 2024-04-17T19:53:52.107 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/16' 2024-04-17T19:53:52.107 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/32' 2024-04-17T19:53:52.107 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/30' 2024-04-17T19:53:52.107 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/14' 2024-04-17T19:53:52.108 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/18' 2024-04-17T19:53:52.108 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/13' 2024-04-17T19:53:52.108 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/33' 2024-04-17T19:53:52.108 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/17' 2024-04-17T19:53:52.108 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/35' 2024-04-17T19:53:52.108 INFO:tasks.workunit.client.0.smithi029.stdout:removed './120/37' 2024-04-17T19:53:52.108 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './120' 2024-04-17T19:53:52.108 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/19' 2024-04-17T19:53:52.108 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/15' 2024-04-17T19:53:52.108 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/31' 2024-04-17T19:53:52.108 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/36' 2024-04-17T19:53:52.108 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/20' 2024-04-17T19:53:52.108 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/12' 2024-04-17T19:53:52.108 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/39' 2024-04-17T19:53:52.108 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/10' 2024-04-17T19:53:52.108 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/34' 2024-04-17T19:53:52.108 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/38' 2024-04-17T19:53:52.109 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/11' 2024-04-17T19:53:52.109 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/16' 2024-04-17T19:53:52.109 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/32' 2024-04-17T19:53:52.109 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/30' 2024-04-17T19:53:52.109 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/14' 2024-04-17T19:53:52.109 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/18' 2024-04-17T19:53:52.109 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/13' 2024-04-17T19:53:52.109 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/33' 2024-04-17T19:53:52.109 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/17' 2024-04-17T19:53:52.109 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/35' 2024-04-17T19:53:52.109 INFO:tasks.workunit.client.0.smithi029.stdout:removed './121/37' 2024-04-17T19:53:52.109 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './121' 2024-04-17T19:53:52.109 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/19' 2024-04-17T19:53:52.109 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/15' 2024-04-17T19:53:52.109 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/31' 2024-04-17T19:53:52.109 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/36' 2024-04-17T19:53:52.110 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/20' 2024-04-17T19:53:52.110 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/12' 2024-04-17T19:53:52.110 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/39' 2024-04-17T19:53:52.110 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/10' 2024-04-17T19:53:52.110 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/34' 2024-04-17T19:53:52.110 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/38' 2024-04-17T19:53:52.110 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/11' 2024-04-17T19:53:52.110 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/16' 2024-04-17T19:53:52.110 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/32' 2024-04-17T19:53:52.110 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/30' 2024-04-17T19:53:52.110 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/14' 2024-04-17T19:53:52.110 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/18' 2024-04-17T19:53:52.110 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/13' 2024-04-17T19:53:52.110 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/33' 2024-04-17T19:53:52.110 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/17' 2024-04-17T19:53:52.111 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/35' 2024-04-17T19:53:52.111 INFO:tasks.workunit.client.0.smithi029.stdout:removed './122/37' 2024-04-17T19:53:52.111 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './122' 2024-04-17T19:53:52.111 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/19' 2024-04-17T19:53:52.111 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/15' 2024-04-17T19:53:52.111 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/31' 2024-04-17T19:53:52.111 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/36' 2024-04-17T19:53:52.111 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/20' 2024-04-17T19:53:52.111 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/12' 2024-04-17T19:53:52.111 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/39' 2024-04-17T19:53:52.111 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/10' 2024-04-17T19:53:52.111 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/34' 2024-04-17T19:53:52.111 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/38' 2024-04-17T19:53:52.111 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/11' 2024-04-17T19:53:52.111 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/16' 2024-04-17T19:53:52.111 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/32' 2024-04-17T19:53:52.111 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/30' 2024-04-17T19:53:52.112 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/14' 2024-04-17T19:53:52.112 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/18' 2024-04-17T19:53:52.113 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/13' 2024-04-17T19:53:52.113 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/33' 2024-04-17T19:53:52.113 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/17' 2024-04-17T19:53:52.113 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/35' 2024-04-17T19:53:52.113 INFO:tasks.workunit.client.0.smithi029.stdout:removed './123/37' 2024-04-17T19:53:52.113 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './123' 2024-04-17T19:53:52.113 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/19' 2024-04-17T19:53:52.113 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/15' 2024-04-17T19:53:52.113 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/31' 2024-04-17T19:53:52.113 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/36' 2024-04-17T19:53:52.113 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/20' 2024-04-17T19:53:52.114 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/12' 2024-04-17T19:53:52.114 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/39' 2024-04-17T19:53:52.114 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/10' 2024-04-17T19:53:52.114 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/34' 2024-04-17T19:53:52.114 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/38' 2024-04-17T19:53:52.114 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/11' 2024-04-17T19:53:52.114 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/16' 2024-04-17T19:53:52.114 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/32' 2024-04-17T19:53:52.114 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/30' 2024-04-17T19:53:52.114 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/14' 2024-04-17T19:53:52.114 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/18' 2024-04-17T19:53:52.114 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/13' 2024-04-17T19:53:52.114 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/33' 2024-04-17T19:53:52.115 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/17' 2024-04-17T19:53:52.115 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/35' 2024-04-17T19:53:52.115 INFO:tasks.workunit.client.0.smithi029.stdout:removed './124/37' 2024-04-17T19:53:52.115 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './124' 2024-04-17T19:53:52.115 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/19' 2024-04-17T19:53:52.115 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/15' 2024-04-17T19:53:52.115 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/31' 2024-04-17T19:53:52.115 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/36' 2024-04-17T19:53:52.115 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/20' 2024-04-17T19:53:52.115 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/12' 2024-04-17T19:53:52.115 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/39' 2024-04-17T19:53:52.115 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/10' 2024-04-17T19:53:52.115 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/34' 2024-04-17T19:53:52.115 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/38' 2024-04-17T19:53:52.115 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/11' 2024-04-17T19:53:52.116 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/16' 2024-04-17T19:53:52.116 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/32' 2024-04-17T19:53:52.116 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/30' 2024-04-17T19:53:52.116 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/14' 2024-04-17T19:53:52.116 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/18' 2024-04-17T19:53:52.116 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/13' 2024-04-17T19:53:52.116 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/33' 2024-04-17T19:53:52.116 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/17' 2024-04-17T19:53:52.116 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/35' 2024-04-17T19:53:52.116 INFO:tasks.workunit.client.0.smithi029.stdout:removed './125/37' 2024-04-17T19:53:52.116 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './125' 2024-04-17T19:53:52.116 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/19' 2024-04-17T19:53:52.116 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/15' 2024-04-17T19:53:52.116 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/31' 2024-04-17T19:53:52.116 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/36' 2024-04-17T19:53:52.117 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/20' 2024-04-17T19:53:52.117 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/12' 2024-04-17T19:53:52.117 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/39' 2024-04-17T19:53:52.117 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/10' 2024-04-17T19:53:52.117 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/34' 2024-04-17T19:53:52.117 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/38' 2024-04-17T19:53:52.117 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/11' 2024-04-17T19:53:52.117 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/16' 2024-04-17T19:53:52.117 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/32' 2024-04-17T19:53:52.117 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/30' 2024-04-17T19:53:52.117 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/14' 2024-04-17T19:53:52.117 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/18' 2024-04-17T19:53:52.117 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/13' 2024-04-17T19:53:52.117 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/33' 2024-04-17T19:53:52.117 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/17' 2024-04-17T19:53:52.118 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/35' 2024-04-17T19:53:52.118 INFO:tasks.workunit.client.0.smithi029.stdout:removed './126/37' 2024-04-17T19:53:52.118 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './126' 2024-04-17T19:53:52.118 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/19' 2024-04-17T19:53:52.118 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/15' 2024-04-17T19:53:52.118 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/31' 2024-04-17T19:53:52.118 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/36' 2024-04-17T19:53:52.118 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/20' 2024-04-17T19:53:52.118 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/12' 2024-04-17T19:53:52.118 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/39' 2024-04-17T19:53:52.118 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/10' 2024-04-17T19:53:52.118 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/34' 2024-04-17T19:53:52.118 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/38' 2024-04-17T19:53:52.118 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/11' 2024-04-17T19:53:52.118 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/16' 2024-04-17T19:53:52.119 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/32' 2024-04-17T19:53:52.119 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/30' 2024-04-17T19:53:52.119 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/14' 2024-04-17T19:53:52.119 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/18' 2024-04-17T19:53:52.119 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/13' 2024-04-17T19:53:52.119 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/33' 2024-04-17T19:53:52.119 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/17' 2024-04-17T19:53:52.119 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/35' 2024-04-17T19:53:52.119 INFO:tasks.workunit.client.0.smithi029.stdout:removed './127/37' 2024-04-17T19:53:52.119 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './127' 2024-04-17T19:53:52.119 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/19' 2024-04-17T19:53:52.119 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/15' 2024-04-17T19:53:52.119 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/31' 2024-04-17T19:53:52.119 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/36' 2024-04-17T19:53:52.120 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/20' 2024-04-17T19:53:52.120 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/12' 2024-04-17T19:53:52.120 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/39' 2024-04-17T19:53:52.120 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/10' 2024-04-17T19:53:52.120 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/34' 2024-04-17T19:53:52.120 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/38' 2024-04-17T19:53:52.120 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/11' 2024-04-17T19:53:52.120 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/16' 2024-04-17T19:53:52.120 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/32' 2024-04-17T19:53:52.120 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/30' 2024-04-17T19:53:52.120 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/14' 2024-04-17T19:53:52.120 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/18' 2024-04-17T19:53:52.120 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/13' 2024-04-17T19:53:52.120 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/33' 2024-04-17T19:53:52.121 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/17' 2024-04-17T19:53:52.121 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/35' 2024-04-17T19:53:54.394 INFO:tasks.workunit.client.0.smithi029.stdout:removed './128/37' 2024-04-17T19:53:54.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './128' 2024-04-17T19:53:54.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/19' 2024-04-17T19:53:54.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/15' 2024-04-17T19:53:54.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/31' 2024-04-17T19:53:54.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/36' 2024-04-17T19:53:54.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/20' 2024-04-17T19:53:54.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/12' 2024-04-17T19:53:54.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/39' 2024-04-17T19:53:54.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/10' 2024-04-17T19:53:54.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/34' 2024-04-17T19:53:54.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/38' 2024-04-17T19:53:54.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/11' 2024-04-17T19:53:54.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/16' 2024-04-17T19:53:54.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/32' 2024-04-17T19:53:54.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/30' 2024-04-17T19:53:54.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/14' 2024-04-17T19:53:54.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/18' 2024-04-17T19:53:54.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/13' 2024-04-17T19:53:54.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/33' 2024-04-17T19:53:54.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/17' 2024-04-17T19:53:54.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/35' 2024-04-17T19:53:54.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './129/37' 2024-04-17T19:53:54.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './129' 2024-04-17T19:53:54.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/19' 2024-04-17T19:53:54.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/15' 2024-04-17T19:53:54.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/31' 2024-04-17T19:53:54.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/36' 2024-04-17T19:53:54.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/20' 2024-04-17T19:53:54.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/12' 2024-04-17T19:53:54.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/39' 2024-04-17T19:53:54.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/10' 2024-04-17T19:53:54.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/34' 2024-04-17T19:53:54.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/38' 2024-04-17T19:53:54.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/11' 2024-04-17T19:53:54.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/16' 2024-04-17T19:53:54.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/32' 2024-04-17T19:53:54.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/30' 2024-04-17T19:53:54.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/14' 2024-04-17T19:53:54.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/18' 2024-04-17T19:53:54.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/13' 2024-04-17T19:53:54.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/33' 2024-04-17T19:53:54.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/17' 2024-04-17T19:53:54.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/35' 2024-04-17T19:53:54.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './130/37' 2024-04-17T19:53:54.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './130' 2024-04-17T19:53:54.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/19' 2024-04-17T19:53:54.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/15' 2024-04-17T19:53:54.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/31' 2024-04-17T19:53:54.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/36' 2024-04-17T19:53:54.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/20' 2024-04-17T19:53:54.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/12' 2024-04-17T19:53:54.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/39' 2024-04-17T19:53:54.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/10' 2024-04-17T19:53:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/34' 2024-04-17T19:53:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/38' 2024-04-17T19:53:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/11' 2024-04-17T19:53:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/16' 2024-04-17T19:53:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/32' 2024-04-17T19:53:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/30' 2024-04-17T19:53:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/14' 2024-04-17T19:53:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/18' 2024-04-17T19:53:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/13' 2024-04-17T19:53:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/33' 2024-04-17T19:53:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/17' 2024-04-17T19:53:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/35' 2024-04-17T19:53:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './131/37' 2024-04-17T19:53:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './131' 2024-04-17T19:53:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/19' 2024-04-17T19:53:54.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/15' 2024-04-17T19:53:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/31' 2024-04-17T19:53:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/36' 2024-04-17T19:53:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/20' 2024-04-17T19:53:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/12' 2024-04-17T19:53:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/39' 2024-04-17T19:53:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/10' 2024-04-17T19:53:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/34' 2024-04-17T19:53:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/38' 2024-04-17T19:53:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/11' 2024-04-17T19:53:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/16' 2024-04-17T19:53:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/32' 2024-04-17T19:53:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/30' 2024-04-17T19:53:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/14' 2024-04-17T19:53:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/18' 2024-04-17T19:53:54.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/13' 2024-04-17T19:53:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/33' 2024-04-17T19:53:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/17' 2024-04-17T19:53:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/35' 2024-04-17T19:53:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './132/37' 2024-04-17T19:53:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './132' 2024-04-17T19:53:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/19' 2024-04-17T19:53:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/15' 2024-04-17T19:53:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/31' 2024-04-17T19:53:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/36' 2024-04-17T19:53:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/20' 2024-04-17T19:53:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/12' 2024-04-17T19:53:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/39' 2024-04-17T19:53:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/10' 2024-04-17T19:53:54.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/34' 2024-04-17T19:53:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/38' 2024-04-17T19:53:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/11' 2024-04-17T19:53:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/16' 2024-04-17T19:53:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/32' 2024-04-17T19:53:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/30' 2024-04-17T19:53:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/14' 2024-04-17T19:53:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/18' 2024-04-17T19:53:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/13' 2024-04-17T19:53:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/33' 2024-04-17T19:53:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/17' 2024-04-17T19:53:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/35' 2024-04-17T19:53:54.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './133/37' 2024-04-17T19:53:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './133' 2024-04-17T19:53:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/19' 2024-04-17T19:53:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/15' 2024-04-17T19:53:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/31' 2024-04-17T19:53:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/36' 2024-04-17T19:53:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/20' 2024-04-17T19:53:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/12' 2024-04-17T19:53:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/39' 2024-04-17T19:53:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/10' 2024-04-17T19:53:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/34' 2024-04-17T19:53:54.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/38' 2024-04-17T19:53:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/11' 2024-04-17T19:53:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/16' 2024-04-17T19:53:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/32' 2024-04-17T19:53:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/30' 2024-04-17T19:53:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/14' 2024-04-17T19:53:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/18' 2024-04-17T19:53:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/13' 2024-04-17T19:53:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/33' 2024-04-17T19:53:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/17' 2024-04-17T19:53:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/35' 2024-04-17T19:53:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './134/37' 2024-04-17T19:53:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './134' 2024-04-17T19:53:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/19' 2024-04-17T19:53:54.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/15' 2024-04-17T19:53:54.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/31' 2024-04-17T19:53:54.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/36' 2024-04-17T19:53:54.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/20' 2024-04-17T19:53:54.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/12' 2024-04-17T19:53:54.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/39' 2024-04-17T19:53:54.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/10' 2024-04-17T19:53:54.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/34' 2024-04-17T19:53:54.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/38' 2024-04-17T19:53:54.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/11' 2024-04-17T19:53:54.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/16' 2024-04-17T19:53:54.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/32' 2024-04-17T19:53:54.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/30' 2024-04-17T19:53:54.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/14' 2024-04-17T19:53:54.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/18' 2024-04-17T19:53:54.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/13' 2024-04-17T19:53:54.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/33' 2024-04-17T19:53:54.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/17' 2024-04-17T19:53:54.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/35' 2024-04-17T19:53:54.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './135/37' 2024-04-17T19:53:54.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './135' 2024-04-17T19:53:54.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/19' 2024-04-17T19:53:54.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/15' 2024-04-17T19:53:54.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/31' 2024-04-17T19:53:54.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/36' 2024-04-17T19:53:54.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/20' 2024-04-17T19:53:54.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/12' 2024-04-17T19:53:54.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/39' 2024-04-17T19:53:54.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/10' 2024-04-17T19:53:54.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/34' 2024-04-17T19:53:54.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/38' 2024-04-17T19:53:54.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/11' 2024-04-17T19:53:54.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/16' 2024-04-17T19:53:54.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/32' 2024-04-17T19:53:54.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/30' 2024-04-17T19:53:54.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/14' 2024-04-17T19:53:54.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/18' 2024-04-17T19:53:54.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/13' 2024-04-17T19:53:54.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/33' 2024-04-17T19:53:54.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/17' 2024-04-17T19:53:54.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/35' 2024-04-17T19:53:54.408 INFO:tasks.workunit.client.0.smithi029.stdout:removed './136/37' 2024-04-17T19:53:54.408 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './136' 2024-04-17T19:53:54.408 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/19' 2024-04-17T19:53:54.408 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/15' 2024-04-17T19:53:54.408 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/31' 2024-04-17T19:53:54.408 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/36' 2024-04-17T19:53:54.408 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/20' 2024-04-17T19:53:54.408 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/12' 2024-04-17T19:53:54.408 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/39' 2024-04-17T19:53:54.408 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/10' 2024-04-17T19:53:54.408 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/34' 2024-04-17T19:53:54.408 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/38' 2024-04-17T19:53:54.408 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/11' 2024-04-17T19:53:54.408 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/16' 2024-04-17T19:53:54.408 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/32' 2024-04-17T19:53:54.409 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/30' 2024-04-17T19:53:54.409 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/14' 2024-04-17T19:53:54.409 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/18' 2024-04-17T19:53:54.409 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/13' 2024-04-17T19:53:54.409 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/33' 2024-04-17T19:53:54.409 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/17' 2024-04-17T19:53:54.409 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/35' 2024-04-17T19:53:54.409 INFO:tasks.workunit.client.0.smithi029.stdout:removed './137/37' 2024-04-17T19:53:54.409 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './137' 2024-04-17T19:53:54.409 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/19' 2024-04-17T19:53:54.409 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/15' 2024-04-17T19:53:54.409 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/31' 2024-04-17T19:53:54.409 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/36' 2024-04-17T19:53:54.409 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/20' 2024-04-17T19:53:54.409 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/12' 2024-04-17T19:53:54.410 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/39' 2024-04-17T19:53:54.410 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/10' 2024-04-17T19:53:54.410 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/34' 2024-04-17T19:53:54.410 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/38' 2024-04-17T19:53:54.410 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/11' 2024-04-17T19:53:54.410 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/16' 2024-04-17T19:53:56.561 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/32' 2024-04-17T19:53:56.562 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/30' 2024-04-17T19:53:56.562 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/14' 2024-04-17T19:53:56.562 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/18' 2024-04-17T19:53:56.562 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/13' 2024-04-17T19:53:56.562 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/33' 2024-04-17T19:53:56.562 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/17' 2024-04-17T19:53:56.562 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/35' 2024-04-17T19:53:56.562 INFO:tasks.workunit.client.0.smithi029.stdout:removed './138/37' 2024-04-17T19:53:56.562 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './138' 2024-04-17T19:53:56.562 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/19' 2024-04-17T19:53:56.562 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/15' 2024-04-17T19:53:56.562 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/31' 2024-04-17T19:53:56.562 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/36' 2024-04-17T19:53:56.563 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/20' 2024-04-17T19:53:56.563 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/12' 2024-04-17T19:53:56.563 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/39' 2024-04-17T19:53:56.563 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/10' 2024-04-17T19:53:56.563 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/34' 2024-04-17T19:53:56.563 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/38' 2024-04-17T19:53:56.563 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/11' 2024-04-17T19:53:56.563 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/16' 2024-04-17T19:53:56.563 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/32' 2024-04-17T19:53:56.563 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/30' 2024-04-17T19:53:56.563 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/14' 2024-04-17T19:53:56.563 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/18' 2024-04-17T19:53:56.563 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/13' 2024-04-17T19:53:56.563 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/33' 2024-04-17T19:53:56.563 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/17' 2024-04-17T19:53:56.564 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/35' 2024-04-17T19:53:56.564 INFO:tasks.workunit.client.0.smithi029.stdout:removed './139/37' 2024-04-17T19:53:56.564 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './139' 2024-04-17T19:53:56.564 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/19' 2024-04-17T19:53:56.564 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/15' 2024-04-17T19:53:56.564 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/31' 2024-04-17T19:53:56.564 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/36' 2024-04-17T19:53:56.564 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/20' 2024-04-17T19:53:56.564 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/12' 2024-04-17T19:53:56.564 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/39' 2024-04-17T19:53:56.564 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/10' 2024-04-17T19:53:56.564 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/34' 2024-04-17T19:53:56.564 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/38' 2024-04-17T19:53:56.564 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/11' 2024-04-17T19:53:56.564 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/16' 2024-04-17T19:53:56.564 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/32' 2024-04-17T19:53:56.565 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/30' 2024-04-17T19:53:56.565 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/14' 2024-04-17T19:53:56.565 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/18' 2024-04-17T19:53:56.565 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/13' 2024-04-17T19:53:56.565 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/33' 2024-04-17T19:53:56.565 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/17' 2024-04-17T19:53:56.565 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/35' 2024-04-17T19:53:56.565 INFO:tasks.workunit.client.0.smithi029.stdout:removed './140/37' 2024-04-17T19:53:56.565 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './140' 2024-04-17T19:53:56.565 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/19' 2024-04-17T19:53:56.565 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/15' 2024-04-17T19:53:56.565 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/31' 2024-04-17T19:53:56.565 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/36' 2024-04-17T19:53:56.565 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/20' 2024-04-17T19:53:56.565 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/12' 2024-04-17T19:53:56.565 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/39' 2024-04-17T19:53:56.566 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/10' 2024-04-17T19:53:56.566 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/34' 2024-04-17T19:53:56.566 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/38' 2024-04-17T19:53:56.566 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/11' 2024-04-17T19:53:56.566 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/16' 2024-04-17T19:53:56.566 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/32' 2024-04-17T19:53:56.566 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/30' 2024-04-17T19:53:56.566 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/14' 2024-04-17T19:53:56.566 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/18' 2024-04-17T19:53:56.566 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/13' 2024-04-17T19:53:56.566 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/33' 2024-04-17T19:53:56.566 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/17' 2024-04-17T19:53:56.566 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/35' 2024-04-17T19:53:56.566 INFO:tasks.workunit.client.0.smithi029.stdout:removed './141/37' 2024-04-17T19:53:56.567 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './141' 2024-04-17T19:53:56.567 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/19' 2024-04-17T19:53:56.567 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/15' 2024-04-17T19:53:56.567 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/31' 2024-04-17T19:53:56.567 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/36' 2024-04-17T19:53:56.567 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/20' 2024-04-17T19:53:56.567 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/12' 2024-04-17T19:53:56.567 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/39' 2024-04-17T19:53:56.567 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/10' 2024-04-17T19:53:56.567 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/34' 2024-04-17T19:53:56.567 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/38' 2024-04-17T19:53:56.567 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/11' 2024-04-17T19:53:56.567 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/16' 2024-04-17T19:53:56.567 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/32' 2024-04-17T19:53:56.567 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/30' 2024-04-17T19:53:56.567 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/14' 2024-04-17T19:53:56.568 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/18' 2024-04-17T19:53:56.568 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/13' 2024-04-17T19:53:56.568 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/33' 2024-04-17T19:53:56.568 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/17' 2024-04-17T19:53:56.568 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/35' 2024-04-17T19:53:56.568 INFO:tasks.workunit.client.0.smithi029.stdout:removed './142/37' 2024-04-17T19:53:56.568 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './142' 2024-04-17T19:53:56.568 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/19' 2024-04-17T19:53:56.568 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/15' 2024-04-17T19:53:56.568 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/31' 2024-04-17T19:53:56.568 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/36' 2024-04-17T19:53:56.568 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/20' 2024-04-17T19:53:56.568 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/12' 2024-04-17T19:53:56.568 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/39' 2024-04-17T19:53:56.568 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/10' 2024-04-17T19:53:56.568 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/34' 2024-04-17T19:53:56.569 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/38' 2024-04-17T19:53:56.569 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/11' 2024-04-17T19:53:56.569 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/16' 2024-04-17T19:53:56.569 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/32' 2024-04-17T19:53:56.569 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/30' 2024-04-17T19:53:56.569 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/14' 2024-04-17T19:53:56.569 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/18' 2024-04-17T19:53:56.569 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/13' 2024-04-17T19:53:56.569 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/33' 2024-04-17T19:53:56.569 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/17' 2024-04-17T19:53:56.569 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/35' 2024-04-17T19:53:56.569 INFO:tasks.workunit.client.0.smithi029.stdout:removed './143/37' 2024-04-17T19:53:56.569 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './143' 2024-04-17T19:53:56.570 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/19' 2024-04-17T19:53:56.570 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/15' 2024-04-17T19:53:56.570 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/31' 2024-04-17T19:53:56.570 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/36' 2024-04-17T19:53:56.570 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/20' 2024-04-17T19:53:56.570 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/12' 2024-04-17T19:53:56.570 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/39' 2024-04-17T19:53:56.570 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/10' 2024-04-17T19:53:56.570 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/34' 2024-04-17T19:53:56.570 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/38' 2024-04-17T19:53:56.570 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/11' 2024-04-17T19:53:56.570 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/16' 2024-04-17T19:53:56.570 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/32' 2024-04-17T19:53:56.570 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/30' 2024-04-17T19:53:56.570 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/14' 2024-04-17T19:53:56.570 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/18' 2024-04-17T19:53:56.571 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/13' 2024-04-17T19:53:56.571 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/33' 2024-04-17T19:53:56.571 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/17' 2024-04-17T19:53:56.571 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/35' 2024-04-17T19:53:56.571 INFO:tasks.workunit.client.0.smithi029.stdout:removed './144/37' 2024-04-17T19:53:56.571 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './144' 2024-04-17T19:53:56.571 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/19' 2024-04-17T19:53:56.571 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/15' 2024-04-17T19:53:56.571 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/31' 2024-04-17T19:53:56.571 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/36' 2024-04-17T19:53:56.571 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/20' 2024-04-17T19:53:56.571 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/12' 2024-04-17T19:53:56.571 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/39' 2024-04-17T19:53:56.571 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/10' 2024-04-17T19:53:56.571 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/34' 2024-04-17T19:53:56.571 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/38' 2024-04-17T19:53:56.571 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/11' 2024-04-17T19:53:56.572 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/16' 2024-04-17T19:53:56.572 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/32' 2024-04-17T19:53:56.572 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/30' 2024-04-17T19:53:56.572 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/14' 2024-04-17T19:53:56.572 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/18' 2024-04-17T19:53:56.572 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/13' 2024-04-17T19:53:56.572 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/33' 2024-04-17T19:53:56.572 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/17' 2024-04-17T19:53:56.572 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/35' 2024-04-17T19:53:56.572 INFO:tasks.workunit.client.0.smithi029.stdout:removed './145/37' 2024-04-17T19:53:56.572 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './145' 2024-04-17T19:53:56.572 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/19' 2024-04-17T19:53:56.572 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/15' 2024-04-17T19:53:56.572 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/31' 2024-04-17T19:53:56.572 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/36' 2024-04-17T19:53:56.572 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/20' 2024-04-17T19:53:56.573 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/12' 2024-04-17T19:53:56.573 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/39' 2024-04-17T19:53:56.573 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/10' 2024-04-17T19:53:56.573 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/34' 2024-04-17T19:53:56.573 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/38' 2024-04-17T19:53:56.573 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/11' 2024-04-17T19:53:56.573 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/16' 2024-04-17T19:53:56.573 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/32' 2024-04-17T19:53:56.573 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/30' 2024-04-17T19:53:56.573 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/14' 2024-04-17T19:53:56.573 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/18' 2024-04-17T19:53:56.573 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/13' 2024-04-17T19:53:56.573 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/33' 2024-04-17T19:53:56.573 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/17' 2024-04-17T19:53:56.573 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/35' 2024-04-17T19:53:56.573 INFO:tasks.workunit.client.0.smithi029.stdout:removed './146/37' 2024-04-17T19:53:56.574 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './146' 2024-04-17T19:53:56.574 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/19' 2024-04-17T19:53:56.574 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/15' 2024-04-17T19:53:56.574 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/31' 2024-04-17T19:53:56.574 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/36' 2024-04-17T19:53:56.574 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/20' 2024-04-17T19:53:56.574 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/12' 2024-04-17T19:53:56.574 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/39' 2024-04-17T19:53:56.574 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/10' 2024-04-17T19:53:56.574 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/34' 2024-04-17T19:53:56.574 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/38' 2024-04-17T19:53:56.574 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/11' 2024-04-17T19:53:56.574 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/16' 2024-04-17T19:53:56.574 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/32' 2024-04-17T19:53:56.575 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/30' 2024-04-17T19:53:56.575 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/14' 2024-04-17T19:53:56.575 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/18' 2024-04-17T19:53:56.575 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/13' 2024-04-17T19:53:56.575 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/33' 2024-04-17T19:53:56.575 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/17' 2024-04-17T19:53:56.575 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/35' 2024-04-17T19:53:56.575 INFO:tasks.workunit.client.0.smithi029.stdout:removed './147/37' 2024-04-17T19:53:56.575 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './147' 2024-04-17T19:53:56.575 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/19' 2024-04-17T19:53:56.575 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/15' 2024-04-17T19:53:56.575 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/31' 2024-04-17T19:53:56.575 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/36' 2024-04-17T19:53:58.280 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/20' 2024-04-17T19:53:58.281 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/12' 2024-04-17T19:53:58.281 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/39' 2024-04-17T19:53:58.281 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/10' 2024-04-17T19:53:58.281 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/34' 2024-04-17T19:53:58.281 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/38' 2024-04-17T19:53:58.281 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/11' 2024-04-17T19:53:58.281 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/16' 2024-04-17T19:53:58.281 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/32' 2024-04-17T19:53:58.281 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/30' 2024-04-17T19:53:58.281 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/14' 2024-04-17T19:53:58.281 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/18' 2024-04-17T19:53:58.281 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/13' 2024-04-17T19:53:58.282 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/33' 2024-04-17T19:53:58.282 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/17' 2024-04-17T19:53:58.282 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/35' 2024-04-17T19:53:58.282 INFO:tasks.workunit.client.0.smithi029.stdout:removed './148/37' 2024-04-17T19:53:58.282 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './148' 2024-04-17T19:53:58.282 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/19' 2024-04-17T19:53:58.282 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/15' 2024-04-17T19:53:58.282 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/31' 2024-04-17T19:53:58.282 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/36' 2024-04-17T19:53:58.282 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/20' 2024-04-17T19:53:58.282 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/12' 2024-04-17T19:53:58.282 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/39' 2024-04-17T19:53:58.282 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/10' 2024-04-17T19:53:58.283 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/34' 2024-04-17T19:53:58.283 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/38' 2024-04-17T19:53:58.283 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/11' 2024-04-17T19:53:58.283 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/16' 2024-04-17T19:53:58.283 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/32' 2024-04-17T19:53:58.283 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/30' 2024-04-17T19:53:58.283 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/14' 2024-04-17T19:53:58.283 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/18' 2024-04-17T19:53:58.283 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/13' 2024-04-17T19:53:58.283 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/33' 2024-04-17T19:53:58.283 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/17' 2024-04-17T19:53:58.283 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/35' 2024-04-17T19:53:58.283 INFO:tasks.workunit.client.0.smithi029.stdout:removed './149/37' 2024-04-17T19:53:58.283 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './149' 2024-04-17T19:53:58.283 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/19' 2024-04-17T19:53:58.283 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/15' 2024-04-17T19:53:58.284 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/31' 2024-04-17T19:53:58.284 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/36' 2024-04-17T19:53:58.284 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/20' 2024-04-17T19:53:58.284 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/12' 2024-04-17T19:53:58.284 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/39' 2024-04-17T19:53:58.284 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/10' 2024-04-17T19:53:58.284 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/34' 2024-04-17T19:53:58.284 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/38' 2024-04-17T19:53:58.284 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/11' 2024-04-17T19:53:58.284 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/16' 2024-04-17T19:53:58.284 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/32' 2024-04-17T19:53:58.284 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/30' 2024-04-17T19:53:58.284 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/14' 2024-04-17T19:53:58.284 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/18' 2024-04-17T19:53:58.284 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/13' 2024-04-17T19:53:58.284 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/33' 2024-04-17T19:53:58.285 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/17' 2024-04-17T19:53:58.285 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/35' 2024-04-17T19:53:58.285 INFO:tasks.workunit.client.0.smithi029.stdout:removed './150/37' 2024-04-17T19:53:58.285 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './150' 2024-04-17T19:53:58.285 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/19' 2024-04-17T19:53:58.285 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/15' 2024-04-17T19:53:58.285 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/31' 2024-04-17T19:53:58.285 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/36' 2024-04-17T19:53:58.285 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/20' 2024-04-17T19:53:58.285 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/12' 2024-04-17T19:53:58.285 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/39' 2024-04-17T19:53:58.285 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/10' 2024-04-17T19:53:58.285 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/34' 2024-04-17T19:53:58.285 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/38' 2024-04-17T19:53:58.285 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/11' 2024-04-17T19:53:58.285 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/16' 2024-04-17T19:53:58.286 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/32' 2024-04-17T19:53:58.286 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/30' 2024-04-17T19:53:58.286 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/14' 2024-04-17T19:53:58.286 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/18' 2024-04-17T19:53:58.286 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/13' 2024-04-17T19:53:58.286 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/33' 2024-04-17T19:53:58.286 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/17' 2024-04-17T19:53:58.286 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/35' 2024-04-17T19:53:58.286 INFO:tasks.workunit.client.0.smithi029.stdout:removed './151/37' 2024-04-17T19:53:58.286 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './151' 2024-04-17T19:53:58.286 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/19' 2024-04-17T19:53:58.286 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/15' 2024-04-17T19:53:58.286 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/31' 2024-04-17T19:53:58.286 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/36' 2024-04-17T19:53:58.286 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/20' 2024-04-17T19:53:58.286 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/12' 2024-04-17T19:53:58.287 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/39' 2024-04-17T19:53:58.287 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/10' 2024-04-17T19:53:58.287 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/34' 2024-04-17T19:53:58.287 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/38' 2024-04-17T19:53:58.287 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/11' 2024-04-17T19:53:58.287 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/16' 2024-04-17T19:53:58.287 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/32' 2024-04-17T19:53:58.287 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/30' 2024-04-17T19:53:58.287 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/14' 2024-04-17T19:53:58.287 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/18' 2024-04-17T19:53:58.287 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/13' 2024-04-17T19:53:58.287 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/33' 2024-04-17T19:53:58.287 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/17' 2024-04-17T19:53:58.287 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/35' 2024-04-17T19:53:58.287 INFO:tasks.workunit.client.0.smithi029.stdout:removed './152/37' 2024-04-17T19:53:58.287 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './152' 2024-04-17T19:53:58.288 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/19' 2024-04-17T19:53:58.288 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/15' 2024-04-17T19:53:58.288 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/31' 2024-04-17T19:53:58.288 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/36' 2024-04-17T19:53:58.288 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/20' 2024-04-17T19:53:58.288 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/12' 2024-04-17T19:53:58.288 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/39' 2024-04-17T19:53:58.288 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/10' 2024-04-17T19:53:58.288 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/34' 2024-04-17T19:53:58.288 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/38' 2024-04-17T19:53:58.288 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/11' 2024-04-17T19:53:58.288 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/16' 2024-04-17T19:53:58.288 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/32' 2024-04-17T19:53:58.288 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/30' 2024-04-17T19:53:58.288 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/14' 2024-04-17T19:53:58.289 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/18' 2024-04-17T19:53:58.289 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/13' 2024-04-17T19:53:58.289 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/33' 2024-04-17T19:53:58.289 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/17' 2024-04-17T19:53:58.289 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/35' 2024-04-17T19:53:58.289 INFO:tasks.workunit.client.0.smithi029.stdout:removed './153/37' 2024-04-17T19:53:58.289 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './153' 2024-04-17T19:53:58.289 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/19' 2024-04-17T19:53:58.289 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/15' 2024-04-17T19:53:58.289 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/31' 2024-04-17T19:53:58.289 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/36' 2024-04-17T19:53:58.289 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/20' 2024-04-17T19:53:58.289 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/12' 2024-04-17T19:53:58.289 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/39' 2024-04-17T19:53:58.289 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/10' 2024-04-17T19:53:58.289 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/34' 2024-04-17T19:53:58.289 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/38' 2024-04-17T19:53:58.290 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/11' 2024-04-17T19:53:58.290 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/16' 2024-04-17T19:53:58.290 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/32' 2024-04-17T19:53:58.290 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/30' 2024-04-17T19:53:58.290 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/14' 2024-04-17T19:53:58.290 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/18' 2024-04-17T19:53:58.290 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/13' 2024-04-17T19:53:58.290 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/33' 2024-04-17T19:53:58.290 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/17' 2024-04-17T19:53:58.290 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/35' 2024-04-17T19:53:58.290 INFO:tasks.workunit.client.0.smithi029.stdout:removed './154/37' 2024-04-17T19:53:58.290 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './154' 2024-04-17T19:53:58.290 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/19' 2024-04-17T19:53:58.290 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/15' 2024-04-17T19:53:58.290 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/31' 2024-04-17T19:53:58.291 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/36' 2024-04-17T19:53:58.291 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/20' 2024-04-17T19:53:58.291 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/12' 2024-04-17T19:53:58.291 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/39' 2024-04-17T19:53:58.291 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/10' 2024-04-17T19:53:58.291 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/34' 2024-04-17T19:53:58.291 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/38' 2024-04-17T19:53:58.291 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/11' 2024-04-17T19:53:58.291 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/16' 2024-04-17T19:53:58.291 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/32' 2024-04-17T19:53:58.291 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/30' 2024-04-17T19:53:58.291 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/14' 2024-04-17T19:53:58.291 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/18' 2024-04-17T19:53:58.291 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/13' 2024-04-17T19:53:58.291 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/33' 2024-04-17T19:53:58.292 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/17' 2024-04-17T19:53:58.292 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/35' 2024-04-17T19:53:58.292 INFO:tasks.workunit.client.0.smithi029.stdout:removed './155/37' 2024-04-17T19:53:58.292 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './155' 2024-04-17T19:53:58.292 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/19' 2024-04-17T19:53:58.292 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/15' 2024-04-17T19:53:58.292 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/31' 2024-04-17T19:53:58.292 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/36' 2024-04-17T19:53:58.292 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/20' 2024-04-17T19:53:58.292 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/12' 2024-04-17T19:53:58.292 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/39' 2024-04-17T19:53:58.292 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/10' 2024-04-17T19:53:58.292 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/34' 2024-04-17T19:53:58.292 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/38' 2024-04-17T19:53:58.292 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/11' 2024-04-17T19:53:58.292 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/16' 2024-04-17T19:53:58.293 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/32' 2024-04-17T19:53:58.293 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/30' 2024-04-17T19:53:58.293 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/14' 2024-04-17T19:53:58.293 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/18' 2024-04-17T19:53:58.293 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/13' 2024-04-17T19:53:58.293 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/33' 2024-04-17T19:53:58.293 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/17' 2024-04-17T19:53:58.293 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/35' 2024-04-17T19:53:58.293 INFO:tasks.workunit.client.0.smithi029.stdout:removed './156/37' 2024-04-17T19:53:58.293 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './156' 2024-04-17T19:53:58.293 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/19' 2024-04-17T19:53:58.293 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/15' 2024-04-17T19:53:58.293 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/31' 2024-04-17T19:53:58.293 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/36' 2024-04-17T19:53:58.293 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/20' 2024-04-17T19:53:58.294 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/12' 2024-04-17T19:53:58.294 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/39' 2024-04-17T19:53:58.294 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/10' 2024-04-17T19:53:58.294 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/34' 2024-04-17T19:53:58.294 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/38' 2024-04-17T19:53:58.294 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/11' 2024-04-17T19:53:58.294 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/16' 2024-04-17T19:53:58.294 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/32' 2024-04-17T19:53:58.294 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/30' 2024-04-17T19:53:58.294 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/14' 2024-04-17T19:53:58.294 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/18' 2024-04-17T19:53:58.294 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/13' 2024-04-17T19:53:58.294 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/33' 2024-04-17T19:54:02.500 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/17' 2024-04-17T19:54:02.500 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/35' 2024-04-17T19:54:02.500 INFO:tasks.workunit.client.0.smithi029.stdout:removed './157/37' 2024-04-17T19:54:02.501 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './157' 2024-04-17T19:54:02.501 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/19' 2024-04-17T19:54:02.501 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/15' 2024-04-17T19:54:02.501 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/31' 2024-04-17T19:54:02.501 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/36' 2024-04-17T19:54:02.501 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/20' 2024-04-17T19:54:02.501 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/12' 2024-04-17T19:54:02.501 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/39' 2024-04-17T19:54:02.501 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/10' 2024-04-17T19:54:02.501 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/34' 2024-04-17T19:54:02.501 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/38' 2024-04-17T19:54:02.501 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/11' 2024-04-17T19:54:02.501 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/16' 2024-04-17T19:54:02.501 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/32' 2024-04-17T19:54:02.501 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/30' 2024-04-17T19:54:02.501 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/14' 2024-04-17T19:54:02.502 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/18' 2024-04-17T19:54:02.502 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/13' 2024-04-17T19:54:02.502 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/33' 2024-04-17T19:54:02.502 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/17' 2024-04-17T19:54:02.502 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/35' 2024-04-17T19:54:02.502 INFO:tasks.workunit.client.0.smithi029.stdout:removed './158/37' 2024-04-17T19:54:02.502 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './158' 2024-04-17T19:54:02.502 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/19' 2024-04-17T19:54:02.502 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/15' 2024-04-17T19:54:02.502 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/31' 2024-04-17T19:54:02.502 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/36' 2024-04-17T19:54:02.502 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/20' 2024-04-17T19:54:02.503 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/12' 2024-04-17T19:54:02.503 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/39' 2024-04-17T19:54:02.503 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/10' 2024-04-17T19:54:02.503 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/34' 2024-04-17T19:54:02.503 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/38' 2024-04-17T19:54:02.503 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/11' 2024-04-17T19:54:02.503 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/16' 2024-04-17T19:54:02.503 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/32' 2024-04-17T19:54:02.503 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/30' 2024-04-17T19:54:02.503 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/14' 2024-04-17T19:54:02.503 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/18' 2024-04-17T19:54:02.503 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/13' 2024-04-17T19:54:02.503 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/33' 2024-04-17T19:54:02.503 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/17' 2024-04-17T19:54:02.504 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/35' 2024-04-17T19:54:02.504 INFO:tasks.workunit.client.0.smithi029.stdout:removed './159/37' 2024-04-17T19:54:02.504 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './159' 2024-04-17T19:54:02.504 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/19' 2024-04-17T19:54:02.504 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/15' 2024-04-17T19:54:02.504 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/31' 2024-04-17T19:54:02.504 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/36' 2024-04-17T19:54:02.504 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/20' 2024-04-17T19:54:02.504 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/12' 2024-04-17T19:54:02.504 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/39' 2024-04-17T19:54:02.504 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/10' 2024-04-17T19:54:02.504 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/34' 2024-04-17T19:54:02.504 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/38' 2024-04-17T19:54:02.504 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/11' 2024-04-17T19:54:02.504 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/16' 2024-04-17T19:54:02.504 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/32' 2024-04-17T19:54:02.505 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/30' 2024-04-17T19:54:02.505 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/14' 2024-04-17T19:54:02.505 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/18' 2024-04-17T19:54:02.505 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/13' 2024-04-17T19:54:02.505 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/33' 2024-04-17T19:54:02.505 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/17' 2024-04-17T19:54:02.505 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/35' 2024-04-17T19:54:02.505 INFO:tasks.workunit.client.0.smithi029.stdout:removed './160/37' 2024-04-17T19:54:02.505 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './160' 2024-04-17T19:54:02.505 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/19' 2024-04-17T19:54:02.505 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/15' 2024-04-17T19:54:02.505 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/31' 2024-04-17T19:54:02.505 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/36' 2024-04-17T19:54:02.505 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/20' 2024-04-17T19:54:02.506 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/12' 2024-04-17T19:54:02.506 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/39' 2024-04-17T19:54:02.506 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/10' 2024-04-17T19:54:02.506 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/34' 2024-04-17T19:54:02.506 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/38' 2024-04-17T19:54:02.506 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/11' 2024-04-17T19:54:02.506 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/16' 2024-04-17T19:54:02.506 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/32' 2024-04-17T19:54:02.506 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/30' 2024-04-17T19:54:02.506 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/14' 2024-04-17T19:54:02.506 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/18' 2024-04-17T19:54:02.506 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/13' 2024-04-17T19:54:02.506 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/33' 2024-04-17T19:54:02.506 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/17' 2024-04-17T19:54:02.507 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/35' 2024-04-17T19:54:02.507 INFO:tasks.workunit.client.0.smithi029.stdout:removed './161/37' 2024-04-17T19:54:02.507 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './161' 2024-04-17T19:54:02.507 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/19' 2024-04-17T19:54:02.507 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/15' 2024-04-17T19:54:02.507 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/31' 2024-04-17T19:54:02.507 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/36' 2024-04-17T19:54:02.507 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/20' 2024-04-17T19:54:02.507 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/12' 2024-04-17T19:54:02.507 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/39' 2024-04-17T19:54:02.507 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/10' 2024-04-17T19:54:02.507 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/34' 2024-04-17T19:54:02.507 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/38' 2024-04-17T19:54:02.507 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/11' 2024-04-17T19:54:02.507 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/16' 2024-04-17T19:54:02.508 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/32' 2024-04-17T19:54:02.508 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/30' 2024-04-17T19:54:02.508 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/14' 2024-04-17T19:54:02.508 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/18' 2024-04-17T19:54:02.508 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/13' 2024-04-17T19:54:02.508 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/33' 2024-04-17T19:54:02.508 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/17' 2024-04-17T19:54:02.508 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/35' 2024-04-17T19:54:02.508 INFO:tasks.workunit.client.0.smithi029.stdout:removed './162/37' 2024-04-17T19:54:02.508 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './162' 2024-04-17T19:54:02.508 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/19' 2024-04-17T19:54:02.508 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/15' 2024-04-17T19:54:02.508 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/31' 2024-04-17T19:54:02.508 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/36' 2024-04-17T19:54:02.508 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/20' 2024-04-17T19:54:02.508 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/12' 2024-04-17T19:54:02.509 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/39' 2024-04-17T19:54:02.509 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/10' 2024-04-17T19:54:02.509 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/34' 2024-04-17T19:54:02.509 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/38' 2024-04-17T19:54:02.509 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/11' 2024-04-17T19:54:02.509 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/16' 2024-04-17T19:54:02.509 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/32' 2024-04-17T19:54:02.509 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/30' 2024-04-17T19:54:02.509 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/14' 2024-04-17T19:54:02.509 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/18' 2024-04-17T19:54:02.509 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/13' 2024-04-17T19:54:02.509 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/33' 2024-04-17T19:54:02.509 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/17' 2024-04-17T19:54:02.509 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/35' 2024-04-17T19:54:02.509 INFO:tasks.workunit.client.0.smithi029.stdout:removed './163/37' 2024-04-17T19:54:02.509 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './163' 2024-04-17T19:54:02.510 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/19' 2024-04-17T19:54:02.510 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/15' 2024-04-17T19:54:02.510 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/31' 2024-04-17T19:54:02.510 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/36' 2024-04-17T19:54:02.510 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/20' 2024-04-17T19:54:02.510 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/12' 2024-04-17T19:54:02.510 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/39' 2024-04-17T19:54:02.510 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/10' 2024-04-17T19:54:02.510 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/34' 2024-04-17T19:54:02.510 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/38' 2024-04-17T19:54:02.510 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/11' 2024-04-17T19:54:02.510 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/16' 2024-04-17T19:54:02.510 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/32' 2024-04-17T19:54:02.510 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/30' 2024-04-17T19:54:02.510 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/14' 2024-04-17T19:54:02.510 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/18' 2024-04-17T19:54:02.511 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/13' 2024-04-17T19:54:02.511 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/33' 2024-04-17T19:54:02.511 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/17' 2024-04-17T19:54:02.511 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/35' 2024-04-17T19:54:02.511 INFO:tasks.workunit.client.0.smithi029.stdout:removed './164/37' 2024-04-17T19:54:02.511 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './164' 2024-04-17T19:54:02.511 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/19' 2024-04-17T19:54:02.511 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/15' 2024-04-17T19:54:02.511 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/31' 2024-04-17T19:54:02.511 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/36' 2024-04-17T19:54:02.511 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/20' 2024-04-17T19:54:02.511 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/12' 2024-04-17T19:54:02.511 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/39' 2024-04-17T19:54:02.511 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/10' 2024-04-17T19:54:02.511 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/34' 2024-04-17T19:54:02.512 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/38' 2024-04-17T19:54:02.512 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/11' 2024-04-17T19:54:02.512 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/16' 2024-04-17T19:54:02.512 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/32' 2024-04-17T19:54:02.512 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/30' 2024-04-17T19:54:02.512 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/14' 2024-04-17T19:54:02.512 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/18' 2024-04-17T19:54:02.512 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/13' 2024-04-17T19:54:02.512 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/33' 2024-04-17T19:54:02.512 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/17' 2024-04-17T19:54:02.512 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/35' 2024-04-17T19:54:02.512 INFO:tasks.workunit.client.0.smithi029.stdout:removed './165/37' 2024-04-17T19:54:02.512 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './165' 2024-04-17T19:54:02.512 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/19' 2024-04-17T19:54:02.512 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/15' 2024-04-17T19:54:02.513 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/31' 2024-04-17T19:54:02.513 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/36' 2024-04-17T19:54:02.513 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/20' 2024-04-17T19:54:02.513 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/12' 2024-04-17T19:54:02.513 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/39' 2024-04-17T19:54:02.513 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/10' 2024-04-17T19:54:02.513 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/34' 2024-04-17T19:54:02.513 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/38' 2024-04-17T19:54:02.513 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/11' 2024-04-17T19:54:02.513 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/16' 2024-04-17T19:54:02.513 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/32' 2024-04-17T19:54:02.513 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/30' 2024-04-17T19:54:02.513 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/14' 2024-04-17T19:54:02.513 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/18' 2024-04-17T19:54:02.513 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/13' 2024-04-17T19:54:02.513 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/33' 2024-04-17T19:54:02.514 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/17' 2024-04-17T19:54:02.514 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/35' 2024-04-17T19:54:02.514 INFO:tasks.workunit.client.0.smithi029.stdout:removed './166/37' 2024-04-17T19:54:02.514 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './166' 2024-04-17T19:54:02.514 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/19' 2024-04-17T19:54:02.514 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/15' 2024-04-17T19:54:02.514 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/31' 2024-04-17T19:54:02.514 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/36' 2024-04-17T19:54:02.514 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/20' 2024-04-17T19:54:02.514 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/12' 2024-04-17T19:54:02.514 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/39' 2024-04-17T19:54:02.514 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/10' 2024-04-17T19:54:02.514 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/34' 2024-04-17T19:54:02.514 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/38' 2024-04-17T19:54:04.805 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/11' 2024-04-17T19:54:04.805 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/16' 2024-04-17T19:54:04.805 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/32' 2024-04-17T19:54:04.805 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/30' 2024-04-17T19:54:04.805 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/14' 2024-04-17T19:54:04.806 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/18' 2024-04-17T19:54:04.806 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/13' 2024-04-17T19:54:04.806 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/33' 2024-04-17T19:54:04.806 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/17' 2024-04-17T19:54:04.806 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/35' 2024-04-17T19:54:04.806 INFO:tasks.workunit.client.0.smithi029.stdout:removed './167/37' 2024-04-17T19:54:04.806 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './167' 2024-04-17T19:54:04.806 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/19' 2024-04-17T19:54:04.806 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/15' 2024-04-17T19:54:04.806 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/31' 2024-04-17T19:54:04.806 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/36' 2024-04-17T19:54:04.806 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/20' 2024-04-17T19:54:04.806 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/12' 2024-04-17T19:54:04.806 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/39' 2024-04-17T19:54:04.806 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/10' 2024-04-17T19:54:04.807 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/34' 2024-04-17T19:54:04.807 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/38' 2024-04-17T19:54:04.807 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/11' 2024-04-17T19:54:04.807 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/16' 2024-04-17T19:54:04.807 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/32' 2024-04-17T19:54:04.807 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/30' 2024-04-17T19:54:04.807 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/14' 2024-04-17T19:54:04.807 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/18' 2024-04-17T19:54:04.807 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/13' 2024-04-17T19:54:04.807 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/33' 2024-04-17T19:54:04.807 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/17' 2024-04-17T19:54:04.807 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/35' 2024-04-17T19:54:04.807 INFO:tasks.workunit.client.0.smithi029.stdout:removed './168/37' 2024-04-17T19:54:04.807 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './168' 2024-04-17T19:54:04.807 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/19' 2024-04-17T19:54:04.807 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/15' 2024-04-17T19:54:04.807 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/31' 2024-04-17T19:54:04.808 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/36' 2024-04-17T19:54:04.808 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/20' 2024-04-17T19:54:04.808 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/12' 2024-04-17T19:54:04.808 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/39' 2024-04-17T19:54:04.808 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/10' 2024-04-17T19:54:04.808 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/34' 2024-04-17T19:54:04.808 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/38' 2024-04-17T19:54:04.808 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/11' 2024-04-17T19:54:04.808 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/16' 2024-04-17T19:54:04.808 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/32' 2024-04-17T19:54:04.808 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/30' 2024-04-17T19:54:04.808 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/14' 2024-04-17T19:54:04.808 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/18' 2024-04-17T19:54:04.808 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/13' 2024-04-17T19:54:04.808 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/33' 2024-04-17T19:54:04.809 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/17' 2024-04-17T19:54:04.809 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/35' 2024-04-17T19:54:04.809 INFO:tasks.workunit.client.0.smithi029.stdout:removed './169/37' 2024-04-17T19:54:04.809 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './169' 2024-04-17T19:54:04.809 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/19' 2024-04-17T19:54:04.809 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/15' 2024-04-17T19:54:04.809 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/31' 2024-04-17T19:54:04.809 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/36' 2024-04-17T19:54:04.809 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/20' 2024-04-17T19:54:04.809 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/12' 2024-04-17T19:54:04.809 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/39' 2024-04-17T19:54:04.809 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/10' 2024-04-17T19:54:04.809 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/34' 2024-04-17T19:54:04.809 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/38' 2024-04-17T19:54:04.809 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/11' 2024-04-17T19:54:04.810 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/16' 2024-04-17T19:54:04.810 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/32' 2024-04-17T19:54:04.810 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/30' 2024-04-17T19:54:04.810 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/14' 2024-04-17T19:54:04.810 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/18' 2024-04-17T19:54:04.810 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/13' 2024-04-17T19:54:04.810 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/33' 2024-04-17T19:54:04.810 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/17' 2024-04-17T19:54:04.810 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/35' 2024-04-17T19:54:04.810 INFO:tasks.workunit.client.0.smithi029.stdout:removed './170/37' 2024-04-17T19:54:04.810 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './170' 2024-04-17T19:54:04.810 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/19' 2024-04-17T19:54:04.810 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/15' 2024-04-17T19:54:04.810 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/31' 2024-04-17T19:54:04.810 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/36' 2024-04-17T19:54:04.810 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/20' 2024-04-17T19:54:04.811 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/12' 2024-04-17T19:54:04.811 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/39' 2024-04-17T19:54:04.811 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/10' 2024-04-17T19:54:04.811 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/34' 2024-04-17T19:54:04.811 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/38' 2024-04-17T19:54:04.811 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/11' 2024-04-17T19:54:04.811 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/16' 2024-04-17T19:54:04.811 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/32' 2024-04-17T19:54:04.811 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/30' 2024-04-17T19:54:04.811 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/14' 2024-04-17T19:54:04.811 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/18' 2024-04-17T19:54:04.811 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/13' 2024-04-17T19:54:04.811 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/33' 2024-04-17T19:54:04.811 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/17' 2024-04-17T19:54:04.811 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/35' 2024-04-17T19:54:04.812 INFO:tasks.workunit.client.0.smithi029.stdout:removed './171/37' 2024-04-17T19:54:04.812 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './171' 2024-04-17T19:54:04.812 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/19' 2024-04-17T19:54:04.812 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/15' 2024-04-17T19:54:04.812 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/31' 2024-04-17T19:54:04.812 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/36' 2024-04-17T19:54:04.812 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/20' 2024-04-17T19:54:04.812 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/12' 2024-04-17T19:54:04.812 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/39' 2024-04-17T19:54:04.812 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/10' 2024-04-17T19:54:04.812 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/34' 2024-04-17T19:54:04.812 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/38' 2024-04-17T19:54:04.812 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/11' 2024-04-17T19:54:04.812 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/16' 2024-04-17T19:54:04.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/32' 2024-04-17T19:54:04.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/30' 2024-04-17T19:54:04.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/14' 2024-04-17T19:54:04.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/18' 2024-04-17T19:54:04.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/13' 2024-04-17T19:54:04.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/33' 2024-04-17T19:54:04.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/17' 2024-04-17T19:54:04.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/35' 2024-04-17T19:54:04.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './172/37' 2024-04-17T19:54:04.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './172' 2024-04-17T19:54:04.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/19' 2024-04-17T19:54:04.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/15' 2024-04-17T19:54:04.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/31' 2024-04-17T19:54:04.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/36' 2024-04-17T19:54:04.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/20' 2024-04-17T19:54:04.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/12' 2024-04-17T19:54:04.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/39' 2024-04-17T19:54:04.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/10' 2024-04-17T19:54:04.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/34' 2024-04-17T19:54:04.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/38' 2024-04-17T19:54:04.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/11' 2024-04-17T19:54:04.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/16' 2024-04-17T19:54:04.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/32' 2024-04-17T19:54:04.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/30' 2024-04-17T19:54:04.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/14' 2024-04-17T19:54:04.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/18' 2024-04-17T19:54:04.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/13' 2024-04-17T19:54:04.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/33' 2024-04-17T19:54:04.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/17' 2024-04-17T19:54:04.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/35' 2024-04-17T19:54:04.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './173/37' 2024-04-17T19:54:04.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './173' 2024-04-17T19:54:04.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/19' 2024-04-17T19:54:04.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/15' 2024-04-17T19:54:04.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/31' 2024-04-17T19:54:04.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/36' 2024-04-17T19:54:04.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/20' 2024-04-17T19:54:04.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/12' 2024-04-17T19:54:04.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/39' 2024-04-17T19:54:04.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/10' 2024-04-17T19:54:04.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/34' 2024-04-17T19:54:04.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/38' 2024-04-17T19:54:04.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/11' 2024-04-17T19:54:04.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/16' 2024-04-17T19:54:04.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/32' 2024-04-17T19:54:04.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/30' 2024-04-17T19:54:04.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/14' 2024-04-17T19:54:04.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/18' 2024-04-17T19:54:04.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/13' 2024-04-17T19:54:04.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/33' 2024-04-17T19:54:04.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/17' 2024-04-17T19:54:04.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/35' 2024-04-17T19:54:04.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './174/37' 2024-04-17T19:54:04.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './174' 2024-04-17T19:54:04.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/19' 2024-04-17T19:54:04.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/15' 2024-04-17T19:54:04.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/31' 2024-04-17T19:54:04.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/36' 2024-04-17T19:54:04.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/20' 2024-04-17T19:54:04.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/12' 2024-04-17T19:54:04.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/39' 2024-04-17T19:54:04.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/10' 2024-04-17T19:54:04.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/34' 2024-04-17T19:54:04.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/38' 2024-04-17T19:54:04.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/11' 2024-04-17T19:54:04.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/16' 2024-04-17T19:54:04.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/32' 2024-04-17T19:54:04.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/30' 2024-04-17T19:54:04.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/14' 2024-04-17T19:54:04.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/18' 2024-04-17T19:54:04.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/13' 2024-04-17T19:54:04.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/33' 2024-04-17T19:54:04.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/17' 2024-04-17T19:54:04.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/35' 2024-04-17T19:54:04.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './175/37' 2024-04-17T19:54:04.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './175' 2024-04-17T19:54:04.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/19' 2024-04-17T19:54:04.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/15' 2024-04-17T19:54:04.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/31' 2024-04-17T19:54:04.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/36' 2024-04-17T19:54:04.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/20' 2024-04-17T19:54:04.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/12' 2024-04-17T19:54:04.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/39' 2024-04-17T19:54:04.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/10' 2024-04-17T19:54:04.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/34' 2024-04-17T19:54:04.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/38' 2024-04-17T19:54:04.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/11' 2024-04-17T19:54:04.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/16' 2024-04-17T19:54:04.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/32' 2024-04-17T19:54:04.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/30' 2024-04-17T19:54:04.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/14' 2024-04-17T19:54:04.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/18' 2024-04-17T19:54:04.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/13' 2024-04-17T19:54:04.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/33' 2024-04-17T19:54:04.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/17' 2024-04-17T19:54:04.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/35' 2024-04-17T19:54:04.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './176/37' 2024-04-17T19:54:04.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './176' 2024-04-17T19:54:04.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/19' 2024-04-17T19:54:04.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/15' 2024-04-17T19:54:06.154 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/31' 2024-04-17T19:54:06.154 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/36' 2024-04-17T19:54:06.154 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/20' 2024-04-17T19:54:06.154 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/12' 2024-04-17T19:54:06.154 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/39' 2024-04-17T19:54:06.154 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/10' 2024-04-17T19:54:06.154 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/34' 2024-04-17T19:54:06.155 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/38' 2024-04-17T19:54:06.155 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/11' 2024-04-17T19:54:06.155 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/16' 2024-04-17T19:54:06.155 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/32' 2024-04-17T19:54:06.155 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/30' 2024-04-17T19:54:06.155 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/14' 2024-04-17T19:54:06.155 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/18' 2024-04-17T19:54:06.155 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/13' 2024-04-17T19:54:06.155 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/33' 2024-04-17T19:54:06.155 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/17' 2024-04-17T19:54:06.155 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/35' 2024-04-17T19:54:06.155 INFO:tasks.workunit.client.0.smithi029.stdout:removed './177/37' 2024-04-17T19:54:06.155 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './177' 2024-04-17T19:54:06.155 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/19' 2024-04-17T19:54:06.155 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/15' 2024-04-17T19:54:06.155 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/31' 2024-04-17T19:54:06.155 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/36' 2024-04-17T19:54:06.156 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/20' 2024-04-17T19:54:06.156 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/12' 2024-04-17T19:54:06.156 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/39' 2024-04-17T19:54:06.156 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/10' 2024-04-17T19:54:06.156 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/34' 2024-04-17T19:54:06.156 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/38' 2024-04-17T19:54:06.156 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/11' 2024-04-17T19:54:06.156 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/16' 2024-04-17T19:54:06.156 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/32' 2024-04-17T19:54:06.156 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/30' 2024-04-17T19:54:06.156 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/14' 2024-04-17T19:54:06.156 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/18' 2024-04-17T19:54:06.156 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/13' 2024-04-17T19:54:06.156 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/33' 2024-04-17T19:54:06.156 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/17' 2024-04-17T19:54:06.156 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/35' 2024-04-17T19:54:06.156 INFO:tasks.workunit.client.0.smithi029.stdout:removed './178/37' 2024-04-17T19:54:06.157 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './178' 2024-04-17T19:54:06.157 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/19' 2024-04-17T19:54:06.157 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/15' 2024-04-17T19:54:06.157 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/31' 2024-04-17T19:54:06.157 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/36' 2024-04-17T19:54:06.157 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/20' 2024-04-17T19:54:06.157 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/12' 2024-04-17T19:54:06.157 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/39' 2024-04-17T19:54:06.157 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/10' 2024-04-17T19:54:06.157 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/34' 2024-04-17T19:54:06.157 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/38' 2024-04-17T19:54:06.157 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/11' 2024-04-17T19:54:06.157 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/16' 2024-04-17T19:54:06.157 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/32' 2024-04-17T19:54:06.157 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/30' 2024-04-17T19:54:06.157 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/14' 2024-04-17T19:54:06.157 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/18' 2024-04-17T19:54:06.158 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/13' 2024-04-17T19:54:06.158 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/33' 2024-04-17T19:54:06.158 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/17' 2024-04-17T19:54:06.158 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/35' 2024-04-17T19:54:06.158 INFO:tasks.workunit.client.0.smithi029.stdout:removed './179/37' 2024-04-17T19:54:06.158 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './179' 2024-04-17T19:54:06.158 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/19' 2024-04-17T19:54:06.158 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/15' 2024-04-17T19:54:06.158 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/31' 2024-04-17T19:54:06.158 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/36' 2024-04-17T19:54:06.158 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/20' 2024-04-17T19:54:06.158 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/12' 2024-04-17T19:54:06.158 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/39' 2024-04-17T19:54:06.158 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/10' 2024-04-17T19:54:06.159 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/34' 2024-04-17T19:54:06.159 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/38' 2024-04-17T19:54:06.159 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/11' 2024-04-17T19:54:06.159 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/16' 2024-04-17T19:54:06.159 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/32' 2024-04-17T19:54:06.159 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/30' 2024-04-17T19:54:06.159 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/14' 2024-04-17T19:54:06.159 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/18' 2024-04-17T19:54:06.159 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/13' 2024-04-17T19:54:06.159 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/33' 2024-04-17T19:54:06.159 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/17' 2024-04-17T19:54:06.159 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/35' 2024-04-17T19:54:06.159 INFO:tasks.workunit.client.0.smithi029.stdout:removed './180/37' 2024-04-17T19:54:06.159 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './180' 2024-04-17T19:54:06.159 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/19' 2024-04-17T19:54:06.159 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/15' 2024-04-17T19:54:06.159 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/31' 2024-04-17T19:54:06.160 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/36' 2024-04-17T19:54:06.160 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/20' 2024-04-17T19:54:06.160 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/12' 2024-04-17T19:54:06.160 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/39' 2024-04-17T19:54:06.160 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/10' 2024-04-17T19:54:06.160 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/34' 2024-04-17T19:54:06.160 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/38' 2024-04-17T19:54:06.160 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/11' 2024-04-17T19:54:06.160 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/16' 2024-04-17T19:54:06.160 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/32' 2024-04-17T19:54:06.160 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/30' 2024-04-17T19:54:06.160 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/14' 2024-04-17T19:54:06.160 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/18' 2024-04-17T19:54:06.160 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/13' 2024-04-17T19:54:06.160 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/33' 2024-04-17T19:54:06.160 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/17' 2024-04-17T19:54:06.160 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/35' 2024-04-17T19:54:06.161 INFO:tasks.workunit.client.0.smithi029.stdout:removed './181/37' 2024-04-17T19:54:06.161 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './181' 2024-04-17T19:54:06.161 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/19' 2024-04-17T19:54:06.161 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/15' 2024-04-17T19:54:06.161 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/31' 2024-04-17T19:54:06.161 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/36' 2024-04-17T19:54:06.161 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/20' 2024-04-17T19:54:06.161 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/12' 2024-04-17T19:54:06.161 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/39' 2024-04-17T19:54:06.161 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/10' 2024-04-17T19:54:06.161 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/34' 2024-04-17T19:54:06.161 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/38' 2024-04-17T19:54:06.161 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/11' 2024-04-17T19:54:06.161 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/16' 2024-04-17T19:54:06.161 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/32' 2024-04-17T19:54:06.161 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/30' 2024-04-17T19:54:06.161 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/14' 2024-04-17T19:54:06.162 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/18' 2024-04-17T19:54:06.162 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/13' 2024-04-17T19:54:06.162 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/33' 2024-04-17T19:54:06.162 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/17' 2024-04-17T19:54:06.162 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/35' 2024-04-17T19:54:06.162 INFO:tasks.workunit.client.0.smithi029.stdout:removed './182/37' 2024-04-17T19:54:06.162 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './182' 2024-04-17T19:54:06.162 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/19' 2024-04-17T19:54:06.162 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/15' 2024-04-17T19:54:06.162 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/31' 2024-04-17T19:54:06.162 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/36' 2024-04-17T19:54:06.162 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/20' 2024-04-17T19:54:06.162 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/12' 2024-04-17T19:54:06.162 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/39' 2024-04-17T19:54:06.163 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/10' 2024-04-17T19:54:06.163 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/34' 2024-04-17T19:54:06.163 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/38' 2024-04-17T19:54:06.163 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/11' 2024-04-17T19:54:06.163 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/16' 2024-04-17T19:54:06.163 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/32' 2024-04-17T19:54:06.163 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/30' 2024-04-17T19:54:06.163 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/14' 2024-04-17T19:54:06.163 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/18' 2024-04-17T19:54:06.163 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/13' 2024-04-17T19:54:06.163 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/33' 2024-04-17T19:54:06.163 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/17' 2024-04-17T19:54:06.163 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/35' 2024-04-17T19:54:06.163 INFO:tasks.workunit.client.0.smithi029.stdout:removed './183/37' 2024-04-17T19:54:06.163 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './183' 2024-04-17T19:54:06.163 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/19' 2024-04-17T19:54:06.164 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/15' 2024-04-17T19:54:06.164 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/31' 2024-04-17T19:54:06.164 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/36' 2024-04-17T19:54:06.164 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/20' 2024-04-17T19:54:06.164 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/12' 2024-04-17T19:54:06.164 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/39' 2024-04-17T19:54:06.164 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/10' 2024-04-17T19:54:06.164 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/34' 2024-04-17T19:54:06.164 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/38' 2024-04-17T19:54:06.164 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/11' 2024-04-17T19:54:06.164 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/16' 2024-04-17T19:54:06.164 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/32' 2024-04-17T19:54:06.164 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/30' 2024-04-17T19:54:06.164 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/14' 2024-04-17T19:54:06.165 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/18' 2024-04-17T19:54:06.165 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/13' 2024-04-17T19:54:06.165 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/33' 2024-04-17T19:54:06.165 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/17' 2024-04-17T19:54:06.165 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/35' 2024-04-17T19:54:06.165 INFO:tasks.workunit.client.0.smithi029.stdout:removed './184/37' 2024-04-17T19:54:06.165 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './184' 2024-04-17T19:54:06.165 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/19' 2024-04-17T19:54:06.165 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/15' 2024-04-17T19:54:06.165 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/31' 2024-04-17T19:54:06.165 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/36' 2024-04-17T19:54:06.165 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/20' 2024-04-17T19:54:06.165 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/12' 2024-04-17T19:54:06.165 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/39' 2024-04-17T19:54:06.165 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/10' 2024-04-17T19:54:06.165 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/34' 2024-04-17T19:54:06.165 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/38' 2024-04-17T19:54:06.166 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/11' 2024-04-17T19:54:06.166 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/16' 2024-04-17T19:54:06.166 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/32' 2024-04-17T19:54:06.166 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/30' 2024-04-17T19:54:06.166 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/14' 2024-04-17T19:54:06.166 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/18' 2024-04-17T19:54:06.166 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/13' 2024-04-17T19:54:06.166 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/33' 2024-04-17T19:54:06.166 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/17' 2024-04-17T19:54:06.166 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/35' 2024-04-17T19:54:06.166 INFO:tasks.workunit.client.0.smithi029.stdout:removed './185/37' 2024-04-17T19:54:06.166 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './185' 2024-04-17T19:54:06.166 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/19' 2024-04-17T19:54:06.166 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/15' 2024-04-17T19:54:06.166 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/31' 2024-04-17T19:54:06.166 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/36' 2024-04-17T19:54:06.167 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/20' 2024-04-17T19:54:06.167 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/12' 2024-04-17T19:54:06.167 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/39' 2024-04-17T19:54:06.167 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/10' 2024-04-17T19:54:06.167 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/34' 2024-04-17T19:54:06.167 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/38' 2024-04-17T19:54:06.167 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/11' 2024-04-17T19:54:06.167 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/16' 2024-04-17T19:54:06.167 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/32' 2024-04-17T19:54:06.167 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/30' 2024-04-17T19:54:06.167 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/14' 2024-04-17T19:54:06.167 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/18' 2024-04-17T19:54:09.857 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/13' 2024-04-17T19:54:09.857 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/33' 2024-04-17T19:54:09.857 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/17' 2024-04-17T19:54:09.857 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/35' 2024-04-17T19:54:09.857 INFO:tasks.workunit.client.0.smithi029.stdout:removed './186/37' 2024-04-17T19:54:09.857 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './186' 2024-04-17T19:54:09.857 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/19' 2024-04-17T19:54:09.857 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/15' 2024-04-17T19:54:09.857 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/31' 2024-04-17T19:54:09.857 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/36' 2024-04-17T19:54:09.857 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/20' 2024-04-17T19:54:09.858 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/12' 2024-04-17T19:54:09.858 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/39' 2024-04-17T19:54:09.858 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/10' 2024-04-17T19:54:09.858 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/34' 2024-04-17T19:54:09.858 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/38' 2024-04-17T19:54:09.858 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/11' 2024-04-17T19:54:09.858 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/16' 2024-04-17T19:54:09.858 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/32' 2024-04-17T19:54:09.858 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/30' 2024-04-17T19:54:09.858 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/14' 2024-04-17T19:54:09.858 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/18' 2024-04-17T19:54:09.858 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/13' 2024-04-17T19:54:09.858 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/33' 2024-04-17T19:54:09.858 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/17' 2024-04-17T19:54:09.858 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/35' 2024-04-17T19:54:09.859 INFO:tasks.workunit.client.0.smithi029.stdout:removed './187/37' 2024-04-17T19:54:09.859 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './187' 2024-04-17T19:54:09.859 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/19' 2024-04-17T19:54:09.859 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/15' 2024-04-17T19:54:09.859 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/31' 2024-04-17T19:54:09.859 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/36' 2024-04-17T19:54:09.859 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/20' 2024-04-17T19:54:09.859 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/12' 2024-04-17T19:54:09.859 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/39' 2024-04-17T19:54:09.859 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/10' 2024-04-17T19:54:09.859 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/34' 2024-04-17T19:54:09.859 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/38' 2024-04-17T19:54:09.859 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/11' 2024-04-17T19:54:09.859 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/16' 2024-04-17T19:54:09.859 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/32' 2024-04-17T19:54:09.859 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/30' 2024-04-17T19:54:09.860 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/14' 2024-04-17T19:54:09.860 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/18' 2024-04-17T19:54:09.860 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/13' 2024-04-17T19:54:09.860 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/33' 2024-04-17T19:54:09.860 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/17' 2024-04-17T19:54:09.860 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/35' 2024-04-17T19:54:09.860 INFO:tasks.workunit.client.0.smithi029.stdout:removed './188/37' 2024-04-17T19:54:09.860 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './188' 2024-04-17T19:54:09.860 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/19' 2024-04-17T19:54:09.860 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/15' 2024-04-17T19:54:09.860 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/31' 2024-04-17T19:54:09.860 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/36' 2024-04-17T19:54:09.860 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/20' 2024-04-17T19:54:09.860 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/12' 2024-04-17T19:54:09.860 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/39' 2024-04-17T19:54:09.861 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/10' 2024-04-17T19:54:09.861 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/34' 2024-04-17T19:54:09.861 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/38' 2024-04-17T19:54:09.861 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/11' 2024-04-17T19:54:09.861 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/16' 2024-04-17T19:54:09.861 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/32' 2024-04-17T19:54:09.861 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/30' 2024-04-17T19:54:09.861 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/14' 2024-04-17T19:54:09.861 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/18' 2024-04-17T19:54:09.861 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/13' 2024-04-17T19:54:09.861 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/33' 2024-04-17T19:54:09.861 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/17' 2024-04-17T19:54:09.861 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/35' 2024-04-17T19:54:09.861 INFO:tasks.workunit.client.0.smithi029.stdout:removed './189/37' 2024-04-17T19:54:09.862 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './189' 2024-04-17T19:54:09.862 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/19' 2024-04-17T19:54:09.862 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/15' 2024-04-17T19:54:09.862 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/31' 2024-04-17T19:54:09.862 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/36' 2024-04-17T19:54:09.862 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/20' 2024-04-17T19:54:09.862 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/12' 2024-04-17T19:54:09.862 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/39' 2024-04-17T19:54:09.862 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/10' 2024-04-17T19:54:09.862 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/34' 2024-04-17T19:54:09.862 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/38' 2024-04-17T19:54:09.862 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/11' 2024-04-17T19:54:09.862 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/16' 2024-04-17T19:54:09.862 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/32' 2024-04-17T19:54:09.862 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/30' 2024-04-17T19:54:09.862 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/14' 2024-04-17T19:54:09.863 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/18' 2024-04-17T19:54:09.863 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/13' 2024-04-17T19:54:09.863 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/33' 2024-04-17T19:54:09.863 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/17' 2024-04-17T19:54:09.863 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/35' 2024-04-17T19:54:09.863 INFO:tasks.workunit.client.0.smithi029.stdout:removed './190/37' 2024-04-17T19:54:09.863 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './190' 2024-04-17T19:54:09.863 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/19' 2024-04-17T19:54:09.863 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/15' 2024-04-17T19:54:09.863 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/31' 2024-04-17T19:54:09.863 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/36' 2024-04-17T19:54:09.863 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/20' 2024-04-17T19:54:09.863 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/12' 2024-04-17T19:54:09.863 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/39' 2024-04-17T19:54:09.863 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/10' 2024-04-17T19:54:09.863 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/34' 2024-04-17T19:54:09.863 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/38' 2024-04-17T19:54:09.864 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/11' 2024-04-17T19:54:09.864 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/16' 2024-04-17T19:54:09.864 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/32' 2024-04-17T19:54:09.864 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/30' 2024-04-17T19:54:09.864 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/14' 2024-04-17T19:54:09.864 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/18' 2024-04-17T19:54:09.864 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/13' 2024-04-17T19:54:09.864 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/33' 2024-04-17T19:54:09.864 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/17' 2024-04-17T19:54:09.864 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/35' 2024-04-17T19:54:09.864 INFO:tasks.workunit.client.0.smithi029.stdout:removed './191/37' 2024-04-17T19:54:09.864 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './191' 2024-04-17T19:54:09.864 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/19' 2024-04-17T19:54:09.864 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/15' 2024-04-17T19:54:09.864 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/31' 2024-04-17T19:54:09.864 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/36' 2024-04-17T19:54:09.865 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/20' 2024-04-17T19:54:09.865 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/12' 2024-04-17T19:54:09.865 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/39' 2024-04-17T19:54:09.865 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/10' 2024-04-17T19:54:09.865 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/34' 2024-04-17T19:54:09.865 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/38' 2024-04-17T19:54:09.865 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/11' 2024-04-17T19:54:09.865 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/16' 2024-04-17T19:54:09.865 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/32' 2024-04-17T19:54:09.865 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/30' 2024-04-17T19:54:09.865 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/14' 2024-04-17T19:54:09.865 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/18' 2024-04-17T19:54:09.865 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/13' 2024-04-17T19:54:09.865 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/33' 2024-04-17T19:54:09.865 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/17' 2024-04-17T19:54:09.865 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/35' 2024-04-17T19:54:09.865 INFO:tasks.workunit.client.0.smithi029.stdout:removed './192/37' 2024-04-17T19:54:09.866 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './192' 2024-04-17T19:54:09.866 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/19' 2024-04-17T19:54:09.866 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/15' 2024-04-17T19:54:09.866 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/31' 2024-04-17T19:54:09.866 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/36' 2024-04-17T19:54:09.866 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/20' 2024-04-17T19:54:09.866 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/12' 2024-04-17T19:54:09.866 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/39' 2024-04-17T19:54:09.866 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/10' 2024-04-17T19:54:09.866 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/34' 2024-04-17T19:54:09.866 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/38' 2024-04-17T19:54:09.866 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/11' 2024-04-17T19:54:09.866 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/16' 2024-04-17T19:54:09.866 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/32' 2024-04-17T19:54:09.867 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/30' 2024-04-17T19:54:09.867 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/14' 2024-04-17T19:54:09.867 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/18' 2024-04-17T19:54:09.867 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/13' 2024-04-17T19:54:09.867 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/33' 2024-04-17T19:54:09.867 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/17' 2024-04-17T19:54:09.867 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/35' 2024-04-17T19:54:09.867 INFO:tasks.workunit.client.0.smithi029.stdout:removed './193/37' 2024-04-17T19:54:09.867 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './193' 2024-04-17T19:54:09.867 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/19' 2024-04-17T19:54:09.867 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/15' 2024-04-17T19:54:09.867 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/31' 2024-04-17T19:54:09.867 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/36' 2024-04-17T19:54:09.867 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/20' 2024-04-17T19:54:09.867 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/12' 2024-04-17T19:54:09.867 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/39' 2024-04-17T19:54:09.867 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/10' 2024-04-17T19:54:09.868 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/34' 2024-04-17T19:54:09.868 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/38' 2024-04-17T19:54:09.868 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/11' 2024-04-17T19:54:09.868 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/16' 2024-04-17T19:54:09.868 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/32' 2024-04-17T19:54:09.868 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/30' 2024-04-17T19:54:09.868 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/14' 2024-04-17T19:54:09.868 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/18' 2024-04-17T19:54:09.868 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/13' 2024-04-17T19:54:09.868 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/33' 2024-04-17T19:54:09.868 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/17' 2024-04-17T19:54:09.868 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/35' 2024-04-17T19:54:09.868 INFO:tasks.workunit.client.0.smithi029.stdout:removed './194/37' 2024-04-17T19:54:09.868 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './194' 2024-04-17T19:54:09.868 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/19' 2024-04-17T19:54:09.868 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/15' 2024-04-17T19:54:09.868 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/31' 2024-04-17T19:54:09.869 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/36' 2024-04-17T19:54:09.869 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/20' 2024-04-17T19:54:09.869 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/12' 2024-04-17T19:54:09.869 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/39' 2024-04-17T19:54:09.869 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/10' 2024-04-17T19:54:09.869 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/34' 2024-04-17T19:54:09.869 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/38' 2024-04-17T19:54:09.869 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/11' 2024-04-17T19:54:09.869 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/16' 2024-04-17T19:54:09.869 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/32' 2024-04-17T19:54:09.869 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/30' 2024-04-17T19:54:09.869 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/14' 2024-04-17T19:54:09.869 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/18' 2024-04-17T19:54:09.869 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/13' 2024-04-17T19:54:09.869 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/33' 2024-04-17T19:54:09.869 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/17' 2024-04-17T19:54:09.870 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/35' 2024-04-17T19:54:09.870 INFO:tasks.workunit.client.0.smithi029.stdout:removed './195/37' 2024-04-17T19:54:09.870 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './195' 2024-04-17T19:54:09.870 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/19' 2024-04-17T19:54:09.870 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/15' 2024-04-17T19:54:09.870 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/31' 2024-04-17T19:54:09.870 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/36' 2024-04-17T19:54:09.870 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/20' 2024-04-17T19:54:09.870 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/12' 2024-04-17T19:54:09.870 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/39' 2024-04-17T19:54:09.870 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/10' 2024-04-17T19:54:11.773 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/34' 2024-04-17T19:54:11.773 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/38' 2024-04-17T19:54:11.773 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/11' 2024-04-17T19:54:11.773 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/16' 2024-04-17T19:54:11.773 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/32' 2024-04-17T19:54:11.773 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/30' 2024-04-17T19:54:11.774 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/14' 2024-04-17T19:54:11.774 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/18' 2024-04-17T19:54:11.774 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/13' 2024-04-17T19:54:11.774 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/33' 2024-04-17T19:54:11.774 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/17' 2024-04-17T19:54:11.774 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/35' 2024-04-17T19:54:11.774 INFO:tasks.workunit.client.0.smithi029.stdout:removed './196/37' 2024-04-17T19:54:11.774 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './196' 2024-04-17T19:54:11.774 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/19' 2024-04-17T19:54:11.775 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/15' 2024-04-17T19:54:11.775 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/31' 2024-04-17T19:54:11.775 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/36' 2024-04-17T19:54:11.775 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/20' 2024-04-17T19:54:11.775 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/12' 2024-04-17T19:54:11.775 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/39' 2024-04-17T19:54:11.775 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/10' 2024-04-17T19:54:11.775 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/34' 2024-04-17T19:54:11.775 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/38' 2024-04-17T19:54:11.775 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/11' 2024-04-17T19:54:11.775 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/16' 2024-04-17T19:54:11.775 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/32' 2024-04-17T19:54:11.776 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/30' 2024-04-17T19:54:11.776 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/14' 2024-04-17T19:54:11.776 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/18' 2024-04-17T19:54:11.776 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/13' 2024-04-17T19:54:11.776 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/33' 2024-04-17T19:54:11.776 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/17' 2024-04-17T19:54:11.776 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/35' 2024-04-17T19:54:11.776 INFO:tasks.workunit.client.0.smithi029.stdout:removed './197/37' 2024-04-17T19:54:11.776 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './197' 2024-04-17T19:54:11.776 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/19' 2024-04-17T19:54:11.776 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/15' 2024-04-17T19:54:11.777 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/31' 2024-04-17T19:54:11.777 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/36' 2024-04-17T19:54:11.777 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/20' 2024-04-17T19:54:11.777 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/12' 2024-04-17T19:54:11.777 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/39' 2024-04-17T19:54:11.777 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/10' 2024-04-17T19:54:11.777 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/34' 2024-04-17T19:54:11.777 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/38' 2024-04-17T19:54:11.777 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/11' 2024-04-17T19:54:11.777 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/16' 2024-04-17T19:54:11.777 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/32' 2024-04-17T19:54:11.777 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/30' 2024-04-17T19:54:11.778 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/14' 2024-04-17T19:54:11.778 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/18' 2024-04-17T19:54:11.778 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/13' 2024-04-17T19:54:11.778 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/33' 2024-04-17T19:54:11.778 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/17' 2024-04-17T19:54:11.778 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/35' 2024-04-17T19:54:11.778 INFO:tasks.workunit.client.0.smithi029.stdout:removed './198/37' 2024-04-17T19:54:11.778 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './198' 2024-04-17T19:54:11.778 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/19' 2024-04-17T19:54:11.778 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/15' 2024-04-17T19:54:11.778 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/31' 2024-04-17T19:54:11.779 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/36' 2024-04-17T19:54:11.779 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/20' 2024-04-17T19:54:11.779 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/12' 2024-04-17T19:54:11.779 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/39' 2024-04-17T19:54:11.779 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/10' 2024-04-17T19:54:11.779 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/34' 2024-04-17T19:54:11.779 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/38' 2024-04-17T19:54:11.779 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/11' 2024-04-17T19:54:11.779 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/16' 2024-04-17T19:54:11.779 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/32' 2024-04-17T19:54:11.779 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/30' 2024-04-17T19:54:11.779 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/14' 2024-04-17T19:54:11.780 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/18' 2024-04-17T19:54:11.780 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/13' 2024-04-17T19:54:11.780 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/33' 2024-04-17T19:54:11.780 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/17' 2024-04-17T19:54:11.780 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/35' 2024-04-17T19:54:11.780 INFO:tasks.workunit.client.0.smithi029.stdout:removed './199/37' 2024-04-17T19:54:11.780 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './199' 2024-04-17T19:54:11.780 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/21' 2024-04-17T19:54:11.780 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/27' 2024-04-17T19:54:11.780 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/31' 2024-04-17T19:54:11.780 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/36' 2024-04-17T19:54:11.780 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/20' 2024-04-17T19:54:11.781 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/39' 2024-04-17T19:54:11.781 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/34' 2024-04-17T19:54:11.781 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/38' 2024-04-17T19:54:11.781 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/28' 2024-04-17T19:54:11.781 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/24' 2024-04-17T19:54:11.781 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/29' 2024-04-17T19:54:11.781 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/32' 2024-04-17T19:54:11.781 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/22' 2024-04-17T19:54:11.781 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/25' 2024-04-17T19:54:11.781 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/30' 2024-04-17T19:54:11.781 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/33' 2024-04-17T19:54:11.782 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/23' 2024-04-17T19:54:11.782 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/35' 2024-04-17T19:54:11.782 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/26' 2024-04-17T19:54:11.782 INFO:tasks.workunit.client.0.smithi029.stdout:removed './200/37' 2024-04-17T19:54:11.782 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './200' 2024-04-17T19:54:11.782 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/21' 2024-04-17T19:54:11.782 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/27' 2024-04-17T19:54:11.782 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/31' 2024-04-17T19:54:11.782 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/36' 2024-04-17T19:54:11.782 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/20' 2024-04-17T19:54:11.783 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/39' 2024-04-17T19:54:11.783 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/34' 2024-04-17T19:54:11.783 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/38' 2024-04-17T19:54:11.783 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/28' 2024-04-17T19:54:11.783 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/24' 2024-04-17T19:54:11.783 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/29' 2024-04-17T19:54:11.783 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/32' 2024-04-17T19:54:11.783 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/22' 2024-04-17T19:54:11.783 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/25' 2024-04-17T19:54:11.783 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/30' 2024-04-17T19:54:11.783 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/33' 2024-04-17T19:54:11.784 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/23' 2024-04-17T19:54:11.784 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/35' 2024-04-17T19:54:11.784 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/26' 2024-04-17T19:54:11.784 INFO:tasks.workunit.client.0.smithi029.stdout:removed './201/37' 2024-04-17T19:54:11.784 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './201' 2024-04-17T19:54:11.784 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/21' 2024-04-17T19:54:11.784 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/27' 2024-04-17T19:54:11.784 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/31' 2024-04-17T19:54:11.784 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/36' 2024-04-17T19:54:11.784 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/20' 2024-04-17T19:54:11.784 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/39' 2024-04-17T19:54:11.785 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/34' 2024-04-17T19:54:11.785 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/38' 2024-04-17T19:54:11.785 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/28' 2024-04-17T19:54:11.785 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/24' 2024-04-17T19:54:11.785 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/29' 2024-04-17T19:54:11.785 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/32' 2024-04-17T19:54:11.785 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/22' 2024-04-17T19:54:11.785 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/25' 2024-04-17T19:54:11.785 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/30' 2024-04-17T19:54:11.785 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/33' 2024-04-17T19:54:11.785 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/23' 2024-04-17T19:54:11.786 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/35' 2024-04-17T19:54:11.786 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/26' 2024-04-17T19:54:11.786 INFO:tasks.workunit.client.0.smithi029.stdout:removed './202/37' 2024-04-17T19:54:11.786 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './202' 2024-04-17T19:54:11.786 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/21' 2024-04-17T19:54:11.786 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/27' 2024-04-17T19:54:11.786 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/31' 2024-04-17T19:54:11.786 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/36' 2024-04-17T19:54:11.786 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/20' 2024-04-17T19:54:11.786 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/39' 2024-04-17T19:54:11.786 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/34' 2024-04-17T19:54:11.787 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/38' 2024-04-17T19:54:11.787 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/28' 2024-04-17T19:54:11.787 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/24' 2024-04-17T19:54:11.787 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/29' 2024-04-17T19:54:11.787 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/32' 2024-04-17T19:54:11.787 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/22' 2024-04-17T19:54:11.787 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/25' 2024-04-17T19:54:11.787 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/30' 2024-04-17T19:54:11.787 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/33' 2024-04-17T19:54:11.787 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/23' 2024-04-17T19:54:11.787 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/35' 2024-04-17T19:54:11.788 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/26' 2024-04-17T19:54:11.788 INFO:tasks.workunit.client.0.smithi029.stdout:removed './203/37' 2024-04-17T19:54:11.788 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './203' 2024-04-17T19:54:11.788 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/21' 2024-04-17T19:54:11.788 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/27' 2024-04-17T19:54:11.788 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/31' 2024-04-17T19:54:11.788 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/36' 2024-04-17T19:54:11.788 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/20' 2024-04-17T19:54:11.788 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/39' 2024-04-17T19:54:11.788 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/34' 2024-04-17T19:54:11.788 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/38' 2024-04-17T19:54:11.788 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/28' 2024-04-17T19:54:11.789 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/24' 2024-04-17T19:54:11.789 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/29' 2024-04-17T19:54:11.789 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/32' 2024-04-17T19:54:11.789 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/22' 2024-04-17T19:54:11.789 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/25' 2024-04-17T19:54:11.789 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/30' 2024-04-17T19:54:11.789 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/33' 2024-04-17T19:54:11.789 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/23' 2024-04-17T19:54:11.789 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/35' 2024-04-17T19:54:11.789 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/26' 2024-04-17T19:54:11.789 INFO:tasks.workunit.client.0.smithi029.stdout:removed './204/37' 2024-04-17T19:54:11.789 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './204' 2024-04-17T19:54:11.789 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/21' 2024-04-17T19:54:11.790 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/27' 2024-04-17T19:54:11.790 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/31' 2024-04-17T19:54:11.790 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/36' 2024-04-17T19:54:11.790 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/20' 2024-04-17T19:54:11.790 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/39' 2024-04-17T19:54:11.790 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/34' 2024-04-17T19:54:11.790 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/38' 2024-04-17T19:54:11.790 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/28' 2024-04-17T19:54:11.790 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/24' 2024-04-17T19:54:11.790 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/29' 2024-04-17T19:54:11.790 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/32' 2024-04-17T19:54:11.790 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/22' 2024-04-17T19:54:11.791 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/25' 2024-04-17T19:54:11.791 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/30' 2024-04-17T19:54:11.791 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/33' 2024-04-17T19:54:11.791 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/23' 2024-04-17T19:54:11.791 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/35' 2024-04-17T19:54:11.791 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/26' 2024-04-17T19:54:11.791 INFO:tasks.workunit.client.0.smithi029.stdout:removed './205/37' 2024-04-17T19:54:11.791 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './205' 2024-04-17T19:54:11.791 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/21' 2024-04-17T19:54:11.791 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/27' 2024-04-17T19:54:11.791 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/31' 2024-04-17T19:54:11.791 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/36' 2024-04-17T19:54:11.792 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/20' 2024-04-17T19:54:11.792 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/39' 2024-04-17T19:54:14.392 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/34' 2024-04-17T19:54:14.392 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/38' 2024-04-17T19:54:14.392 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/28' 2024-04-17T19:54:14.392 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/24' 2024-04-17T19:54:14.392 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/29' 2024-04-17T19:54:14.392 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/32' 2024-04-17T19:54:14.392 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/22' 2024-04-17T19:54:14.393 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/25' 2024-04-17T19:54:14.393 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/30' 2024-04-17T19:54:14.393 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/33' 2024-04-17T19:54:14.393 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/23' 2024-04-17T19:54:14.393 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/35' 2024-04-17T19:54:14.393 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/26' 2024-04-17T19:54:14.393 INFO:tasks.workunit.client.0.smithi029.stdout:removed './206/37' 2024-04-17T19:54:14.393 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './206' 2024-04-17T19:54:14.393 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/21' 2024-04-17T19:54:14.393 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/27' 2024-04-17T19:54:14.393 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/31' 2024-04-17T19:54:14.393 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/36' 2024-04-17T19:54:14.393 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/20' 2024-04-17T19:54:14.394 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/39' 2024-04-17T19:54:14.394 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/34' 2024-04-17T19:54:14.394 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/38' 2024-04-17T19:54:14.394 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/28' 2024-04-17T19:54:14.394 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/24' 2024-04-17T19:54:14.394 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/29' 2024-04-17T19:54:14.394 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/32' 2024-04-17T19:54:14.394 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/22' 2024-04-17T19:54:14.394 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/25' 2024-04-17T19:54:14.394 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/30' 2024-04-17T19:54:14.394 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/33' 2024-04-17T19:54:14.394 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/23' 2024-04-17T19:54:14.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/35' 2024-04-17T19:54:14.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/26' 2024-04-17T19:54:14.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './207/37' 2024-04-17T19:54:14.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './207' 2024-04-17T19:54:14.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/21' 2024-04-17T19:54:14.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/27' 2024-04-17T19:54:14.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/31' 2024-04-17T19:54:14.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/36' 2024-04-17T19:54:14.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/20' 2024-04-17T19:54:14.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/39' 2024-04-17T19:54:14.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/34' 2024-04-17T19:54:14.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/38' 2024-04-17T19:54:14.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/28' 2024-04-17T19:54:14.395 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/24' 2024-04-17T19:54:14.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/29' 2024-04-17T19:54:14.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/32' 2024-04-17T19:54:14.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/22' 2024-04-17T19:54:14.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/25' 2024-04-17T19:54:14.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/30' 2024-04-17T19:54:14.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/33' 2024-04-17T19:54:14.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/23' 2024-04-17T19:54:14.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/35' 2024-04-17T19:54:14.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/26' 2024-04-17T19:54:14.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './208/37' 2024-04-17T19:54:14.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './208' 2024-04-17T19:54:14.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/21' 2024-04-17T19:54:14.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/27' 2024-04-17T19:54:14.396 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/31' 2024-04-17T19:54:14.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/36' 2024-04-17T19:54:14.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/20' 2024-04-17T19:54:14.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/39' 2024-04-17T19:54:14.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/34' 2024-04-17T19:54:14.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/38' 2024-04-17T19:54:14.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/28' 2024-04-17T19:54:14.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/24' 2024-04-17T19:54:14.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/29' 2024-04-17T19:54:14.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/32' 2024-04-17T19:54:14.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/22' 2024-04-17T19:54:14.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/25' 2024-04-17T19:54:14.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/30' 2024-04-17T19:54:14.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/33' 2024-04-17T19:54:14.397 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/23' 2024-04-17T19:54:14.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/35' 2024-04-17T19:54:14.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/26' 2024-04-17T19:54:14.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './209/37' 2024-04-17T19:54:14.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './209' 2024-04-17T19:54:14.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/21' 2024-04-17T19:54:14.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/27' 2024-04-17T19:54:14.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/31' 2024-04-17T19:54:14.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/36' 2024-04-17T19:54:14.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/20' 2024-04-17T19:54:14.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/39' 2024-04-17T19:54:14.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/34' 2024-04-17T19:54:14.398 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/38' 2024-04-17T19:54:14.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/28' 2024-04-17T19:54:14.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/24' 2024-04-17T19:54:14.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/29' 2024-04-17T19:54:14.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/32' 2024-04-17T19:54:14.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/22' 2024-04-17T19:54:14.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/25' 2024-04-17T19:54:14.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/30' 2024-04-17T19:54:14.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/33' 2024-04-17T19:54:14.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/23' 2024-04-17T19:54:14.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/35' 2024-04-17T19:54:14.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/26' 2024-04-17T19:54:14.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './210/37' 2024-04-17T19:54:14.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './210' 2024-04-17T19:54:14.399 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/21' 2024-04-17T19:54:14.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/27' 2024-04-17T19:54:14.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/31' 2024-04-17T19:54:14.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/36' 2024-04-17T19:54:14.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/20' 2024-04-17T19:54:14.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/39' 2024-04-17T19:54:14.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/34' 2024-04-17T19:54:14.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/38' 2024-04-17T19:54:14.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/28' 2024-04-17T19:54:14.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/24' 2024-04-17T19:54:14.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/29' 2024-04-17T19:54:14.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/32' 2024-04-17T19:54:14.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/22' 2024-04-17T19:54:14.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/25' 2024-04-17T19:54:14.400 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/30' 2024-04-17T19:54:14.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/33' 2024-04-17T19:54:14.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/23' 2024-04-17T19:54:14.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/35' 2024-04-17T19:54:14.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/26' 2024-04-17T19:54:14.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './211/37' 2024-04-17T19:54:14.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './211' 2024-04-17T19:54:14.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/21' 2024-04-17T19:54:14.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/27' 2024-04-17T19:54:14.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/31' 2024-04-17T19:54:14.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/36' 2024-04-17T19:54:14.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/20' 2024-04-17T19:54:14.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/39' 2024-04-17T19:54:14.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/34' 2024-04-17T19:54:14.401 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/38' 2024-04-17T19:54:14.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/28' 2024-04-17T19:54:14.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/24' 2024-04-17T19:54:14.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/29' 2024-04-17T19:54:14.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/32' 2024-04-17T19:54:14.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/22' 2024-04-17T19:54:14.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/25' 2024-04-17T19:54:14.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/30' 2024-04-17T19:54:14.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/33' 2024-04-17T19:54:14.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/23' 2024-04-17T19:54:14.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/35' 2024-04-17T19:54:14.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/26' 2024-04-17T19:54:14.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed './212/37' 2024-04-17T19:54:14.402 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './212' 2024-04-17T19:54:14.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/21' 2024-04-17T19:54:14.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/27' 2024-04-17T19:54:14.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/31' 2024-04-17T19:54:14.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/36' 2024-04-17T19:54:14.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/20' 2024-04-17T19:54:14.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/39' 2024-04-17T19:54:14.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/34' 2024-04-17T19:54:14.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/38' 2024-04-17T19:54:14.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/28' 2024-04-17T19:54:14.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/24' 2024-04-17T19:54:14.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/29' 2024-04-17T19:54:14.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/32' 2024-04-17T19:54:14.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/22' 2024-04-17T19:54:14.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/25' 2024-04-17T19:54:14.403 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/30' 2024-04-17T19:54:14.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/33' 2024-04-17T19:54:14.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/23' 2024-04-17T19:54:14.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/35' 2024-04-17T19:54:14.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/26' 2024-04-17T19:54:14.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './213/37' 2024-04-17T19:54:14.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './213' 2024-04-17T19:54:14.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/21' 2024-04-17T19:54:14.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/27' 2024-04-17T19:54:14.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/31' 2024-04-17T19:54:14.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/36' 2024-04-17T19:54:14.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/20' 2024-04-17T19:54:14.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/39' 2024-04-17T19:54:14.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/34' 2024-04-17T19:54:14.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/38' 2024-04-17T19:54:14.404 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/28' 2024-04-17T19:54:14.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/24' 2024-04-17T19:54:14.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/29' 2024-04-17T19:54:14.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/32' 2024-04-17T19:54:14.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/22' 2024-04-17T19:54:14.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/25' 2024-04-17T19:54:14.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/30' 2024-04-17T19:54:14.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/33' 2024-04-17T19:54:14.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/23' 2024-04-17T19:54:14.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/35' 2024-04-17T19:54:14.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/26' 2024-04-17T19:54:14.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './214/37' 2024-04-17T19:54:14.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './214' 2024-04-17T19:54:14.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/21' 2024-04-17T19:54:14.405 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/27' 2024-04-17T19:54:14.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/31' 2024-04-17T19:54:14.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/36' 2024-04-17T19:54:14.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/20' 2024-04-17T19:54:14.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/39' 2024-04-17T19:54:14.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/34' 2024-04-17T19:54:14.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/38' 2024-04-17T19:54:14.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/28' 2024-04-17T19:54:14.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/24' 2024-04-17T19:54:14.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/29' 2024-04-17T19:54:14.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/32' 2024-04-17T19:54:14.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/22' 2024-04-17T19:54:14.406 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/25' 2024-04-17T19:54:14.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/30' 2024-04-17T19:54:14.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/33' 2024-04-17T19:54:14.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/23' 2024-04-17T19:54:14.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/35' 2024-04-17T19:54:14.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/26' 2024-04-17T19:54:14.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './215/37' 2024-04-17T19:54:14.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './215' 2024-04-17T19:54:14.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/21' 2024-04-17T19:54:14.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/27' 2024-04-17T19:54:14.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/31' 2024-04-17T19:54:14.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/36' 2024-04-17T19:54:14.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/20' 2024-04-17T19:54:14.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/39' 2024-04-17T19:54:14.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/34' 2024-04-17T19:54:14.407 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/38' 2024-04-17T19:54:15.537 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/28' 2024-04-17T19:54:15.537 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/24' 2024-04-17T19:54:15.537 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/29' 2024-04-17T19:54:15.537 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/32' 2024-04-17T19:54:15.537 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/22' 2024-04-17T19:54:15.537 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/25' 2024-04-17T19:54:15.537 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/30' 2024-04-17T19:54:15.538 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/33' 2024-04-17T19:54:15.538 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/23' 2024-04-17T19:54:15.538 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/35' 2024-04-17T19:54:15.538 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/26' 2024-04-17T19:54:15.538 INFO:tasks.workunit.client.0.smithi029.stdout:removed './216/37' 2024-04-17T19:54:15.538 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './216' 2024-04-17T19:54:15.538 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/21' 2024-04-17T19:54:15.538 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/27' 2024-04-17T19:54:15.538 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/31' 2024-04-17T19:54:15.538 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/36' 2024-04-17T19:54:15.538 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/20' 2024-04-17T19:54:15.538 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/39' 2024-04-17T19:54:15.538 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/34' 2024-04-17T19:54:15.538 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/38' 2024-04-17T19:54:15.538 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/28' 2024-04-17T19:54:15.539 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/24' 2024-04-17T19:54:15.539 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/29' 2024-04-17T19:54:15.539 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/32' 2024-04-17T19:54:15.539 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/22' 2024-04-17T19:54:15.539 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/25' 2024-04-17T19:54:15.539 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/30' 2024-04-17T19:54:15.539 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/33' 2024-04-17T19:54:15.539 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/23' 2024-04-17T19:54:15.539 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/35' 2024-04-17T19:54:15.539 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/26' 2024-04-17T19:54:15.539 INFO:tasks.workunit.client.0.smithi029.stdout:removed './217/37' 2024-04-17T19:54:15.539 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './217' 2024-04-17T19:54:15.539 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/21' 2024-04-17T19:54:15.539 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/27' 2024-04-17T19:54:15.539 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/31' 2024-04-17T19:54:15.540 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/36' 2024-04-17T19:54:15.540 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/20' 2024-04-17T19:54:15.540 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/39' 2024-04-17T19:54:15.540 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/34' 2024-04-17T19:54:15.540 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/38' 2024-04-17T19:54:15.540 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/28' 2024-04-17T19:54:15.540 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/24' 2024-04-17T19:54:15.540 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/29' 2024-04-17T19:54:15.540 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/32' 2024-04-17T19:54:15.540 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/22' 2024-04-17T19:54:15.540 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/25' 2024-04-17T19:54:15.540 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/30' 2024-04-17T19:54:15.540 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/33' 2024-04-17T19:54:15.540 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/23' 2024-04-17T19:54:15.540 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/35' 2024-04-17T19:54:15.540 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/26' 2024-04-17T19:54:15.541 INFO:tasks.workunit.client.0.smithi029.stdout:removed './218/37' 2024-04-17T19:54:15.541 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './218' 2024-04-17T19:54:15.541 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/21' 2024-04-17T19:54:15.541 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/27' 2024-04-17T19:54:15.541 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/31' 2024-04-17T19:54:15.541 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/36' 2024-04-17T19:54:15.541 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/20' 2024-04-17T19:54:15.541 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/39' 2024-04-17T19:54:15.541 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/34' 2024-04-17T19:54:15.541 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/38' 2024-04-17T19:54:15.541 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/28' 2024-04-17T19:54:15.541 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/24' 2024-04-17T19:54:15.541 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/29' 2024-04-17T19:54:15.542 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/32' 2024-04-17T19:54:15.542 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/22' 2024-04-17T19:54:15.542 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/25' 2024-04-17T19:54:15.542 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/30' 2024-04-17T19:54:15.542 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/33' 2024-04-17T19:54:15.542 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/23' 2024-04-17T19:54:15.542 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/35' 2024-04-17T19:54:15.542 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/26' 2024-04-17T19:54:15.542 INFO:tasks.workunit.client.0.smithi029.stdout:removed './219/37' 2024-04-17T19:54:15.542 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './219' 2024-04-17T19:54:15.542 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/21' 2024-04-17T19:54:15.542 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/27' 2024-04-17T19:54:15.542 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/31' 2024-04-17T19:54:15.542 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/36' 2024-04-17T19:54:15.543 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/20' 2024-04-17T19:54:15.543 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/39' 2024-04-17T19:54:15.543 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/34' 2024-04-17T19:54:15.543 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/38' 2024-04-17T19:54:15.543 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/28' 2024-04-17T19:54:15.543 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/24' 2024-04-17T19:54:15.543 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/29' 2024-04-17T19:54:15.543 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/32' 2024-04-17T19:54:15.543 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/22' 2024-04-17T19:54:15.543 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/25' 2024-04-17T19:54:15.543 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/30' 2024-04-17T19:54:15.543 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/33' 2024-04-17T19:54:15.543 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/23' 2024-04-17T19:54:15.543 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/35' 2024-04-17T19:54:15.543 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/26' 2024-04-17T19:54:15.543 INFO:tasks.workunit.client.0.smithi029.stdout:removed './220/37' 2024-04-17T19:54:15.544 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './220' 2024-04-17T19:54:15.544 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/21' 2024-04-17T19:54:15.544 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/27' 2024-04-17T19:54:15.544 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/31' 2024-04-17T19:54:15.544 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/36' 2024-04-17T19:54:15.544 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/20' 2024-04-17T19:54:15.544 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/39' 2024-04-17T19:54:15.544 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/34' 2024-04-17T19:54:15.544 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/38' 2024-04-17T19:54:15.544 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/28' 2024-04-17T19:54:15.544 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/24' 2024-04-17T19:54:15.544 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/29' 2024-04-17T19:54:15.544 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/32' 2024-04-17T19:54:15.544 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/22' 2024-04-17T19:54:15.545 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/25' 2024-04-17T19:54:15.545 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/30' 2024-04-17T19:54:15.545 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/33' 2024-04-17T19:54:15.545 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/23' 2024-04-17T19:54:15.545 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/35' 2024-04-17T19:54:15.545 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/26' 2024-04-17T19:54:15.545 INFO:tasks.workunit.client.0.smithi029.stdout:removed './221/37' 2024-04-17T19:54:15.545 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './221' 2024-04-17T19:54:15.545 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/21' 2024-04-17T19:54:15.545 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/27' 2024-04-17T19:54:15.545 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/31' 2024-04-17T19:54:15.545 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/36' 2024-04-17T19:54:15.545 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/20' 2024-04-17T19:54:15.545 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/39' 2024-04-17T19:54:15.545 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/34' 2024-04-17T19:54:15.545 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/38' 2024-04-17T19:54:15.546 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/28' 2024-04-17T19:54:15.546 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/24' 2024-04-17T19:54:15.546 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/29' 2024-04-17T19:54:15.546 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/32' 2024-04-17T19:54:15.546 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/22' 2024-04-17T19:54:15.546 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/25' 2024-04-17T19:54:15.546 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/30' 2024-04-17T19:54:15.546 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/33' 2024-04-17T19:54:15.546 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/23' 2024-04-17T19:54:15.546 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/35' 2024-04-17T19:54:15.546 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/26' 2024-04-17T19:54:15.546 INFO:tasks.workunit.client.0.smithi029.stdout:removed './222/37' 2024-04-17T19:54:15.546 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './222' 2024-04-17T19:54:15.547 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/21' 2024-04-17T19:54:15.547 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/27' 2024-04-17T19:54:15.547 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/31' 2024-04-17T19:54:15.547 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/36' 2024-04-17T19:54:15.547 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/20' 2024-04-17T19:54:15.547 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/39' 2024-04-17T19:54:15.547 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/34' 2024-04-17T19:54:15.547 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/38' 2024-04-17T19:54:15.547 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/28' 2024-04-17T19:54:15.547 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/24' 2024-04-17T19:54:15.547 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/29' 2024-04-17T19:54:15.547 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/32' 2024-04-17T19:54:15.547 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/22' 2024-04-17T19:54:15.547 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/25' 2024-04-17T19:54:15.547 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/30' 2024-04-17T19:54:15.548 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/33' 2024-04-17T19:54:15.548 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/23' 2024-04-17T19:54:15.548 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/35' 2024-04-17T19:54:15.548 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/26' 2024-04-17T19:54:15.548 INFO:tasks.workunit.client.0.smithi029.stdout:removed './223/37' 2024-04-17T19:54:15.548 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './223' 2024-04-17T19:54:15.548 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/21' 2024-04-17T19:54:15.548 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/27' 2024-04-17T19:54:15.548 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/31' 2024-04-17T19:54:15.548 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/36' 2024-04-17T19:54:15.548 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/20' 2024-04-17T19:54:15.548 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/39' 2024-04-17T19:54:15.548 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/34' 2024-04-17T19:54:15.548 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/38' 2024-04-17T19:54:15.548 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/28' 2024-04-17T19:54:15.548 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/24' 2024-04-17T19:54:15.549 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/29' 2024-04-17T19:54:15.549 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/32' 2024-04-17T19:54:15.549 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/22' 2024-04-17T19:54:15.549 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/25' 2024-04-17T19:54:15.549 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/30' 2024-04-17T19:54:15.549 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/33' 2024-04-17T19:54:15.549 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/23' 2024-04-17T19:54:15.549 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/35' 2024-04-17T19:54:15.549 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/26' 2024-04-17T19:54:15.549 INFO:tasks.workunit.client.0.smithi029.stdout:removed './224/37' 2024-04-17T19:54:15.549 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './224' 2024-04-17T19:54:15.549 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/21' 2024-04-17T19:54:15.549 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/27' 2024-04-17T19:54:15.549 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/31' 2024-04-17T19:54:15.549 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/36' 2024-04-17T19:54:15.549 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/20' 2024-04-17T19:54:15.550 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/39' 2024-04-17T19:54:15.550 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/34' 2024-04-17T19:54:15.550 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/38' 2024-04-17T19:54:15.550 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/28' 2024-04-17T19:54:15.550 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/24' 2024-04-17T19:54:15.550 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/29' 2024-04-17T19:54:15.550 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/32' 2024-04-17T19:54:15.550 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/22' 2024-04-17T19:54:15.550 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/25' 2024-04-17T19:54:15.550 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/30' 2024-04-17T19:54:15.550 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/33' 2024-04-17T19:54:15.550 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/23' 2024-04-17T19:54:15.550 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/35' 2024-04-17T19:54:15.550 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/26' 2024-04-17T19:54:15.550 INFO:tasks.workunit.client.0.smithi029.stdout:removed './225/37' 2024-04-17T19:54:15.550 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './225' 2024-04-17T19:54:15.551 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/21' 2024-04-17T19:54:15.551 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/27' 2024-04-17T19:54:15.551 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/31' 2024-04-17T19:54:15.551 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/36' 2024-04-17T19:54:15.551 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/20' 2024-04-17T19:54:15.551 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/39' 2024-04-17T19:54:15.551 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/34' 2024-04-17T19:54:15.551 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/38' 2024-04-17T19:54:15.551 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/28' 2024-04-17T19:54:15.551 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/24' 2024-04-17T19:54:16.656 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/29' 2024-04-17T19:54:16.656 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/32' 2024-04-17T19:54:16.656 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/22' 2024-04-17T19:54:16.656 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/25' 2024-04-17T19:54:16.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/30' 2024-04-17T19:54:16.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/33' 2024-04-17T19:54:16.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/23' 2024-04-17T19:54:16.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/35' 2024-04-17T19:54:16.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/26' 2024-04-17T19:54:16.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './226/37' 2024-04-17T19:54:16.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './226' 2024-04-17T19:54:16.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/21' 2024-04-17T19:54:16.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/27' 2024-04-17T19:54:16.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/31' 2024-04-17T19:54:16.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/36' 2024-04-17T19:54:16.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/20' 2024-04-17T19:54:16.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/39' 2024-04-17T19:54:16.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/34' 2024-04-17T19:54:16.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/38' 2024-04-17T19:54:16.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/28' 2024-04-17T19:54:16.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/24' 2024-04-17T19:54:16.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/29' 2024-04-17T19:54:16.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/32' 2024-04-17T19:54:16.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/22' 2024-04-17T19:54:16.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/25' 2024-04-17T19:54:16.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/30' 2024-04-17T19:54:16.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/33' 2024-04-17T19:54:16.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/23' 2024-04-17T19:54:16.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/35' 2024-04-17T19:54:16.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/26' 2024-04-17T19:54:16.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './227/37' 2024-04-17T19:54:16.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './227' 2024-04-17T19:54:16.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/21' 2024-04-17T19:54:16.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/27' 2024-04-17T19:54:16.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/31' 2024-04-17T19:54:16.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/36' 2024-04-17T19:54:16.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/20' 2024-04-17T19:54:16.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/39' 2024-04-17T19:54:16.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/34' 2024-04-17T19:54:16.660 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/38' 2024-04-17T19:54:16.660 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/28' 2024-04-17T19:54:16.660 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/24' 2024-04-17T19:54:16.660 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/29' 2024-04-17T19:54:16.660 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/32' 2024-04-17T19:54:16.660 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/22' 2024-04-17T19:54:16.660 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/25' 2024-04-17T19:54:16.660 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/30' 2024-04-17T19:54:16.660 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/33' 2024-04-17T19:54:16.660 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/23' 2024-04-17T19:54:16.660 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/35' 2024-04-17T19:54:16.661 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/26' 2024-04-17T19:54:16.661 INFO:tasks.workunit.client.0.smithi029.stdout:removed './228/37' 2024-04-17T19:54:16.661 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './228' 2024-04-17T19:54:16.661 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/21' 2024-04-17T19:54:16.661 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/27' 2024-04-17T19:54:16.661 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/31' 2024-04-17T19:54:16.661 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/36' 2024-04-17T19:54:16.661 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/20' 2024-04-17T19:54:16.661 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/39' 2024-04-17T19:54:16.661 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/34' 2024-04-17T19:54:16.661 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/38' 2024-04-17T19:54:16.662 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/28' 2024-04-17T19:54:16.662 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/24' 2024-04-17T19:54:16.662 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/29' 2024-04-17T19:54:16.662 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/32' 2024-04-17T19:54:16.662 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/22' 2024-04-17T19:54:16.662 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/25' 2024-04-17T19:54:16.662 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/30' 2024-04-17T19:54:16.662 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/33' 2024-04-17T19:54:16.662 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/23' 2024-04-17T19:54:16.662 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/35' 2024-04-17T19:54:16.662 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/26' 2024-04-17T19:54:16.663 INFO:tasks.workunit.client.0.smithi029.stdout:removed './229/37' 2024-04-17T19:54:16.663 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './229' 2024-04-17T19:54:16.663 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/21' 2024-04-17T19:54:16.663 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/27' 2024-04-17T19:54:16.663 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/31' 2024-04-17T19:54:16.663 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/36' 2024-04-17T19:54:16.663 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/20' 2024-04-17T19:54:16.663 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/39' 2024-04-17T19:54:16.663 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/34' 2024-04-17T19:54:16.663 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/38' 2024-04-17T19:54:16.663 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/28' 2024-04-17T19:54:16.663 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/24' 2024-04-17T19:54:16.664 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/29' 2024-04-17T19:54:16.664 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/32' 2024-04-17T19:54:16.664 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/22' 2024-04-17T19:54:16.664 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/25' 2024-04-17T19:54:16.664 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/30' 2024-04-17T19:54:16.664 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/33' 2024-04-17T19:54:16.664 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/23' 2024-04-17T19:54:16.664 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/35' 2024-04-17T19:54:16.664 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/26' 2024-04-17T19:54:16.664 INFO:tasks.workunit.client.0.smithi029.stdout:removed './230/37' 2024-04-17T19:54:16.664 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './230' 2024-04-17T19:54:16.664 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/21' 2024-04-17T19:54:16.664 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/27' 2024-04-17T19:54:16.665 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/31' 2024-04-17T19:54:16.665 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/36' 2024-04-17T19:54:16.665 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/20' 2024-04-17T19:54:16.665 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/39' 2024-04-17T19:54:16.665 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/34' 2024-04-17T19:54:16.665 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/38' 2024-04-17T19:54:16.665 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/28' 2024-04-17T19:54:16.665 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/24' 2024-04-17T19:54:16.665 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/29' 2024-04-17T19:54:16.665 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/32' 2024-04-17T19:54:16.665 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/22' 2024-04-17T19:54:16.665 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/25' 2024-04-17T19:54:16.666 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/30' 2024-04-17T19:54:16.666 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/33' 2024-04-17T19:54:16.666 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/23' 2024-04-17T19:54:16.666 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/35' 2024-04-17T19:54:16.666 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/26' 2024-04-17T19:54:16.666 INFO:tasks.workunit.client.0.smithi029.stdout:removed './231/37' 2024-04-17T19:54:16.666 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './231' 2024-04-17T19:54:16.666 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/21' 2024-04-17T19:54:16.666 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/27' 2024-04-17T19:54:16.666 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/31' 2024-04-17T19:54:16.666 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/36' 2024-04-17T19:54:16.666 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/20' 2024-04-17T19:54:16.667 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/39' 2024-04-17T19:54:16.667 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/34' 2024-04-17T19:54:16.667 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/38' 2024-04-17T19:54:16.667 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/28' 2024-04-17T19:54:16.667 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/24' 2024-04-17T19:54:16.667 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/29' 2024-04-17T19:54:16.667 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/32' 2024-04-17T19:54:16.667 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/22' 2024-04-17T19:54:16.667 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/25' 2024-04-17T19:54:16.667 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/30' 2024-04-17T19:54:16.667 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/33' 2024-04-17T19:54:16.668 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/23' 2024-04-17T19:54:16.668 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/35' 2024-04-17T19:54:16.668 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/26' 2024-04-17T19:54:16.668 INFO:tasks.workunit.client.0.smithi029.stdout:removed './232/37' 2024-04-17T19:54:16.668 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './232' 2024-04-17T19:54:16.668 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/21' 2024-04-17T19:54:16.668 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/27' 2024-04-17T19:54:16.668 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/31' 2024-04-17T19:54:16.668 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/36' 2024-04-17T19:54:16.668 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/20' 2024-04-17T19:54:16.668 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/39' 2024-04-17T19:54:16.668 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/34' 2024-04-17T19:54:16.669 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/38' 2024-04-17T19:54:16.669 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/28' 2024-04-17T19:54:16.669 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/24' 2024-04-17T19:54:16.669 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/29' 2024-04-17T19:54:16.669 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/32' 2024-04-17T19:54:16.669 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/22' 2024-04-17T19:54:16.669 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/25' 2024-04-17T19:54:16.669 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/30' 2024-04-17T19:54:16.669 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/33' 2024-04-17T19:54:16.669 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/23' 2024-04-17T19:54:16.669 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/35' 2024-04-17T19:54:16.669 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/26' 2024-04-17T19:54:16.669 INFO:tasks.workunit.client.0.smithi029.stdout:removed './233/37' 2024-04-17T19:54:16.670 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './233' 2024-04-17T19:54:16.670 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/21' 2024-04-17T19:54:16.670 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/27' 2024-04-17T19:54:16.670 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/31' 2024-04-17T19:54:16.670 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/36' 2024-04-17T19:54:16.670 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/20' 2024-04-17T19:54:16.670 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/39' 2024-04-17T19:54:16.670 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/34' 2024-04-17T19:54:16.670 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/38' 2024-04-17T19:54:16.670 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/28' 2024-04-17T19:54:16.670 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/24' 2024-04-17T19:54:16.670 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/29' 2024-04-17T19:54:16.670 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/32' 2024-04-17T19:54:16.671 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/22' 2024-04-17T19:54:16.671 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/25' 2024-04-17T19:54:16.671 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/30' 2024-04-17T19:54:16.671 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/33' 2024-04-17T19:54:16.671 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/23' 2024-04-17T19:54:16.671 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/35' 2024-04-17T19:54:16.671 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/26' 2024-04-17T19:54:16.671 INFO:tasks.workunit.client.0.smithi029.stdout:removed './234/37' 2024-04-17T19:54:16.671 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './234' 2024-04-17T19:54:16.671 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/21' 2024-04-17T19:54:16.671 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/27' 2024-04-17T19:54:16.671 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/31' 2024-04-17T19:54:16.672 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/36' 2024-04-17T19:54:16.672 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/20' 2024-04-17T19:54:16.672 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/39' 2024-04-17T19:54:16.672 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/34' 2024-04-17T19:54:16.672 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/38' 2024-04-17T19:54:16.672 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/28' 2024-04-17T19:54:16.672 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/24' 2024-04-17T19:54:16.672 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/29' 2024-04-17T19:54:16.672 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/32' 2024-04-17T19:54:16.672 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/22' 2024-04-17T19:54:16.672 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/25' 2024-04-17T19:54:16.672 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/30' 2024-04-17T19:54:16.672 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/33' 2024-04-17T19:54:16.672 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/23' 2024-04-17T19:54:16.673 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/35' 2024-04-17T19:54:16.673 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/26' 2024-04-17T19:54:16.673 INFO:tasks.workunit.client.0.smithi029.stdout:removed './235/37' 2024-04-17T19:54:16.673 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './235' 2024-04-17T19:54:16.673 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/21' 2024-04-17T19:54:16.673 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/27' 2024-04-17T19:54:16.673 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/31' 2024-04-17T19:54:16.673 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/36' 2024-04-17T19:54:16.673 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/20' 2024-04-17T19:54:16.673 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/39' 2024-04-17T19:54:16.673 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/34' 2024-04-17T19:54:16.673 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/38' 2024-04-17T19:54:16.673 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/28' 2024-04-17T19:54:16.674 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/24' 2024-04-17T19:54:16.674 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/29' 2024-04-17T19:54:16.674 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/32' 2024-04-17T19:54:18.872 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/22' 2024-04-17T19:54:18.872 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/25' 2024-04-17T19:54:18.872 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/30' 2024-04-17T19:54:18.872 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/33' 2024-04-17T19:54:18.873 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/23' 2024-04-17T19:54:18.873 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/35' 2024-04-17T19:54:18.873 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/26' 2024-04-17T19:54:18.873 INFO:tasks.workunit.client.0.smithi029.stdout:removed './236/37' 2024-04-17T19:54:18.874 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './236' 2024-04-17T19:54:18.874 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/21' 2024-04-17T19:54:18.874 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/27' 2024-04-17T19:54:18.874 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/31' 2024-04-17T19:54:18.875 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/36' 2024-04-17T19:54:18.875 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/20' 2024-04-17T19:54:18.875 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/39' 2024-04-17T19:54:18.875 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/34' 2024-04-17T19:54:18.875 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/38' 2024-04-17T19:54:18.876 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/28' 2024-04-17T19:54:18.876 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/24' 2024-04-17T19:54:18.876 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/29' 2024-04-17T19:54:18.876 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/32' 2024-04-17T19:54:18.876 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/22' 2024-04-17T19:54:18.877 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/25' 2024-04-17T19:54:18.877 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/30' 2024-04-17T19:54:18.877 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/33' 2024-04-17T19:54:18.877 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/23' 2024-04-17T19:54:18.878 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/35' 2024-04-17T19:54:18.878 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/26' 2024-04-17T19:54:18.878 INFO:tasks.workunit.client.0.smithi029.stdout:removed './237/37' 2024-04-17T19:54:18.878 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './237' 2024-04-17T19:54:18.878 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/21' 2024-04-17T19:54:18.879 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/27' 2024-04-17T19:54:18.879 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/31' 2024-04-17T19:54:18.879 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/36' 2024-04-17T19:54:18.879 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/20' 2024-04-17T19:54:18.880 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/39' 2024-04-17T19:54:18.880 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/34' 2024-04-17T19:54:18.880 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/38' 2024-04-17T19:54:18.880 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/28' 2024-04-17T19:54:18.880 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/24' 2024-04-17T19:54:18.881 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/29' 2024-04-17T19:54:18.881 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/32' 2024-04-17T19:54:18.881 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/22' 2024-04-17T19:54:18.881 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/25' 2024-04-17T19:54:18.881 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/30' 2024-04-17T19:54:18.882 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/33' 2024-04-17T19:54:18.882 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/23' 2024-04-17T19:54:18.882 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/35' 2024-04-17T19:54:18.882 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/26' 2024-04-17T19:54:18.882 INFO:tasks.workunit.client.0.smithi029.stdout:removed './238/37' 2024-04-17T19:54:18.882 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './238' 2024-04-17T19:54:18.882 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/21' 2024-04-17T19:54:18.882 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/27' 2024-04-17T19:54:18.883 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/31' 2024-04-17T19:54:18.883 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/36' 2024-04-17T19:54:18.883 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/20' 2024-04-17T19:54:18.883 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/39' 2024-04-17T19:54:18.883 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/34' 2024-04-17T19:54:18.883 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/38' 2024-04-17T19:54:18.883 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/28' 2024-04-17T19:54:18.883 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/24' 2024-04-17T19:54:18.883 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/29' 2024-04-17T19:54:18.884 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/32' 2024-04-17T19:54:18.884 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/22' 2024-04-17T19:54:18.884 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/25' 2024-04-17T19:54:18.884 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/30' 2024-04-17T19:54:18.884 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/33' 2024-04-17T19:54:18.884 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/23' 2024-04-17T19:54:18.884 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/35' 2024-04-17T19:54:18.884 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/26' 2024-04-17T19:54:18.884 INFO:tasks.workunit.client.0.smithi029.stdout:removed './239/37' 2024-04-17T19:54:18.885 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './239' 2024-04-17T19:54:18.885 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/21' 2024-04-17T19:54:18.885 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/27' 2024-04-17T19:54:18.885 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/31' 2024-04-17T19:54:18.885 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/36' 2024-04-17T19:54:18.885 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/20' 2024-04-17T19:54:18.885 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/39' 2024-04-17T19:54:18.885 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/34' 2024-04-17T19:54:18.885 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/38' 2024-04-17T19:54:18.886 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/28' 2024-04-17T19:54:18.886 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/24' 2024-04-17T19:54:18.886 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/29' 2024-04-17T19:54:18.886 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/32' 2024-04-17T19:54:18.886 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/22' 2024-04-17T19:54:18.886 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/25' 2024-04-17T19:54:18.886 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/30' 2024-04-17T19:54:18.886 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/33' 2024-04-17T19:54:18.886 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/23' 2024-04-17T19:54:18.886 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/35' 2024-04-17T19:54:18.887 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/26' 2024-04-17T19:54:18.887 INFO:tasks.workunit.client.0.smithi029.stdout:removed './240/37' 2024-04-17T19:54:18.887 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './240' 2024-04-17T19:54:18.887 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/21' 2024-04-17T19:54:18.887 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/27' 2024-04-17T19:54:18.887 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/31' 2024-04-17T19:54:18.887 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/36' 2024-04-17T19:54:18.887 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/20' 2024-04-17T19:54:18.888 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/39' 2024-04-17T19:54:18.888 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/34' 2024-04-17T19:54:18.888 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/38' 2024-04-17T19:54:18.888 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/28' 2024-04-17T19:54:18.888 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/24' 2024-04-17T19:54:18.888 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/29' 2024-04-17T19:54:18.888 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/32' 2024-04-17T19:54:18.888 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/22' 2024-04-17T19:54:18.888 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/25' 2024-04-17T19:54:18.888 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/30' 2024-04-17T19:54:18.889 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/33' 2024-04-17T19:54:18.889 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/23' 2024-04-17T19:54:18.889 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/35' 2024-04-17T19:54:18.889 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/26' 2024-04-17T19:54:18.889 INFO:tasks.workunit.client.0.smithi029.stdout:removed './241/37' 2024-04-17T19:54:18.889 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './241' 2024-04-17T19:54:18.889 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/21' 2024-04-17T19:54:18.889 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/27' 2024-04-17T19:54:18.889 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/31' 2024-04-17T19:54:18.889 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/36' 2024-04-17T19:54:18.890 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/20' 2024-04-17T19:54:18.890 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/39' 2024-04-17T19:54:18.890 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/34' 2024-04-17T19:54:18.890 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/38' 2024-04-17T19:54:18.890 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/28' 2024-04-17T19:54:18.890 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/24' 2024-04-17T19:54:18.890 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/29' 2024-04-17T19:54:18.890 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/32' 2024-04-17T19:54:18.890 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/22' 2024-04-17T19:54:18.891 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/25' 2024-04-17T19:54:18.891 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/30' 2024-04-17T19:54:18.891 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/33' 2024-04-17T19:54:18.891 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/23' 2024-04-17T19:54:18.891 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/35' 2024-04-17T19:54:18.891 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/26' 2024-04-17T19:54:18.891 INFO:tasks.workunit.client.0.smithi029.stdout:removed './242/37' 2024-04-17T19:54:18.891 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './242' 2024-04-17T19:54:18.891 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/21' 2024-04-17T19:54:18.891 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/27' 2024-04-17T19:54:18.892 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/31' 2024-04-17T19:54:18.892 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/36' 2024-04-17T19:54:18.892 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/20' 2024-04-17T19:54:18.892 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/39' 2024-04-17T19:54:18.892 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/34' 2024-04-17T19:54:18.892 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/38' 2024-04-17T19:54:18.892 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/28' 2024-04-17T19:54:18.892 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/24' 2024-04-17T19:54:18.892 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/29' 2024-04-17T19:54:18.892 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/32' 2024-04-17T19:54:18.893 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/22' 2024-04-17T19:54:18.893 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/25' 2024-04-17T19:54:18.893 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/30' 2024-04-17T19:54:18.893 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/33' 2024-04-17T19:54:18.893 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/23' 2024-04-17T19:54:18.893 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/35' 2024-04-17T19:54:18.893 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/26' 2024-04-17T19:54:18.893 INFO:tasks.workunit.client.0.smithi029.stdout:removed './243/37' 2024-04-17T19:54:18.893 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './243' 2024-04-17T19:54:18.893 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/21' 2024-04-17T19:54:18.894 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/27' 2024-04-17T19:54:18.894 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/31' 2024-04-17T19:54:18.894 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/36' 2024-04-17T19:54:18.894 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/20' 2024-04-17T19:54:18.894 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/39' 2024-04-17T19:54:18.894 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/34' 2024-04-17T19:54:18.894 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/38' 2024-04-17T19:54:18.894 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/28' 2024-04-17T19:54:18.894 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/24' 2024-04-17T19:54:18.894 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/29' 2024-04-17T19:54:18.895 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/32' 2024-04-17T19:54:18.895 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/22' 2024-04-17T19:54:18.895 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/25' 2024-04-17T19:54:18.895 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/30' 2024-04-17T19:54:18.895 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/33' 2024-04-17T19:54:18.895 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/23' 2024-04-17T19:54:18.895 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/35' 2024-04-17T19:54:18.895 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/26' 2024-04-17T19:54:18.895 INFO:tasks.workunit.client.0.smithi029.stdout:removed './244/37' 2024-04-17T19:54:18.895 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './244' 2024-04-17T19:54:18.896 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/21' 2024-04-17T19:54:18.896 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/27' 2024-04-17T19:54:18.896 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/31' 2024-04-17T19:54:18.896 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/36' 2024-04-17T19:54:18.896 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/20' 2024-04-17T19:54:18.896 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/39' 2024-04-17T19:54:18.896 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/34' 2024-04-17T19:54:18.896 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/38' 2024-04-17T19:54:18.896 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/28' 2024-04-17T19:54:18.897 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/24' 2024-04-17T19:54:18.897 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/29' 2024-04-17T19:54:18.897 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/32' 2024-04-17T19:54:18.897 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/22' 2024-04-17T19:54:18.897 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/25' 2024-04-17T19:54:18.897 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/30' 2024-04-17T19:54:18.897 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/33' 2024-04-17T19:54:18.897 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/23' 2024-04-17T19:54:18.897 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/35' 2024-04-17T19:54:18.897 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/26' 2024-04-17T19:54:18.897 INFO:tasks.workunit.client.0.smithi029.stdout:removed './245/37' 2024-04-17T19:54:18.898 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './245' 2024-04-17T19:54:18.898 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/21' 2024-04-17T19:54:18.898 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/27' 2024-04-17T19:54:18.898 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/31' 2024-04-17T19:54:18.898 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/36' 2024-04-17T19:54:18.898 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/20' 2024-04-17T19:54:18.898 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/39' 2024-04-17T19:54:18.898 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/34' 2024-04-17T19:54:18.898 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/38' 2024-04-17T19:54:18.898 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/28' 2024-04-17T19:54:18.899 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/24' 2024-04-17T19:54:18.899 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/29' 2024-04-17T19:54:18.899 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/32' 2024-04-17T19:54:18.899 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/22' 2024-04-17T19:54:21.294 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/25' 2024-04-17T19:54:21.294 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/30' 2024-04-17T19:54:21.294 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/33' 2024-04-17T19:54:21.294 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/23' 2024-04-17T19:54:21.295 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/35' 2024-04-17T19:54:21.295 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/26' 2024-04-17T19:54:21.295 INFO:tasks.workunit.client.0.smithi029.stdout:removed './246/37' 2024-04-17T19:54:21.295 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './246' 2024-04-17T19:54:21.295 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/21' 2024-04-17T19:54:21.295 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/27' 2024-04-17T19:54:21.295 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/31' 2024-04-17T19:54:21.295 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/36' 2024-04-17T19:54:21.295 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/20' 2024-04-17T19:54:21.295 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/39' 2024-04-17T19:54:21.295 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/34' 2024-04-17T19:54:21.295 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/38' 2024-04-17T19:54:21.296 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/28' 2024-04-17T19:54:21.296 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/24' 2024-04-17T19:54:21.296 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/29' 2024-04-17T19:54:21.296 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/32' 2024-04-17T19:54:21.296 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/22' 2024-04-17T19:54:21.296 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/25' 2024-04-17T19:54:21.296 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/30' 2024-04-17T19:54:21.296 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/33' 2024-04-17T19:54:21.296 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/23' 2024-04-17T19:54:21.296 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/35' 2024-04-17T19:54:21.296 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/26' 2024-04-17T19:54:21.297 INFO:tasks.workunit.client.0.smithi029.stdout:removed './247/37' 2024-04-17T19:54:21.297 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './247' 2024-04-17T19:54:21.297 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/21' 2024-04-17T19:54:21.297 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/27' 2024-04-17T19:54:21.297 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/31' 2024-04-17T19:54:21.297 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/36' 2024-04-17T19:54:21.297 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/20' 2024-04-17T19:54:21.297 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/39' 2024-04-17T19:54:21.297 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/34' 2024-04-17T19:54:21.297 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/38' 2024-04-17T19:54:21.297 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/28' 2024-04-17T19:54:21.297 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/24' 2024-04-17T19:54:21.297 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/29' 2024-04-17T19:54:21.298 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/32' 2024-04-17T19:54:21.298 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/22' 2024-04-17T19:54:21.298 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/25' 2024-04-17T19:54:21.298 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/30' 2024-04-17T19:54:21.298 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/33' 2024-04-17T19:54:21.298 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/23' 2024-04-17T19:54:21.298 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/35' 2024-04-17T19:54:21.298 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/26' 2024-04-17T19:54:21.298 INFO:tasks.workunit.client.0.smithi029.stdout:removed './248/37' 2024-04-17T19:54:21.298 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './248' 2024-04-17T19:54:21.299 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/21' 2024-04-17T19:54:21.299 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/27' 2024-04-17T19:54:21.299 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/31' 2024-04-17T19:54:21.299 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/36' 2024-04-17T19:54:21.299 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/20' 2024-04-17T19:54:21.299 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/39' 2024-04-17T19:54:21.299 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/34' 2024-04-17T19:54:21.299 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/38' 2024-04-17T19:54:21.299 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/28' 2024-04-17T19:54:21.299 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/24' 2024-04-17T19:54:21.299 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/29' 2024-04-17T19:54:21.299 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/32' 2024-04-17T19:54:21.299 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/22' 2024-04-17T19:54:21.300 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/25' 2024-04-17T19:54:21.300 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/30' 2024-04-17T19:54:21.300 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/33' 2024-04-17T19:54:21.300 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/23' 2024-04-17T19:54:21.300 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/35' 2024-04-17T19:54:21.300 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/26' 2024-04-17T19:54:21.300 INFO:tasks.workunit.client.0.smithi029.stdout:removed './249/37' 2024-04-17T19:54:21.300 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './249' 2024-04-17T19:54:21.300 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/21' 2024-04-17T19:54:21.300 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/27' 2024-04-17T19:54:21.300 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/31' 2024-04-17T19:54:21.300 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/36' 2024-04-17T19:54:21.300 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/20' 2024-04-17T19:54:21.301 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/39' 2024-04-17T19:54:21.301 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/34' 2024-04-17T19:54:21.301 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/38' 2024-04-17T19:54:21.301 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/28' 2024-04-17T19:54:21.301 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/24' 2024-04-17T19:54:21.301 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/29' 2024-04-17T19:54:21.301 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/32' 2024-04-17T19:54:21.301 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/22' 2024-04-17T19:54:21.301 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/25' 2024-04-17T19:54:21.301 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/30' 2024-04-17T19:54:21.301 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/33' 2024-04-17T19:54:21.301 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/23' 2024-04-17T19:54:21.302 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/35' 2024-04-17T19:54:21.302 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/26' 2024-04-17T19:54:21.302 INFO:tasks.workunit.client.0.smithi029.stdout:removed './250/37' 2024-04-17T19:54:21.302 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './250' 2024-04-17T19:54:21.302 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/21' 2024-04-17T19:54:21.302 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/27' 2024-04-17T19:54:21.302 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/31' 2024-04-17T19:54:21.302 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/36' 2024-04-17T19:54:21.302 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/20' 2024-04-17T19:54:21.302 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/39' 2024-04-17T19:54:21.302 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/34' 2024-04-17T19:54:21.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/38' 2024-04-17T19:54:21.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/28' 2024-04-17T19:54:21.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/24' 2024-04-17T19:54:21.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/29' 2024-04-17T19:54:21.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/32' 2024-04-17T19:54:21.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/22' 2024-04-17T19:54:21.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/25' 2024-04-17T19:54:21.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/30' 2024-04-17T19:54:21.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/33' 2024-04-17T19:54:21.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/23' 2024-04-17T19:54:21.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/35' 2024-04-17T19:54:21.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/26' 2024-04-17T19:54:21.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './251/37' 2024-04-17T19:54:21.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './251' 2024-04-17T19:54:21.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/21' 2024-04-17T19:54:21.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/27' 2024-04-17T19:54:21.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/31' 2024-04-17T19:54:21.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/36' 2024-04-17T19:54:21.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/20' 2024-04-17T19:54:21.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/39' 2024-04-17T19:54:21.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/34' 2024-04-17T19:54:21.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/38' 2024-04-17T19:54:21.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/28' 2024-04-17T19:54:21.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/24' 2024-04-17T19:54:21.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/29' 2024-04-17T19:54:21.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/32' 2024-04-17T19:54:21.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/22' 2024-04-17T19:54:21.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/25' 2024-04-17T19:54:21.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/30' 2024-04-17T19:54:21.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/33' 2024-04-17T19:54:21.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/23' 2024-04-17T19:54:21.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/35' 2024-04-17T19:54:21.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/26' 2024-04-17T19:54:21.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './252/37' 2024-04-17T19:54:21.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './252' 2024-04-17T19:54:21.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/21' 2024-04-17T19:54:21.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/27' 2024-04-17T19:54:21.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/31' 2024-04-17T19:54:21.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/36' 2024-04-17T19:54:21.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/20' 2024-04-17T19:54:21.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/39' 2024-04-17T19:54:21.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/34' 2024-04-17T19:54:21.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/38' 2024-04-17T19:54:21.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/28' 2024-04-17T19:54:21.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/24' 2024-04-17T19:54:21.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/29' 2024-04-17T19:54:21.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/32' 2024-04-17T19:54:21.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/22' 2024-04-17T19:54:21.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/25' 2024-04-17T19:54:21.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/30' 2024-04-17T19:54:21.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/33' 2024-04-17T19:54:21.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/23' 2024-04-17T19:54:21.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/35' 2024-04-17T19:54:21.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/26' 2024-04-17T19:54:21.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './253/37' 2024-04-17T19:54:21.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './253' 2024-04-17T19:54:21.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/21' 2024-04-17T19:54:21.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/27' 2024-04-17T19:54:21.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/31' 2024-04-17T19:54:21.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/36' 2024-04-17T19:54:21.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/20' 2024-04-17T19:54:21.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/39' 2024-04-17T19:54:21.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/34' 2024-04-17T19:54:21.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/38' 2024-04-17T19:54:21.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/28' 2024-04-17T19:54:21.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/24' 2024-04-17T19:54:21.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/29' 2024-04-17T19:54:21.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/32' 2024-04-17T19:54:21.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/22' 2024-04-17T19:54:21.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/25' 2024-04-17T19:54:21.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/30' 2024-04-17T19:54:21.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/33' 2024-04-17T19:54:21.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/23' 2024-04-17T19:54:21.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/35' 2024-04-17T19:54:21.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/26' 2024-04-17T19:54:21.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './254/37' 2024-04-17T19:54:21.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './254' 2024-04-17T19:54:21.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/21' 2024-04-17T19:54:21.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/27' 2024-04-17T19:54:21.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/31' 2024-04-17T19:54:21.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/36' 2024-04-17T19:54:21.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/20' 2024-04-17T19:54:21.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/39' 2024-04-17T19:54:21.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/34' 2024-04-17T19:54:21.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/38' 2024-04-17T19:54:21.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/28' 2024-04-17T19:54:21.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/24' 2024-04-17T19:54:21.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/29' 2024-04-17T19:54:21.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/32' 2024-04-17T19:54:21.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/22' 2024-04-17T19:54:21.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/25' 2024-04-17T19:54:21.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/30' 2024-04-17T19:54:21.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/33' 2024-04-17T19:54:21.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/23' 2024-04-17T19:54:21.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/35' 2024-04-17T19:54:21.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/26' 2024-04-17T19:54:21.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './255/37' 2024-04-17T19:54:21.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './255' 2024-04-17T19:54:21.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/21' 2024-04-17T19:54:21.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/27' 2024-04-17T19:54:21.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/31' 2024-04-17T19:54:21.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/36' 2024-04-17T19:54:21.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/20' 2024-04-17T19:54:21.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/39' 2024-04-17T19:54:21.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/34' 2024-04-17T19:54:21.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/38' 2024-04-17T19:54:21.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/28' 2024-04-17T19:54:21.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/24' 2024-04-17T19:54:21.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/29' 2024-04-17T19:54:21.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/32' 2024-04-17T19:54:21.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/22' 2024-04-17T19:54:21.312 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/25' 2024-04-17T19:54:21.312 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/30' 2024-04-17T19:54:21.821 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:54:21.825 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:54:23.824 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/33' 2024-04-17T19:54:23.824 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/23' 2024-04-17T19:54:23.824 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/35' 2024-04-17T19:54:23.824 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/26' 2024-04-17T19:54:23.824 INFO:tasks.workunit.client.0.smithi029.stdout:removed './256/37' 2024-04-17T19:54:23.825 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './256' 2024-04-17T19:54:23.825 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/21' 2024-04-17T19:54:23.825 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/27' 2024-04-17T19:54:23.825 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/31' 2024-04-17T19:54:23.825 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/36' 2024-04-17T19:54:23.825 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/20' 2024-04-17T19:54:23.825 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/39' 2024-04-17T19:54:23.825 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/34' 2024-04-17T19:54:23.825 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/38' 2024-04-17T19:54:23.826 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/28' 2024-04-17T19:54:23.826 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/24' 2024-04-17T19:54:23.826 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/29' 2024-04-17T19:54:23.826 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/32' 2024-04-17T19:54:23.826 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/22' 2024-04-17T19:54:23.826 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/25' 2024-04-17T19:54:23.826 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/30' 2024-04-17T19:54:23.826 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/33' 2024-04-17T19:54:23.826 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/23' 2024-04-17T19:54:23.826 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/35' 2024-04-17T19:54:23.827 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/26' 2024-04-17T19:54:23.827 INFO:tasks.workunit.client.0.smithi029.stdout:removed './257/37' 2024-04-17T19:54:23.827 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './257' 2024-04-17T19:54:23.827 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/21' 2024-04-17T19:54:23.827 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/27' 2024-04-17T19:54:23.827 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/31' 2024-04-17T19:54:23.827 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/36' 2024-04-17T19:54:23.827 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/20' 2024-04-17T19:54:23.827 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/39' 2024-04-17T19:54:23.827 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/34' 2024-04-17T19:54:23.828 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/38' 2024-04-17T19:54:23.828 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/28' 2024-04-17T19:54:23.828 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/24' 2024-04-17T19:54:23.828 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/29' 2024-04-17T19:54:23.828 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/32' 2024-04-17T19:54:23.828 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/22' 2024-04-17T19:54:23.828 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/25' 2024-04-17T19:54:23.828 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/30' 2024-04-17T19:54:23.828 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/33' 2024-04-17T19:54:23.828 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/23' 2024-04-17T19:54:23.828 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/35' 2024-04-17T19:54:23.829 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/26' 2024-04-17T19:54:23.829 INFO:tasks.workunit.client.0.smithi029.stdout:removed './258/37' 2024-04-17T19:54:23.829 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './258' 2024-04-17T19:54:23.829 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/21' 2024-04-17T19:54:23.829 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/27' 2024-04-17T19:54:23.829 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/31' 2024-04-17T19:54:23.829 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/36' 2024-04-17T19:54:23.829 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/20' 2024-04-17T19:54:23.829 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/39' 2024-04-17T19:54:23.829 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/34' 2024-04-17T19:54:23.829 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/38' 2024-04-17T19:54:23.830 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/28' 2024-04-17T19:54:23.830 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/24' 2024-04-17T19:54:23.830 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/29' 2024-04-17T19:54:23.830 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/32' 2024-04-17T19:54:23.830 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/22' 2024-04-17T19:54:23.830 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/25' 2024-04-17T19:54:23.830 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/30' 2024-04-17T19:54:23.830 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/33' 2024-04-17T19:54:23.830 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/23' 2024-04-17T19:54:23.830 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/35' 2024-04-17T19:54:23.830 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/26' 2024-04-17T19:54:23.831 INFO:tasks.workunit.client.0.smithi029.stdout:removed './259/37' 2024-04-17T19:54:23.831 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './259' 2024-04-17T19:54:23.831 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/21' 2024-04-17T19:54:23.831 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/27' 2024-04-17T19:54:23.831 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/31' 2024-04-17T19:54:23.831 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/36' 2024-04-17T19:54:23.831 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/20' 2024-04-17T19:54:23.831 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/39' 2024-04-17T19:54:23.831 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/34' 2024-04-17T19:54:23.831 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/38' 2024-04-17T19:54:23.832 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/28' 2024-04-17T19:54:23.832 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/24' 2024-04-17T19:54:23.832 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/29' 2024-04-17T19:54:23.832 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/32' 2024-04-17T19:54:23.832 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/22' 2024-04-17T19:54:23.832 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/25' 2024-04-17T19:54:23.832 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/30' 2024-04-17T19:54:23.832 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/33' 2024-04-17T19:54:23.832 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/23' 2024-04-17T19:54:23.832 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/35' 2024-04-17T19:54:23.833 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/26' 2024-04-17T19:54:23.833 INFO:tasks.workunit.client.0.smithi029.stdout:removed './260/37' 2024-04-17T19:54:23.833 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './260' 2024-04-17T19:54:23.833 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/21' 2024-04-17T19:54:23.833 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/27' 2024-04-17T19:54:23.833 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/31' 2024-04-17T19:54:23.833 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/36' 2024-04-17T19:54:23.833 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/20' 2024-04-17T19:54:23.833 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/39' 2024-04-17T19:54:23.833 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/34' 2024-04-17T19:54:23.833 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/38' 2024-04-17T19:54:23.834 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/28' 2024-04-17T19:54:23.834 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/24' 2024-04-17T19:54:23.834 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/29' 2024-04-17T19:54:23.834 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/32' 2024-04-17T19:54:23.834 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/22' 2024-04-17T19:54:23.834 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/25' 2024-04-17T19:54:23.834 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/30' 2024-04-17T19:54:23.834 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/33' 2024-04-17T19:54:23.834 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/23' 2024-04-17T19:54:23.834 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/35' 2024-04-17T19:54:23.835 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/26' 2024-04-17T19:54:23.835 INFO:tasks.workunit.client.0.smithi029.stdout:removed './261/37' 2024-04-17T19:54:23.835 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './261' 2024-04-17T19:54:23.835 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/21' 2024-04-17T19:54:23.835 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/27' 2024-04-17T19:54:23.835 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/31' 2024-04-17T19:54:23.835 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/36' 2024-04-17T19:54:23.835 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/20' 2024-04-17T19:54:23.835 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/39' 2024-04-17T19:54:23.835 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/34' 2024-04-17T19:54:23.836 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/38' 2024-04-17T19:54:23.836 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/28' 2024-04-17T19:54:23.836 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/24' 2024-04-17T19:54:23.836 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/29' 2024-04-17T19:54:23.836 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/32' 2024-04-17T19:54:23.836 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/22' 2024-04-17T19:54:23.836 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/25' 2024-04-17T19:54:23.836 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/30' 2024-04-17T19:54:23.836 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/33' 2024-04-17T19:54:23.836 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/23' 2024-04-17T19:54:23.836 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/35' 2024-04-17T19:54:23.836 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/26' 2024-04-17T19:54:23.837 INFO:tasks.workunit.client.0.smithi029.stdout:removed './262/37' 2024-04-17T19:54:23.837 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './262' 2024-04-17T19:54:23.837 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/21' 2024-04-17T19:54:23.837 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/27' 2024-04-17T19:54:23.837 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/31' 2024-04-17T19:54:23.837 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/36' 2024-04-17T19:54:23.837 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/20' 2024-04-17T19:54:23.837 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/39' 2024-04-17T19:54:23.837 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/34' 2024-04-17T19:54:23.837 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/38' 2024-04-17T19:54:23.837 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/28' 2024-04-17T19:54:23.837 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/24' 2024-04-17T19:54:23.838 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/29' 2024-04-17T19:54:23.838 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/32' 2024-04-17T19:54:23.838 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/22' 2024-04-17T19:54:23.838 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/25' 2024-04-17T19:54:23.838 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/30' 2024-04-17T19:54:23.838 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/33' 2024-04-17T19:54:23.838 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/23' 2024-04-17T19:54:23.838 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/35' 2024-04-17T19:54:23.838 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/26' 2024-04-17T19:54:23.838 INFO:tasks.workunit.client.0.smithi029.stdout:removed './263/37' 2024-04-17T19:54:23.839 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './263' 2024-04-17T19:54:23.839 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/21' 2024-04-17T19:54:23.839 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/27' 2024-04-17T19:54:23.839 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/31' 2024-04-17T19:54:23.839 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/36' 2024-04-17T19:54:23.839 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/20' 2024-04-17T19:54:23.839 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/39' 2024-04-17T19:54:23.839 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/34' 2024-04-17T19:54:23.839 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/38' 2024-04-17T19:54:23.839 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/28' 2024-04-17T19:54:23.840 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/24' 2024-04-17T19:54:23.840 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/29' 2024-04-17T19:54:23.840 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/32' 2024-04-17T19:54:23.840 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/22' 2024-04-17T19:54:23.840 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/25' 2024-04-17T19:54:23.840 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/30' 2024-04-17T19:54:23.840 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/33' 2024-04-17T19:54:23.840 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/23' 2024-04-17T19:54:23.840 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/35' 2024-04-17T19:54:23.840 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/26' 2024-04-17T19:54:23.840 INFO:tasks.workunit.client.0.smithi029.stdout:removed './264/37' 2024-04-17T19:54:23.841 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './264' 2024-04-17T19:54:23.841 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/21' 2024-04-17T19:54:23.841 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/27' 2024-04-17T19:54:23.841 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/31' 2024-04-17T19:54:23.841 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/36' 2024-04-17T19:54:23.841 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/20' 2024-04-17T19:54:23.841 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/39' 2024-04-17T19:54:23.841 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/34' 2024-04-17T19:54:23.841 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/38' 2024-04-17T19:54:23.841 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/28' 2024-04-17T19:54:23.841 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/24' 2024-04-17T19:54:23.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/29' 2024-04-17T19:54:23.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/32' 2024-04-17T19:54:23.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/22' 2024-04-17T19:54:23.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/25' 2024-04-17T19:54:23.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/30' 2024-04-17T19:54:23.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/33' 2024-04-17T19:54:23.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/23' 2024-04-17T19:54:23.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/35' 2024-04-17T19:54:23.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/26' 2024-04-17T19:54:23.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './265/37' 2024-04-17T19:54:23.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './265' 2024-04-17T19:54:23.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/21' 2024-04-17T19:54:23.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/27' 2024-04-17T19:54:23.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/31' 2024-04-17T19:54:23.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/36' 2024-04-17T19:54:23.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/20' 2024-04-17T19:54:23.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/39' 2024-04-17T19:54:23.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/34' 2024-04-17T19:54:23.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/38' 2024-04-17T19:54:23.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/28' 2024-04-17T19:54:23.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/24' 2024-04-17T19:54:23.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/29' 2024-04-17T19:54:23.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/32' 2024-04-17T19:54:23.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/22' 2024-04-17T19:54:23.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/25' 2024-04-17T19:54:23.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/30' 2024-04-17T19:54:23.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/33' 2024-04-17T19:54:23.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/23' 2024-04-17T19:54:25.451 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/35' 2024-04-17T19:54:25.451 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/26' 2024-04-17T19:54:25.451 INFO:tasks.workunit.client.0.smithi029.stdout:removed './266/37' 2024-04-17T19:54:25.451 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './266' 2024-04-17T19:54:25.451 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/21' 2024-04-17T19:54:25.451 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/27' 2024-04-17T19:54:25.451 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/31' 2024-04-17T19:54:25.451 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/36' 2024-04-17T19:54:25.451 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/20' 2024-04-17T19:54:25.451 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/39' 2024-04-17T19:54:25.451 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/34' 2024-04-17T19:54:25.451 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/38' 2024-04-17T19:54:25.452 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/28' 2024-04-17T19:54:25.452 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/24' 2024-04-17T19:54:25.452 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/29' 2024-04-17T19:54:25.452 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/32' 2024-04-17T19:54:25.452 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/22' 2024-04-17T19:54:25.452 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/25' 2024-04-17T19:54:25.452 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/30' 2024-04-17T19:54:25.452 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/33' 2024-04-17T19:54:25.452 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/23' 2024-04-17T19:54:25.452 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/35' 2024-04-17T19:54:25.452 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/26' 2024-04-17T19:54:25.452 INFO:tasks.workunit.client.0.smithi029.stdout:removed './267/37' 2024-04-17T19:54:25.452 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './267' 2024-04-17T19:54:25.452 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/21' 2024-04-17T19:54:25.452 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/27' 2024-04-17T19:54:25.452 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/31' 2024-04-17T19:54:25.453 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/36' 2024-04-17T19:54:25.453 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/20' 2024-04-17T19:54:25.453 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/39' 2024-04-17T19:54:25.453 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/34' 2024-04-17T19:54:25.453 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/38' 2024-04-17T19:54:25.453 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/28' 2024-04-17T19:54:25.453 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/24' 2024-04-17T19:54:25.453 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/29' 2024-04-17T19:54:25.453 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/32' 2024-04-17T19:54:25.453 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/22' 2024-04-17T19:54:25.453 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/25' 2024-04-17T19:54:25.453 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/30' 2024-04-17T19:54:25.453 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/33' 2024-04-17T19:54:25.453 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/23' 2024-04-17T19:54:25.453 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/35' 2024-04-17T19:54:25.453 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/26' 2024-04-17T19:54:25.453 INFO:tasks.workunit.client.0.smithi029.stdout:removed './268/37' 2024-04-17T19:54:25.454 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './268' 2024-04-17T19:54:25.454 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/21' 2024-04-17T19:54:25.454 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/27' 2024-04-17T19:54:25.454 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/31' 2024-04-17T19:54:25.454 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/36' 2024-04-17T19:54:25.454 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/20' 2024-04-17T19:54:25.454 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/39' 2024-04-17T19:54:25.454 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/34' 2024-04-17T19:54:25.454 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/38' 2024-04-17T19:54:25.454 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/28' 2024-04-17T19:54:25.454 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/24' 2024-04-17T19:54:25.454 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/29' 2024-04-17T19:54:25.454 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/32' 2024-04-17T19:54:25.454 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/22' 2024-04-17T19:54:25.454 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/25' 2024-04-17T19:54:25.455 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/30' 2024-04-17T19:54:25.455 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/33' 2024-04-17T19:54:25.455 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/23' 2024-04-17T19:54:25.455 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/35' 2024-04-17T19:54:25.455 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/26' 2024-04-17T19:54:25.455 INFO:tasks.workunit.client.0.smithi029.stdout:removed './269/37' 2024-04-17T19:54:25.455 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './269' 2024-04-17T19:54:25.455 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/21' 2024-04-17T19:54:25.455 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/27' 2024-04-17T19:54:25.455 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/31' 2024-04-17T19:54:25.455 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/36' 2024-04-17T19:54:25.455 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/20' 2024-04-17T19:54:25.455 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/39' 2024-04-17T19:54:25.455 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/34' 2024-04-17T19:54:25.455 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/38' 2024-04-17T19:54:25.455 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/28' 2024-04-17T19:54:25.455 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/24' 2024-04-17T19:54:25.455 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/29' 2024-04-17T19:54:25.456 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/32' 2024-04-17T19:54:25.456 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/22' 2024-04-17T19:54:25.456 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/25' 2024-04-17T19:54:25.456 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/30' 2024-04-17T19:54:25.456 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/33' 2024-04-17T19:54:25.456 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/23' 2024-04-17T19:54:25.456 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/35' 2024-04-17T19:54:25.456 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/26' 2024-04-17T19:54:25.456 INFO:tasks.workunit.client.0.smithi029.stdout:removed './270/37' 2024-04-17T19:54:25.456 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './270' 2024-04-17T19:54:25.456 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/21' 2024-04-17T19:54:25.456 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/27' 2024-04-17T19:54:25.456 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/31' 2024-04-17T19:54:25.456 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/36' 2024-04-17T19:54:25.456 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/20' 2024-04-17T19:54:25.456 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/39' 2024-04-17T19:54:25.456 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/34' 2024-04-17T19:54:25.457 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/38' 2024-04-17T19:54:25.457 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/28' 2024-04-17T19:54:25.457 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/24' 2024-04-17T19:54:25.457 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/29' 2024-04-17T19:54:25.457 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/32' 2024-04-17T19:54:25.457 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/22' 2024-04-17T19:54:25.457 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/25' 2024-04-17T19:54:25.457 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/30' 2024-04-17T19:54:25.457 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/33' 2024-04-17T19:54:25.457 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/23' 2024-04-17T19:54:25.457 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/35' 2024-04-17T19:54:25.457 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/26' 2024-04-17T19:54:25.457 INFO:tasks.workunit.client.0.smithi029.stdout:removed './271/37' 2024-04-17T19:54:25.457 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './271' 2024-04-17T19:54:25.457 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/21' 2024-04-17T19:54:25.457 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/27' 2024-04-17T19:54:25.458 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/31' 2024-04-17T19:54:25.458 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/36' 2024-04-17T19:54:25.458 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/20' 2024-04-17T19:54:25.458 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/39' 2024-04-17T19:54:25.458 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/34' 2024-04-17T19:54:25.458 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/38' 2024-04-17T19:54:25.458 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/28' 2024-04-17T19:54:25.458 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/24' 2024-04-17T19:54:25.458 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/29' 2024-04-17T19:54:25.458 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/32' 2024-04-17T19:54:25.458 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/22' 2024-04-17T19:54:25.458 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/25' 2024-04-17T19:54:25.458 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/30' 2024-04-17T19:54:25.458 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/33' 2024-04-17T19:54:25.458 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/23' 2024-04-17T19:54:25.458 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/35' 2024-04-17T19:54:25.459 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/26' 2024-04-17T19:54:25.459 INFO:tasks.workunit.client.0.smithi029.stdout:removed './272/37' 2024-04-17T19:54:25.459 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './272' 2024-04-17T19:54:25.459 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/21' 2024-04-17T19:54:25.459 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/27' 2024-04-17T19:54:25.459 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/31' 2024-04-17T19:54:25.459 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/36' 2024-04-17T19:54:25.459 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/20' 2024-04-17T19:54:25.459 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/39' 2024-04-17T19:54:25.459 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/34' 2024-04-17T19:54:25.459 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/38' 2024-04-17T19:54:25.459 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/28' 2024-04-17T19:54:25.459 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/24' 2024-04-17T19:54:25.459 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/29' 2024-04-17T19:54:25.459 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/32' 2024-04-17T19:54:25.459 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/22' 2024-04-17T19:54:25.459 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/25' 2024-04-17T19:54:25.460 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/30' 2024-04-17T19:54:25.460 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/33' 2024-04-17T19:54:25.460 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/23' 2024-04-17T19:54:25.460 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/35' 2024-04-17T19:54:25.460 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/26' 2024-04-17T19:54:25.460 INFO:tasks.workunit.client.0.smithi029.stdout:removed './273/37' 2024-04-17T19:54:25.460 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './273' 2024-04-17T19:54:25.460 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/21' 2024-04-17T19:54:25.460 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/27' 2024-04-17T19:54:25.460 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/31' 2024-04-17T19:54:25.460 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/36' 2024-04-17T19:54:25.460 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/20' 2024-04-17T19:54:25.460 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/39' 2024-04-17T19:54:25.460 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/34' 2024-04-17T19:54:25.460 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/38' 2024-04-17T19:54:25.460 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/28' 2024-04-17T19:54:25.460 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/24' 2024-04-17T19:54:25.461 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/29' 2024-04-17T19:54:25.461 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/32' 2024-04-17T19:54:25.461 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/22' 2024-04-17T19:54:25.461 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/25' 2024-04-17T19:54:25.461 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/30' 2024-04-17T19:54:25.461 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/33' 2024-04-17T19:54:25.461 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/23' 2024-04-17T19:54:25.461 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/35' 2024-04-17T19:54:25.461 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/26' 2024-04-17T19:54:25.461 INFO:tasks.workunit.client.0.smithi029.stdout:removed './274/37' 2024-04-17T19:54:25.461 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './274' 2024-04-17T19:54:25.461 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/21' 2024-04-17T19:54:25.461 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/27' 2024-04-17T19:54:25.461 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/31' 2024-04-17T19:54:25.461 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/36' 2024-04-17T19:54:25.461 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/20' 2024-04-17T19:54:25.461 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/39' 2024-04-17T19:54:25.462 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/34' 2024-04-17T19:54:25.462 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/38' 2024-04-17T19:54:25.462 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/28' 2024-04-17T19:54:25.462 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/24' 2024-04-17T19:54:25.462 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/29' 2024-04-17T19:54:25.462 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/32' 2024-04-17T19:54:25.462 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/22' 2024-04-17T19:54:25.462 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/25' 2024-04-17T19:54:25.462 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/30' 2024-04-17T19:54:25.462 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/33' 2024-04-17T19:54:25.462 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/23' 2024-04-17T19:54:25.462 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/35' 2024-04-17T19:54:25.462 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/26' 2024-04-17T19:54:25.462 INFO:tasks.workunit.client.0.smithi029.stdout:removed './275/37' 2024-04-17T19:54:25.462 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './275' 2024-04-17T19:54:25.462 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/21' 2024-04-17T19:54:25.463 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/27' 2024-04-17T19:54:25.463 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/31' 2024-04-17T19:54:25.463 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/36' 2024-04-17T19:54:25.463 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/20' 2024-04-17T19:54:25.463 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/39' 2024-04-17T19:54:25.463 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/34' 2024-04-17T19:54:25.463 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/38' 2024-04-17T19:54:25.463 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/28' 2024-04-17T19:54:25.463 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/24' 2024-04-17T19:54:25.463 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/29' 2024-04-17T19:54:25.463 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/32' 2024-04-17T19:54:25.463 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/22' 2024-04-17T19:54:25.463 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/25' 2024-04-17T19:54:25.463 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/30' 2024-04-17T19:54:25.463 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/33' 2024-04-17T19:54:25.463 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/23' 2024-04-17T19:54:25.464 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/35' 2024-04-17T19:54:25.464 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/26' 2024-04-17T19:54:27.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './276/37' 2024-04-17T19:54:27.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './276' 2024-04-17T19:54:27.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/21' 2024-04-17T19:54:27.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/27' 2024-04-17T19:54:27.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/31' 2024-04-17T19:54:27.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/36' 2024-04-17T19:54:27.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/20' 2024-04-17T19:54:27.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/39' 2024-04-17T19:54:27.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/34' 2024-04-17T19:54:27.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/38' 2024-04-17T19:54:27.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/28' 2024-04-17T19:54:27.303 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/24' 2024-04-17T19:54:27.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/29' 2024-04-17T19:54:27.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/32' 2024-04-17T19:54:27.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/22' 2024-04-17T19:54:27.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/25' 2024-04-17T19:54:27.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/30' 2024-04-17T19:54:27.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/33' 2024-04-17T19:54:27.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/23' 2024-04-17T19:54:27.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/35' 2024-04-17T19:54:27.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/26' 2024-04-17T19:54:27.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './277/37' 2024-04-17T19:54:27.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './277' 2024-04-17T19:54:27.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/21' 2024-04-17T19:54:27.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/27' 2024-04-17T19:54:27.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/31' 2024-04-17T19:54:27.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/36' 2024-04-17T19:54:27.304 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/20' 2024-04-17T19:54:27.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/39' 2024-04-17T19:54:27.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/34' 2024-04-17T19:54:27.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/38' 2024-04-17T19:54:27.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/28' 2024-04-17T19:54:27.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/24' 2024-04-17T19:54:27.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/29' 2024-04-17T19:54:27.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/32' 2024-04-17T19:54:27.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/22' 2024-04-17T19:54:27.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/25' 2024-04-17T19:54:27.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/30' 2024-04-17T19:54:27.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/33' 2024-04-17T19:54:27.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/23' 2024-04-17T19:54:27.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/35' 2024-04-17T19:54:27.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/26' 2024-04-17T19:54:27.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './278/37' 2024-04-17T19:54:27.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './278' 2024-04-17T19:54:27.305 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/21' 2024-04-17T19:54:27.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/27' 2024-04-17T19:54:27.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/31' 2024-04-17T19:54:27.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/36' 2024-04-17T19:54:27.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/20' 2024-04-17T19:54:27.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/39' 2024-04-17T19:54:27.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/34' 2024-04-17T19:54:27.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/38' 2024-04-17T19:54:27.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/28' 2024-04-17T19:54:27.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/24' 2024-04-17T19:54:27.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/29' 2024-04-17T19:54:27.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/32' 2024-04-17T19:54:27.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/22' 2024-04-17T19:54:27.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/25' 2024-04-17T19:54:27.306 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/30' 2024-04-17T19:54:27.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/33' 2024-04-17T19:54:27.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/23' 2024-04-17T19:54:27.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/35' 2024-04-17T19:54:27.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/26' 2024-04-17T19:54:27.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './279/37' 2024-04-17T19:54:27.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './279' 2024-04-17T19:54:27.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/21' 2024-04-17T19:54:27.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/27' 2024-04-17T19:54:27.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/31' 2024-04-17T19:54:27.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/36' 2024-04-17T19:54:27.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/20' 2024-04-17T19:54:27.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/39' 2024-04-17T19:54:27.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/34' 2024-04-17T19:54:27.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/38' 2024-04-17T19:54:27.307 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/28' 2024-04-17T19:54:27.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/24' 2024-04-17T19:54:27.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/29' 2024-04-17T19:54:27.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/32' 2024-04-17T19:54:27.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/22' 2024-04-17T19:54:27.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/25' 2024-04-17T19:54:27.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/30' 2024-04-17T19:54:27.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/33' 2024-04-17T19:54:27.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/23' 2024-04-17T19:54:27.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/35' 2024-04-17T19:54:27.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/26' 2024-04-17T19:54:27.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './280/37' 2024-04-17T19:54:27.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './280' 2024-04-17T19:54:27.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/21' 2024-04-17T19:54:27.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/27' 2024-04-17T19:54:27.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/31' 2024-04-17T19:54:27.308 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/36' 2024-04-17T19:54:27.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/20' 2024-04-17T19:54:27.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/39' 2024-04-17T19:54:27.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/34' 2024-04-17T19:54:27.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/38' 2024-04-17T19:54:27.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/28' 2024-04-17T19:54:27.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/24' 2024-04-17T19:54:27.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/29' 2024-04-17T19:54:27.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/32' 2024-04-17T19:54:27.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/22' 2024-04-17T19:54:27.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/25' 2024-04-17T19:54:27.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/30' 2024-04-17T19:54:27.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/33' 2024-04-17T19:54:27.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/23' 2024-04-17T19:54:27.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/35' 2024-04-17T19:54:27.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/26' 2024-04-17T19:54:27.309 INFO:tasks.workunit.client.0.smithi029.stdout:removed './281/37' 2024-04-17T19:54:27.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './281' 2024-04-17T19:54:27.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/21' 2024-04-17T19:54:27.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/27' 2024-04-17T19:54:27.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/31' 2024-04-17T19:54:27.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/36' 2024-04-17T19:54:27.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/20' 2024-04-17T19:54:27.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/39' 2024-04-17T19:54:27.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/34' 2024-04-17T19:54:27.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/38' 2024-04-17T19:54:27.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/28' 2024-04-17T19:54:27.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/24' 2024-04-17T19:54:27.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/29' 2024-04-17T19:54:27.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/32' 2024-04-17T19:54:27.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/22' 2024-04-17T19:54:27.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/25' 2024-04-17T19:54:27.310 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/30' 2024-04-17T19:54:27.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/33' 2024-04-17T19:54:27.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/23' 2024-04-17T19:54:27.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/35' 2024-04-17T19:54:27.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/26' 2024-04-17T19:54:27.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './282/37' 2024-04-17T19:54:27.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './282' 2024-04-17T19:54:27.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/21' 2024-04-17T19:54:27.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/27' 2024-04-17T19:54:27.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/31' 2024-04-17T19:54:27.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/36' 2024-04-17T19:54:27.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/20' 2024-04-17T19:54:27.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/39' 2024-04-17T19:54:27.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/34' 2024-04-17T19:54:27.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/38' 2024-04-17T19:54:27.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/28' 2024-04-17T19:54:27.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/24' 2024-04-17T19:54:27.311 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/29' 2024-04-17T19:54:27.312 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/32' 2024-04-17T19:54:27.312 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/22' 2024-04-17T19:54:27.312 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/25' 2024-04-17T19:54:27.312 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/30' 2024-04-17T19:54:27.312 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/33' 2024-04-17T19:54:27.312 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/23' 2024-04-17T19:54:27.312 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/35' 2024-04-17T19:54:27.312 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/26' 2024-04-17T19:54:27.312 INFO:tasks.workunit.client.0.smithi029.stdout:removed './283/37' 2024-04-17T19:54:27.312 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './283' 2024-04-17T19:54:27.312 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/21' 2024-04-17T19:54:27.312 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/27' 2024-04-17T19:54:27.312 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/31' 2024-04-17T19:54:27.312 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/36' 2024-04-17T19:54:27.312 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/20' 2024-04-17T19:54:27.312 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/39' 2024-04-17T19:54:27.313 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/34' 2024-04-17T19:54:27.313 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/38' 2024-04-17T19:54:27.313 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/28' 2024-04-17T19:54:27.313 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/24' 2024-04-17T19:54:27.313 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/29' 2024-04-17T19:54:27.313 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/32' 2024-04-17T19:54:27.313 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/22' 2024-04-17T19:54:27.313 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/25' 2024-04-17T19:54:27.313 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/30' 2024-04-17T19:54:27.313 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/33' 2024-04-17T19:54:27.313 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/23' 2024-04-17T19:54:27.313 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/35' 2024-04-17T19:54:27.313 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/26' 2024-04-17T19:54:27.313 INFO:tasks.workunit.client.0.smithi029.stdout:removed './284/37' 2024-04-17T19:54:27.313 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './284' 2024-04-17T19:54:27.313 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/21' 2024-04-17T19:54:27.314 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/27' 2024-04-17T19:54:27.314 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/31' 2024-04-17T19:54:27.314 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/36' 2024-04-17T19:54:27.314 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/20' 2024-04-17T19:54:27.314 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/39' 2024-04-17T19:54:27.314 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/34' 2024-04-17T19:54:27.314 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/38' 2024-04-17T19:54:27.314 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/28' 2024-04-17T19:54:27.314 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/24' 2024-04-17T19:54:27.314 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/29' 2024-04-17T19:54:27.314 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/32' 2024-04-17T19:54:27.314 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/22' 2024-04-17T19:54:27.314 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/25' 2024-04-17T19:54:27.314 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/30' 2024-04-17T19:54:27.314 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/33' 2024-04-17T19:54:27.314 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/23' 2024-04-17T19:54:27.315 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/35' 2024-04-17T19:54:27.315 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/26' 2024-04-17T19:54:27.315 INFO:tasks.workunit.client.0.smithi029.stdout:removed './285/37' 2024-04-17T19:54:27.315 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './285' 2024-04-17T19:54:27.315 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/21' 2024-04-17T19:54:27.315 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/27' 2024-04-17T19:54:27.315 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/31' 2024-04-17T19:54:27.315 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/36' 2024-04-17T19:54:27.315 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/20' 2024-04-17T19:54:27.315 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/39' 2024-04-17T19:54:27.315 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/34' 2024-04-17T19:54:27.315 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/38' 2024-04-17T19:54:27.315 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/28' 2024-04-17T19:54:27.315 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/24' 2024-04-17T19:54:27.315 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/29' 2024-04-17T19:54:27.316 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/32' 2024-04-17T19:54:27.316 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/22' 2024-04-17T19:54:27.316 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/25' 2024-04-17T19:54:27.316 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/30' 2024-04-17T19:54:27.316 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/33' 2024-04-17T19:54:27.316 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/23' 2024-04-17T19:54:27.316 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/35' 2024-04-17T19:54:27.316 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/26' 2024-04-17T19:54:27.316 INFO:tasks.workunit.client.0.smithi029.stdout:removed './286/37' 2024-04-17T19:54:27.316 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './286' 2024-04-17T19:54:28.232 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/21' 2024-04-17T19:54:28.232 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/27' 2024-04-17T19:54:28.232 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/31' 2024-04-17T19:54:28.232 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/36' 2024-04-17T19:54:28.233 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/20' 2024-04-17T19:54:28.233 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/39' 2024-04-17T19:54:28.233 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/34' 2024-04-17T19:54:28.233 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/38' 2024-04-17T19:54:28.233 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/28' 2024-04-17T19:54:28.233 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/24' 2024-04-17T19:54:28.233 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/29' 2024-04-17T19:54:28.233 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/32' 2024-04-17T19:54:28.233 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/22' 2024-04-17T19:54:28.233 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/25' 2024-04-17T19:54:28.233 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/30' 2024-04-17T19:54:28.233 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/33' 2024-04-17T19:54:28.233 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/23' 2024-04-17T19:54:28.233 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/35' 2024-04-17T19:54:28.234 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/26' 2024-04-17T19:54:28.234 INFO:tasks.workunit.client.0.smithi029.stdout:removed './287/37' 2024-04-17T19:54:28.234 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './287' 2024-04-17T19:54:28.234 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/21' 2024-04-17T19:54:28.234 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/27' 2024-04-17T19:54:28.234 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/31' 2024-04-17T19:54:28.234 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/36' 2024-04-17T19:54:28.234 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/20' 2024-04-17T19:54:28.234 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/39' 2024-04-17T19:54:28.234 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/34' 2024-04-17T19:54:28.234 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/38' 2024-04-17T19:54:28.234 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/28' 2024-04-17T19:54:28.234 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/24' 2024-04-17T19:54:28.234 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/29' 2024-04-17T19:54:28.235 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/32' 2024-04-17T19:54:28.235 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/22' 2024-04-17T19:54:28.235 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/25' 2024-04-17T19:54:28.235 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/30' 2024-04-17T19:54:28.235 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/33' 2024-04-17T19:54:28.235 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/23' 2024-04-17T19:54:28.235 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/35' 2024-04-17T19:54:28.235 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/26' 2024-04-17T19:54:28.235 INFO:tasks.workunit.client.0.smithi029.stdout:removed './288/37' 2024-04-17T19:54:28.235 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './288' 2024-04-17T19:54:28.235 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/21' 2024-04-17T19:54:28.235 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/27' 2024-04-17T19:54:28.235 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/31' 2024-04-17T19:54:28.235 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/36' 2024-04-17T19:54:28.235 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/20' 2024-04-17T19:54:28.235 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/39' 2024-04-17T19:54:28.236 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/34' 2024-04-17T19:54:28.236 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/38' 2024-04-17T19:54:28.236 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/28' 2024-04-17T19:54:28.236 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/24' 2024-04-17T19:54:28.236 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/29' 2024-04-17T19:54:28.236 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/32' 2024-04-17T19:54:28.236 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/22' 2024-04-17T19:54:28.236 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/25' 2024-04-17T19:54:28.236 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/30' 2024-04-17T19:54:28.236 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/33' 2024-04-17T19:54:28.236 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/23' 2024-04-17T19:54:28.236 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/35' 2024-04-17T19:54:28.236 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/26' 2024-04-17T19:54:28.236 INFO:tasks.workunit.client.0.smithi029.stdout:removed './289/37' 2024-04-17T19:54:28.236 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './289' 2024-04-17T19:54:28.236 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/21' 2024-04-17T19:54:28.236 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/27' 2024-04-17T19:54:28.237 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/31' 2024-04-17T19:54:28.237 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/36' 2024-04-17T19:54:28.237 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/20' 2024-04-17T19:54:28.237 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/39' 2024-04-17T19:54:28.237 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/34' 2024-04-17T19:54:28.237 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/38' 2024-04-17T19:54:28.237 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/28' 2024-04-17T19:54:28.237 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/24' 2024-04-17T19:54:28.237 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/29' 2024-04-17T19:54:28.237 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/32' 2024-04-17T19:54:28.237 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/22' 2024-04-17T19:54:28.237 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/25' 2024-04-17T19:54:28.237 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/30' 2024-04-17T19:54:28.237 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/33' 2024-04-17T19:54:28.237 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/23' 2024-04-17T19:54:28.237 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/35' 2024-04-17T19:54:28.237 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/26' 2024-04-17T19:54:28.238 INFO:tasks.workunit.client.0.smithi029.stdout:removed './290/37' 2024-04-17T19:54:28.238 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './290' 2024-04-17T19:54:28.238 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/21' 2024-04-17T19:54:28.238 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/27' 2024-04-17T19:54:28.238 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/31' 2024-04-17T19:54:28.238 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/36' 2024-04-17T19:54:28.238 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/20' 2024-04-17T19:54:28.238 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/39' 2024-04-17T19:54:28.238 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/34' 2024-04-17T19:54:28.238 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/38' 2024-04-17T19:54:28.238 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/28' 2024-04-17T19:54:28.238 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/24' 2024-04-17T19:54:28.238 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/29' 2024-04-17T19:54:28.238 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/32' 2024-04-17T19:54:28.238 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/22' 2024-04-17T19:54:28.238 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/25' 2024-04-17T19:54:28.239 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/30' 2024-04-17T19:54:28.239 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/33' 2024-04-17T19:54:28.239 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/23' 2024-04-17T19:54:28.239 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/35' 2024-04-17T19:54:28.239 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/26' 2024-04-17T19:54:28.239 INFO:tasks.workunit.client.0.smithi029.stdout:removed './291/37' 2024-04-17T19:54:28.239 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './291' 2024-04-17T19:54:28.239 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/21' 2024-04-17T19:54:28.239 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/27' 2024-04-17T19:54:28.239 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/31' 2024-04-17T19:54:28.239 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/36' 2024-04-17T19:54:28.239 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/20' 2024-04-17T19:54:28.239 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/39' 2024-04-17T19:54:28.239 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/34' 2024-04-17T19:54:28.239 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/38' 2024-04-17T19:54:28.239 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/28' 2024-04-17T19:54:28.240 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/24' 2024-04-17T19:54:28.240 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/29' 2024-04-17T19:54:28.240 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/32' 2024-04-17T19:54:28.240 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/22' 2024-04-17T19:54:28.240 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/25' 2024-04-17T19:54:28.240 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/30' 2024-04-17T19:54:28.240 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/33' 2024-04-17T19:54:28.240 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/23' 2024-04-17T19:54:28.240 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/35' 2024-04-17T19:54:28.240 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/26' 2024-04-17T19:54:28.240 INFO:tasks.workunit.client.0.smithi029.stdout:removed './292/37' 2024-04-17T19:54:28.240 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './292' 2024-04-17T19:54:28.240 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/21' 2024-04-17T19:54:28.240 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/27' 2024-04-17T19:54:28.240 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/31' 2024-04-17T19:54:28.240 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/36' 2024-04-17T19:54:28.241 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/20' 2024-04-17T19:54:28.241 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/39' 2024-04-17T19:54:28.241 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/34' 2024-04-17T19:54:28.241 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/38' 2024-04-17T19:54:28.241 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/28' 2024-04-17T19:54:28.241 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/24' 2024-04-17T19:54:28.241 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/29' 2024-04-17T19:54:28.241 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/32' 2024-04-17T19:54:28.241 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/22' 2024-04-17T19:54:28.241 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/25' 2024-04-17T19:54:28.241 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/30' 2024-04-17T19:54:28.241 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/33' 2024-04-17T19:54:28.241 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/23' 2024-04-17T19:54:28.241 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/35' 2024-04-17T19:54:28.241 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/26' 2024-04-17T19:54:28.242 INFO:tasks.workunit.client.0.smithi029.stdout:removed './293/37' 2024-04-17T19:54:28.242 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './293' 2024-04-17T19:54:28.242 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/21' 2024-04-17T19:54:28.242 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/27' 2024-04-17T19:54:28.242 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/31' 2024-04-17T19:54:28.242 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/36' 2024-04-17T19:54:28.242 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/20' 2024-04-17T19:54:28.242 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/39' 2024-04-17T19:54:28.242 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/34' 2024-04-17T19:54:28.242 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/38' 2024-04-17T19:54:28.242 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/28' 2024-04-17T19:54:28.242 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/24' 2024-04-17T19:54:28.242 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/29' 2024-04-17T19:54:28.242 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/32' 2024-04-17T19:54:28.242 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/22' 2024-04-17T19:54:28.243 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/25' 2024-04-17T19:54:28.243 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/30' 2024-04-17T19:54:28.243 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/33' 2024-04-17T19:54:28.243 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/23' 2024-04-17T19:54:28.243 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/35' 2024-04-17T19:54:28.243 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/26' 2024-04-17T19:54:28.243 INFO:tasks.workunit.client.0.smithi029.stdout:removed './294/37' 2024-04-17T19:54:28.243 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './294' 2024-04-17T19:54:28.243 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/21' 2024-04-17T19:54:28.243 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/27' 2024-04-17T19:54:28.243 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/31' 2024-04-17T19:54:28.243 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/36' 2024-04-17T19:54:28.243 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/20' 2024-04-17T19:54:28.243 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/39' 2024-04-17T19:54:28.243 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/34' 2024-04-17T19:54:28.243 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/38' 2024-04-17T19:54:28.243 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/28' 2024-04-17T19:54:28.244 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/24' 2024-04-17T19:54:28.244 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/29' 2024-04-17T19:54:28.244 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/32' 2024-04-17T19:54:28.244 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/22' 2024-04-17T19:54:28.244 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/25' 2024-04-17T19:54:28.244 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/30' 2024-04-17T19:54:28.244 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/33' 2024-04-17T19:54:28.244 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/23' 2024-04-17T19:54:28.244 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/35' 2024-04-17T19:54:28.244 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/26' 2024-04-17T19:54:28.244 INFO:tasks.workunit.client.0.smithi029.stdout:removed './295/37' 2024-04-17T19:54:28.244 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './295' 2024-04-17T19:54:28.244 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/21' 2024-04-17T19:54:28.244 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/27' 2024-04-17T19:54:28.244 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/31' 2024-04-17T19:54:28.244 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/36' 2024-04-17T19:54:28.244 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/20' 2024-04-17T19:54:28.244 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/39' 2024-04-17T19:54:28.245 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/34' 2024-04-17T19:54:28.245 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/38' 2024-04-17T19:54:28.245 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/28' 2024-04-17T19:54:28.245 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/24' 2024-04-17T19:54:28.245 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/29' 2024-04-17T19:54:28.245 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/32' 2024-04-17T19:54:28.245 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/22' 2024-04-17T19:54:28.245 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/25' 2024-04-17T19:54:28.245 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/30' 2024-04-17T19:54:28.245 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/33' 2024-04-17T19:54:28.245 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/23' 2024-04-17T19:54:28.245 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/35' 2024-04-17T19:54:28.245 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/26' 2024-04-17T19:54:28.246 INFO:tasks.workunit.client.0.smithi029.stdout:removed './296/37' 2024-04-17T19:54:28.246 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './296' 2024-04-17T19:54:28.246 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/21' 2024-04-17T19:54:28.246 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/27' 2024-04-17T19:54:28.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/31' 2024-04-17T19:54:28.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/36' 2024-04-17T19:54:28.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/20' 2024-04-17T19:54:28.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/39' 2024-04-17T19:54:28.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/34' 2024-04-17T19:54:28.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/38' 2024-04-17T19:54:28.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/28' 2024-04-17T19:54:28.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/24' 2024-04-17T19:54:28.842 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/29' 2024-04-17T19:54:28.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/32' 2024-04-17T19:54:28.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/22' 2024-04-17T19:54:28.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/25' 2024-04-17T19:54:28.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/30' 2024-04-17T19:54:28.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/33' 2024-04-17T19:54:28.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/23' 2024-04-17T19:54:28.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/35' 2024-04-17T19:54:28.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/26' 2024-04-17T19:54:28.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './297/37' 2024-04-17T19:54:28.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './297' 2024-04-17T19:54:28.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/21' 2024-04-17T19:54:28.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/27' 2024-04-17T19:54:28.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/31' 2024-04-17T19:54:28.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/36' 2024-04-17T19:54:28.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/20' 2024-04-17T19:54:28.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/39' 2024-04-17T19:54:28.843 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/34' 2024-04-17T19:54:28.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/38' 2024-04-17T19:54:28.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/28' 2024-04-17T19:54:28.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/24' 2024-04-17T19:54:28.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/29' 2024-04-17T19:54:28.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/32' 2024-04-17T19:54:28.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/22' 2024-04-17T19:54:28.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/25' 2024-04-17T19:54:28.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/30' 2024-04-17T19:54:28.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/33' 2024-04-17T19:54:28.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/23' 2024-04-17T19:54:28.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/35' 2024-04-17T19:54:28.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/26' 2024-04-17T19:54:28.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './298/37' 2024-04-17T19:54:28.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './298' 2024-04-17T19:54:28.844 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/21' 2024-04-17T19:54:28.845 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/27' 2024-04-17T19:54:28.845 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/31' 2024-04-17T19:54:28.845 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/36' 2024-04-17T19:54:28.845 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/20' 2024-04-17T19:54:28.845 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/39' 2024-04-17T19:54:28.845 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/34' 2024-04-17T19:54:28.845 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/38' 2024-04-17T19:54:28.845 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/28' 2024-04-17T19:54:28.845 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/24' 2024-04-17T19:54:28.845 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/29' 2024-04-17T19:54:28.845 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/32' 2024-04-17T19:54:28.845 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/22' 2024-04-17T19:54:28.845 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/25' 2024-04-17T19:54:28.845 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/30' 2024-04-17T19:54:28.845 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/33' 2024-04-17T19:54:28.846 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/23' 2024-04-17T19:54:28.846 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/35' 2024-04-17T19:54:28.846 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/26' 2024-04-17T19:54:28.846 INFO:tasks.workunit.client.0.smithi029.stdout:removed './299/37' 2024-04-17T19:54:28.846 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './299' 2024-04-17T19:54:28.846 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/166/19' 2024-04-17T19:54:28.846 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/166/15' 2024-04-17T19:54:28.846 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/166/20' 2024-04-17T19:54:28.846 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/166/12' 2024-04-17T19:54:28.846 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/166/10' 2024-04-17T19:54:28.846 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/166/11' 2024-04-17T19:54:28.846 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/166/16' 2024-04-17T19:54:28.846 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/166/14' 2024-04-17T19:54:28.846 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/166/18' 2024-04-17T19:54:28.846 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/166/13' 2024-04-17T19:54:28.846 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/166/17' 2024-04-17T19:54:28.847 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/166' 2024-04-17T19:54:28.847 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/147/19' 2024-04-17T19:54:28.847 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/147/15' 2024-04-17T19:54:28.847 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/147/20' 2024-04-17T19:54:28.847 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/147/12' 2024-04-17T19:54:28.847 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/147/10' 2024-04-17T19:54:28.847 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/147/11' 2024-04-17T19:54:28.847 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/147/16' 2024-04-17T19:54:28.847 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/147/14' 2024-04-17T19:54:28.847 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/147/18' 2024-04-17T19:54:28.847 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/147/13' 2024-04-17T19:54:28.847 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/147/17' 2024-04-17T19:54:28.847 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/147' 2024-04-17T19:54:28.847 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/151/19' 2024-04-17T19:54:28.847 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/151/15' 2024-04-17T19:54:28.847 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/151/20' 2024-04-17T19:54:28.847 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/151/12' 2024-04-17T19:54:28.848 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/151/10' 2024-04-17T19:54:28.848 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/151/11' 2024-04-17T19:54:28.848 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/151/16' 2024-04-17T19:54:28.848 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/151/14' 2024-04-17T19:54:28.848 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/151/18' 2024-04-17T19:54:28.848 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/151/13' 2024-04-17T19:54:28.848 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/151/17' 2024-04-17T19:54:28.848 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/151' 2024-04-17T19:54:28.848 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/170/19' 2024-04-17T19:54:28.848 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/170/15' 2024-04-17T19:54:28.848 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/170/20' 2024-04-17T19:54:28.848 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/170/12' 2024-04-17T19:54:28.848 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/170/10' 2024-04-17T19:54:28.848 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/170/11' 2024-04-17T19:54:28.848 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/170/16' 2024-04-17T19:54:28.848 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/170/14' 2024-04-17T19:54:28.849 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/170/18' 2024-04-17T19:54:28.849 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/170/13' 2024-04-17T19:54:28.849 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/170/17' 2024-04-17T19:54:28.849 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/170' 2024-04-17T19:54:28.849 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/167/19' 2024-04-17T19:54:28.849 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/167/15' 2024-04-17T19:54:28.849 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/167/20' 2024-04-17T19:54:28.849 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/167/12' 2024-04-17T19:54:28.849 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/167/10' 2024-04-17T19:54:28.849 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/167/11' 2024-04-17T19:54:28.849 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/167/16' 2024-04-17T19:54:28.849 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/167/14' 2024-04-17T19:54:28.849 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/167/18' 2024-04-17T19:54:28.849 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/167/13' 2024-04-17T19:54:28.849 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/167/17' 2024-04-17T19:54:28.849 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/167' 2024-04-17T19:54:28.849 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/140/19' 2024-04-17T19:54:28.850 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/140/15' 2024-04-17T19:54:28.850 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/140/20' 2024-04-17T19:54:28.850 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/140/12' 2024-04-17T19:54:28.850 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/140/10' 2024-04-17T19:54:28.850 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/140/11' 2024-04-17T19:54:28.850 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/140/16' 2024-04-17T19:54:28.850 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/140/14' 2024-04-17T19:54:28.850 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/140/18' 2024-04-17T19:54:28.850 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/140/13' 2024-04-17T19:54:28.850 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/140/17' 2024-04-17T19:54:28.850 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/140' 2024-04-17T19:54:28.850 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/142/19' 2024-04-17T19:54:28.850 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/142/15' 2024-04-17T19:54:28.851 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/142/20' 2024-04-17T19:54:28.851 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/142/12' 2024-04-17T19:54:28.851 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/142/10' 2024-04-17T19:54:29.330 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/142/11' 2024-04-17T19:54:29.330 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/142/16' 2024-04-17T19:54:29.330 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/142/14' 2024-04-17T19:54:29.330 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/142/18' 2024-04-17T19:54:29.330 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/142/13' 2024-04-17T19:54:29.330 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/142/17' 2024-04-17T19:54:29.330 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/142' 2024-04-17T19:54:29.330 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/150/19' 2024-04-17T19:54:29.330 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/150/15' 2024-04-17T19:54:29.330 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/150/20' 2024-04-17T19:54:29.330 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/150/12' 2024-04-17T19:54:29.330 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/150/10' 2024-04-17T19:54:29.331 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/150/11' 2024-04-17T19:54:29.331 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/150/16' 2024-04-17T19:54:29.331 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/150/14' 2024-04-17T19:54:29.331 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/150/18' 2024-04-17T19:54:29.331 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/150/13' 2024-04-17T19:54:29.331 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/150/17' 2024-04-17T19:54:29.331 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/150' 2024-04-17T19:54:29.331 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/105/19' 2024-04-17T19:54:29.331 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/105/15' 2024-04-17T19:54:29.331 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/105/20' 2024-04-17T19:54:29.331 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/105/12' 2024-04-17T19:54:29.331 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/105/10' 2024-04-17T19:54:29.331 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/105/11' 2024-04-17T19:54:29.331 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/105/16' 2024-04-17T19:54:29.332 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/105/14' 2024-04-17T19:54:29.332 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/105/18' 2024-04-17T19:54:29.332 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/105/13' 2024-04-17T19:54:29.332 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/105/17' 2024-04-17T19:54:29.332 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/105' 2024-04-17T19:54:29.332 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/149/19' 2024-04-17T19:54:29.332 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/149/15' 2024-04-17T19:54:29.332 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/149/20' 2024-04-17T19:54:29.332 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/149/12' 2024-04-17T19:54:29.332 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/149/10' 2024-04-17T19:54:29.332 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/149/11' 2024-04-17T19:54:29.332 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/149/16' 2024-04-17T19:54:29.333 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/149/14' 2024-04-17T19:54:29.333 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/149/18' 2024-04-17T19:54:29.333 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/149/13' 2024-04-17T19:54:29.333 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/149/17' 2024-04-17T19:54:29.333 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/149' 2024-04-17T19:54:29.333 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/109/19' 2024-04-17T19:54:29.333 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/109/15' 2024-04-17T19:54:29.333 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/109/20' 2024-04-17T19:54:29.333 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/109/12' 2024-04-17T19:54:29.333 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/109/10' 2024-04-17T19:54:29.333 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/109/11' 2024-04-17T19:54:29.333 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/109/16' 2024-04-17T19:54:29.333 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/109/14' 2024-04-17T19:54:29.333 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/109/18' 2024-04-17T19:54:29.333 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/109/13' 2024-04-17T19:54:29.334 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/109/17' 2024-04-17T19:54:29.334 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/109' 2024-04-17T19:54:29.334 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/158/19' 2024-04-17T19:54:29.334 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/158/15' 2024-04-17T19:54:29.334 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/158/20' 2024-04-17T19:54:29.334 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/158/12' 2024-04-17T19:54:29.334 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/158/10' 2024-04-17T19:54:29.334 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/158/11' 2024-04-17T19:54:29.334 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/158/16' 2024-04-17T19:54:29.334 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/158/14' 2024-04-17T19:54:29.334 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/158/18' 2024-04-17T19:54:29.334 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/158/13' 2024-04-17T19:54:29.334 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/158/17' 2024-04-17T19:54:29.335 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/158' 2024-04-17T19:54:29.335 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/106/19' 2024-04-17T19:54:29.335 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/106/15' 2024-04-17T19:54:29.335 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/106/20' 2024-04-17T19:54:29.335 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/106/12' 2024-04-17T19:54:29.335 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/106/10' 2024-04-17T19:54:29.335 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/106/11' 2024-04-17T19:54:29.335 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/106/16' 2024-04-17T19:54:29.335 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/106/14' 2024-04-17T19:54:29.335 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/106/18' 2024-04-17T19:54:29.335 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/106/13' 2024-04-17T19:54:29.335 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/106/17' 2024-04-17T19:54:29.335 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/106' 2024-04-17T19:54:29.335 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/189/19' 2024-04-17T19:54:29.336 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/189/15' 2024-04-17T19:54:29.336 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/189/20' 2024-04-17T19:54:29.336 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/189/12' 2024-04-17T19:54:29.336 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/189/10' 2024-04-17T19:54:29.336 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/189/11' 2024-04-17T19:54:29.336 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/189/16' 2024-04-17T19:54:29.336 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/189/14' 2024-04-17T19:54:29.336 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/189/18' 2024-04-17T19:54:29.336 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/189/13' 2024-04-17T19:54:29.336 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/189/17' 2024-04-17T19:54:29.336 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/189' 2024-04-17T19:54:29.336 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/133/19' 2024-04-17T19:54:29.336 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/133/15' 2024-04-17T19:54:29.337 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/133/20' 2024-04-17T19:54:29.337 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/133/12' 2024-04-17T19:54:29.337 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/133/10' 2024-04-17T19:54:29.337 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/133/11' 2024-04-17T19:54:29.337 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/133/16' 2024-04-17T19:54:29.337 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/133/14' 2024-04-17T19:54:29.337 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/133/18' 2024-04-17T19:54:29.337 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/133/13' 2024-04-17T19:54:29.337 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/133/17' 2024-04-17T19:54:29.337 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/133' 2024-04-17T19:54:29.337 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/173/19' 2024-04-17T19:54:29.337 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/173/15' 2024-04-17T19:54:29.337 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/173/20' 2024-04-17T19:54:29.337 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/173/12' 2024-04-17T19:54:29.338 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/173/10' 2024-04-17T19:54:29.338 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/173/11' 2024-04-17T19:54:31.676 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/173/16' 2024-04-17T19:54:31.676 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/173/14' 2024-04-17T19:54:31.676 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/173/18' 2024-04-17T19:54:31.676 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/173/13' 2024-04-17T19:54:31.676 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/173/17' 2024-04-17T19:54:31.676 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/173' 2024-04-17T19:54:31.676 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/174/19' 2024-04-17T19:54:31.676 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/174/15' 2024-04-17T19:54:31.677 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/174/20' 2024-04-17T19:54:31.677 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/174/12' 2024-04-17T19:54:31.677 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/174/10' 2024-04-17T19:54:31.677 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/174/11' 2024-04-17T19:54:31.677 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/174/16' 2024-04-17T19:54:31.677 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/174/14' 2024-04-17T19:54:31.677 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/174/18' 2024-04-17T19:54:31.677 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/174/13' 2024-04-17T19:54:31.677 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/174/17' 2024-04-17T19:54:31.677 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/174' 2024-04-17T19:54:31.677 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/155/19' 2024-04-17T19:54:31.677 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/155/15' 2024-04-17T19:54:31.677 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/155/20' 2024-04-17T19:54:31.677 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/155/12' 2024-04-17T19:54:31.677 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/155/10' 2024-04-17T19:54:31.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/155/11' 2024-04-17T19:54:31.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/155/16' 2024-04-17T19:54:31.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/155/14' 2024-04-17T19:54:31.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/155/18' 2024-04-17T19:54:31.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/155/13' 2024-04-17T19:54:31.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/155/17' 2024-04-17T19:54:31.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/155' 2024-04-17T19:54:31.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/114/19' 2024-04-17T19:54:31.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/114/15' 2024-04-17T19:54:31.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/114/20' 2024-04-17T19:54:31.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/114/12' 2024-04-17T19:54:31.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/114/10' 2024-04-17T19:54:31.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/114/11' 2024-04-17T19:54:31.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/114/16' 2024-04-17T19:54:31.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/114/14' 2024-04-17T19:54:31.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/114/18' 2024-04-17T19:54:31.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/114/13' 2024-04-17T19:54:31.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/114/17' 2024-04-17T19:54:31.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/114' 2024-04-17T19:54:31.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/159/19' 2024-04-17T19:54:31.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/159/15' 2024-04-17T19:54:31.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/159/20' 2024-04-17T19:54:31.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/159/12' 2024-04-17T19:54:31.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/159/10' 2024-04-17T19:54:31.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/159/11' 2024-04-17T19:54:31.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/159/16' 2024-04-17T19:54:31.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/159/14' 2024-04-17T19:54:31.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/159/18' 2024-04-17T19:54:31.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/159/13' 2024-04-17T19:54:31.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/159/17' 2024-04-17T19:54:31.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/159' 2024-04-17T19:54:31.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/187/19' 2024-04-17T19:54:31.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/187/15' 2024-04-17T19:54:31.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/187/20' 2024-04-17T19:54:31.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/187/12' 2024-04-17T19:54:31.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/187/10' 2024-04-17T19:54:31.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/187/11' 2024-04-17T19:54:31.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/187/16' 2024-04-17T19:54:31.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/187/14' 2024-04-17T19:54:31.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/187/18' 2024-04-17T19:54:31.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/187/13' 2024-04-17T19:54:31.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/187/17' 2024-04-17T19:54:31.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/187' 2024-04-17T19:54:31.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/185/19' 2024-04-17T19:54:31.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/185/15' 2024-04-17T19:54:31.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/185/20' 2024-04-17T19:54:31.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/185/12' 2024-04-17T19:54:31.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/185/10' 2024-04-17T19:54:31.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/185/11' 2024-04-17T19:54:31.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/185/16' 2024-04-17T19:54:31.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/185/14' 2024-04-17T19:54:31.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/185/18' 2024-04-17T19:54:31.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/185/13' 2024-04-17T19:54:31.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/185/17' 2024-04-17T19:54:31.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/185' 2024-04-17T19:54:31.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/120/19' 2024-04-17T19:54:31.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/120/15' 2024-04-17T19:54:31.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/120/20' 2024-04-17T19:54:31.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/120/12' 2024-04-17T19:54:31.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/120/10' 2024-04-17T19:54:31.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/120/11' 2024-04-17T19:54:31.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/120/16' 2024-04-17T19:54:31.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/120/14' 2024-04-17T19:54:31.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/120/18' 2024-04-17T19:54:31.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/120/13' 2024-04-17T19:54:31.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/120/17' 2024-04-17T19:54:31.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/120' 2024-04-17T19:54:31.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/110/19' 2024-04-17T19:54:31.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/110/15' 2024-04-17T19:54:31.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/110/20' 2024-04-17T19:54:31.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/110/12' 2024-04-17T19:54:31.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/110/10' 2024-04-17T19:54:31.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/110/11' 2024-04-17T19:54:31.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/110/16' 2024-04-17T19:54:31.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/110/14' 2024-04-17T19:54:31.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/110/18' 2024-04-17T19:54:31.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/110/13' 2024-04-17T19:54:31.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/110/17' 2024-04-17T19:54:31.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/110' 2024-04-17T19:54:31.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/137/19' 2024-04-17T19:54:31.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/137/15' 2024-04-17T19:54:31.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/137/20' 2024-04-17T19:54:31.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/137/12' 2024-04-17T19:54:31.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/137/10' 2024-04-17T19:54:31.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/137/11' 2024-04-17T19:54:31.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/137/16' 2024-04-17T19:54:32.677 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/137/14' 2024-04-17T19:54:32.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/137/18' 2024-04-17T19:54:32.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/137/13' 2024-04-17T19:54:32.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/137/17' 2024-04-17T19:54:32.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/137' 2024-04-17T19:54:32.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/198/19' 2024-04-17T19:54:32.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/198/15' 2024-04-17T19:54:32.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/198/20' 2024-04-17T19:54:32.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/198/12' 2024-04-17T19:54:32.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/198/10' 2024-04-17T19:54:32.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/198/11' 2024-04-17T19:54:32.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/198/16' 2024-04-17T19:54:32.678 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/198/14' 2024-04-17T19:54:32.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/198/18' 2024-04-17T19:54:32.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/198/13' 2024-04-17T19:54:32.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/198/17' 2024-04-17T19:54:32.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/198' 2024-04-17T19:54:32.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/136/19' 2024-04-17T19:54:32.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/136/15' 2024-04-17T19:54:32.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/136/20' 2024-04-17T19:54:32.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/136/12' 2024-04-17T19:54:32.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/136/10' 2024-04-17T19:54:32.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/136/11' 2024-04-17T19:54:32.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/136/16' 2024-04-17T19:54:32.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/136/14' 2024-04-17T19:54:32.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/136/18' 2024-04-17T19:54:32.679 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/136/13' 2024-04-17T19:54:32.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/136/17' 2024-04-17T19:54:32.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/136' 2024-04-17T19:54:32.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/125/19' 2024-04-17T19:54:32.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/125/15' 2024-04-17T19:54:32.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/125/20' 2024-04-17T19:54:32.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/125/12' 2024-04-17T19:54:32.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/125/10' 2024-04-17T19:54:32.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/125/11' 2024-04-17T19:54:32.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/125/16' 2024-04-17T19:54:32.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/125/14' 2024-04-17T19:54:32.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/125/18' 2024-04-17T19:54:32.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/125/13' 2024-04-17T19:54:32.680 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/125/17' 2024-04-17T19:54:32.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/125' 2024-04-17T19:54:32.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/194/19' 2024-04-17T19:54:32.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/194/15' 2024-04-17T19:54:32.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/194/20' 2024-04-17T19:54:32.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/194/12' 2024-04-17T19:54:32.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/194/10' 2024-04-17T19:54:32.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/194/11' 2024-04-17T19:54:32.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/194/16' 2024-04-17T19:54:32.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/194/14' 2024-04-17T19:54:32.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/194/18' 2024-04-17T19:54:32.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/194/13' 2024-04-17T19:54:32.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/194/17' 2024-04-17T19:54:32.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/194' 2024-04-17T19:54:32.681 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/119/19' 2024-04-17T19:54:32.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/119/15' 2024-04-17T19:54:32.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/119/20' 2024-04-17T19:54:32.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/119/12' 2024-04-17T19:54:32.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/119/10' 2024-04-17T19:54:32.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/119/11' 2024-04-17T19:54:32.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/119/16' 2024-04-17T19:54:32.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/119/14' 2024-04-17T19:54:32.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/119/18' 2024-04-17T19:54:32.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/119/13' 2024-04-17T19:54:32.682 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/119/17' 2024-04-17T19:54:32.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/119' 2024-04-17T19:54:32.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/157/19' 2024-04-17T19:54:32.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/157/15' 2024-04-17T19:54:32.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/157/20' 2024-04-17T19:54:32.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/157/12' 2024-04-17T19:54:32.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/157/10' 2024-04-17T19:54:32.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/157/11' 2024-04-17T19:54:32.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/157/16' 2024-04-17T19:54:32.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/157/14' 2024-04-17T19:54:32.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/157/18' 2024-04-17T19:54:32.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/157/13' 2024-04-17T19:54:32.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/157/17' 2024-04-17T19:54:32.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/157' 2024-04-17T19:54:32.683 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/172/19' 2024-04-17T19:54:32.684 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/172/15' 2024-04-17T19:54:32.684 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/172/20' 2024-04-17T19:54:32.684 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/172/12' 2024-04-17T19:54:32.684 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/172/10' 2024-04-17T19:54:32.684 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/172/11' 2024-04-17T19:54:32.684 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/172/16' 2024-04-17T19:54:32.684 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/172/14' 2024-04-17T19:54:32.684 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/172/18' 2024-04-17T19:54:32.684 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/172/13' 2024-04-17T19:54:32.684 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/172/17' 2024-04-17T19:54:32.684 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/172' 2024-04-17T19:54:32.684 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/186/19' 2024-04-17T19:54:32.684 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/186/15' 2024-04-17T19:54:32.684 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/186/20' 2024-04-17T19:54:32.685 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/186/12' 2024-04-17T19:54:32.685 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/186/10' 2024-04-17T19:54:32.685 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/186/11' 2024-04-17T19:54:32.685 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/186/16' 2024-04-17T19:54:32.685 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/186/14' 2024-04-17T19:54:32.685 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/186/18' 2024-04-17T19:54:32.685 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/186/13' 2024-04-17T19:54:32.685 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/186/17' 2024-04-17T19:54:32.685 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/186' 2024-04-17T19:54:32.685 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/190/19' 2024-04-17T19:54:32.685 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/190/15' 2024-04-17T19:54:32.685 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/190/20' 2024-04-17T19:54:32.685 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/190/12' 2024-04-17T19:54:32.685 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/190/10' 2024-04-17T19:54:32.685 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/190/11' 2024-04-17T19:54:32.686 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/190/16' 2024-04-17T19:54:32.686 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/190/14' 2024-04-17T19:54:32.944 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/190/18' 2024-04-17T19:54:32.944 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/190/13' 2024-04-17T19:54:32.944 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/190/17' 2024-04-17T19:54:32.944 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/190' 2024-04-17T19:54:32.944 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/104/19' 2024-04-17T19:54:32.945 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/104/15' 2024-04-17T19:54:32.945 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/104/20' 2024-04-17T19:54:32.945 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/104/12' 2024-04-17T19:54:32.945 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/104/10' 2024-04-17T19:54:32.945 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/104/11' 2024-04-17T19:54:32.945 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/104/16' 2024-04-17T19:54:32.945 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/104/14' 2024-04-17T19:54:32.945 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/104/18' 2024-04-17T19:54:32.945 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/104/13' 2024-04-17T19:54:32.945 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/104/17' 2024-04-17T19:54:32.945 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/104' 2024-04-17T19:54:32.945 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/164/19' 2024-04-17T19:54:32.945 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/164/15' 2024-04-17T19:54:32.945 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/164/20' 2024-04-17T19:54:32.945 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/164/12' 2024-04-17T19:54:32.946 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/164/10' 2024-04-17T19:54:32.946 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/164/11' 2024-04-17T19:54:32.946 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/164/16' 2024-04-17T19:54:32.946 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/164/14' 2024-04-17T19:54:32.946 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/164/18' 2024-04-17T19:54:32.946 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/164/13' 2024-04-17T19:54:32.946 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/164/17' 2024-04-17T19:54:32.946 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/164' 2024-04-17T19:54:32.946 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/112/19' 2024-04-17T19:54:32.946 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/112/15' 2024-04-17T19:54:32.946 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/112/20' 2024-04-17T19:54:32.946 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/112/12' 2024-04-17T19:54:32.946 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/112/10' 2024-04-17T19:54:32.947 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/112/11' 2024-04-17T19:54:32.947 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/112/16' 2024-04-17T19:54:32.947 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/112/14' 2024-04-17T19:54:32.947 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/112/18' 2024-04-17T19:54:32.947 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/112/13' 2024-04-17T19:54:32.947 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/112/17' 2024-04-17T19:54:32.947 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/112' 2024-04-17T19:54:32.947 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/100/19' 2024-04-17T19:54:32.947 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/100/15' 2024-04-17T19:54:32.947 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/100/20' 2024-04-17T19:54:32.947 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/100/12' 2024-04-17T19:54:32.947 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/100/10' 2024-04-17T19:54:32.947 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/100/11' 2024-04-17T19:54:32.947 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/100/16' 2024-04-17T19:54:32.948 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/100/14' 2024-04-17T19:54:32.948 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/100/18' 2024-04-17T19:54:32.948 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/100/13' 2024-04-17T19:54:32.948 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/100/17' 2024-04-17T19:54:32.948 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/100' 2024-04-17T19:54:32.948 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/134/19' 2024-04-17T19:54:32.948 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/134/15' 2024-04-17T19:54:32.948 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/134/20' 2024-04-17T19:54:32.948 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/134/12' 2024-04-17T19:54:32.948 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/134/10' 2024-04-17T19:54:32.948 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/134/11' 2024-04-17T19:54:32.948 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/134/16' 2024-04-17T19:54:32.948 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/134/14' 2024-04-17T19:54:32.948 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/134/18' 2024-04-17T19:54:32.949 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/134/13' 2024-04-17T19:54:32.949 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/134/17' 2024-04-17T19:54:32.949 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/134' 2024-04-17T19:54:32.949 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/115/19' 2024-04-17T19:54:32.949 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/115/15' 2024-04-17T19:54:32.949 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/115/20' 2024-04-17T19:54:32.949 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/115/12' 2024-04-17T19:54:32.949 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/115/10' 2024-04-17T19:54:32.949 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/115/11' 2024-04-17T19:54:32.949 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/115/16' 2024-04-17T19:54:32.949 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/115/14' 2024-04-17T19:54:32.949 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/115/18' 2024-04-17T19:54:32.949 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/115/13' 2024-04-17T19:54:32.949 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/115/17' 2024-04-17T19:54:32.949 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/115' 2024-04-17T19:54:32.949 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/116/19' 2024-04-17T19:54:32.950 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/116/15' 2024-04-17T19:54:32.950 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/116/20' 2024-04-17T19:54:32.950 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/116/12' 2024-04-17T19:54:32.950 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/116/10' 2024-04-17T19:54:32.950 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/116/11' 2024-04-17T19:54:32.950 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/116/16' 2024-04-17T19:54:32.950 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/116/14' 2024-04-17T19:54:32.950 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/116/18' 2024-04-17T19:54:32.950 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/116/13' 2024-04-17T19:54:32.950 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/116/17' 2024-04-17T19:54:32.950 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/116' 2024-04-17T19:54:32.950 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/195/19' 2024-04-17T19:54:32.950 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/195/15' 2024-04-17T19:54:32.950 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/195/20' 2024-04-17T19:54:32.950 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/195/12' 2024-04-17T19:54:32.951 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/195/10' 2024-04-17T19:54:32.951 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/195/11' 2024-04-17T19:54:32.951 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/195/16' 2024-04-17T19:54:32.951 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/195/14' 2024-04-17T19:54:32.951 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/195/18' 2024-04-17T19:54:32.951 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/195/13' 2024-04-17T19:54:32.951 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/195/17' 2024-04-17T19:54:32.951 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/195' 2024-04-17T19:54:32.951 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/138/19' 2024-04-17T19:54:32.951 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/138/15' 2024-04-17T19:54:32.951 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/138/20' 2024-04-17T19:54:32.951 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/138/12' 2024-04-17T19:54:32.951 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/138/10' 2024-04-17T19:54:32.951 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/138/11' 2024-04-17T19:54:32.951 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/138/16' 2024-04-17T19:54:32.951 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/138/14' 2024-04-17T19:54:32.951 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/138/18' 2024-04-17T19:54:34.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/138/13' 2024-04-17T19:54:34.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/138/17' 2024-04-17T19:54:34.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/138' 2024-04-17T19:54:34.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/135/19' 2024-04-17T19:54:34.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/135/15' 2024-04-17T19:54:34.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/135/20' 2024-04-17T19:54:34.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/135/12' 2024-04-17T19:54:34.813 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/135/10' 2024-04-17T19:54:34.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/135/11' 2024-04-17T19:54:34.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/135/16' 2024-04-17T19:54:34.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/135/14' 2024-04-17T19:54:34.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/135/18' 2024-04-17T19:54:34.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/135/13' 2024-04-17T19:54:34.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/135/17' 2024-04-17T19:54:34.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/135' 2024-04-17T19:54:34.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/117/19' 2024-04-17T19:54:34.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/117/15' 2024-04-17T19:54:34.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/117/20' 2024-04-17T19:54:34.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/117/12' 2024-04-17T19:54:34.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/117/10' 2024-04-17T19:54:34.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/117/11' 2024-04-17T19:54:34.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/117/16' 2024-04-17T19:54:34.814 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/117/14' 2024-04-17T19:54:34.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/117/18' 2024-04-17T19:54:34.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/117/13' 2024-04-17T19:54:34.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/117/17' 2024-04-17T19:54:34.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/117' 2024-04-17T19:54:34.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/143/19' 2024-04-17T19:54:34.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/143/15' 2024-04-17T19:54:34.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/143/20' 2024-04-17T19:54:34.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/143/12' 2024-04-17T19:54:34.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/143/10' 2024-04-17T19:54:34.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/143/11' 2024-04-17T19:54:34.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/143/16' 2024-04-17T19:54:34.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/143/14' 2024-04-17T19:54:34.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/143/18' 2024-04-17T19:54:34.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/143/13' 2024-04-17T19:54:34.815 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/143/17' 2024-04-17T19:54:34.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/143' 2024-04-17T19:54:34.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/180/19' 2024-04-17T19:54:34.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/180/15' 2024-04-17T19:54:34.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/180/20' 2024-04-17T19:54:34.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/180/12' 2024-04-17T19:54:34.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/180/10' 2024-04-17T19:54:34.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/180/11' 2024-04-17T19:54:34.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/180/16' 2024-04-17T19:54:34.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/180/14' 2024-04-17T19:54:34.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/180/18' 2024-04-17T19:54:34.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/180/13' 2024-04-17T19:54:34.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/180/17' 2024-04-17T19:54:34.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/180' 2024-04-17T19:54:34.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/126/19' 2024-04-17T19:54:34.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/126/15' 2024-04-17T19:54:34.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/126/20' 2024-04-17T19:54:34.816 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/126/12' 2024-04-17T19:54:34.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/126/10' 2024-04-17T19:54:34.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/126/11' 2024-04-17T19:54:34.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/126/16' 2024-04-17T19:54:34.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/126/14' 2024-04-17T19:54:34.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/126/18' 2024-04-17T19:54:34.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/126/13' 2024-04-17T19:54:34.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/126/17' 2024-04-17T19:54:34.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/126' 2024-04-17T19:54:34.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/161/19' 2024-04-17T19:54:34.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/161/15' 2024-04-17T19:54:34.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/161/20' 2024-04-17T19:54:34.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/161/12' 2024-04-17T19:54:34.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/161/10' 2024-04-17T19:54:34.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/161/11' 2024-04-17T19:54:34.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/161/16' 2024-04-17T19:54:34.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/161/14' 2024-04-17T19:54:34.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/161/18' 2024-04-17T19:54:34.817 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/161/13' 2024-04-17T19:54:34.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/161/17' 2024-04-17T19:54:34.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/161' 2024-04-17T19:54:34.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/176/19' 2024-04-17T19:54:34.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/176/15' 2024-04-17T19:54:34.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/176/20' 2024-04-17T19:54:34.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/176/12' 2024-04-17T19:54:34.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/176/10' 2024-04-17T19:54:34.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/176/11' 2024-04-17T19:54:34.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/176/16' 2024-04-17T19:54:34.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/176/14' 2024-04-17T19:54:34.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/176/18' 2024-04-17T19:54:34.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/176/13' 2024-04-17T19:54:34.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/176/17' 2024-04-17T19:54:34.818 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/176' 2024-04-17T19:54:34.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/111/19' 2024-04-17T19:54:34.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/111/15' 2024-04-17T19:54:34.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/111/20' 2024-04-17T19:54:34.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/111/12' 2024-04-17T19:54:34.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/111/10' 2024-04-17T19:54:34.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/111/11' 2024-04-17T19:54:34.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/111/16' 2024-04-17T19:54:34.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/111/14' 2024-04-17T19:54:34.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/111/18' 2024-04-17T19:54:34.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/111/13' 2024-04-17T19:54:34.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/111/17' 2024-04-17T19:54:34.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/111' 2024-04-17T19:54:34.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/191/19' 2024-04-17T19:54:34.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/191/15' 2024-04-17T19:54:34.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/191/20' 2024-04-17T19:54:34.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/191/12' 2024-04-17T19:54:34.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/191/10' 2024-04-17T19:54:34.819 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/191/11' 2024-04-17T19:54:34.820 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/191/16' 2024-04-17T19:54:34.820 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/191/14' 2024-04-17T19:54:34.820 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/191/18' 2024-04-17T19:54:34.820 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/191/13' 2024-04-17T19:54:36.319 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/191/17' 2024-04-17T19:54:36.319 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/191' 2024-04-17T19:54:36.319 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/184/19' 2024-04-17T19:54:36.319 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/184/15' 2024-04-17T19:54:36.319 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/184/20' 2024-04-17T19:54:36.319 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/184/12' 2024-04-17T19:54:36.320 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/184/10' 2024-04-17T19:54:36.320 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/184/11' 2024-04-17T19:54:36.320 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/184/16' 2024-04-17T19:54:36.320 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/184/14' 2024-04-17T19:54:36.320 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/184/18' 2024-04-17T19:54:36.320 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/184/13' 2024-04-17T19:54:36.320 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/184/17' 2024-04-17T19:54:36.320 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/184' 2024-04-17T19:54:36.320 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/192/19' 2024-04-17T19:54:36.320 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/192/15' 2024-04-17T19:54:36.320 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/192/20' 2024-04-17T19:54:36.320 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/192/12' 2024-04-17T19:54:36.320 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/192/10' 2024-04-17T19:54:36.321 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/192/11' 2024-04-17T19:54:36.321 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/192/16' 2024-04-17T19:54:36.321 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/192/14' 2024-04-17T19:54:36.321 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/192/18' 2024-04-17T19:54:36.321 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/192/13' 2024-04-17T19:54:36.321 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/192/17' 2024-04-17T19:54:36.321 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/192' 2024-04-17T19:54:36.321 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/103/19' 2024-04-17T19:54:36.321 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/103/15' 2024-04-17T19:54:36.321 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/103/20' 2024-04-17T19:54:36.321 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/103/12' 2024-04-17T19:54:36.321 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/103/10' 2024-04-17T19:54:36.321 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/103/11' 2024-04-17T19:54:36.322 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/103/16' 2024-04-17T19:54:36.322 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/103/14' 2024-04-17T19:54:36.322 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/103/18' 2024-04-17T19:54:36.322 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/103/13' 2024-04-17T19:54:36.322 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/103/17' 2024-04-17T19:54:36.322 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/103' 2024-04-17T19:54:36.322 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/128/19' 2024-04-17T19:54:36.322 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/128/15' 2024-04-17T19:54:36.322 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/128/20' 2024-04-17T19:54:36.322 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/128/12' 2024-04-17T19:54:36.322 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/128/10' 2024-04-17T19:54:36.322 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/128/11' 2024-04-17T19:54:36.322 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/128/16' 2024-04-17T19:54:36.322 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/128/14' 2024-04-17T19:54:36.322 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/128/18' 2024-04-17T19:54:36.323 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/128/13' 2024-04-17T19:54:36.323 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/128/17' 2024-04-17T19:54:36.323 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/128' 2024-04-17T19:54:36.323 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/144/19' 2024-04-17T19:54:36.323 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/144/15' 2024-04-17T19:54:36.323 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/144/20' 2024-04-17T19:54:36.323 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/144/12' 2024-04-17T19:54:36.323 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/144/10' 2024-04-17T19:54:36.323 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/144/11' 2024-04-17T19:54:36.323 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/144/16' 2024-04-17T19:54:36.323 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/144/14' 2024-04-17T19:54:36.323 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/144/18' 2024-04-17T19:54:36.323 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/144/13' 2024-04-17T19:54:36.323 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/144/17' 2024-04-17T19:54:36.323 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/144' 2024-04-17T19:54:36.324 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/146/19' 2024-04-17T19:54:36.324 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/146/15' 2024-04-17T19:54:36.324 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/146/20' 2024-04-17T19:54:36.324 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/146/12' 2024-04-17T19:54:36.324 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/146/10' 2024-04-17T19:54:36.324 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/146/11' 2024-04-17T19:54:36.324 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/146/16' 2024-04-17T19:54:36.324 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/146/14' 2024-04-17T19:54:36.324 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/146/18' 2024-04-17T19:54:36.324 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/146/13' 2024-04-17T19:54:36.324 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/146/17' 2024-04-17T19:54:36.324 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/146' 2024-04-17T19:54:36.324 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/141/19' 2024-04-17T19:54:36.324 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/141/15' 2024-04-17T19:54:36.324 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/141/20' 2024-04-17T19:54:36.324 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/141/12' 2024-04-17T19:54:36.325 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/141/10' 2024-04-17T19:54:36.325 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/141/11' 2024-04-17T19:54:36.325 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/141/16' 2024-04-17T19:54:36.325 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/141/14' 2024-04-17T19:54:36.325 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/141/18' 2024-04-17T19:54:36.325 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/141/13' 2024-04-17T19:54:36.325 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/141/17' 2024-04-17T19:54:36.325 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/141' 2024-04-17T19:54:36.325 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/182/19' 2024-04-17T19:54:36.325 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/182/15' 2024-04-17T19:54:36.325 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/182/20' 2024-04-17T19:54:36.325 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/182/12' 2024-04-17T19:54:36.325 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/182/10' 2024-04-17T19:54:36.325 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/182/11' 2024-04-17T19:54:36.326 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/182/16' 2024-04-17T19:54:36.326 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/182/14' 2024-04-17T19:54:36.326 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/182/18' 2024-04-17T19:54:36.326 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/182/13' 2024-04-17T19:54:36.326 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/182/17' 2024-04-17T19:54:36.326 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/182' 2024-04-17T19:54:36.326 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/107/19' 2024-04-17T19:54:36.326 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/107/15' 2024-04-17T19:54:36.326 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/107/20' 2024-04-17T19:54:36.326 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/107/12' 2024-04-17T19:54:36.326 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/107/10' 2024-04-17T19:54:36.326 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/107/11' 2024-04-17T19:54:36.326 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/107/16' 2024-04-17T19:54:36.327 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/107/14' 2024-04-17T19:54:36.327 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/107/18' 2024-04-17T19:54:36.327 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/107/13' 2024-04-17T19:54:36.327 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/107/17' 2024-04-17T19:54:37.651 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/107' 2024-04-17T19:54:37.651 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/178/19' 2024-04-17T19:54:37.652 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/178/15' 2024-04-17T19:54:37.652 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/178/20' 2024-04-17T19:54:37.652 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/178/12' 2024-04-17T19:54:37.652 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/178/10' 2024-04-17T19:54:37.652 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/178/11' 2024-04-17T19:54:37.652 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/178/16' 2024-04-17T19:54:37.652 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/178/14' 2024-04-17T19:54:37.652 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/178/18' 2024-04-17T19:54:37.652 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/178/13' 2024-04-17T19:54:37.652 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/178/17' 2024-04-17T19:54:37.652 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/178' 2024-04-17T19:54:37.652 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/156/19' 2024-04-17T19:54:37.652 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/156/15' 2024-04-17T19:54:37.653 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/156/20' 2024-04-17T19:54:37.653 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/156/12' 2024-04-17T19:54:37.653 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/156/10' 2024-04-17T19:54:37.653 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/156/11' 2024-04-17T19:54:37.653 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/156/16' 2024-04-17T19:54:37.653 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/156/14' 2024-04-17T19:54:37.653 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/156/18' 2024-04-17T19:54:37.653 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/156/13' 2024-04-17T19:54:37.653 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/156/17' 2024-04-17T19:54:37.653 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/156' 2024-04-17T19:54:37.653 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/131/19' 2024-04-17T19:54:37.653 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/131/15' 2024-04-17T19:54:37.653 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/131/20' 2024-04-17T19:54:37.654 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/131/12' 2024-04-17T19:54:37.654 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/131/10' 2024-04-17T19:54:37.654 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/131/11' 2024-04-17T19:54:37.654 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/131/16' 2024-04-17T19:54:37.654 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/131/14' 2024-04-17T19:54:37.654 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/131/18' 2024-04-17T19:54:37.654 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/131/13' 2024-04-17T19:54:37.654 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/131/17' 2024-04-17T19:54:37.654 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/131' 2024-04-17T19:54:37.654 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/123/19' 2024-04-17T19:54:37.654 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/123/15' 2024-04-17T19:54:37.654 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/123/20' 2024-04-17T19:54:37.654 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/123/12' 2024-04-17T19:54:37.654 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/123/10' 2024-04-17T19:54:37.655 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/123/11' 2024-04-17T19:54:37.655 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/123/16' 2024-04-17T19:54:37.655 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/123/14' 2024-04-17T19:54:37.655 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/123/18' 2024-04-17T19:54:37.655 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/123/13' 2024-04-17T19:54:37.655 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/123/17' 2024-04-17T19:54:37.655 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/123' 2024-04-17T19:54:37.655 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/160/19' 2024-04-17T19:54:37.655 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/160/15' 2024-04-17T19:54:37.655 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/160/20' 2024-04-17T19:54:37.655 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/160/12' 2024-04-17T19:54:37.655 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/160/10' 2024-04-17T19:54:37.655 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/160/11' 2024-04-17T19:54:37.655 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/160/16' 2024-04-17T19:54:37.656 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/160/14' 2024-04-17T19:54:37.656 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/160/18' 2024-04-17T19:54:37.656 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/160/13' 2024-04-17T19:54:37.656 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/160/17' 2024-04-17T19:54:37.656 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/160' 2024-04-17T19:54:37.656 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/165/19' 2024-04-17T19:54:37.656 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/165/15' 2024-04-17T19:54:37.656 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/165/20' 2024-04-17T19:54:37.656 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/165/12' 2024-04-17T19:54:37.656 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/165/10' 2024-04-17T19:54:37.656 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/165/11' 2024-04-17T19:54:37.656 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/165/16' 2024-04-17T19:54:37.656 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/165/14' 2024-04-17T19:54:37.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/165/18' 2024-04-17T19:54:37.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/165/13' 2024-04-17T19:54:37.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/165/17' 2024-04-17T19:54:37.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/165' 2024-04-17T19:54:37.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/101/19' 2024-04-17T19:54:37.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/101/15' 2024-04-17T19:54:37.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/101/20' 2024-04-17T19:54:37.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/101/12' 2024-04-17T19:54:37.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/101/10' 2024-04-17T19:54:37.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/101/11' 2024-04-17T19:54:37.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/101/16' 2024-04-17T19:54:37.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/101/14' 2024-04-17T19:54:37.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/101/18' 2024-04-17T19:54:37.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/101/13' 2024-04-17T19:54:37.657 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/101/17' 2024-04-17T19:54:37.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/101' 2024-04-17T19:54:37.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/148/19' 2024-04-17T19:54:37.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/148/15' 2024-04-17T19:54:37.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/148/20' 2024-04-17T19:54:37.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/148/12' 2024-04-17T19:54:37.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/148/10' 2024-04-17T19:54:37.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/148/11' 2024-04-17T19:54:37.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/148/16' 2024-04-17T19:54:37.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/148/14' 2024-04-17T19:54:37.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/148/18' 2024-04-17T19:54:37.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/148/13' 2024-04-17T19:54:37.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/148/17' 2024-04-17T19:54:37.658 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/148' 2024-04-17T19:54:37.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/168/19' 2024-04-17T19:54:37.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/168/15' 2024-04-17T19:54:37.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/168/20' 2024-04-17T19:54:37.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/168/12' 2024-04-17T19:54:37.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/168/10' 2024-04-17T19:54:37.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/168/11' 2024-04-17T19:54:37.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/168/16' 2024-04-17T19:54:37.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/168/14' 2024-04-17T19:54:37.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/168/18' 2024-04-17T19:54:37.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/168/13' 2024-04-17T19:54:37.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/168/17' 2024-04-17T19:54:37.659 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/168' 2024-04-17T19:54:37.899 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/179/19' 2024-04-17T19:54:37.899 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/179/15' 2024-04-17T19:54:37.899 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/179/20' 2024-04-17T19:54:37.899 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/179/12' 2024-04-17T19:54:37.899 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/179/10' 2024-04-17T19:54:37.899 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/179/11' 2024-04-17T19:54:37.899 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/179/16' 2024-04-17T19:54:37.900 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/179/14' 2024-04-17T19:54:37.900 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/179/18' 2024-04-17T19:54:37.900 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/179/13' 2024-04-17T19:54:37.900 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/179/17' 2024-04-17T19:54:37.900 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/179' 2024-04-17T19:54:37.900 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/132/19' 2024-04-17T19:54:37.900 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/132/15' 2024-04-17T19:54:37.900 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/132/20' 2024-04-17T19:54:37.900 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/132/12' 2024-04-17T19:54:37.900 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/132/10' 2024-04-17T19:54:37.900 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/132/11' 2024-04-17T19:54:37.900 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/132/16' 2024-04-17T19:54:37.900 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/132/14' 2024-04-17T19:54:37.900 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/132/18' 2024-04-17T19:54:37.900 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/132/13' 2024-04-17T19:54:37.901 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/132/17' 2024-04-17T19:54:37.901 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/132' 2024-04-17T19:54:37.901 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/199/19' 2024-04-17T19:54:37.901 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/199/15' 2024-04-17T19:54:37.901 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/199/20' 2024-04-17T19:54:37.901 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/199/12' 2024-04-17T19:54:37.901 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/199/10' 2024-04-17T19:54:37.901 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/199/11' 2024-04-17T19:54:37.901 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/199/16' 2024-04-17T19:54:37.901 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/199/14' 2024-04-17T19:54:37.901 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/199/18' 2024-04-17T19:54:37.901 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/199/13' 2024-04-17T19:54:37.901 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/199/17' 2024-04-17T19:54:37.901 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/199' 2024-04-17T19:54:37.901 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/163/19' 2024-04-17T19:54:37.902 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/163/15' 2024-04-17T19:54:37.902 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/163/20' 2024-04-17T19:54:37.902 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/163/12' 2024-04-17T19:54:37.902 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/163/10' 2024-04-17T19:54:37.902 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/163/11' 2024-04-17T19:54:37.902 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/163/16' 2024-04-17T19:54:37.902 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/163/14' 2024-04-17T19:54:37.902 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/163/18' 2024-04-17T19:54:37.902 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/163/13' 2024-04-17T19:54:37.902 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/163/17' 2024-04-17T19:54:37.902 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/163' 2024-04-17T19:54:37.902 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/153/19' 2024-04-17T19:54:37.902 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/153/15' 2024-04-17T19:54:37.903 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/153/20' 2024-04-17T19:54:37.903 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/153/12' 2024-04-17T19:54:37.903 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/153/10' 2024-04-17T19:54:37.903 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/153/11' 2024-04-17T19:54:37.903 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/153/16' 2024-04-17T19:54:37.903 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/153/14' 2024-04-17T19:54:37.903 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/153/18' 2024-04-17T19:54:37.903 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/153/13' 2024-04-17T19:54:37.903 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/153/17' 2024-04-17T19:54:37.903 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/153' 2024-04-17T19:54:37.903 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/122/19' 2024-04-17T19:54:37.903 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/122/15' 2024-04-17T19:54:37.903 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/122/20' 2024-04-17T19:54:37.904 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/122/12' 2024-04-17T19:54:37.904 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/122/10' 2024-04-17T19:54:37.904 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/122/11' 2024-04-17T19:54:37.904 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/122/16' 2024-04-17T19:54:37.904 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/122/14' 2024-04-17T19:54:37.904 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/122/18' 2024-04-17T19:54:37.904 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/122/13' 2024-04-17T19:54:37.904 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/122/17' 2024-04-17T19:54:37.904 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/122' 2024-04-17T19:54:37.904 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/162/19' 2024-04-17T19:54:37.904 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/162/15' 2024-04-17T19:54:37.904 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/162/20' 2024-04-17T19:54:37.904 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/162/12' 2024-04-17T19:54:37.904 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/162/10' 2024-04-17T19:54:37.904 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/162/11' 2024-04-17T19:54:37.905 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/162/16' 2024-04-17T19:54:37.905 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/162/14' 2024-04-17T19:54:37.905 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/162/18' 2024-04-17T19:54:37.905 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/162/13' 2024-04-17T19:54:37.905 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/162/17' 2024-04-17T19:54:37.905 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/162' 2024-04-17T19:54:37.905 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/197/19' 2024-04-17T19:54:37.905 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/197/15' 2024-04-17T19:54:37.905 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/197/20' 2024-04-17T19:54:37.905 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/197/12' 2024-04-17T19:54:37.905 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/197/10' 2024-04-17T19:54:37.905 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/197/11' 2024-04-17T19:54:37.905 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/197/16' 2024-04-17T19:54:37.905 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/197/14' 2024-04-17T19:54:37.905 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/197/18' 2024-04-17T19:54:37.906 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/197/13' 2024-04-17T19:54:37.906 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/197/17' 2024-04-17T19:54:37.906 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/197' 2024-04-17T19:54:37.906 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/113/19' 2024-04-17T19:54:37.906 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/113/15' 2024-04-17T19:54:37.906 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/113/20' 2024-04-17T19:54:37.906 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/113/12' 2024-04-17T19:54:37.906 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/113/10' 2024-04-17T19:54:37.906 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/113/11' 2024-04-17T19:54:37.906 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/113/16' 2024-04-17T19:54:37.906 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/113/14' 2024-04-17T19:54:37.906 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/113/18' 2024-04-17T19:54:37.906 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/113/13' 2024-04-17T19:54:37.906 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/113/17' 2024-04-17T19:54:37.907 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/113' 2024-04-17T19:54:37.907 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/152/19' 2024-04-17T19:54:39.907 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/152/15' 2024-04-17T19:54:39.908 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/152/20' 2024-04-17T19:54:39.908 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/152/12' 2024-04-17T19:54:39.908 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/152/10' 2024-04-17T19:54:39.908 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/152/11' 2024-04-17T19:54:39.908 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/152/16' 2024-04-17T19:54:39.908 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/152/14' 2024-04-17T19:54:39.908 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/152/18' 2024-04-17T19:54:39.908 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/152/13' 2024-04-17T19:54:39.908 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/152/17' 2024-04-17T19:54:39.908 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/152' 2024-04-17T19:54:39.908 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/121/19' 2024-04-17T19:54:39.908 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/121/15' 2024-04-17T19:54:39.908 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/121/20' 2024-04-17T19:54:39.908 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/121/12' 2024-04-17T19:54:39.909 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/121/10' 2024-04-17T19:54:39.909 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/121/11' 2024-04-17T19:54:39.909 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/121/16' 2024-04-17T19:54:39.909 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/121/14' 2024-04-17T19:54:39.909 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/121/18' 2024-04-17T19:54:39.909 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/121/13' 2024-04-17T19:54:39.909 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/121/17' 2024-04-17T19:54:39.909 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/121' 2024-04-17T19:54:39.909 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/118/19' 2024-04-17T19:54:39.909 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/118/15' 2024-04-17T19:54:39.909 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/118/20' 2024-04-17T19:54:39.909 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/118/12' 2024-04-17T19:54:39.909 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/118/10' 2024-04-17T19:54:39.909 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/118/11' 2024-04-17T19:54:39.909 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/118/16' 2024-04-17T19:54:39.909 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/118/14' 2024-04-17T19:54:39.909 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/118/18' 2024-04-17T19:54:39.910 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/118/13' 2024-04-17T19:54:39.910 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/118/17' 2024-04-17T19:54:39.910 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/118' 2024-04-17T19:54:39.910 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/196/19' 2024-04-17T19:54:39.910 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/196/15' 2024-04-17T19:54:39.910 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/196/20' 2024-04-17T19:54:39.910 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/196/12' 2024-04-17T19:54:39.910 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/196/10' 2024-04-17T19:54:39.910 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/196/11' 2024-04-17T19:54:39.910 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/196/16' 2024-04-17T19:54:39.910 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/196/14' 2024-04-17T19:54:39.910 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/196/18' 2024-04-17T19:54:39.910 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/196/13' 2024-04-17T19:54:39.910 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/196/17' 2024-04-17T19:54:39.910 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/196' 2024-04-17T19:54:39.911 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/124/19' 2024-04-17T19:54:39.911 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/124/15' 2024-04-17T19:54:39.911 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/124/20' 2024-04-17T19:54:39.911 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/124/12' 2024-04-17T19:54:39.911 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/124/10' 2024-04-17T19:54:39.911 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/124/11' 2024-04-17T19:54:39.911 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/124/16' 2024-04-17T19:54:39.911 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/124/14' 2024-04-17T19:54:39.911 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/124/18' 2024-04-17T19:54:39.911 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/124/13' 2024-04-17T19:54:39.911 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/124/17' 2024-04-17T19:54:39.911 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/124' 2024-04-17T19:54:39.911 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/188/19' 2024-04-17T19:54:39.911 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/188/15' 2024-04-17T19:54:39.911 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/188/20' 2024-04-17T19:54:39.912 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/188/12' 2024-04-17T19:54:39.912 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/188/10' 2024-04-17T19:54:39.912 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/188/11' 2024-04-17T19:54:39.912 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/188/16' 2024-04-17T19:54:39.912 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/188/14' 2024-04-17T19:54:39.912 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/188/18' 2024-04-17T19:54:39.912 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/188/13' 2024-04-17T19:54:39.912 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/188/17' 2024-04-17T19:54:39.912 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/188' 2024-04-17T19:54:39.912 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/145/19' 2024-04-17T19:54:39.912 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/145/15' 2024-04-17T19:54:39.912 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/145/20' 2024-04-17T19:54:39.912 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/145/12' 2024-04-17T19:54:39.912 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/145/10' 2024-04-17T19:54:39.912 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/145/11' 2024-04-17T19:54:39.913 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/145/16' 2024-04-17T19:54:39.913 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/145/14' 2024-04-17T19:54:39.913 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/145/18' 2024-04-17T19:54:39.913 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/145/13' 2024-04-17T19:54:39.913 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/145/17' 2024-04-17T19:54:39.913 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/145' 2024-04-17T19:54:39.913 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/130/19' 2024-04-17T19:54:39.913 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/130/15' 2024-04-17T19:54:39.913 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/130/20' 2024-04-17T19:54:39.913 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/130/12' 2024-04-17T19:54:39.913 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/130/10' 2024-04-17T19:54:39.913 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/130/11' 2024-04-17T19:54:39.913 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/130/16' 2024-04-17T19:54:39.913 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/130/14' 2024-04-17T19:54:39.913 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/130/18' 2024-04-17T19:54:39.914 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/130/13' 2024-04-17T19:54:39.914 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/130/17' 2024-04-17T19:54:39.914 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/130' 2024-04-17T19:54:39.914 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/193/19' 2024-04-17T19:54:39.914 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/193/15' 2024-04-17T19:54:39.914 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/193/20' 2024-04-17T19:54:39.914 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/193/12' 2024-04-17T19:54:39.914 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/193/10' 2024-04-17T19:54:39.914 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/193/11' 2024-04-17T19:54:39.914 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/193/16' 2024-04-17T19:54:39.914 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/193/14' 2024-04-17T19:54:39.914 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/193/18' 2024-04-17T19:54:39.914 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/193/13' 2024-04-17T19:54:39.914 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/193/17' 2024-04-17T19:54:39.914 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/193' 2024-04-17T19:54:39.915 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/175/19' 2024-04-17T19:54:39.915 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/175/15' 2024-04-17T19:54:40.424 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/175/20' 2024-04-17T19:54:40.424 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/175/12' 2024-04-17T19:54:40.424 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/175/10' 2024-04-17T19:54:40.424 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/175/11' 2024-04-17T19:54:40.424 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/175/16' 2024-04-17T19:54:40.424 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/175/14' 2024-04-17T19:54:40.424 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/175/18' 2024-04-17T19:54:40.424 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/175/13' 2024-04-17T19:54:40.424 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/175/17' 2024-04-17T19:54:40.424 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/175' 2024-04-17T19:54:40.424 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/181/19' 2024-04-17T19:54:40.424 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/181/15' 2024-04-17T19:54:40.424 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/181/20' 2024-04-17T19:54:40.425 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/181/12' 2024-04-17T19:54:40.425 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/181/10' 2024-04-17T19:54:40.425 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/181/11' 2024-04-17T19:54:40.425 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/181/16' 2024-04-17T19:54:40.425 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/181/14' 2024-04-17T19:54:40.425 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/181/18' 2024-04-17T19:54:40.425 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/181/13' 2024-04-17T19:54:40.425 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/181/17' 2024-04-17T19:54:40.425 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/181' 2024-04-17T19:54:40.425 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/127/19' 2024-04-17T19:54:40.425 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/127/15' 2024-04-17T19:54:40.425 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/127/20' 2024-04-17T19:54:40.425 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/127/12' 2024-04-17T19:54:40.425 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/127/10' 2024-04-17T19:54:40.425 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/127/11' 2024-04-17T19:54:40.426 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/127/16' 2024-04-17T19:54:40.426 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/127/14' 2024-04-17T19:54:40.426 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/127/18' 2024-04-17T19:54:40.426 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/127/13' 2024-04-17T19:54:40.426 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/127/17' 2024-04-17T19:54:40.426 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/127' 2024-04-17T19:54:40.426 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/108/19' 2024-04-17T19:54:40.426 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/108/15' 2024-04-17T19:54:40.426 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/108/20' 2024-04-17T19:54:40.426 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/108/12' 2024-04-17T19:54:40.426 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/108/10' 2024-04-17T19:54:40.426 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/108/11' 2024-04-17T19:54:40.426 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/108/16' 2024-04-17T19:54:40.426 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/108/14' 2024-04-17T19:54:40.426 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/108/18' 2024-04-17T19:54:40.427 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/108/13' 2024-04-17T19:54:40.427 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/108/17' 2024-04-17T19:54:40.427 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/108' 2024-04-17T19:54:40.427 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/183/19' 2024-04-17T19:54:40.427 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/183/15' 2024-04-17T19:54:40.427 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/183/20' 2024-04-17T19:54:40.427 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/183/12' 2024-04-17T19:54:40.427 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/183/10' 2024-04-17T19:54:40.427 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/183/11' 2024-04-17T19:54:40.427 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/183/16' 2024-04-17T19:54:40.427 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/183/14' 2024-04-17T19:54:40.427 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/183/18' 2024-04-17T19:54:40.427 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/183/13' 2024-04-17T19:54:40.427 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/183/17' 2024-04-17T19:54:40.427 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/183' 2024-04-17T19:54:40.427 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/177/19' 2024-04-17T19:54:40.428 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/177/15' 2024-04-17T19:54:40.428 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/177/20' 2024-04-17T19:54:40.428 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/177/12' 2024-04-17T19:54:40.428 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/177/10' 2024-04-17T19:54:40.428 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/177/11' 2024-04-17T19:54:40.428 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/177/16' 2024-04-17T19:54:40.428 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/177/14' 2024-04-17T19:54:40.428 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/177/18' 2024-04-17T19:54:40.428 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/177/13' 2024-04-17T19:54:40.428 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/177/17' 2024-04-17T19:54:40.428 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/177' 2024-04-17T19:54:40.428 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/169/19' 2024-04-17T19:54:40.428 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/169/15' 2024-04-17T19:54:40.428 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/169/20' 2024-04-17T19:54:40.429 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/169/12' 2024-04-17T19:54:40.429 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/169/10' 2024-04-17T19:54:40.429 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/169/11' 2024-04-17T19:54:40.429 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/169/16' 2024-04-17T19:54:40.429 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/169/14' 2024-04-17T19:54:40.429 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/169/18' 2024-04-17T19:54:40.429 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/169/13' 2024-04-17T19:54:40.429 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/169/17' 2024-04-17T19:54:40.429 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/169' 2024-04-17T19:54:40.429 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/154/19' 2024-04-17T19:54:40.429 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/154/15' 2024-04-17T19:54:40.429 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/154/20' 2024-04-17T19:54:40.429 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/154/12' 2024-04-17T19:54:40.429 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/154/10' 2024-04-17T19:54:40.429 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/154/11' 2024-04-17T19:54:40.429 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/154/16' 2024-04-17T19:54:40.430 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/154/14' 2024-04-17T19:54:40.430 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/154/18' 2024-04-17T19:54:40.430 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/154/13' 2024-04-17T19:54:40.430 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/154/17' 2024-04-17T19:54:40.430 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/154' 2024-04-17T19:54:40.430 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/171/19' 2024-04-17T19:54:40.430 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/171/15' 2024-04-17T19:54:40.430 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/171/20' 2024-04-17T19:54:40.430 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/171/12' 2024-04-17T19:54:40.430 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/171/10' 2024-04-17T19:54:40.430 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/171/11' 2024-04-17T19:54:40.430 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/171/16' 2024-04-17T19:54:40.430 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/171/14' 2024-04-17T19:54:40.430 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/171/18' 2024-04-17T19:54:40.430 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/171/13' 2024-04-17T19:54:40.431 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/171/17' 2024-04-17T19:54:40.431 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/171' 2024-04-17T19:54:40.431 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/139/19' 2024-04-17T19:54:40.431 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/139/15' 2024-04-17T19:54:40.431 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/139/20' 2024-04-17T19:54:40.518 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/139/12' 2024-04-17T19:54:40.518 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/139/10' 2024-04-17T19:54:40.518 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/139/11' 2024-04-17T19:54:40.518 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/139/16' 2024-04-17T19:54:40.518 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/139/14' 2024-04-17T19:54:40.518 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/139/18' 2024-04-17T19:54:40.518 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/139/13' 2024-04-17T19:54:40.519 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/139/17' 2024-04-17T19:54:40.519 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/139' 2024-04-17T19:54:40.519 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/102/19' 2024-04-17T19:54:40.519 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/102/15' 2024-04-17T19:54:40.519 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/102/20' 2024-04-17T19:54:40.519 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/102/12' 2024-04-17T19:54:40.519 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/102/10' 2024-04-17T19:54:40.519 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/102/11' 2024-04-17T19:54:40.519 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/102/16' 2024-04-17T19:54:40.519 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/102/14' 2024-04-17T19:54:40.519 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/102/18' 2024-04-17T19:54:40.519 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/102/13' 2024-04-17T19:54:40.519 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/102/17' 2024-04-17T19:54:40.519 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/102' 2024-04-17T19:54:40.519 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/129/19' 2024-04-17T19:54:40.520 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/129/15' 2024-04-17T19:54:40.520 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/129/20' 2024-04-17T19:54:40.520 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/129/12' 2024-04-17T19:54:40.520 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/129/10' 2024-04-17T19:54:40.520 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/129/11' 2024-04-17T19:54:40.520 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/129/16' 2024-04-17T19:54:40.520 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/129/14' 2024-04-17T19:54:40.520 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/129/18' 2024-04-17T19:54:40.520 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/129/13' 2024-04-17T19:54:40.520 INFO:tasks.workunit.client.0.smithi029.stdout:removed './copyofsnap1/test1/129/17' 2024-04-17T19:54:40.520 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1/129' 2024-04-17T19:54:40.520 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1/test1' 2024-04-17T19:54:40.520 INFO:tasks.workunit.client.0.smithi029.stdout:removed directory './copyofsnap1' 2024-04-17T19:54:40.522 INFO:tasks.workunit.client.0.smithi029.stdout:OK 2024-04-17T19:54:40.522 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-17T19:54:40.522 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-17T19:54:40.553 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2024-04-17T19:54:40.553 DEBUG:teuthology.orchestra.run.smithi029:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f3a98d3f110319c567ec86bf70955c141495ccf6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2024-04-17T19:54:44.973 INFO:tasks.workunit.client.0.smithi029.stderr:+ set -e 2024-04-17T19:54:44.973 INFO:tasks.workunit.client.0.smithi029.stderr:+ touch foo 2024-04-17T19:54:44.975 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod +x foo 2024-04-17T19:54:44.976 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/s 2024-04-17T19:54:45.112 INFO:tasks.workunit.client.0.smithi029.stderr:+ find .snap/s/foo -executable 2024-04-17T19:54:45.112 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep foo 2024-04-17T19:54:45.117 INFO:tasks.workunit.client.0.smithi029.stdout:.snap/s/foo 2024-04-17T19:54:45.117 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir .snap/s 2024-04-17T19:54:45.130 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm foo 2024-04-17T19:54:45.636 INFO:tasks.workunit.client.0.smithi029.stdout:OK 2024-04-17T19:54:45.637 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo OK 2024-04-17T19:54:45.637 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-17T19:54:45.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-17T19:54:46.522 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2024-04-17T19:54:46.523 DEBUG:teuthology.orchestra.run.smithi029:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f3a98d3f110319c567ec86bf70955c141495ccf6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2024-04-17T19:54:49.960 INFO:tasks.workunit.client.0.smithi029.stderr:+ set -e 2024-04-17T19:54:49.961 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir foo 2024-04-17T19:54:49.964 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo x 2024-04-17T19:54:49.964 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:54:51.147 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir foo/.snap/ss 2024-04-17T19:54:51.174 INFO:tasks.workunit.client.0.smithi029.stderr:+ ln foo/x foo/xx 2024-04-17T19:54:51.177 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat foo/.snap/ss/x 2024-04-17T19:54:51.187 INFO:tasks.workunit.client.0.smithi029.stdout:x 2024-04-17T19:54:51.188 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir foo/.snap/ss 2024-04-17T19:54:51.200 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo a 2024-04-17T19:54:51.201 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo b 2024-04-17T19:54:51.202 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir foo/.snap/s 2024-04-17T19:54:51.217 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat foo/.snap/s/a 2024-04-17T19:54:51.231 INFO:tasks.workunit.client.0.smithi029.stderr:+ r=a 2024-04-17T19:54:51.231 INFO:tasks.workunit.client.0.smithi029.stderr:+ [ -z a ] 2024-04-17T19:54:51.232 INFO:tasks.workunit.client.0.smithi029.stderr:+ ln foo/b foo/b2 2024-04-17T19:54:51.234 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat foo/.snap/s/b 2024-04-17T19:54:51.255 INFO:tasks.workunit.client.0.smithi029.stdout:b 2024-04-17T19:54:51.256 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo this used to hang: 2024-04-17T19:54:51.256 INFO:tasks.workunit.client.0.smithi029.stdout:this used to hang: 2024-04-17T19:54:51.256 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo more 2024-04-17T19:54:51.257 INFO:tasks.workunit.client.0.smithi029.stdout:oh, it didn't hang! good job. 2024-04-17T19:54:51.257 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo oh, it didn't hang! good job. 2024-04-17T19:54:51.257 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat foo/b 2024-04-17T19:54:51.257 INFO:tasks.workunit.client.0.smithi029.stdout:b 2024-04-17T19:54:51.257 INFO:tasks.workunit.client.0.smithi029.stdout:more 2024-04-17T19:54:51.258 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir foo/.snap/s 2024-04-17T19:54:51.268 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm -r foo 2024-04-17T19:54:51.288 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo OK 2024-04-17T19:54:51.289 INFO:tasks.workunit.client.0.smithi029.stdout:OK 2024-04-17T19:54:51.289 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-17T19:54:51.289 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-17T19:54:51.324 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2024-04-17T19:54:51.324 DEBUG:teuthology.orchestra.run.smithi029:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f3a98d3f110319c567ec86bf70955c141495ccf6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2024-04-17T19:54:51.875 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:54:51.878 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:54:55.400 INFO:tasks.workunit.client.0.smithi029.stderr:+ set -e 2024-04-17T19:54:55.400 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:54:55.402 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/bar 2024-04-17T19:54:55.863 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/foo 2024-04-17T19:54:55.876 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/foo 2024-04-17T19:54:55.886 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:54:55.890 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/bar/a 2024-04-17T19:54:59.946 INFO:tasks.workunit.client.0.smithi029.stdout: File: .snap/bar/a 2024-04-17T19:54:59.947 INFO:tasks.workunit.client.0.smithi029.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-17T19:54:59.947 INFO:tasks.workunit.client.0.smithi029.stdout:Device: feh/254d Inode: 1099511633797 Links: 2 2024-04-17T19:54:59.947 INFO:tasks.workunit.client.0.smithi029.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1271/ ubuntu) 2024-04-17T19:54:59.947 INFO:tasks.workunit.client.0.smithi029.stdout:Access: 2024-04-17 19:54:55.409002187 +0000 2024-04-17T19:54:59.947 INFO:tasks.workunit.client.0.smithi029.stdout:Modify: 2024-04-17 19:54:55.409002187 +0000 2024-04-17T19:54:59.947 INFO:tasks.workunit.client.0.smithi029.stdout:Change: 2024-04-17 19:54:55.409002187 +0000 2024-04-17T19:54:59.947 INFO:tasks.workunit.client.0.smithi029.stdout: Birth: 2024-04-17 19:54:55.409002187 +0000 2024-04-17T19:54:59.947 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir .snap/bar 2024-04-17T19:54:59.959 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo OK 2024-04-17T19:54:59.959 INFO:tasks.workunit.client.0.smithi029.stdout:OK 2024-04-17T19:54:59.960 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-17T19:54:59.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-17T19:54:59.990 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2024-04-17T19:54:59.990 DEBUG:teuthology.orchestra.run.smithi029:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f3a98d3f110319c567ec86bf70955c141495ccf6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2024-04-17T19:55:00.061 INFO:tasks.workunit.client.0.smithi029.stderr:+ set -e 2024-04-17T19:55:00.061 INFO:tasks.workunit.client.0.smithi029.stderr:+ seq 1 20 2024-04-17T19:55:00.062 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:00.065 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:00.067 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:00.081 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:00.092 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:00.092 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:00.674 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:00.690 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:00.691 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:00.691 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:00.691 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:01 sshd 2024-04-17T19:55:00.691 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:00.691 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:00.691 INFO:tasks.workunit.client.0.smithi029.stdout: 38835 ? 00:00:00 ps 2024-04-17T19:55:00.691 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:00.713 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:00.716 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:00.726 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:00.728 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:00.730 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:00.745 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:00.756 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:00.757 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:00.864 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:00.881 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:00.882 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:00.882 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:00.882 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:01 sshd 2024-04-17T19:55:00.882 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:00.882 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:00.882 INFO:tasks.workunit.client.0.smithi029.stdout: 38845 ? 00:00:00 ps 2024-04-17T19:55:00.883 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:00.904 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:00.908 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:00.918 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:00.920 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:00.922 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:00.938 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:00.949 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:00.950 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:01.057 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:01.072 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:01.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:01.073 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:01.073 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:01 sshd 2024-04-17T19:55:01.073 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:01.073 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:01.073 INFO:tasks.workunit.client.0.smithi029.stdout: 38855 ? 00:00:00 ps 2024-04-17T19:55:01.073 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:01.093 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:01.451 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:01.461 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:01.464 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:01.466 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:01.480 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:01.937 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:02.025 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:02.026 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:02.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.108+0000 7f246c800640 1 Processor -- start 2024-04-17T19:55:02.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.108+0000 7f246c800640 1 -- start start 2024-04-17T19:55:02.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.108+0000 7f246c800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f24640c9b20 0x7f24641560d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:02.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.108+0000 7f246c800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f246406e300 0x7f246406e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:02.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.108+0000 7f246c800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f246415be30 0x7f246415c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:02.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.108+0000 7f246c800640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f24640c8fa0 con 0x7f246406e300 2024-04-17T19:55:02.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.108+0000 7f246c800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f24640c55e0 con 0x7f246415be30 2024-04-17T19:55:02.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.108+0000 7f246c800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f24640c9f30 con 0x7f24640c9b20 2024-04-17T19:55:02.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.108+0000 7f2469400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f246406e300 0x7f246406e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:02.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.108+0000 7f2469400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f246406e300 0x7f246406e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37014/0 (socket says 172.21.15.29:37014) 2024-04-17T19:55:02.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.108+0000 7f2469400640 1 -- 172.21.15.29:0/875192035 learned_addr learned my addr 172.21.15.29:0/875192035 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:02.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.112+0000 7f246a800640 1 --2- 172.21.15.29:0/875192035 >> v2:172.21.15.37:3300/0 conn(0x7f246415be30 0x7f246415c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:02.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.112+0000 7f2469e00640 1 --2- 172.21.15.29:0/875192035 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f24640c9b20 0x7f24641560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:02.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.112+0000 7f2469400640 1 -- 172.21.15.29:0/875192035 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f24640c9b20 msgr2=0x7f24641560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:02.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.112+0000 7f2469400640 1 --2- 172.21.15.29:0/875192035 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f24640c9b20 0x7f24641560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.112+0000 7f2469400640 1 -- 172.21.15.29:0/875192035 >> v2:172.21.15.37:3300/0 conn(0x7f246415be30 msgr2=0x7f246415c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:02.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.112+0000 7f2469400640 1 --2- 172.21.15.29:0/875192035 >> v2:172.21.15.37:3300/0 conn(0x7f246415be30 0x7f246415c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.112+0000 7f2469400640 1 -- 172.21.15.29:0/875192035 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24640ca1d0 con 0x7f246406e300 2024-04-17T19:55:02.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.112+0000 7f2469400640 1 --2- 172.21.15.29:0/875192035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f246406e300 0x7f246406e710 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f24640cb780 tx=0x7f245401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6896c6276337a95 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:02.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.112+0000 7f2468a00640 1 -- 172.21.15.29:0/875192035 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24540b5070 con 0x7f246406e300 2024-04-17T19:55:02.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.112+0000 7f2468a00640 1 -- 172.21.15.29:0/875192035 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24540a89d0 con 0x7f246406e300 2024-04-17T19:55:02.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.112+0000 7f2468a00640 1 -- 172.21.15.29:0/875192035 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24540a8da0 con 0x7f246406e300 2024-04-17T19:55:02.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.116+0000 7f246c800640 1 -- 172.21.15.29:0/875192035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f246406e300 msgr2=0x7f246406e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:02.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.116+0000 7f246c800640 1 --2- 172.21.15.29:0/875192035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f246406e300 0x7f246406e710 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f24640cb780 tx=0x7f245401ed10 comp rx=0 tx=0).stop 2024-04-17T19:55:02.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.116+0000 7f246c800640 1 -- 172.21.15.29:0/875192035 shutdown_connections 2024-04-17T19:55:02.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.116+0000 7f246c800640 1 --2- 172.21.15.29:0/875192035 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f246406e300 0x7f246406e710 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.116+0000 7f246c800640 1 --2- 172.21.15.29:0/875192035 >> v2:172.21.15.37:3300/0 conn(0x7f246415be30 0x7f246415c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.116+0000 7f246c800640 1 --2- 172.21.15.29:0/875192035 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f24640c9b20 0x7f24641560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.116+0000 7f246c800640 1 -- 172.21.15.29:0/875192035 >> 172.21.15.29:0/875192035 conn(0x7f246406c3c0 msgr2=0x7f24640ca3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:02.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.116+0000 7f246c800640 1 -- 172.21.15.29:0/875192035 shutdown_connections 2024-04-17T19:55:02.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.116+0000 7f246c800640 1 -- 172.21.15.29:0/875192035 wait complete. 2024-04-17T19:55:02.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.116+0000 7f246c800640 1 Processor -- start 2024-04-17T19:55:02.112 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.116+0000 7f246c800640 1 -- start start 2024-04-17T19:55:02.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f246c800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f246406e300 0x7f24641e4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:02.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f246c800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f24640c9b20 0x7f24640c7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:02.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f246c800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f246415be30 0x7f2464147050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:02.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f246c800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f24640c8670 con 0x7f246415be30 2024-04-17T19:55:02.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f246c800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f24640c8390 con 0x7f246406e300 2024-04-17T19:55:02.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f246c800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f24640c8950 con 0x7f24640c9b20 2024-04-17T19:55:02.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f246a800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f246415be30 0x7f2464147050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:02.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f246a800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f246415be30 0x7f2464147050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37022/0 (socket says 172.21.15.29:37022) 2024-04-17T19:55:02.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f246a800640 1 -- 172.21.15.29:0/166410997 learned_addr learned my addr 172.21.15.29:0/166410997 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:02.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f2469e00640 1 --2- 172.21.15.29:0/166410997 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f246406e300 0x7f24641e4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:02.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f2469400640 1 --2- 172.21.15.29:0/166410997 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f24640c9b20 0x7f24640c7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:02.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f246a800640 1 -- 172.21.15.29:0/166410997 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f24640c9b20 msgr2=0x7f24640c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:02.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f246a800640 1 --2- 172.21.15.29:0/166410997 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f24640c9b20 0x7f24640c7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f246a800640 1 -- 172.21.15.29:0/166410997 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f246406e300 msgr2=0x7f24641e4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:02.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f246a800640 1 --2- 172.21.15.29:0/166410997 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f246406e300 0x7f24641e4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f246a800640 1 -- 172.21.15.29:0/166410997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24641e2090 con 0x7f246415be30 2024-04-17T19:55:02.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f2469e00640 1 --2- 172.21.15.29:0/166410997 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f246406e300 0x7f24641e4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:55:02.116 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f246a800640 1 --2- 172.21.15.29:0/166410997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f246415be30 0x7f2464147050 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f246000ef90 tx=0x7f246000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:02.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f245e000640 1 -- 172.21.15.29:0/166410997 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2460018020 con 0x7f246415be30 2024-04-17T19:55:02.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f245e000640 1 -- 172.21.15.29:0/166410997 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2460014480 con 0x7f246415be30 2024-04-17T19:55:02.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f246c800640 1 -- 172.21.15.29:0/166410997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24641e2330 con 0x7f246415be30 2024-04-17T19:55:02.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f245e000640 1 -- 172.21.15.29:0/166410997 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2460009e60 con 0x7f246415be30 2024-04-17T19:55:02.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f246c800640 1 -- 172.21.15.29:0/166410997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24641e25f0 con 0x7f246415be30 2024-04-17T19:55:02.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.120+0000 7f246c800640 1 -- 172.21.15.29:0/166410997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2464069b90 con 0x7f246415be30 2024-04-17T19:55:02.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.124+0000 7f245e000640 1 -- 172.21.15.29:0/166410997 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f246001b020 con 0x7f246415be30 2024-04-17T19:55:02.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.124+0000 7f245e000640 1 --2- 172.21.15.29:0/166410997 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2448068380 0x7f244806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:02.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.124+0000 7f245e000640 1 -- 172.21.15.29:0/166410997 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7105+0+0 (secure 0 0 0) 0x7f2460087d70 con 0x7f246415be30 2024-04-17T19:55:02.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.124+0000 7f2469e00640 1 --2- 172.21.15.29:0/166410997 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2448068380 0x7f244806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:02.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.124+0000 7f2469e00640 1 --2- 172.21.15.29:0/166410997 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2448068380 0x7f244806a890 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f245800ba50 tx=0x7f2458008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:02.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.128+0000 7f245e000640 1 -- 172.21.15.29:0/166410997 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2460022020 con 0x7f246415be30 2024-04-17T19:55:02.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.324+0000 7f246c800640 1 -- 172.21.15.29:0/166410997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2464075f50 con 0x7f246415be30 2024-04-17T19:55:02.321 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:02.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.328+0000 7f245e000640 1 -- 172.21.15.29:0/166410997 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6904 (secure 0 0 0) 0x7f2460010040 con 0x7f246415be30 2024-04-17T19:55:02.322 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:02.322 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5144,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:34.909002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,4305,5212,4302,4259]},"id":1}]} 2024-04-17T19:55:02.322 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-04-17T19:55:02.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.328+0000 7f246c800640 1 -- 172.21.15.29:0/166410997 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2448068380 msgr2=0x7f244806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:02.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.328+0000 7f246c800640 1 --2- 172.21.15.29:0/166410997 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2448068380 0x7f244806a890 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f245800ba50 tx=0x7f2458008040 comp rx=0 tx=0).stop 2024-04-17T19:55:02.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.328+0000 7f246c800640 1 -- 172.21.15.29:0/166410997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f246415be30 msgr2=0x7f2464147050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:02.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.328+0000 7f246c800640 1 --2- 172.21.15.29:0/166410997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f246415be30 0x7f2464147050 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f246000ef90 tx=0x7f246000c490 comp rx=0 tx=0).stop 2024-04-17T19:55:02.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.328+0000 7f246c800640 1 -- 172.21.15.29:0/166410997 shutdown_connections 2024-04-17T19:55:02.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.328+0000 7f246c800640 1 --2- 172.21.15.29:0/166410997 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2448068380 0x7f244806a890 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.328+0000 7f246c800640 1 --2- 172.21.15.29:0/166410997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f246415be30 0x7f2464147050 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.328+0000 7f246c800640 1 --2- 172.21.15.29:0/166410997 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f24640c9b20 0x7f24640c7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.328+0000 7f246c800640 1 --2- 172.21.15.29:0/166410997 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f246406e300 0x7f24641e4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.328+0000 7f246c800640 1 -- 172.21.15.29:0/166410997 >> 172.21.15.29:0/166410997 conn(0x7f246406c3c0 msgr2=0x7f246415a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:02.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.328+0000 7f246c800640 1 -- 172.21.15.29:0/166410997 shutdown_connections 2024-04-17T19:55:02.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.328+0000 7f246c800640 1 -- 172.21.15.29:0/166410997 wait complete. 2024-04-17T19:55:02.346 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:02.346 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:02.346 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:02.346 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:01 sshd 2024-04-17T19:55:02.346 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:02.346 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:02.346 INFO:tasks.workunit.client.0.smithi029.stdout: 38893 ? 00:00:00 ps 2024-04-17T19:55:02.347 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:02.351 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=2) 2024-04-17T19:55:02.351 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T19:55:02.351 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T19:55:02.371 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T19:55:02.371 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:02.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.568+0000 7fc9adc00640 1 Processor -- start 2024-04-17T19:55:02.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.568+0000 7fc9adc00640 1 -- start start 2024-04-17T19:55:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.568+0000 7fc9adc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9a80c6f70 0x7fc9a80c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.568+0000 7fc9adc00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc9a806c660 0x7fc9a806ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.568+0000 7fc9adc00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9a815a0a0 con 0x7fc9a8159c90 2024-04-17T19:55:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.568+0000 7fc9adc00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fc9a815abe0 con 0x7fc9a806c660 2024-04-17T19:55:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.568+0000 7fc9adc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc9a815a340 con 0x7fc9a80c6f70 2024-04-17T19:55:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.572+0000 7fc9a7400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc9a806c660 0x7fc9a806ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.572+0000 7fc9a7400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc9a806c660 0x7fc9a806ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37030/0 (socket says 172.21.15.29:37030) 2024-04-17T19:55:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.572+0000 7fc9a7400640 1 -- 172.21.15.29:0/3842323457 learned_addr learned my addr 172.21.15.29:0/3842323457 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.572+0000 7fc99fe00640 1 --2- 172.21.15.29:0/3842323457 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9a80c6f70 0x7fc9a80c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.572+0000 7fc9a7400640 1 -- 172.21.15.29:0/3842323457 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9a80c6f70 msgr2=0x7fc9a80c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.572+0000 7fc9a7400640 1 --2- 172.21.15.29:0/3842323457 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9a80c6f70 0x7fc9a80c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.572+0000 7fc9a7400640 1 -- 172.21.15.29:0/3842323457 >> v1:172.21.15.37:6789/0 conn(0x7fc9a8159c90 legacy=0x7fc9a80ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:02.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.572+0000 7fc9a7400640 1 -- 172.21.15.29:0/3842323457 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a815a620 con 0x7fc9a806c660 2024-04-17T19:55:02.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.572+0000 7fc99fe00640 1 --2- 172.21.15.29:0/3842323457 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9a80c6f70 0x7fc9a80c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:55:02.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.572+0000 7fc9a7400640 1 --2- 172.21.15.29:0/3842323457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9a806c660 0x7fc9a806ca70 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009e50 tx=0x7fc9a001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ac2a425566b01a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:02.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.572+0000 7fc9a6000640 1 -- 172.21.15.29:0/3842323457 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a00a5700 con 0x7fc9a806c660 2024-04-17T19:55:02.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.572+0000 7fc9a6000640 1 -- 172.21.15.29:0/3842323457 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9a00a58a0 con 0x7fc9a806c660 2024-04-17T19:55:02.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.572+0000 7fc9a6000640 1 -- 172.21.15.29:0/3842323457 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9a00a5c70 con 0x7fc9a806c660 2024-04-17T19:55:02.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.576+0000 7fc9adc00640 1 -- 172.21.15.29:0/3842323457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9a806c660 msgr2=0x7fc9a806ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:02.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.576+0000 7fc9adc00640 1 --2- 172.21.15.29:0/3842323457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9a806c660 0x7fc9a806ca70 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0009e50 tx=0x7fc9a001ca30 comp rx=0 tx=0).stop 2024-04-17T19:55:02.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.576+0000 7fc9adc00640 1 -- 172.21.15.29:0/3842323457 shutdown_connections 2024-04-17T19:55:02.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.576+0000 7fc9adc00640 1 --2- 172.21.15.29:0/3842323457 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9a806c660 0x7fc9a806ca70 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.576+0000 7fc9adc00640 1 --2- 172.21.15.29:0/3842323457 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9a80c6f70 0x7fc9a80c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.576+0000 7fc9adc00640 1 -- 172.21.15.29:0/3842323457 >> 172.21.15.29:0/3842323457 conn(0x7fc9a806fde0 msgr2=0x7fc9a80701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:02.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.576+0000 7fc9adc00640 1 -- 172.21.15.29:0/3842323457 shutdown_connections 2024-04-17T19:55:02.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.576+0000 7fc9adc00640 1 -- 172.21.15.29:0/3842323457 wait complete. 2024-04-17T19:55:02.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.576+0000 7fc9adc00640 1 Processor -- start 2024-04-17T19:55:02.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.576+0000 7fc9adc00640 1 -- start start 2024-04-17T19:55:02.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9adc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9a806c660 0x7fc9a81f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:02.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9adc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9a80c6f70 0x7fc9a81466c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:02.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9adc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9a8159c90 0x7fc9a80cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:02.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9adc00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a80ca9c0 con 0x7fc9a806c660 2024-04-17T19:55:02.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9adc00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a806cfb0 con 0x7fc9a80c6f70 2024-04-17T19:55:02.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9adc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc9a80cab40 con 0x7fc9a8159c90 2024-04-17T19:55:02.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9a6a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9a806c660 0x7fc9a81f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:02.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9a6a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9a806c660 0x7fc9a81f51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37042/0 (socket says 172.21.15.29:37042) 2024-04-17T19:55:02.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9a6a00640 1 -- 172.21.15.29:0/3933253873 learned_addr learned my addr 172.21.15.29:0/3933253873 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:02.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc99fe00640 1 --2- 172.21.15.29:0/3933253873 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9a80c6f70 0x7fc9a81466c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:02.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9a7400640 1 --2- 172.21.15.29:0/3933253873 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9a8159c90 0x7fc9a80cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:02.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9a6a00640 1 -- 172.21.15.29:0/3933253873 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9a8159c90 msgr2=0x7fc9a80cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:02.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9a6a00640 1 --2- 172.21.15.29:0/3933253873 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9a8159c90 0x7fc9a80cb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9a6a00640 1 -- 172.21.15.29:0/3933253873 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9a80c6f70 msgr2=0x7fc9a81466c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:02.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9a6a00640 1 --2- 172.21.15.29:0/3933253873 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9a80c6f70 0x7fc9a81466c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9a6a00640 1 -- 172.21.15.29:0/3933253873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a81492b0 con 0x7fc9a806c660 2024-04-17T19:55:02.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9a6a00640 1 --2- 172.21.15.29:0/3933253873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9a806c660 0x7fc9a81f51b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc99800c3c0 tx=0x7fc99800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:02.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc99ea00640 1 -- 172.21.15.29:0/3933253873 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc998017070 con 0x7fc9a806c660 2024-04-17T19:55:02.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc99ea00640 1 -- 172.21.15.29:0/3933253873 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9980052f0 con 0x7fc9a806c660 2024-04-17T19:55:02.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9adc00640 1 -- 172.21.15.29:0/3933253873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9a81518b0 con 0x7fc9a806c660 2024-04-17T19:55:02.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc99ea00640 1 -- 172.21.15.29:0/3933253873 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc998003860 con 0x7fc9a806c660 2024-04-17T19:55:02.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9adc00640 1 -- 172.21.15.29:0/3933253873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9a8150dd0 con 0x7fc9a806c660 2024-04-17T19:55:02.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc9adc00640 1 -- 172.21.15.29:0/3933253873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9a8069b90 con 0x7fc9a806c660 2024-04-17T19:55:02.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.580+0000 7fc99ea00640 1 -- 172.21.15.29:0/3933253873 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc99801a070 con 0x7fc9a806c660 2024-04-17T19:55:02.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.584+0000 7fc99ea00640 1 --2- 172.21.15.29:0/3933253873 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc984068380 0x7fc98406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:02.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.584+0000 7fc99ea00640 1 -- 172.21.15.29:0/3933253873 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 7105+0+0 (secure 0 0 0) 0x7fc998086170 con 0x7fc9a806c660 2024-04-17T19:55:02.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.584+0000 7fc99fe00640 1 --2- 172.21.15.29:0/3933253873 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc984068380 0x7fc98406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:02.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.584+0000 7fc99fe00640 1 --2- 172.21.15.29:0/3933253873 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc984068380 0x7fc98406a890 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc9a815e310 tx=0x7fc990008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:02.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.588+0000 7fc99ea00640 1 -- 172.21.15.29:0/3933253873 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc998055ac0 con 0x7fc9a806c660 2024-04-17T19:55:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.788+0000 7fc9adc00640 1 -- 172.21.15.29:0/3933253873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9a8153830 con 0x7fc9a806c660 2024-04-17T19:55:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.788+0000 7fc99ea00640 1 -- 172.21.15.29:0/3933253873 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6904 (secure 0 0 0) 0x7fc998059dc0 con 0x7fc9a806c660 2024-04-17T19:55:02.783 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:02.783 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5144,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:34.909002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,4305,5212,4302,4259]},"id":1}]} 2024-04-17T19:55:02.783 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-04-17T19:55:02.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.788+0000 7fc9adc00640 1 -- 172.21.15.29:0/3933253873 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc984068380 msgr2=0x7fc98406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:02.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.788+0000 7fc9adc00640 1 --2- 172.21.15.29:0/3933253873 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc984068380 0x7fc98406a890 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc9a815e310 tx=0x7fc990008040 comp rx=0 tx=0).stop 2024-04-17T19:55:02.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.788+0000 7fc9adc00640 1 -- 172.21.15.29:0/3933253873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9a806c660 msgr2=0x7fc9a81f51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:02.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.788+0000 7fc9adc00640 1 --2- 172.21.15.29:0/3933253873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9a806c660 0x7fc9a81f51b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fc99800c3c0 tx=0x7fc99800c880 comp rx=0 tx=0).stop 2024-04-17T19:55:02.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.788+0000 7fc9adc00640 1 -- 172.21.15.29:0/3933253873 shutdown_connections 2024-04-17T19:55:02.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.788+0000 7fc9adc00640 1 --2- 172.21.15.29:0/3933253873 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc984068380 0x7fc98406a890 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.788+0000 7fc9adc00640 1 --2- 172.21.15.29:0/3933253873 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc9a8159c90 0x7fc9a80cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.788+0000 7fc9adc00640 1 --2- 172.21.15.29:0/3933253873 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc9a80c6f70 0x7fc9a81466c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.788+0000 7fc9adc00640 1 --2- 172.21.15.29:0/3933253873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9a806c660 0x7fc9a81f51b0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:02.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.788+0000 7fc9adc00640 1 -- 172.21.15.29:0/3933253873 >> 172.21.15.29:0/3933253873 conn(0x7fc9a806fde0 msgr2=0x7fc9a80701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:02.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.788+0000 7fc9adc00640 1 -- 172.21.15.29:0/3933253873 shutdown_connections 2024-04-17T19:55:02.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:02.788+0000 7fc9adc00640 1 -- 172.21.15.29:0/3933253873 wait complete. 2024-04-17T19:55:02.804 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-17T19:55:04.805 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:05.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.048+0000 7ff5f0c00640 1 Processor -- start 2024-04-17T19:55:05.043 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.048+0000 7ff5f0c00640 1 -- start start 2024-04-17T19:55:05.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.048+0000 7ff5f0c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff5e40dd0e0 0x7ff5e4034ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:05.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.048+0000 7ff5f0c00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff5e4009170 0x7ff5e4007890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:05.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.048+0000 7ff5f0c00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5e40dd720 con 0x7ff5e4007480 2024-04-17T19:55:05.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.048+0000 7ff5f0c00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff5e40dd4f0 con 0x7ff5e4009170 2024-04-17T19:55:05.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.048+0000 7ff5f0c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e40ddfc0 con 0x7ff5e40dd0e0 2024-04-17T19:55:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.048+0000 7ff5eb400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff5e40dd0e0 0x7ff5e4034ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.048+0000 7ff5eaa00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff5e4007480 0x7ff5e4008a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60656/0 (socket says 172.21.15.29:60656) 2024-04-17T19:55:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.048+0000 7ff5eaa00640 1 -- 172.21.15.29:0/2129787640 learned_addr learned my addr 172.21.15.29:0/2129787640 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:55:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.052+0000 7ff5ebe00640 1 --2- 172.21.15.29:0/2129787640 >> v2:172.21.15.37:3300/0 conn(0x7ff5e4009170 0x7ff5e4007890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.052+0000 7ff5ea000640 1 -- 172.21.15.29:0/2129787640 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 859351786 0 0) 0x7ff5e40dd720 con 0x7ff5e4007480 2024-04-17T19:55:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.052+0000 7ff5ea000640 1 -- 172.21.15.29:0/2129787640 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5d4003530 con 0x7ff5e4007480 2024-04-17T19:55:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.052+0000 7ff5ebe00640 1 -- 172.21.15.29:0/2129787640 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff5e40dd0e0 msgr2=0x7ff5e4034ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.052+0000 7ff5ebe00640 1 --2- 172.21.15.29:0/2129787640 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff5e40dd0e0 0x7ff5e4034ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.052+0000 7ff5ebe00640 1 -- 172.21.15.29:0/2129787640 >> v1:172.21.15.29:6789/0 conn(0x7ff5e4007480 legacy=0x7ff5e4008a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.052+0000 7ff5ebe00640 1 -- 172.21.15.29:0/2129787640 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e40dda00 con 0x7ff5e4009170 2024-04-17T19:55:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.052+0000 7ff5eb400640 1 --2- 172.21.15.29:0/2129787640 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff5e40dd0e0 0x7ff5e4034ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:55:05.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.052+0000 7ff5ebe00640 1 --2- 172.21.15.29:0/2129787640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5e4009170 0x7ff5e4007890 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec065810 tx=0x7ff5ec09c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61a1bc607c2781c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:05.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.052+0000 7ff5ea000640 1 -- 172.21.15.29:0/2129787640 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec125ce0 con 0x7ff5e4009170 2024-04-17T19:55:05.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.052+0000 7ff5ea000640 1 -- 172.21.15.29:0/2129787640 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5ec12d440 con 0x7ff5e4009170 2024-04-17T19:55:05.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.052+0000 7ff5ea000640 1 -- 172.21.15.29:0/2129787640 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec12d870 con 0x7ff5e4009170 2024-04-17T19:55:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.056+0000 7ff5f0c00640 1 -- 172.21.15.29:0/2129787640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5e4009170 msgr2=0x7ff5e4007890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.056+0000 7ff5f0c00640 1 --2- 172.21.15.29:0/2129787640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5e4009170 0x7ff5e4007890 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec065810 tx=0x7ff5ec09c000 comp rx=0 tx=0).stop 2024-04-17T19:55:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.056+0000 7ff5f0c00640 1 -- 172.21.15.29:0/2129787640 shutdown_connections 2024-04-17T19:55:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.056+0000 7ff5f0c00640 1 --2- 172.21.15.29:0/2129787640 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5e4009170 0x7ff5e4007890 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.056+0000 7ff5f0c00640 1 --2- 172.21.15.29:0/2129787640 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff5e40dd0e0 0x7ff5e4034ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.056+0000 7ff5f0c00640 1 -- 172.21.15.29:0/2129787640 >> 172.21.15.29:0/2129787640 conn(0x7ff5e40514d0 msgr2=0x7ff5e40518e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.056+0000 7ff5f0c00640 1 -- 172.21.15.29:0/2129787640 shutdown_connections 2024-04-17T19:55:05.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.056+0000 7ff5f0c00640 1 -- 172.21.15.29:0/2129787640 wait complete. 2024-04-17T19:55:05.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.056+0000 7ff5f0c00640 1 Processor -- start 2024-04-17T19:55:05.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.056+0000 7ff5f0c00640 1 -- start start 2024-04-17T19:55:05.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5f0c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff5e4007480 0x7ff5e4176170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:05.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5f0c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5e4009170 0x7ff5e4004f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:05.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5f0c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5e40dd0e0 0x7ff5e40cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:05.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5f0c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e4008540 con 0x7ff5e40dd0e0 2024-04-17T19:55:05.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5f0c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e4008260 con 0x7ff5e4009170 2024-04-17T19:55:05.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5f0c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e4036750 con 0x7ff5e4007480 2024-04-17T19:55:05.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5ebe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5e40dd0e0 0x7ff5e40cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:05.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5ebe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5e40dd0e0 0x7ff5e40cf0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37058/0 (socket says 172.21.15.29:37058) 2024-04-17T19:55:05.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5ebe00640 1 -- 172.21.15.29:0/897624110 learned_addr learned my addr 172.21.15.29:0/897624110 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:05.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5eaa00640 1 --2- 172.21.15.29:0/897624110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5e4009170 0x7ff5e4004f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:05.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5eb400640 1 --2- 172.21.15.29:0/897624110 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff5e4007480 0x7ff5e4176170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:05.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5ebe00640 1 -- 172.21.15.29:0/897624110 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff5e4007480 msgr2=0x7ff5e4176170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:05.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5ebe00640 1 --2- 172.21.15.29:0/897624110 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff5e4007480 0x7ff5e4176170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:05.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5ebe00640 1 -- 172.21.15.29:0/897624110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5e4009170 msgr2=0x7ff5e4004f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:05.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5ebe00640 1 --2- 172.21.15.29:0/897624110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5e4009170 0x7ff5e4004f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5ebe00640 1 -- 172.21.15.29:0/897624110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e41759e0 con 0x7ff5e40dd0e0 2024-04-17T19:55:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5ebe00640 1 --2- 172.21.15.29:0/897624110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5e40dd0e0 0x7ff5e40cf0d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0518f0 tx=0x7ff5ec09a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5df400640 1 -- 172.21.15.29:0/897624110 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec12d8e0 con 0x7ff5e40dd0e0 2024-04-17T19:55:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5df400640 1 -- 172.21.15.29:0/897624110 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5ec12da80 con 0x7ff5e40dd0e0 2024-04-17T19:55:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5f0c00640 1 -- 172.21.15.29:0/897624110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e41766b0 con 0x7ff5e40dd0e0 2024-04-17T19:55:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5df400640 1 -- 172.21.15.29:0/897624110 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec12dc30 con 0x7ff5e40dd0e0 2024-04-17T19:55:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5f0c00640 1 -- 172.21.15.29:0/897624110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5e4048c00 con 0x7ff5e40dd0e0 2024-04-17T19:55:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5f0c00640 1 -- 172.21.15.29:0/897624110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5e4005900 con 0x7ff5e40dd0e0 2024-04-17T19:55:05.057 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.060+0000 7ff5df400640 1 -- 172.21.15.29:0/897624110 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff5ec13c030 con 0x7ff5e40dd0e0 2024-04-17T19:55:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.064+0000 7ff5df400640 1 --2- 172.21.15.29:0/897624110 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff5cc068380 0x7ff5cc06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.064+0000 7ff5df400640 1 -- 172.21.15.29:0/897624110 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7105+0+0 (secure 0 0 0) 0x7ff5ec1a0b60 con 0x7ff5e40dd0e0 2024-04-17T19:55:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.064+0000 7ff5eb400640 1 --2- 172.21.15.29:0/897624110 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff5cc068380 0x7ff5cc06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:05.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.064+0000 7ff5eb400640 1 --2- 172.21.15.29:0/897624110 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff5cc068380 0x7ff5cc06a890 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0009580 tx=0x7ff5e0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:05.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.068+0000 7ff5df400640 1 -- 172.21.15.29:0/897624110 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff5ec143020 con 0x7ff5e40dd0e0 2024-04-17T19:55:05.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.300+0000 7ff5f0c00640 1 -- 172.21.15.29:0/897624110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5e4034aa0 con 0x7ff5e40dd0e0 2024-04-17T19:55:05.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.300+0000 7ff5df400640 1 -- 172.21.15.29:0/897624110 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6904 (secure 0 0 0) 0x7ff5ec095c30 con 0x7ff5e40dd0e0 2024-04-17T19:55:05.296 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:05.296 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5144,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:34.909002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,4305,5212,4302,4259]},"id":1}]} 2024-04-17T19:55:05.296 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-04-17T19:55:05.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.304+0000 7ff5f0c00640 1 -- 172.21.15.29:0/897624110 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff5cc068380 msgr2=0x7ff5cc06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:05.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.304+0000 7ff5f0c00640 1 --2- 172.21.15.29:0/897624110 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff5cc068380 0x7ff5cc06a890 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0009580 tx=0x7ff5e0009190 comp rx=0 tx=0).stop 2024-04-17T19:55:05.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.304+0000 7ff5f0c00640 1 -- 172.21.15.29:0/897624110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5e40dd0e0 msgr2=0x7ff5e40cf0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:05.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.304+0000 7ff5f0c00640 1 --2- 172.21.15.29:0/897624110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5e40dd0e0 0x7ff5e40cf0d0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec0518f0 tx=0x7ff5ec09a890 comp rx=0 tx=0).stop 2024-04-17T19:55:05.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.304+0000 7ff5f0c00640 1 -- 172.21.15.29:0/897624110 shutdown_connections 2024-04-17T19:55:05.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.304+0000 7ff5f0c00640 1 --2- 172.21.15.29:0/897624110 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff5cc068380 0x7ff5cc06a890 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:05.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.304+0000 7ff5f0c00640 1 --2- 172.21.15.29:0/897624110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5e40dd0e0 0x7ff5e40cf0d0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:05.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.304+0000 7ff5f0c00640 1 --2- 172.21.15.29:0/897624110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff5e4009170 0x7ff5e4004f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:05.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.304+0000 7ff5f0c00640 1 --2- 172.21.15.29:0/897624110 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff5e4007480 0x7ff5e4176170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:05.299 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.304+0000 7ff5f0c00640 1 -- 172.21.15.29:0/897624110 >> 172.21.15.29:0/897624110 conn(0x7ff5e40514d0 msgr2=0x7ff5e40d3290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:05.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.304+0000 7ff5f0c00640 1 -- 172.21.15.29:0/897624110 shutdown_connections 2024-04-17T19:55:05.300 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:05.304+0000 7ff5f0c00640 1 -- 172.21.15.29:0/897624110 wait complete. 2024-04-17T19:55:05.323 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-17T19:55:07.324 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:07.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.496+0000 7ff4d1e00640 1 Processor -- start 2024-04-17T19:55:07.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.496+0000 7ff4d1e00640 1 -- start start 2024-04-17T19:55:07.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.496+0000 7ff4d1e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff4cc0cc8c0 0x7ff4cc0cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:07.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.496+0000 7ff4d1e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff4cc159390 0x7ff4cc1597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:07.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.496+0000 7ff4d1e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4cc0ca330 0x7ff4cc0ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:07.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.496+0000 7ff4d1e00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7ff4cc0c88b0 con 0x7ff4cc159390 2024-04-17T19:55:07.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.496+0000 7ff4d1e00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff4cc0c6ce0 con 0x7ff4cc0cc8c0 2024-04-17T19:55:07.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.496+0000 7ff4d1e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff4cc0c6fc0 con 0x7ff4cc0ca330 2024-04-17T19:55:07.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.496+0000 7ff4ca000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff4cc159390 0x7ff4cc1597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:07.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.496+0000 7ff4ca000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff4cc159390 0x7ff4cc1597a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37064/0 (socket says 172.21.15.29:37064) 2024-04-17T19:55:07.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.496+0000 7ff4caa00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff4cc0cc8c0 0x7ff4cc0cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:07.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.496+0000 7ff4ca000640 1 -- 172.21.15.29:0/1639039632 learned_addr learned my addr 172.21.15.29:0/1639039632 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:07.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.496+0000 7ff4cb400640 1 --2- 172.21.15.29:0/1639039632 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4cc0ca330 0x7ff4cc0ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:07.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.500+0000 7ff4ca000640 1 -- 172.21.15.29:0/1639039632 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4cc0ca330 msgr2=0x7ff4cc0ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:07.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.500+0000 7ff4ca000640 1 --2- 172.21.15.29:0/1639039632 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4cc0ca330 0x7ff4cc0ca740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:07.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.500+0000 7ff4ca000640 1 -- 172.21.15.29:0/1639039632 >> v2:172.21.15.37:3300/0 conn(0x7ff4cc0cc8c0 msgr2=0x7ff4cc0cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:07.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.500+0000 7ff4ca000640 1 --2- 172.21.15.29:0/1639039632 >> v2:172.21.15.37:3300/0 conn(0x7ff4cc0cc8c0 0x7ff4cc0cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:07.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.500+0000 7ff4ca000640 1 -- 172.21.15.29:0/1639039632 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4cc0c72a0 con 0x7ff4cc159390 2024-04-17T19:55:07.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.500+0000 7ff4ca000640 1 --2- 172.21.15.29:0/1639039632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4cc159390 0x7ff4cc1597a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc0721b0 tx=0x7ff4b801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e47ed25ef42499 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:07.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.500+0000 7ff4c9600640 1 -- 172.21.15.29:0/1639039632 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b80b0070 con 0x7ff4cc159390 2024-04-17T19:55:07.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.500+0000 7ff4c9600640 1 -- 172.21.15.29:0/1639039632 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b80ab600 con 0x7ff4cc159390 2024-04-17T19:55:07.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.500+0000 7ff4c9600640 1 -- 172.21.15.29:0/1639039632 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b80ab9d0 con 0x7ff4cc159390 2024-04-17T19:55:07.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.504+0000 7ff4d1e00640 1 -- 172.21.15.29:0/1639039632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4cc159390 msgr2=0x7ff4cc1597a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:07.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.504+0000 7ff4d1e00640 1 --2- 172.21.15.29:0/1639039632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4cc159390 0x7ff4cc1597a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc0721b0 tx=0x7ff4b801c520 comp rx=0 tx=0).stop 2024-04-17T19:55:07.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.504+0000 7ff4d1e00640 1 -- 172.21.15.29:0/1639039632 shutdown_connections 2024-04-17T19:55:07.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.504+0000 7ff4d1e00640 1 --2- 172.21.15.29:0/1639039632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4cc159390 0x7ff4cc1597a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:07.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.504+0000 7ff4d1e00640 1 --2- 172.21.15.29:0/1639039632 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4cc0ca330 0x7ff4cc0ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:07.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.504+0000 7ff4d1e00640 1 --2- 172.21.15.29:0/1639039632 >> v2:172.21.15.37:3300/0 conn(0x7ff4cc0cc8c0 0x7ff4cc0cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:07.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.504+0000 7ff4d1e00640 1 -- 172.21.15.29:0/1639039632 >> 172.21.15.29:0/1639039632 conn(0x7ff4cc06fde0 msgr2=0x7ff4cc0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:07.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.504+0000 7ff4d1e00640 1 -- 172.21.15.29:0/1639039632 shutdown_connections 2024-04-17T19:55:07.498 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.504+0000 7ff4d1e00640 1 -- 172.21.15.29:0/1639039632 wait complete. 2024-04-17T19:55:07.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.504+0000 7ff4d1e00640 1 Processor -- start 2024-04-17T19:55:07.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.504+0000 7ff4d1e00640 1 -- start start 2024-04-17T19:55:07.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4d1e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4cc0ca330 0x7ff4cc15e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:07.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4d1e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4cc0cc8c0 0x7ff4cc154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:07.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4d1e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4cc159390 0x7ff4cc1f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:07.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4d1e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff4cc15ca60 con 0x7ff4cc0cc8c0 2024-04-17T19:55:07.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4d1e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff4cc0c6a90 con 0x7ff4cc159390 2024-04-17T19:55:07.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4d1e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff4cc15cbe0 con 0x7ff4cc0ca330 2024-04-17T19:55:07.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4ca000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4cc0cc8c0 0x7ff4cc154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:07.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4ca000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4cc0cc8c0 0x7ff4cc154c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37072/0 (socket says 172.21.15.29:37072) 2024-04-17T19:55:07.503 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4ca000640 1 -- 172.21.15.29:0/3590024597 learned_addr learned my addr 172.21.15.29:0/3590024597 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4caa00640 1 --2- 172.21.15.29:0/3590024597 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4cc0ca330 0x7ff4cc15e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4cb400640 1 --2- 172.21.15.29:0/3590024597 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4cc159390 0x7ff4cc1f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4ca000640 1 -- 172.21.15.29:0/3590024597 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4cc0ca330 msgr2=0x7ff4cc15e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4ca000640 1 --2- 172.21.15.29:0/3590024597 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4cc0ca330 0x7ff4cc15e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4ca000640 1 -- 172.21.15.29:0/3590024597 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4cc159390 msgr2=0x7ff4cc1f6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4ca000640 1 --2- 172.21.15.29:0/3590024597 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4cc159390 0x7ff4cc1f6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4ca000640 1 -- 172.21.15.29:0/3590024597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4cc1551b0 con 0x7ff4cc0cc8c0 2024-04-17T19:55:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4caa00640 1 --2- 172.21.15.29:0/3590024597 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4cc0ca330 0x7ff4cc15e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:55:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4ca000640 1 --2- 172.21.15.29:0/3590024597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4cc0cc8c0 0x7ff4cc154c70 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff4b801abd0 tx=0x7ff4b80192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4c3400640 1 -- 172.21.15.29:0/3590024597 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b80b0040 con 0x7ff4cc0cc8c0 2024-04-17T19:55:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4c3400640 1 -- 172.21.15.29:0/3590024597 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b80b4620 con 0x7ff4cc0cc8c0 2024-04-17T19:55:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4c3400640 1 -- 172.21.15.29:0/3590024597 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b80b4a10 con 0x7ff4cc0cc8c0 2024-04-17T19:55:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4d1e00640 1 -- 172.21.15.29:0/3590024597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4cc1f7480 con 0x7ff4cc0cc8c0 2024-04-17T19:55:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4d1e00640 1 -- 172.21.15.29:0/3590024597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4cc15d940 con 0x7ff4cc0cc8c0 2024-04-17T19:55:07.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4d1e00640 1 -- 172.21.15.29:0/3590024597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4cc069b90 con 0x7ff4cc0cc8c0 2024-04-17T19:55:07.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.508+0000 7ff4c3400640 1 -- 172.21.15.29:0/3590024597 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff4b80ab4c0 con 0x7ff4cc0cc8c0 2024-04-17T19:55:07.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.512+0000 7ff4c3400640 1 --2- 172.21.15.29:0/3590024597 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff4ac068380 0x7ff4ac06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:07.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.512+0000 7ff4c3400640 1 -- 172.21.15.29:0/3590024597 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 7105+0+0 (secure 0 0 0) 0x7ff4b8120bb0 con 0x7ff4cc0cc8c0 2024-04-17T19:55:07.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.512+0000 7ff4caa00640 1 --2- 172.21.15.29:0/3590024597 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff4ac068380 0x7ff4ac06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:07.507 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.512+0000 7ff4caa00640 1 --2- 172.21.15.29:0/3590024597 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff4ac068380 0x7ff4ac06a890 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009b10 tx=0x7ff4bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:07.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.516+0000 7ff4c3400640 1 -- 172.21.15.29:0/3590024597 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff4b8016110 con 0x7ff4cc0cc8c0 2024-04-17T19:55:07.710 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.716+0000 7ff4d1e00640 1 -- 172.21.15.29:0/3590024597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4cc154890 con 0x7ff4cc0cc8c0 2024-04-17T19:55:07.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.716+0000 7ff4c3400640 1 -- 172.21.15.29:0/3590024597 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6904 (secure 0 0 0) 0x7ff4b80f0be0 con 0x7ff4cc0cc8c0 2024-04-17T19:55:07.711 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:07.711 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5144,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:34.909002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,4305,5212,4302,4259]},"id":1}]} 2024-04-17T19:55:07.711 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-04-17T19:55:07.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.716+0000 7ff4d1e00640 1 -- 172.21.15.29:0/3590024597 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff4ac068380 msgr2=0x7ff4ac06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:07.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.716+0000 7ff4d1e00640 1 --2- 172.21.15.29:0/3590024597 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff4ac068380 0x7ff4ac06a890 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009b10 tx=0x7ff4bc008040 comp rx=0 tx=0).stop 2024-04-17T19:55:07.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.716+0000 7ff4d1e00640 1 -- 172.21.15.29:0/3590024597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4cc0cc8c0 msgr2=0x7ff4cc154c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:07.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.716+0000 7ff4d1e00640 1 --2- 172.21.15.29:0/3590024597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4cc0cc8c0 0x7ff4cc154c70 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7ff4b801abd0 tx=0x7ff4b80192c0 comp rx=0 tx=0).stop 2024-04-17T19:55:07.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.716+0000 7ff4d1e00640 1 -- 172.21.15.29:0/3590024597 shutdown_connections 2024-04-17T19:55:07.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.716+0000 7ff4d1e00640 1 --2- 172.21.15.29:0/3590024597 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff4ac068380 0x7ff4ac06a890 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:07.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.716+0000 7ff4d1e00640 1 --2- 172.21.15.29:0/3590024597 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff4cc159390 0x7ff4cc1f6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:07.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.716+0000 7ff4d1e00640 1 --2- 172.21.15.29:0/3590024597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4cc0cc8c0 0x7ff4cc154c70 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:07.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.716+0000 7ff4d1e00640 1 --2- 172.21.15.29:0/3590024597 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff4cc0ca330 0x7ff4cc15e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:07.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.716+0000 7ff4d1e00640 1 -- 172.21.15.29:0/3590024597 >> 172.21.15.29:0/3590024597 conn(0x7ff4cc06fde0 msgr2=0x7ff4cc13ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:07.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.716+0000 7ff4d1e00640 1 -- 172.21.15.29:0/3590024597 shutdown_connections 2024-04-17T19:55:07.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:07.716+0000 7ff4d1e00640 1 -- 172.21.15.29:0/3590024597 wait complete. 2024-04-17T19:55:07.733 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-17T19:55:09.734 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:09.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.904+0000 7fb0b7200640 1 Processor -- start 2024-04-17T19:55:09.901 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.904+0000 7fb0b7200640 1 -- start start 2024-04-17T19:55:09.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.908+0000 7fb0b7200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb0b8153c00 0x7fb0b8072200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:09.903 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.908+0000 7fb0b7200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0b806d210 0x7fb0b80c7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.908+0000 7fb0b7200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0b8154010 con 0x7fb0b806d620 2024-04-17T19:55:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.908+0000 7fb0b7200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fb0b8159c30 con 0x7fb0b806d210 2024-04-17T19:55:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.908+0000 7fb0b7200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb0b8071000 con 0x7fb0b8153c00 2024-04-17T19:55:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.908+0000 7fb0afe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0b806d210 0x7fb0b80c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.908+0000 7fb0b5e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb0b8153c00 0x7fb0b8072200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.908+0000 7fb0afe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0b806d210 0x7fb0b80c7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53494/0 (socket says 172.21.15.29:53494) 2024-04-17T19:55:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.908+0000 7fb0afe00640 1 -- 172.21.15.29:0/2916494362 learned_addr learned my addr 172.21.15.29:0/2916494362 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.908+0000 7fb0b6800640 1 --1- 172.21.15.29:0/2916494362 >> v1:172.21.15.37:6789/0 conn(0x7fb0b806d620 0x7fb0b815d110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:60848/0 (socket says 172.21.15.29:60848) 2024-04-17T19:55:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.908+0000 7fb0afe00640 1 -- 172.21.15.29:0/2916494362 >> v2:172.21.15.37:3300/0 conn(0x7fb0b8153c00 msgr2=0x7fb0b8072200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.908+0000 7fb0afe00640 1 --2- 172.21.15.29:0/2916494362 >> v2:172.21.15.37:3300/0 conn(0x7fb0b8153c00 0x7fb0b8072200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.908+0000 7fb0afe00640 1 -- 172.21.15.29:0/2916494362 >> v1:172.21.15.37:6789/0 conn(0x7fb0b806d620 legacy=0x7fb0b815d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:09.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.908+0000 7fb0afe00640 1 -- 172.21.15.29:0/2916494362 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b8072b20 con 0x7fb0b806d210 2024-04-17T19:55:09.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.912+0000 7fb0afe00640 1 --2- 172.21.15.29:0/2916494362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0b806d210 0x7fb0b80c7980 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80cdf50 tx=0x7fb0a001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2059e7af5bbd66a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:09.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.912+0000 7fb0b5400640 1 -- 172.21.15.29:0/2916494362 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a00ab600 con 0x7fb0b806d210 2024-04-17T19:55:09.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.912+0000 7fb0b5400640 1 -- 172.21.15.29:0/2916494362 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0a00b0070 con 0x7fb0b806d210 2024-04-17T19:55:09.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.912+0000 7fb0b5400640 1 -- 172.21.15.29:0/2916494362 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a00ab9c0 con 0x7fb0b806d210 2024-04-17T19:55:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.912+0000 7fb0b7200640 1 -- 172.21.15.29:0/2916494362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0b806d210 msgr2=0x7fb0b80c7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.912+0000 7fb0b7200640 1 --2- 172.21.15.29:0/2916494362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0b806d210 0x7fb0b80c7980 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80cdf50 tx=0x7fb0a001c4f0 comp rx=0 tx=0).stop 2024-04-17T19:55:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.912+0000 7fb0b7200640 1 -- 172.21.15.29:0/2916494362 shutdown_connections 2024-04-17T19:55:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.912+0000 7fb0b7200640 1 --2- 172.21.15.29:0/2916494362 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0b806d210 0x7fb0b80c7980 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.912+0000 7fb0b7200640 1 --2- 172.21.15.29:0/2916494362 >> v2:172.21.15.37:3300/0 conn(0x7fb0b8153c00 0x7fb0b8072200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.912+0000 7fb0b7200640 1 -- 172.21.15.29:0/2916494362 >> 172.21.15.29:0/2916494362 conn(0x7fb0b80ccf80 msgr2=0x7fb0b80cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:09.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.912+0000 7fb0b7200640 1 -- 172.21.15.29:0/2916494362 shutdown_connections 2024-04-17T19:55:09.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.912+0000 7fb0b7200640 1 -- 172.21.15.29:0/2916494362 wait complete. 2024-04-17T19:55:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0b7200640 1 Processor -- start 2024-04-17T19:55:09.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0b7200640 1 -- start start 2024-04-17T19:55:09.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0b7200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0b806d210 0x7fb0b8154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:09.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0b7200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0b806d620 0x7fb0b8146f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0b7200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0b8153c00 0x7fb0b8150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0b7200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b8072f80 con 0x7fb0b8153c00 2024-04-17T19:55:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0b7200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b8072e00 con 0x7fb0b806d210 2024-04-17T19:55:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0b7200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb0b815a9b0 con 0x7fb0b806d620 2024-04-17T19:55:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0b6800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0b8153c00 0x7fb0b8150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0b5e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0b806d210 0x7fb0b8154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0afe00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0b806d620 0x7fb0b8146f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0b6800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0b8153c00 0x7fb0b8150110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53498/0 (socket says 172.21.15.29:53498) 2024-04-17T19:55:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0b6800640 1 -- 172.21.15.29:0/4272038018 learned_addr learned my addr 172.21.15.29:0/4272038018 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0b6800640 1 -- 172.21.15.29:0/4272038018 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0b806d620 msgr2=0x7fb0b8146f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0b6800640 1 --2- 172.21.15.29:0/4272038018 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0b806d620 0x7fb0b8146f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0b6800640 1 -- 172.21.15.29:0/4272038018 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0b806d210 msgr2=0x7fb0b8154b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:09.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0b6800640 1 --2- 172.21.15.29:0/4272038018 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0b806d210 0x7fb0b8154b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:09.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0b6800640 1 -- 172.21.15.29:0/4272038018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b81550c0 con 0x7fb0b8153c00 2024-04-17T19:55:09.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0b6800640 1 --2- 172.21.15.29:0/4272038018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0b8153c00 0x7fb0b8150110 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb0b000c700 tx=0x7fb0b000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:09.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.916+0000 7fb0ae000640 1 -- 172.21.15.29:0/4272038018 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0017070 con 0x7fb0b8153c00 2024-04-17T19:55:09.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.920+0000 7fb0ae000640 1 -- 172.21.15.29:0/4272038018 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0b000ce10 con 0x7fb0b8153c00 2024-04-17T19:55:09.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.920+0000 7fb0ae000640 1 -- 172.21.15.29:0/4272038018 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b000a840 con 0x7fb0b8153c00 2024-04-17T19:55:09.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.920+0000 7fb0b7200640 1 -- 172.21.15.29:0/4272038018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0b80c7220 con 0x7fb0b8153c00 2024-04-17T19:55:09.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.920+0000 7fb0b7200640 1 -- 172.21.15.29:0/4272038018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0b8150cd0 con 0x7fb0b8153c00 2024-04-17T19:55:09.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.920+0000 7fb0b7200640 1 -- 172.21.15.29:0/4272038018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0b804ff10 con 0x7fb0b8153c00 2024-04-17T19:55:09.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.920+0000 7fb0ae000640 1 -- 172.21.15.29:0/4272038018 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb0b001a070 con 0x7fb0b8153c00 2024-04-17T19:55:09.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.920+0000 7fb0ae000640 1 --2- 172.21.15.29:0/4272038018 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb0940683b0 0x7fb09406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:09.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.920+0000 7fb0ae000640 1 -- 172.21.15.29:0/4272038018 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7049+0+0 (secure 0 0 0) 0x7fb0b00863e0 con 0x7fb0b8153c00 2024-04-17T19:55:09.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.920+0000 7fb0b5e00640 1 --2- 172.21.15.29:0/4272038018 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb0940683b0 0x7fb09406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:09.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.924+0000 7fb0b5e00640 1 --2- 172.21.15.29:0/4272038018 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb0940683b0 0x7fb09406a8c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009ac0 tx=0x7fb0a8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:09.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:09.928+0000 7fb0ae000640 1 -- 172.21.15.29:0/4272038018 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb0b0055e00 con 0x7fb0b8153c00 2024-04-17T19:55:10.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:10.128+0000 7fb0b7200640 1 -- 172.21.15.29:0/4272038018 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0b8071dd0 con 0x7fb0b8153c00 2024-04-17T19:55:10.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:10.128+0000 7fb0ae000640 1 -- 172.21.15.29:0/4272038018 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6904 (secure 0 0 0) 0x7fb0b005a100 con 0x7fb0b8153c00 2024-04-17T19:55:10.124 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:10.124 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5144,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:34.909002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,4305,5212,4302,4259]},"id":1}]} 2024-04-17T19:55:10.124 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-04-17T19:55:10.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:10.132+0000 7fb0b7200640 1 -- 172.21.15.29:0/4272038018 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb0940683b0 msgr2=0x7fb09406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:10.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:10.132+0000 7fb0b7200640 1 --2- 172.21.15.29:0/4272038018 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb0940683b0 0x7fb09406a8c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009ac0 tx=0x7fb0a8009190 comp rx=0 tx=0).stop 2024-04-17T19:55:10.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:10.132+0000 7fb0b7200640 1 -- 172.21.15.29:0/4272038018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0b8153c00 msgr2=0x7fb0b8150110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:10.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:10.132+0000 7fb0b7200640 1 --2- 172.21.15.29:0/4272038018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0b8153c00 0x7fb0b8150110 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb0b000c700 tx=0x7fb0b000cbc0 comp rx=0 tx=0).stop 2024-04-17T19:55:10.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:10.132+0000 7fb0b7200640 1 -- 172.21.15.29:0/4272038018 shutdown_connections 2024-04-17T19:55:10.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:10.132+0000 7fb0b7200640 1 --2- 172.21.15.29:0/4272038018 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb0940683b0 0x7fb09406a8c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:10.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:10.132+0000 7fb0b7200640 1 --2- 172.21.15.29:0/4272038018 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0b8153c00 0x7fb0b8150110 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:10.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:10.132+0000 7fb0b7200640 1 --2- 172.21.15.29:0/4272038018 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0b806d620 0x7fb0b8146f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:10.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:10.132+0000 7fb0b7200640 1 --2- 172.21.15.29:0/4272038018 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0b806d210 0x7fb0b8154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:10.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:10.132+0000 7fb0b7200640 1 -- 172.21.15.29:0/4272038018 >> 172.21.15.29:0/4272038018 conn(0x7fb0b80ccf80 msgr2=0x7fb0b8158b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:10.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:10.132+0000 7fb0b7200640 1 -- 172.21.15.29:0/4272038018 shutdown_connections 2024-04-17T19:55:10.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:10.132+0000 7fb0b7200640 1 -- 172.21.15.29:0/4272038018 wait complete. 2024-04-17T19:55:10.146 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-17T19:55:12.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:12.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.320+0000 7f095fe00640 1 Processor -- start 2024-04-17T19:55:12.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.320+0000 7f095fe00640 1 -- start start 2024-04-17T19:55:12.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.320+0000 7f095fe00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0958159c90 0x7f09580ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:12.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.320+0000 7f095fe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f09580ca7f0 0x7f09580c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:12.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.320+0000 7f095fe00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f095815a0a0 con 0x7f09580c74b0 2024-04-17T19:55:12.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.320+0000 7f095fe00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f095815abe0 con 0x7f09580ca7f0 2024-04-17T19:55:12.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.320+0000 7f095fe00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f095815a340 con 0x7f0958159c90 2024-04-17T19:55:12.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.320+0000 7f095ca00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f09580ca7f0 0x7f09580c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:12.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.320+0000 7f095de00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f09580c74b0 0x7f095806c660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32928/0 (socket says 172.21.15.29:32928) 2024-04-17T19:55:12.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.320+0000 7f095de00640 1 -- 172.21.15.29:0/1477659379 learned_addr learned my addr 172.21.15.29:0/1477659379 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:55:12.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.320+0000 7f095d400640 1 --2- 172.21.15.29:0/1477659379 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0958159c90 0x7f09580ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:12.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.320+0000 7f095ca00640 1 -- 172.21.15.29:0/1477659379 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0958159c90 msgr2=0x7f09580ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:12.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.320+0000 7f095ca00640 1 --2- 172.21.15.29:0/1477659379 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0958159c90 0x7f09580ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:12.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.320+0000 7f095ca00640 1 -- 172.21.15.29:0/1477659379 >> v1:172.21.15.29:6789/0 conn(0x7f09580c74b0 legacy=0x7f095806c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:12.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.320+0000 7f095ca00640 1 -- 172.21.15.29:0/1477659379 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f095815a620 con 0x7f09580ca7f0 2024-04-17T19:55:12.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.320+0000 7f0957e00640 1 -- 172.21.15.29:0/1477659379 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2700929998 0 0) 0x7f095815a0a0 con 0x7f09580c74b0 2024-04-17T19:55:12.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.324+0000 7f095ca00640 1 --2- 172.21.15.29:0/1477659379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09580ca7f0 0x7f09580c6f70 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f09440097c0 tx=0x7f094401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6da3d2c4866e13d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:12.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.324+0000 7f0957e00640 1 -- 172.21.15.29:0/1477659379 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09440b1070 con 0x7f09580ca7f0 2024-04-17T19:55:12.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.324+0000 7f0957e00640 1 -- 172.21.15.29:0/1477659379 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09440ac410 con 0x7f09580ca7f0 2024-04-17T19:55:12.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.324+0000 7f0957e00640 1 -- 172.21.15.29:0/1477659379 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09440ac7e0 con 0x7f09580ca7f0 2024-04-17T19:55:12.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.324+0000 7f095fe00640 1 -- 172.21.15.29:0/1477659379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09580ca7f0 msgr2=0x7f09580c6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:12.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.324+0000 7f095fe00640 1 --2- 172.21.15.29:0/1477659379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09580ca7f0 0x7f09580c6f70 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f09440097c0 tx=0x7f094401c4a0 comp rx=0 tx=0).stop 2024-04-17T19:55:12.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.324+0000 7f095fe00640 1 -- 172.21.15.29:0/1477659379 shutdown_connections 2024-04-17T19:55:12.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.324+0000 7f095fe00640 1 --2- 172.21.15.29:0/1477659379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09580ca7f0 0x7f09580c6f70 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:12.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.324+0000 7f095fe00640 1 --2- 172.21.15.29:0/1477659379 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0958159c90 0x7f09580ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:12.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.324+0000 7f095fe00640 1 -- 172.21.15.29:0/1477659379 >> 172.21.15.29:0/1477659379 conn(0x7f095806fde0 msgr2=0x7f09580701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:12.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.324+0000 7f095fe00640 1 -- 172.21.15.29:0/1477659379 shutdown_connections 2024-04-17T19:55:12.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.324+0000 7f095fe00640 1 -- 172.21.15.29:0/1477659379 wait complete. 2024-04-17T19:55:12.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.328+0000 7f095fe00640 1 Processor -- start 2024-04-17T19:55:12.323 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.328+0000 7f095fe00640 1 -- start start 2024-04-17T19:55:12.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.328+0000 7f095fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09580c74b0 0x7f095814d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:12.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.328+0000 7f095fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09580ca7f0 0x7f0958152e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:12.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.328+0000 7f095fe00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0958159c90 0x7f095814bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:12.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.328+0000 7f095fe00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0958166fb0 con 0x7f09580ca7f0 2024-04-17T19:55:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.328+0000 7f095fe00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0958166e30 con 0x7f09580c74b0 2024-04-17T19:55:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.328+0000 7f095fe00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0958167130 con 0x7f0958159c90 2024-04-17T19:55:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.328+0000 7f095ca00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09580ca7f0 0x7f0958152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.328+0000 7f095ca00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09580ca7f0 0x7f0958152e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53514/0 (socket says 172.21.15.29:53514) 2024-04-17T19:55:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.328+0000 7f095ca00640 1 -- 172.21.15.29:0/417691967 learned_addr learned my addr 172.21.15.29:0/417691967 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.328+0000 7f095de00640 1 --2- 172.21.15.29:0/417691967 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0958159c90 0x7f095814bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.328+0000 7f095d400640 1 --2- 172.21.15.29:0/417691967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09580c74b0 0x7f095814d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.328+0000 7f095ca00640 1 -- 172.21.15.29:0/417691967 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0958159c90 msgr2=0x7f095814bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.328+0000 7f095ca00640 1 --2- 172.21.15.29:0/417691967 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0958159c90 0x7f095814bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.328+0000 7f095ca00640 1 -- 172.21.15.29:0/417691967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09580c74b0 msgr2=0x7f095814d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.328+0000 7f095ca00640 1 --2- 172.21.15.29:0/417691967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09580c74b0 0x7f095814d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.328+0000 7f095ca00640 1 -- 172.21.15.29:0/417691967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f095814d620 con 0x7f09580ca7f0 2024-04-17T19:55:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.332+0000 7f095ca00640 1 --2- 172.21.15.29:0/417691967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09580ca7f0 0x7f0958152e40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f094401a0d0 tx=0x7f0944019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:12.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.332+0000 7f0955600640 1 -- 172.21.15.29:0/417691967 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09440b1040 con 0x7f09580ca7f0 2024-04-17T19:55:12.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.332+0000 7f0955600640 1 -- 172.21.15.29:0/417691967 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09440ba070 con 0x7f09580ca7f0 2024-04-17T19:55:12.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.332+0000 7f095fe00640 1 -- 172.21.15.29:0/417691967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09581f5f80 con 0x7f09580ca7f0 2024-04-17T19:55:12.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.332+0000 7f0955600640 1 -- 172.21.15.29:0/417691967 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09440b5650 con 0x7f09580ca7f0 2024-04-17T19:55:12.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.332+0000 7f095fe00640 1 -- 172.21.15.29:0/417691967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09581f6540 con 0x7f09580ca7f0 2024-04-17T19:55:12.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.332+0000 7f095fe00640 1 -- 172.21.15.29:0/417691967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0958069b90 con 0x7f09580ca7f0 2024-04-17T19:55:12.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.332+0000 7f0955600640 1 -- 172.21.15.29:0/417691967 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f09440b57f0 con 0x7f09580ca7f0 2024-04-17T19:55:12.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.332+0000 7f0955600640 1 --2- 172.21.15.29:0/417691967 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f09380683b0 0x7f093806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:12.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.332+0000 7f0955600640 1 -- 172.21.15.29:0/417691967 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7049+0+0 (secure 0 0 0) 0x7f09441209a0 con 0x7f09580ca7f0 2024-04-17T19:55:12.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.332+0000 7f095d400640 1 --2- 172.21.15.29:0/417691967 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f09380683b0 0x7f093806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:12.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.336+0000 7f095d400640 1 --2- 172.21.15.29:0/417691967 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f09380683b0 0x7f093806a8c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f09580c8e90 tx=0x7f094c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:12.333 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.336+0000 7f0955600640 1 -- 172.21.15.29:0/417691967 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f09440c4020 con 0x7f09580ca7f0 2024-04-17T19:55:12.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.536+0000 7f095fe00640 1 -- 172.21.15.29:0/417691967 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09581f5d00 con 0x7f09580ca7f0 2024-04-17T19:55:12.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.536+0000 7f0955600640 1 -- 172.21.15.29:0/417691967 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6904 (secure 0 0 0) 0x7f09581f5d00 con 0x7f09580ca7f0 2024-04-17T19:55:12.533 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:12.534 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5144,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:34.909002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,4305,5212,4302,4259]},"id":1}]} 2024-04-17T19:55:12.534 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-04-17T19:55:12.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.540+0000 7f095fe00640 1 -- 172.21.15.29:0/417691967 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f09380683b0 msgr2=0x7f093806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:12.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.540+0000 7f095fe00640 1 --2- 172.21.15.29:0/417691967 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f09380683b0 0x7f093806a8c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f09580c8e90 tx=0x7f094c008040 comp rx=0 tx=0).stop 2024-04-17T19:55:12.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.540+0000 7f095fe00640 1 -- 172.21.15.29:0/417691967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09580ca7f0 msgr2=0x7f0958152e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:12.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.540+0000 7f095fe00640 1 --2- 172.21.15.29:0/417691967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09580ca7f0 0x7f0958152e40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f094401a0d0 tx=0x7f0944019240 comp rx=0 tx=0).stop 2024-04-17T19:55:12.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.540+0000 7f095fe00640 1 -- 172.21.15.29:0/417691967 shutdown_connections 2024-04-17T19:55:12.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.540+0000 7f095fe00640 1 --2- 172.21.15.29:0/417691967 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f09380683b0 0x7f093806a8c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:12.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.540+0000 7f095fe00640 1 --2- 172.21.15.29:0/417691967 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0958159c90 0x7f095814bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:12.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.540+0000 7f095fe00640 1 --2- 172.21.15.29:0/417691967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09580ca7f0 0x7f0958152e40 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:12.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.540+0000 7f095fe00640 1 --2- 172.21.15.29:0/417691967 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f09580c74b0 0x7f095814d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:12.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.540+0000 7f095fe00640 1 -- 172.21.15.29:0/417691967 >> 172.21.15.29:0/417691967 conn(0x7f095806fde0 msgr2=0x7f0958148db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:12.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.540+0000 7f095fe00640 1 -- 172.21.15.29:0/417691967 shutdown_connections 2024-04-17T19:55:12.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:12.540+0000 7f095fe00640 1 -- 172.21.15.29:0/417691967 wait complete. 2024-04-17T19:55:12.556 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-17T19:55:14.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:14.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b3200640 1 Processor -- start 2024-04-17T19:55:14.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b3200640 1 -- start start 2024-04-17T19:55:14.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b3200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8b406d310 0x7fe8b40c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:14.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b3200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8b4158af0 con 0x7fe8b4141ad0 2024-04-17T19:55:14.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b3200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8b4158dd0 con 0x7fe8b4141ee0 2024-04-17T19:55:14.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b3200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe8b41590b0 con 0x7fe8b406d310 2024-04-17T19:55:14.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b1e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe8b4141ad0 0x7fe8b40c54d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32930/0 (socket says 172.21.15.29:32930) 2024-04-17T19:55:14.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b1e00640 1 -- 172.21.15.29:0/1619698904 learned_addr learned my addr 172.21.15.29:0/1619698904 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:55:14.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b2800640 1 --2- 172.21.15.29:0/1619698904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8b406d310 0x7fe8b40c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:14.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b0a00640 1 -- 172.21.15.29:0/1619698904 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1159843396 0 0) 0x7fe8b4158af0 con 0x7fe8b4141ad0 2024-04-17T19:55:14.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b0a00640 1 -- 172.21.15.29:0/1619698904 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe898003530 con 0x7fe8b4141ad0 2024-04-17T19:55:14.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b0a00640 1 -- 172.21.15.29:0/1619698904 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3163555069 0 0) 0x7fe8b4158dd0 con 0x7fe8b4141ee0 2024-04-17T19:55:14.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b0a00640 1 -- 172.21.15.29:0/1619698904 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8b4158af0 con 0x7fe8b4141ee0 2024-04-17T19:55:14.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b0a00640 1 -- 172.21.15.29:0/1619698904 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 215385635 0 0) 0x7fe898003530 con 0x7fe8b4141ad0 2024-04-17T19:55:14.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b0a00640 1 -- 172.21.15.29:0/1619698904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8b406d310 msgr2=0x7fe8b40c77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:14.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b0a00640 1 --2- 172.21.15.29:0/1619698904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8b406d310 0x7fe8b40c77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:14.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b0a00640 1 -- 172.21.15.29:0/1619698904 >> v1:172.21.15.37:6789/0 conn(0x7fe8b4141ee0 legacy=0x7fe8b406cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:14.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b0a00640 1 -- 172.21.15.29:0/1619698904 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8b4159390 con 0x7fe8b4141ad0 2024-04-17T19:55:14.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b0a00640 1 -- 172.21.15.29:0/1619698904 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fe8a4001680 con 0x7fe8b4141ad0 2024-04-17T19:55:14.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b0a00640 1 -- 172.21.15.29:0/1619698904 >> v1:172.21.15.29:6789/0 conn(0x7fe8b4141ad0 legacy=0x7fe8b40c54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:14.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b0a00640 1 --2- 172.21.15.29:0/1619698904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe898003bf0 0x7fe898004040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:14.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b0a00640 1 --2- 172.21.15.29:0/1619698904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8b406d310 0x7fe898004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:14.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b0a00640 1 --2- 172.21.15.29:0/1619698904 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe898004d50 0x7fe898005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:14.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b0a00640 1 -- 172.21.15.29:0/1619698904 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b41590b0 con 0x7fe898003bf0 2024-04-17T19:55:14.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b0a00640 1 -- 172.21.15.29:0/1619698904 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe89800dda0 con 0x7fe898004d50 2024-04-17T19:55:14.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b0a00640 1 -- 172.21.15.29:0/1619698904 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe89800dfa0 con 0x7fe8b406d310 2024-04-17T19:55:14.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.728+0000 7fe8b1e00640 1 --2- 172.21.15.29:0/1619698904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe898003bf0 0x7fe898004040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:14.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.732+0000 7fe8b2800640 1 --2- 172.21.15.29:0/1619698904 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe898004d50 0x7fe898005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:14.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.732+0000 7fe8b1400640 1 --2- 172.21.15.29:0/1619698904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8b406d310 0x7fe898004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:14.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.732+0000 7fe8b1e00640 1 -- 172.21.15.29:0/1619698904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8b406d310 msgr2=0x7fe898004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:14.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.732+0000 7fe8b1e00640 1 --2- 172.21.15.29:0/1619698904 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8b406d310 0x7fe898004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:14.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.732+0000 7fe8b1e00640 1 -- 172.21.15.29:0/1619698904 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe898004d50 msgr2=0x7fe898005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:14.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.732+0000 7fe8b1e00640 1 --2- 172.21.15.29:0/1619698904 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe898004d50 0x7fe898005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:14.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.732+0000 7fe8b1e00640 1 -- 172.21.15.29:0/1619698904 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe898003530 con 0x7fe898003bf0 2024-04-17T19:55:14.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.732+0000 7fe8b1e00640 1 --2- 172.21.15.29:0/1619698904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe898003bf0 0x7fe898004040 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4007fd0 tx=0x7fe8a401cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cc0458cb32ca77f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:14.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.732+0000 7fe8b0a00640 1 -- 172.21.15.29:0/1619698904 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4007bf0 con 0x7fe898003bf0 2024-04-17T19:55:14.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.732+0000 7fe8b0a00640 1 -- 172.21.15.29:0/1619698904 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8a40a9020 con 0x7fe898003bf0 2024-04-17T19:55:14.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.732+0000 7fe8b0a00640 1 -- 172.21.15.29:0/1619698904 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8a4005b00 con 0x7fe898003bf0 2024-04-17T19:55:14.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.732+0000 7fe8b3200640 1 -- 172.21.15.29:0/1619698904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe898003bf0 msgr2=0x7fe898004040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:14.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.732+0000 7fe8b3200640 1 --2- 172.21.15.29:0/1619698904 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe898003bf0 0x7fe898004040 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe8a4007fd0 tx=0x7fe8a401cb40 comp rx=0 tx=0).stop 2024-04-17T19:55:14.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.736+0000 7fe8b2800640 1 -- 172.21.15.29:0/1619698904 reap_dead start 2024-04-17T19:55:14.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.736+0000 7fe8b3200640 1 -- 172.21.15.29:0/1619698904 shutdown_connections 2024-04-17T19:55:14.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.736+0000 7fe8b3200640 1 -- 172.21.15.29:0/1619698904 >> 172.21.15.29:0/1619698904 conn(0x7fe8b4072db0 msgr2=0x7fe8b40731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:14.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.736+0000 7fe8b3200640 1 -- 172.21.15.29:0/1619698904 shutdown_connections 2024-04-17T19:55:14.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.736+0000 7fe8b3200640 1 -- 172.21.15.29:0/1619698904 wait complete. 2024-04-17T19:55:14.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.736+0000 7fe8b3200640 1 Processor -- start 2024-04-17T19:55:14.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.736+0000 7fe8b3200640 1 -- start start 2024-04-17T19:55:14.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.736+0000 7fe8b3200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe898003bf0 0x7fe8b41e3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:14.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.736+0000 7fe8b3200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8b4155ed0 0x7fe8b41eded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:14.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.736+0000 7fe8b3200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8b41ecf00 0x7fe8b41e2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:14.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.736+0000 7fe8b3200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b40c5d60 con 0x7fe8b41ecf00 2024-04-17T19:55:14.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.736+0000 7fe8b3200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe8b40c5be0 con 0x7fe898003bf0 2024-04-17T19:55:14.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.736+0000 7fe8b3200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe8b40c77d0 con 0x7fe8b4155ed0 2024-04-17T19:55:14.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.740+0000 7fe8b2800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8b41ecf00 0x7fe8b41e2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:14.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.740+0000 7fe8b2800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8b41ecf00 0x7fe8b41e2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53534/0 (socket says 172.21.15.29:53534) 2024-04-17T19:55:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.740+0000 7fe8b2800640 1 -- 172.21.15.29:0/3564412636 learned_addr learned my addr 172.21.15.29:0/3564412636 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.740+0000 7fe8b1e00640 1 --2- 172.21.15.29:0/3564412636 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe898003bf0 0x7fe8b41e3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.740+0000 7fe8b1400640 1 --2- 172.21.15.29:0/3564412636 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8b4155ed0 0x7fe8b41eded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.740+0000 7fe8b2800640 1 -- 172.21.15.29:0/3564412636 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8b4155ed0 msgr2=0x7fe8b41eded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.740+0000 7fe8b2800640 1 --2- 172.21.15.29:0/3564412636 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8b4155ed0 0x7fe8b41eded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.740+0000 7fe8b2800640 1 -- 172.21.15.29:0/3564412636 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe898003bf0 msgr2=0x7fe8b41e3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.740+0000 7fe8b2800640 1 --2- 172.21.15.29:0/3564412636 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe898003bf0 0x7fe8b41e3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.740+0000 7fe8b2800640 1 -- 172.21.15.29:0/3564412636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8b41e2e40 con 0x7fe8b41ecf00 2024-04-17T19:55:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.740+0000 7fe8b2800640 1 --2- 172.21.15.29:0/3564412636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8b41ecf00 0x7fe8b41e2900 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe8b40cf090 tx=0x7fe8ac0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:14.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.740+0000 7fe8aa000640 1 -- 172.21.15.29:0/3564412636 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac011e40 con 0x7fe8b41ecf00 2024-04-17T19:55:14.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.740+0000 7fe8aa000640 1 -- 172.21.15.29:0/3564412636 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8ac01b070 con 0x7fe8b41ecf00 2024-04-17T19:55:14.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.740+0000 7fe8aa000640 1 -- 172.21.15.29:0/3564412636 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8ac0039a0 con 0x7fe8b41ecf00 2024-04-17T19:55:14.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.740+0000 7fe8b3200640 1 -- 172.21.15.29:0/3564412636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8b41ed310 con 0x7fe8b41ecf00 2024-04-17T19:55:14.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.740+0000 7fe8b3200640 1 -- 172.21.15.29:0/3564412636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8b41e4840 con 0x7fe8b41ecf00 2024-04-17T19:55:14.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.740+0000 7fe8b3200640 1 -- 172.21.15.29:0/3564412636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8b404ff10 con 0x7fe8b41ecf00 2024-04-17T19:55:14.737 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.740+0000 7fe8aa000640 1 -- 172.21.15.29:0/3564412636 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe8ac003b40 con 0x7fe8b41ecf00 2024-04-17T19:55:14.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.744+0000 7fe8aa000640 1 --2- 172.21.15.29:0/3564412636 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe894068380 0x7fe89406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:14.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.744+0000 7fe8aa000640 1 -- 172.21.15.29:0/3564412636 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7049+0+0 (secure 0 0 0) 0x7fe8ac087790 con 0x7fe8b41ecf00 2024-04-17T19:55:14.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.744+0000 7fe8b1e00640 1 --2- 172.21.15.29:0/3564412636 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe894068380 0x7fe89406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:14.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.744+0000 7fe8b1e00640 1 --2- 172.21.15.29:0/3564412636 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe894068380 0x7fe89406a890 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe8a40aa000 tx=0x7fe8a40b1040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:14.742 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.748+0000 7fe8aa000640 1 -- 172.21.15.29:0/3564412636 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe8ac025020 con 0x7fe8b41ecf00 2024-04-17T19:55:14.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.952+0000 7fe8b3200640 1 -- 172.21.15.29:0/3564412636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8b41e45d0 con 0x7fe8b41ecf00 2024-04-17T19:55:14.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.952+0000 7fe8aa000640 1 -- 172.21.15.29:0/3564412636 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6904 (secure 0 0 0) 0x7fe8ac057800 con 0x7fe8b41ecf00 2024-04-17T19:55:14.947 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:14.947 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5144,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:34.909002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,4305,5212,4302,4259]},"id":1}]} 2024-04-17T19:55:14.947 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-04-17T19:55:14.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.952+0000 7fe8b3200640 1 -- 172.21.15.29:0/3564412636 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe894068380 msgr2=0x7fe89406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:14.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.952+0000 7fe8b3200640 1 --2- 172.21.15.29:0/3564412636 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe894068380 0x7fe89406a890 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe8a40aa000 tx=0x7fe8a40b1040 comp rx=0 tx=0).stop 2024-04-17T19:55:14.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.952+0000 7fe8b3200640 1 -- 172.21.15.29:0/3564412636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8b41ecf00 msgr2=0x7fe8b41e2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:14.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.952+0000 7fe8b3200640 1 --2- 172.21.15.29:0/3564412636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8b41ecf00 0x7fe8b41e2900 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe8b40cf090 tx=0x7fe8ac0115a0 comp rx=0 tx=0).stop 2024-04-17T19:55:14.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.952+0000 7fe8b3200640 1 -- 172.21.15.29:0/3564412636 shutdown_connections 2024-04-17T19:55:14.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.952+0000 7fe8b3200640 1 --2- 172.21.15.29:0/3564412636 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe894068380 0x7fe89406a890 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:14.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.952+0000 7fe8b3200640 1 --2- 172.21.15.29:0/3564412636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8b41ecf00 0x7fe8b41e2900 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:14.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.952+0000 7fe8b3200640 1 --2- 172.21.15.29:0/3564412636 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe8b4155ed0 0x7fe8b41eded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:14.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.952+0000 7fe8b3200640 1 --2- 172.21.15.29:0/3564412636 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe898003bf0 0x7fe8b41e3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:14.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.952+0000 7fe8b3200640 1 -- 172.21.15.29:0/3564412636 >> 172.21.15.29:0/3564412636 conn(0x7fe8b4072db0 msgr2=0x7fe8b40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:14.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.952+0000 7fe8b3200640 1 -- 172.21.15.29:0/3564412636 shutdown_connections 2024-04-17T19:55:14.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:14.952+0000 7fe8b3200640 1 -- 172.21.15.29:0/3564412636 wait complete. 2024-04-17T19:55:14.969 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-17T19:55:16.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:17.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.140+0000 7ff68be00640 1 Processor -- start 2024-04-17T19:55:17.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.140+0000 7ff68be00640 1 -- start start 2024-04-17T19:55:17.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.140+0000 7ff68be00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff684159c90 0x7ff6840ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:17.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.140+0000 7ff68be00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff68406c660 0x7ff68406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:17.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.140+0000 7ff68be00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff68415a0a0 con 0x7ff6840ca7f0 2024-04-17T19:55:17.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.140+0000 7ff68be00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff68415abe0 con 0x7ff684159c90 2024-04-17T19:55:17.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.140+0000 7ff68be00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff68415a340 con 0x7ff68406c660 2024-04-17T19:55:17.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.140+0000 7ff688a00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff6840ca7f0 0x7ff6840c6f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32942/0 (socket says 172.21.15.29:32942) 2024-04-17T19:55:17.137 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.140+0000 7ff688a00640 1 -- 172.21.15.29:0/514170194 learned_addr learned my addr 172.21.15.29:0/514170194 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:55:17.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.144+0000 7ff689400640 1 --2- 172.21.15.29:0/514170194 >> v2:172.21.15.37:3300/0 conn(0x7ff684159c90 0x7ff6840ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:17.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.144+0000 7ff689e00640 1 --2- 172.21.15.29:0/514170194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff68406c660 0x7ff68406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:17.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.144+0000 7ff683e00640 1 -- 172.21.15.29:0/514170194 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 13897141 0 0) 0x7ff68415a0a0 con 0x7ff6840ca7f0 2024-04-17T19:55:17.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.144+0000 7ff683e00640 1 -- 172.21.15.29:0/514170194 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff66c003530 con 0x7ff6840ca7f0 2024-04-17T19:55:17.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.144+0000 7ff689400640 1 -- 172.21.15.29:0/514170194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff68406c660 msgr2=0x7ff68406ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:17.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.144+0000 7ff689400640 1 --2- 172.21.15.29:0/514170194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff68406c660 0x7ff68406ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:17.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.144+0000 7ff683e00640 1 -- 172.21.15.29:0/514170194 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 490312869 0 0) 0x7ff66c003530 con 0x7ff6840ca7f0 2024-04-17T19:55:17.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.144+0000 7ff689400640 1 -- 172.21.15.29:0/514170194 >> v1:172.21.15.29:6789/0 conn(0x7ff6840ca7f0 legacy=0x7ff6840c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:17.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.144+0000 7ff689400640 1 -- 172.21.15.29:0/514170194 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff68415a620 con 0x7ff684159c90 2024-04-17T19:55:17.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.144+0000 7ff689400640 1 --2- 172.21.15.29:0/514170194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff684159c90 0x7ff6840ca2b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff6780097c0 tx=0x7ff67801c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db7bcaf8caa9889a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:17.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.148+0000 7ff683e00640 1 -- 172.21.15.29:0/514170194 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6780b0070 con 0x7ff684159c90 2024-04-17T19:55:17.141 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.148+0000 7ff683e00640 1 -- 172.21.15.29:0/514170194 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6780ab400 con 0x7ff684159c90 2024-04-17T19:55:17.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.148+0000 7ff683e00640 1 -- 172.21.15.29:0/514170194 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6700016a0 con 0x7ff684159c90 2024-04-17T19:55:17.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.148+0000 7ff68be00640 1 -- 172.21.15.29:0/514170194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff684159c90 msgr2=0x7ff6840ca2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:17.142 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.148+0000 7ff68be00640 1 --2- 172.21.15.29:0/514170194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff684159c90 0x7ff6840ca2b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff6780097c0 tx=0x7ff67801c3c0 comp rx=0 tx=0).stop 2024-04-17T19:55:17.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.148+0000 7ff68be00640 1 -- 172.21.15.29:0/514170194 shutdown_connections 2024-04-17T19:55:17.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.148+0000 7ff68be00640 1 --2- 172.21.15.29:0/514170194 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff68406c660 0x7ff68406ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:17.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.148+0000 7ff68be00640 1 --2- 172.21.15.29:0/514170194 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff684159c90 0x7ff6840ca2b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:17.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.148+0000 7ff68be00640 1 -- 172.21.15.29:0/514170194 >> 172.21.15.29:0/514170194 conn(0x7ff68406fde0 msgr2=0x7ff6840701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:17.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.148+0000 7ff68be00640 1 -- 172.21.15.29:0/514170194 shutdown_connections 2024-04-17T19:55:17.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.148+0000 7ff68be00640 1 -- 172.21.15.29:0/514170194 wait complete. 2024-04-17T19:55:17.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.148+0000 7ff68be00640 1 Processor -- start 2024-04-17T19:55:17.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.148+0000 7ff68be00640 1 -- start start 2024-04-17T19:55:17.146 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff68be00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff68406c660 0x7ff6841f9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff68be00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6840ca7f0 0x7ff68414d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff68be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff684159c90 0x7ff6841519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff68be00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff6840c7680 con 0x7ff684159c90 2024-04-17T19:55:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff68be00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff68406cfb0 con 0x7ff68406c660 2024-04-17T19:55:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff68be00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff6840c7800 con 0x7ff6840ca7f0 2024-04-17T19:55:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff689e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff684159c90 0x7ff6841519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff689e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff684159c90 0x7ff6841519f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53548/0 (socket says 172.21.15.29:53548) 2024-04-17T19:55:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff689e00640 1 -- 172.21.15.29:0/2791739151 learned_addr learned my addr 172.21.15.29:0/2791739151 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff688a00640 1 --2- 172.21.15.29:0/2791739151 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6840ca7f0 0x7ff68414d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff689400640 1 --2- 172.21.15.29:0/2791739151 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff68406c660 0x7ff6841f9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff689e00640 1 -- 172.21.15.29:0/2791739151 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6840ca7f0 msgr2=0x7ff68414d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff689e00640 1 --2- 172.21.15.29:0/2791739151 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6840ca7f0 0x7ff68414d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff689e00640 1 -- 172.21.15.29:0/2791739151 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff68406c660 msgr2=0x7ff6841f9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:17.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff689e00640 1 --2- 172.21.15.29:0/2791739151 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff68406c660 0x7ff6841f9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:17.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff689e00640 1 -- 172.21.15.29:0/2791739151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff684164410 con 0x7ff684159c90 2024-04-17T19:55:17.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff689e00640 1 --2- 172.21.15.29:0/2791739151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff684159c90 0x7ff6841519f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009d30 tx=0x7ff67c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:17.148 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff681600640 1 -- 172.21.15.29:0/2791739151 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67c018020 con 0x7ff684159c90 2024-04-17T19:55:17.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff681600640 1 -- 172.21.15.29:0/2791739151 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff67c014480 con 0x7ff684159c90 2024-04-17T19:55:17.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff68be00640 1 -- 172.21.15.29:0/2791739151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6841f9770 con 0x7ff684159c90 2024-04-17T19:55:17.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff681600640 1 -- 172.21.15.29:0/2791739151 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67c014870 con 0x7ff684159c90 2024-04-17T19:55:17.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff68be00640 1 -- 172.21.15.29:0/2791739151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff684164130 con 0x7ff684159c90 2024-04-17T19:55:17.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.152+0000 7ff68be00640 1 -- 172.21.15.29:0/2791739151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff684069b90 con 0x7ff684159c90 2024-04-17T19:55:17.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.156+0000 7ff681600640 1 -- 172.21.15.29:0/2791739151 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff67c01b020 con 0x7ff684159c90 2024-04-17T19:55:17.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.156+0000 7ff681600640 1 --2- 172.21.15.29:0/2791739151 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff6640683b0 0x7ff66406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:17.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.156+0000 7ff681600640 1 -- 172.21.15.29:0/2791739151 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 7049+0+0 (secure 0 0 0) 0x7ff67c087b70 con 0x7ff684159c90 2024-04-17T19:55:17.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.156+0000 7ff689400640 1 --2- 172.21.15.29:0/2791739151 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff6640683b0 0x7ff66406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:17.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.156+0000 7ff689400640 1 --2- 172.21.15.29:0/2791739151 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff6640683b0 0x7ff66406a8c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff6780ad000 tx=0x7ff6780a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:17.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.160+0000 7ff681600640 1 -- 172.21.15.29:0/2791739151 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff67c022030 con 0x7ff684159c90 2024-04-17T19:55:17.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.364+0000 7ff68be00640 1 -- 172.21.15.29:0/2791739151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff68414dcf0 con 0x7ff684159c90 2024-04-17T19:55:17.359 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.364+0000 7ff681600640 1 -- 172.21.15.29:0/2791739151 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+6904 (secure 0 0 0) 0x7ff67c010040 con 0x7ff684159c90 2024-04-17T19:55:17.359 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:17.359 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5144,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:53:34.909002+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":4259,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4259":{"gid":4259,"name":"a","rank":2,"incarnation":14,"state":"up:active","state_seq":11,"addr":"172.21.15.29:6836/4184160618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":4184160618},{"type":"v1","addr":"172.21.15.29:6836","nonce":4184160618}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,4305,5212,4302,4259]},"id":1}]} 2024-04-17T19:55:17.360 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 37 2024-04-17T19:55:17.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.364+0000 7ff68be00640 1 -- 172.21.15.29:0/2791739151 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff6640683b0 msgr2=0x7ff66406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:17.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.364+0000 7ff68be00640 1 --2- 172.21.15.29:0/2791739151 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff6640683b0 0x7ff66406a8c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff6780ad000 tx=0x7ff6780a6000 comp rx=0 tx=0).stop 2024-04-17T19:55:17.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.364+0000 7ff68be00640 1 -- 172.21.15.29:0/2791739151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff684159c90 msgr2=0x7ff6841519f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:17.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.364+0000 7ff68be00640 1 --2- 172.21.15.29:0/2791739151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff684159c90 0x7ff6841519f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009d30 tx=0x7ff67c00c490 comp rx=0 tx=0).stop 2024-04-17T19:55:17.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.364+0000 7ff68be00640 1 -- 172.21.15.29:0/2791739151 shutdown_connections 2024-04-17T19:55:17.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.364+0000 7ff68be00640 1 --2- 172.21.15.29:0/2791739151 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff6640683b0 0x7ff66406a8c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:17.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.364+0000 7ff68be00640 1 --2- 172.21.15.29:0/2791739151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff684159c90 0x7ff6841519f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:17.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.364+0000 7ff68be00640 1 --2- 172.21.15.29:0/2791739151 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff6840ca7f0 0x7ff68414d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:17.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.364+0000 7ff68be00640 1 --2- 172.21.15.29:0/2791739151 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff68406c660 0x7ff6841f9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:17.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.364+0000 7ff68be00640 1 -- 172.21.15.29:0/2791739151 >> 172.21.15.29:0/2791739151 conn(0x7ff68406fde0 msgr2=0x7ff6840701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:17.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.364+0000 7ff68be00640 1 -- 172.21.15.29:0/2791739151 shutdown_connections 2024-04-17T19:55:17.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:17.368+0000 7ff68be00640 1 -- 172.21.15.29:0/2791739151 wait complete. 2024-04-17T19:55:17.381 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-17T19:55:18.896 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:19.382 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:19.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.552+0000 7f8718200640 1 Processor -- start 2024-04-17T19:55:19.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.552+0000 7f8718200640 1 -- start start 2024-04-17T19:55:19.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.556+0000 7f8718200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87100cccd0 0x7f87100ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:19.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.556+0000 7f8718200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f87100ca870 0x7f87101635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:19.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.556+0000 7f8718200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87100cd0e0 con 0x7f87100cc8c0 2024-04-17T19:55:19.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.556+0000 7f8718200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f87100c88b0 con 0x7f87100ca870 2024-04-17T19:55:19.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.556+0000 7f8718200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f87100c6ce0 con 0x7f87100cccd0 2024-04-17T19:55:19.550 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.556+0000 7f8716200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f87100ca870 0x7f87101635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:19.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.556+0000 7f8716200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f87100ca870 0x7f87101635e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36090/0 (socket says 172.21.15.29:36090) 2024-04-17T19:55:19.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.556+0000 7f8716200640 1 -- 172.21.15.29:0/818089755 learned_addr learned my addr 172.21.15.29:0/818089755 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:19.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.556+0000 7f8714e00640 1 --2- 172.21.15.29:0/818089755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87100cccd0 0x7f87100ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:19.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.556+0000 7f8716200640 1 -- 172.21.15.29:0/818089755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87100cccd0 msgr2=0x7f87100ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:19.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.556+0000 7f8716200640 1 --2- 172.21.15.29:0/818089755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87100cccd0 0x7f87100ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:19.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.556+0000 7f8716200640 1 -- 172.21.15.29:0/818089755 >> v1:172.21.15.37:6789/0 conn(0x7f87100cc8c0 legacy=0x7f8710159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:19.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.556+0000 7f8716200640 1 -- 172.21.15.29:0/818089755 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87100c6fc0 con 0x7f87100ca870 2024-04-17T19:55:19.551 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.556+0000 7f8714e00640 1 --2- 172.21.15.29:0/818089755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87100cccd0 0x7f87100ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:55:19.553 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.556+0000 7f8716200640 1 --2- 172.21.15.29:0/818089755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87100ca870 0x7f87101635e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8708009e50 tx=0x7f870801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d17b068a81ec2ae6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:19.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.560+0000 7f870fe00640 1 -- 172.21.15.29:0/818089755 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87080a5700 con 0x7f87100ca870 2024-04-17T19:55:19.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.560+0000 7f870fe00640 1 -- 172.21.15.29:0/818089755 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87080a58a0 con 0x7f87100ca870 2024-04-17T19:55:19.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.560+0000 7f870fe00640 1 -- 172.21.15.29:0/818089755 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87080a5c70 con 0x7f87100ca870 2024-04-17T19:55:19.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.560+0000 7f8718200640 1 -- 172.21.15.29:0/818089755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87100ca870 msgr2=0x7f87101635e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:19.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.560+0000 7f8718200640 1 --2- 172.21.15.29:0/818089755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87100ca870 0x7f87101635e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8708009e50 tx=0x7f870801ca30 comp rx=0 tx=0).stop 2024-04-17T19:55:19.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.560+0000 7f8718200640 1 -- 172.21.15.29:0/818089755 shutdown_connections 2024-04-17T19:55:19.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.560+0000 7f8718200640 1 --2- 172.21.15.29:0/818089755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87100ca870 0x7f87101635e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:19.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.560+0000 7f8718200640 1 --2- 172.21.15.29:0/818089755 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87100cccd0 0x7f87100ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:19.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.560+0000 7f8718200640 1 -- 172.21.15.29:0/818089755 >> 172.21.15.29:0/818089755 conn(0x7f871006fde0 msgr2=0x7f87100701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:19.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.560+0000 7f8718200640 1 -- 172.21.15.29:0/818089755 shutdown_connections 2024-04-17T19:55:19.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.560+0000 7f8718200640 1 -- 172.21.15.29:0/818089755 wait complete. 2024-04-17T19:55:19.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.560+0000 7f8718200640 1 Processor -- start 2024-04-17T19:55:19.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.560+0000 7f8718200640 1 -- start start 2024-04-17T19:55:19.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8718200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f87100ca870 0x7f871015c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:19.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8718200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87100cc8c0 0x7f871015a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:19.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8718200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87100cccd0 0x7f87101385e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:19.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8718200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8710159c20 con 0x7f87100cccd0 2024-04-17T19:55:19.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8718200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8710159aa0 con 0x7f87100ca870 2024-04-17T19:55:19.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8718200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f87100c6880 con 0x7f87100cc8c0 2024-04-17T19:55:19.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8716200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87100cccd0 0x7f87101385e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:19.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8716200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87100cccd0 0x7f87101385e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36098/0 (socket says 172.21.15.29:36098) 2024-04-17T19:55:19.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8716200640 1 -- 172.21.15.29:0/2910221684 learned_addr learned my addr 172.21.15.29:0/2910221684 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:19.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8715800640 1 --2- 172.21.15.29:0/2910221684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f87100ca870 0x7f871015c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:19.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8714e00640 1 --2- 172.21.15.29:0/2910221684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87100cc8c0 0x7f871015a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:19.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8716200640 1 -- 172.21.15.29:0/2910221684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87100cc8c0 msgr2=0x7f871015a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:19.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8716200640 1 --2- 172.21.15.29:0/2910221684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87100cc8c0 0x7f871015a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:19.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8716200640 1 -- 172.21.15.29:0/2910221684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f87100ca870 msgr2=0x7f871015c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:19.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8716200640 1 --2- 172.21.15.29:0/2910221684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f87100ca870 0x7f871015c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:19.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8716200640 1 -- 172.21.15.29:0/2910221684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87100c4950 con 0x7f87100cccd0 2024-04-17T19:55:19.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8716200640 1 --2- 172.21.15.29:0/2910221684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87100cccd0 0x7f87101385e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f87080af040 tx=0x7f87080197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:19.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f870d600640 1 -- 172.21.15.29:0/2910221684 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87080bc020 con 0x7f87100cccd0 2024-04-17T19:55:19.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f870d600640 1 -- 172.21.15.29:0/2910221684 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87080b8670 con 0x7f87100cccd0 2024-04-17T19:55:19.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8718200640 1 -- 172.21.15.29:0/2910221684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f871013ba40 con 0x7f87100cccd0 2024-04-17T19:55:19.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f870d600640 1 -- 172.21.15.29:0/2910221684 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87080b8a20 con 0x7f87100cccd0 2024-04-17T19:55:19.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8718200640 1 -- 172.21.15.29:0/2910221684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f871013bfc0 con 0x7f87100cccd0 2024-04-17T19:55:19.561 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.564+0000 7f8718200640 1 -- 172.21.15.29:0/2910221684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8710069b90 con 0x7f87100cccd0 2024-04-17T19:55:19.562 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.568+0000 7f870d600640 1 -- 172.21.15.29:0/2910221684 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f87080bf020 con 0x7f87100cccd0 2024-04-17T19:55:19.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.568+0000 7f870d600640 1 --2- 172.21.15.29:0/2910221684 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f86f00683b0 0x7f86f006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:19.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.568+0000 7f870d600640 1 -- 172.21.15.29:0/2910221684 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 7079+0+0 (secure 0 0 0) 0x7f8708121de0 con 0x7f87100cccd0 2024-04-17T19:55:19.563 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.568+0000 7f8715800640 1 --2- 172.21.15.29:0/2910221684 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f86f00683b0 0x7f86f006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:19.564 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.568+0000 7f8715800640 1 --2- 172.21.15.29:0/2910221684 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f86f00683b0 0x7f86f006a8c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8704007fd0 tx=0x7f8704009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:19.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.572+0000 7f870d600640 1 -- 172.21.15.29:0/2910221684 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f87080c6030 con 0x7f87100cccd0 2024-04-17T19:55:19.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.772+0000 7f8718200640 1 -- 172.21.15.29:0/2910221684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f87100c7e10 con 0x7f87100cccd0 2024-04-17T19:55:19.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.772+0000 7f870d600640 1 -- 172.21.15.29:0/2910221684 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6074 (secure 0 0 0) 0x7f8708016620 con 0x7f87100cccd0 2024-04-17T19:55:19.767 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:19.767 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:18.882382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4302,5212,4305,5350]},"id":1}]} 2024-04-17T19:55:19.767 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 38 2024-04-17T19:55:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.772+0000 7f8718200640 1 -- 172.21.15.29:0/2910221684 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f86f00683b0 msgr2=0x7f86f006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.772+0000 7f8718200640 1 --2- 172.21.15.29:0/2910221684 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f86f00683b0 0x7f86f006a8c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8704007fd0 tx=0x7f8704009040 comp rx=0 tx=0).stop 2024-04-17T19:55:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.772+0000 7f8718200640 1 -- 172.21.15.29:0/2910221684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87100cccd0 msgr2=0x7f87101385e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.772+0000 7f8718200640 1 --2- 172.21.15.29:0/2910221684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87100cccd0 0x7f87101385e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f87080af040 tx=0x7f87080197d0 comp rx=0 tx=0).stop 2024-04-17T19:55:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.772+0000 7f8718200640 1 -- 172.21.15.29:0/2910221684 shutdown_connections 2024-04-17T19:55:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.772+0000 7f8718200640 1 --2- 172.21.15.29:0/2910221684 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f86f00683b0 0x7f86f006a8c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.772+0000 7f8718200640 1 --2- 172.21.15.29:0/2910221684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87100cccd0 0x7f87101385e0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.772+0000 7f8718200640 1 --2- 172.21.15.29:0/2910221684 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f87100cc8c0 0x7f871015a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.772+0000 7f8718200640 1 --2- 172.21.15.29:0/2910221684 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f87100ca870 0x7f871015c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.772+0000 7f8718200640 1 -- 172.21.15.29:0/2910221684 >> 172.21.15.29:0/2910221684 conn(0x7f871006fde0 msgr2=0x7f87100cd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.772+0000 7f8718200640 1 -- 172.21.15.29:0/2910221684 shutdown_connections 2024-04-17T19:55:19.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.772+0000 7f8718200640 1 -- 172.21.15.29:0/2910221684 wait complete. 2024-04-17T19:55:19.789 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2024-04-17T19:55:19.789 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-17T19:55:19.789 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:19.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.948+0000 7f42ef400640 1 Processor -- start 2024-04-17T19:55:19.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.948+0000 7f42ef400640 1 -- start start 2024-04-17T19:55:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.948+0000 7f42ef400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42e80cc8c0 0x7f42e80cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.948+0000 7f42ef400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42e8159390 0x7f42e81597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.948+0000 7f42ef400640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42e80c88b0 con 0x7f42e80ca330 2024-04-17T19:55:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.948+0000 7f42ef400640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f42e80c6ce0 con 0x7f42e80cc8c0 2024-04-17T19:55:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.948+0000 7f42ef400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f42e80c6fc0 con 0x7f42e8159390 2024-04-17T19:55:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.948+0000 7f42eca00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42e80cc8c0 0x7f42e80cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.948+0000 7f42eca00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42e80cc8c0 0x7f42e80cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36108/0 (socket says 172.21.15.29:36108) 2024-04-17T19:55:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.948+0000 7f42eca00640 1 -- 172.21.15.29:0/639298192 learned_addr learned my addr 172.21.15.29:0/639298192 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.948+0000 7f42ed400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f42e80ca330 0x7f42e815c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54324/0 (socket says 172.21.15.29:54324) 2024-04-17T19:55:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.948+0000 7f42e7e00640 1 --2- 172.21.15.29:0/639298192 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42e8159390 0x7f42e81597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.948+0000 7f42eca00640 1 -- 172.21.15.29:0/639298192 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42e8159390 msgr2=0x7f42e81597a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.948+0000 7f42eca00640 1 --2- 172.21.15.29:0/639298192 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42e8159390 0x7f42e81597a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:19.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.948+0000 7f42eca00640 1 -- 172.21.15.29:0/639298192 >> v1:172.21.15.29:6789/0 conn(0x7f42e80ca330 legacy=0x7f42e815c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:19.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.948+0000 7f42eca00640 1 -- 172.21.15.29:0/639298192 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e80c72a0 con 0x7f42e80cc8c0 2024-04-17T19:55:19.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.948+0000 7f42e7400640 1 -- 172.21.15.29:0/639298192 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 358329378 0 0) 0x7f42e80c88b0 con 0x7f42e80ca330 2024-04-17T19:55:19.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.952+0000 7f42eca00640 1 --2- 172.21.15.29:0/639298192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e80cc8c0 0x7f42e80cccd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f42dc009870 tx=0x7f42dc01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9006cf1b9df54fba server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:19.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.952+0000 7f42e7400640 1 -- 172.21.15.29:0/639298192 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42dc0b2070 con 0x7f42e80cc8c0 2024-04-17T19:55:19.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.952+0000 7f42e7400640 1 -- 172.21.15.29:0/639298192 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42dc0a9070 con 0x7f42e80cc8c0 2024-04-17T19:55:19.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.952+0000 7f42e7400640 1 -- 172.21.15.29:0/639298192 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42dc0ad880 con 0x7f42e80cc8c0 2024-04-17T19:55:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.952+0000 7f42ef400640 1 -- 172.21.15.29:0/639298192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e80cc8c0 msgr2=0x7f42e80cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.952+0000 7f42ef400640 1 --2- 172.21.15.29:0/639298192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e80cc8c0 0x7f42e80cccd0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f42dc009870 tx=0x7f42dc01c560 comp rx=0 tx=0).stop 2024-04-17T19:55:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.952+0000 7f42ef400640 1 -- 172.21.15.29:0/639298192 shutdown_connections 2024-04-17T19:55:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.952+0000 7f42ef400640 1 --2- 172.21.15.29:0/639298192 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42e8159390 0x7f42e81597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.952+0000 7f42ef400640 1 --2- 172.21.15.29:0/639298192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e80cc8c0 0x7f42e80cccd0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.952+0000 7f42ef400640 1 -- 172.21.15.29:0/639298192 >> 172.21.15.29:0/639298192 conn(0x7f42e806fde0 msgr2=0x7f42e80701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.952+0000 7f42ef400640 1 -- 172.21.15.29:0/639298192 shutdown_connections 2024-04-17T19:55:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.952+0000 7f42ef400640 1 -- 172.21.15.29:0/639298192 wait complete. 2024-04-17T19:55:19.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.956+0000 7f42ef400640 1 Processor -- start 2024-04-17T19:55:19.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.956+0000 7f42ef400640 1 -- start start 2024-04-17T19:55:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.956+0000 7f42ef400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f42e80ca330 0x7f42e815c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.956+0000 7f42ef400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e80cc8c0 0x7f42e815ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.956+0000 7f42ef400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42e8159390 0x7f42e8139c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.956+0000 7f42ef400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f42e80c6a00 con 0x7f42e80cc8c0 2024-04-17T19:55:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.956+0000 7f42ef400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f42e80c6880 con 0x7f42e80ca330 2024-04-17T19:55:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.956+0000 7f42ef400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f42e81645f0 con 0x7f42e8159390 2024-04-17T19:55:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.956+0000 7f42e7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e80cc8c0 0x7f42e815ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.956+0000 7f42e7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e80cc8c0 0x7f42e815ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36122/0 (socket says 172.21.15.29:36122) 2024-04-17T19:55:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.956+0000 7f42e7e00640 1 -- 172.21.15.29:0/3793288717 learned_addr learned my addr 172.21.15.29:0/3793288717 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.956+0000 7f42ed400640 1 --2- 172.21.15.29:0/3793288717 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42e8159390 0x7f42e8139c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.956+0000 7f42eca00640 1 --2- 172.21.15.29:0/3793288717 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f42e80ca330 0x7f42e815c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.956+0000 7f42e7e00640 1 -- 172.21.15.29:0/3793288717 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42e8159390 msgr2=0x7f42e8139c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.956+0000 7f42e7e00640 1 --2- 172.21.15.29:0/3793288717 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42e8159390 0x7f42e8139c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.956+0000 7f42e7e00640 1 -- 172.21.15.29:0/3793288717 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f42e80ca330 msgr2=0x7f42e815c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.956+0000 7f42e7e00640 1 --2- 172.21.15.29:0/3793288717 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f42e80ca330 0x7f42e815c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.956+0000 7f42e7e00640 1 -- 172.21.15.29:0/3793288717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e81587f0 con 0x7f42e80cc8c0 2024-04-17T19:55:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.960+0000 7f42e7e00640 1 --2- 172.21.15.29:0/3793288717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e80cc8c0 0x7f42e815ab50 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f42d400e870 tx=0x7f42d400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.960+0000 7f42e4c00640 1 -- 172.21.15.29:0/3793288717 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d4017070 con 0x7f42e80cc8c0 2024-04-17T19:55:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.960+0000 7f42e4c00640 1 -- 172.21.15.29:0/3793288717 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42d400c940 con 0x7f42e80cc8c0 2024-04-17T19:55:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.960+0000 7f42e4c00640 1 -- 172.21.15.29:0/3793288717 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d400cd30 con 0x7f42e80cc8c0 2024-04-17T19:55:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.960+0000 7f42ef400640 1 -- 172.21.15.29:0/3793288717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42e8158ab0 con 0x7f42e80cc8c0 2024-04-17T19:55:19.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.960+0000 7f42ef400640 1 -- 172.21.15.29:0/3793288717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42e815a350 con 0x7f42e80cc8c0 2024-04-17T19:55:19.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.960+0000 7f42ef400640 1 -- 172.21.15.29:0/3793288717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42e8069b90 con 0x7f42e80cc8c0 2024-04-17T19:55:19.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.960+0000 7f42e4c00640 1 -- 172.21.15.29:0/3793288717 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f42d401a070 con 0x7f42e80cc8c0 2024-04-17T19:55:19.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.960+0000 7f42e4c00640 1 --2- 172.21.15.29:0/3793288717 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f42c8068380 0x7f42c806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:19.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.960+0000 7f42e4c00640 1 -- 172.21.15.29:0/3793288717 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 7079+0+0 (secure 0 0 0) 0x7f42d4087240 con 0x7f42e80cc8c0 2024-04-17T19:55:19.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.960+0000 7f42eca00640 1 --2- 172.21.15.29:0/3793288717 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f42c8068380 0x7f42c806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:19.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.964+0000 7f42eca00640 1 --2- 172.21.15.29:0/3793288717 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f42c8068380 0x7f42c806a890 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f42dc0af000 tx=0x7f42dc0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:19.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:19.964+0000 7f42e4c00640 1 -- 172.21.15.29:0/3793288717 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f42d4057200 con 0x7f42e80cc8c0 2024-04-17T19:55:20.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:20.164+0000 7f42ef400640 1 -- 172.21.15.29:0/3793288717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42e80c7e10 con 0x7f42e80cc8c0 2024-04-17T19:55:20.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:20.164+0000 7f42e4c00640 1 -- 172.21.15.29:0/3793288717 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6074 (secure 0 0 0) 0x7f42d4056b00 con 0x7f42e80cc8c0 2024-04-17T19:55:20.159 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:20.160 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:18.882382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4302,5212,4305,5350]},"id":1}]} 2024-04-17T19:55:20.160 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 38 2024-04-17T19:55:20.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:20.164+0000 7f42ef400640 1 -- 172.21.15.29:0/3793288717 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f42c8068380 msgr2=0x7f42c806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:20.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:20.164+0000 7f42ef400640 1 --2- 172.21.15.29:0/3793288717 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f42c8068380 0x7f42c806a890 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f42dc0af000 tx=0x7f42dc0a7000 comp rx=0 tx=0).stop 2024-04-17T19:55:20.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:20.168+0000 7f42ef400640 1 -- 172.21.15.29:0/3793288717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e80cc8c0 msgr2=0x7f42e815ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:20.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:20.168+0000 7f42ef400640 1 --2- 172.21.15.29:0/3793288717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e80cc8c0 0x7f42e815ab50 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f42d400e870 tx=0x7f42d400ec70 comp rx=0 tx=0).stop 2024-04-17T19:55:20.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:20.168+0000 7f42ef400640 1 -- 172.21.15.29:0/3793288717 shutdown_connections 2024-04-17T19:55:20.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:20.168+0000 7f42ef400640 1 --2- 172.21.15.29:0/3793288717 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f42c8068380 0x7f42c806a890 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:20.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:20.168+0000 7f42ef400640 1 --2- 172.21.15.29:0/3793288717 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f42e8159390 0x7f42e8139c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:20.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:20.168+0000 7f42ef400640 1 --2- 172.21.15.29:0/3793288717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42e80cc8c0 0x7f42e815ab50 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:20.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:20.168+0000 7f42ef400640 1 --2- 172.21.15.29:0/3793288717 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f42e80ca330 0x7f42e815c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:20.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:20.168+0000 7f42ef400640 1 -- 172.21.15.29:0/3793288717 >> 172.21.15.29:0/3793288717 conn(0x7f42e806fde0 msgr2=0x7f42e8164920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:20.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:20.168+0000 7f42ef400640 1 -- 172.21.15.29:0/3793288717 shutdown_connections 2024-04-17T19:55:20.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:20.168+0000 7f42ef400640 1 -- 172.21.15.29:0/3793288717 wait complete. 2024-04-17T19:55:20.182 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 38, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-17T19:47:49.315515+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 38, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4302": { "addr": "172.21.15.29:6835/270402476", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6834", "nonce": 270402476, "type": "v2" }, { "addr": "172.21.15.29:6835", "nonce": 270402476, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 10, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 9 }, "gid_4305": { "addr": "172.21.15.29:6837/2691590158", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6833", "nonce": 2691590158, "type": "v2" }, { "addr": "172.21.15.29:6837", "nonce": 2691590158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4305, "incarnation": 5, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5144": { "addr": "172.21.15.37:6839/858161860", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6836", "nonce": 858161860, "type": "v2" }, { "addr": "172.21.15.37:6839", "nonce": 858161860, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5144, "incarnation": 38, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_5212": { "addr": "172.21.15.37:6838/767681664", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6835", "nonce": 767681664, "type": "v2" }, { "addr": "172.21.15.37:6838", "nonce": 767681664, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5212, "incarnation": 26, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 31 }, "gid_5350": { "addr": "172.21.15.37:6837/2495953425", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6834", "nonce": 2495953425, "type": "v2" }, { "addr": "172.21.15.37:6837", "nonce": 2495953425, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5350, "incarnation": 32, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-17T19:55:18.882382+0000", "qdb_cluster": [ 4302, 5212, 4305, 5350 ], "qdb_leader": 4305, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4305, "mds_1": 4302, "mds_2": 5144, "mds_3": 5350, "mds_4": 5212 } } } ], "standbys": [] } 2024-04-17T19:55:21.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:55:21.955 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:55:22.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:22.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.340+0000 7fcc1b200640 1 Processor -- start 2024-04-17T19:55:22.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc1b200640 1 -- start start 2024-04-17T19:55:22.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc1b200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc140cccd0 0x7fcc140ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:22.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc1b200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc140cd0e0 con 0x7fcc140cc8c0 2024-04-17T19:55:22.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc1b200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc140c88b0 con 0x7fcc140ca870 2024-04-17T19:55:22.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc1b200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcc140c6ce0 con 0x7fcc140cccd0 2024-04-17T19:55:22.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcc140cc8c0 0x7fcc14159390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54334/0 (socket says 172.21.15.29:54334) 2024-04-17T19:55:22.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13e00640 1 -- 172.21.15.29:0/346270571 learned_addr learned my addr 172.21.15.29:0/346270571 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:55:22.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc0be00640 1 --2- 172.21.15.29:0/346270571 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc140cccd0 0x7fcc140ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:22.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13400640 1 -- 172.21.15.29:0/346270571 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3228790252 0 0) 0x7fcc140cd0e0 con 0x7fcc140cc8c0 2024-04-17T19:55:22.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13400640 1 -- 172.21.15.29:0/346270571 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbf8005740 con 0x7fcc140cc8c0 2024-04-17T19:55:22.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13400640 1 -- 172.21.15.29:0/346270571 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1917743770 0 0) 0x7fcc140c88b0 con 0x7fcc140ca870 2024-04-17T19:55:22.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13400640 1 -- 172.21.15.29:0/346270571 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc140cd0e0 con 0x7fcc140ca870 2024-04-17T19:55:22.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13400640 1 -- 172.21.15.29:0/346270571 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1743980439 0 0) 0x7fcbf8005740 con 0x7fcc140cc8c0 2024-04-17T19:55:22.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13400640 1 -- 172.21.15.29:0/346270571 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc140cccd0 msgr2=0x7fcc140ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:22.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13400640 1 --2- 172.21.15.29:0/346270571 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc140cccd0 0x7fcc140ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:22.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13400640 1 -- 172.21.15.29:0/346270571 >> v1:172.21.15.37:6789/0 conn(0x7fcc140ca870 legacy=0x7fcc141635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13400640 1 -- 172.21.15.29:0/346270571 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc140c6fc0 con 0x7fcc140cc8c0 2024-04-17T19:55:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13400640 1 -- 172.21.15.29:0/346270571 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fcc04001680 con 0x7fcc140cc8c0 2024-04-17T19:55:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13400640 1 -- 172.21.15.29:0/346270571 >> v1:172.21.15.29:6789/0 conn(0x7fcc140cc8c0 legacy=0x7fcc14159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13400640 1 --2- 172.21.15.29:0/346270571 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbf8005e00 0x7fcbf8006250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13400640 1 --2- 172.21.15.29:0/346270571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbf8006a20 0x7fcbf8006e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13400640 1 --2- 172.21.15.29:0/346270571 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc140cccd0 0x7fcbf80074e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13400640 1 -- 172.21.15.29:0/346270571 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcc140c6ce0 con 0x7fcbf8006a20 2024-04-17T19:55:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13400640 1 -- 172.21.15.29:0/346270571 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf800ddc0 con 0x7fcbf8005e00 2024-04-17T19:55:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13400640 1 -- 172.21.15.29:0/346270571 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcbf800dfc0 con 0x7fcc140cccd0 2024-04-17T19:55:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc0be00640 1 --2- 172.21.15.29:0/346270571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbf8006a20 0x7fcbf8006e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc13e00640 1 --2- 172.21.15.29:0/346270571 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbf8005e00 0x7fcbf8006250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc19200640 1 --2- 172.21.15.29:0/346270571 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc140cccd0 0x7fcbf80074e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc0be00640 1 -- 172.21.15.29:0/346270571 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc140cccd0 msgr2=0x7fcbf80074e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc0be00640 1 --2- 172.21.15.29:0/346270571 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc140cccd0 0x7fcbf80074e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc0be00640 1 -- 172.21.15.29:0/346270571 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbf8005e00 msgr2=0x7fcbf8006250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:22.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc0be00640 1 --2- 172.21.15.29:0/346270571 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbf8005e00 0x7fcbf8006250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:22.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.344+0000 7fcc0be00640 1 -- 172.21.15.29:0/346270571 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf8005740 con 0x7fcbf8006a20 2024-04-17T19:55:22.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.348+0000 7fcc0be00640 1 --2- 172.21.15.29:0/346270571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbf8006a20 0x7fcbf8006e30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009ab0 tx=0x7fcbfc01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5b5ce61cf2dbd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:22.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.348+0000 7fcc13400640 1 -- 172.21.15.29:0/346270571 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc0b2070 con 0x7fcbf8006a20 2024-04-17T19:55:22.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.348+0000 7fcc13400640 1 -- 172.21.15.29:0/346270571 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbfc0a5660 con 0x7fcbf8006a20 2024-04-17T19:55:22.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.348+0000 7fcc13400640 1 -- 172.21.15.29:0/346270571 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc0096b0 con 0x7fcbf8006a20 2024-04-17T19:55:22.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.348+0000 7fcc1b200640 1 -- 172.21.15.29:0/346270571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbf8006a20 msgr2=0x7fcbf8006e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:22.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.348+0000 7fcc1b200640 1 --2- 172.21.15.29:0/346270571 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbf8006a20 0x7fcbf8006e30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009ab0 tx=0x7fcbfc01ca70 comp rx=0 tx=0).stop 2024-04-17T19:55:22.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.348+0000 7fcc19200640 1 -- 172.21.15.29:0/346270571 reap_dead start 2024-04-17T19:55:22.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.348+0000 7fcc1b200640 1 -- 172.21.15.29:0/346270571 shutdown_connections 2024-04-17T19:55:22.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.348+0000 7fcc1b200640 1 -- 172.21.15.29:0/346270571 >> 172.21.15.29:0/346270571 conn(0x7fcc1406fde0 msgr2=0x7fcc140701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:22.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.348+0000 7fcc1b200640 1 -- 172.21.15.29:0/346270571 shutdown_connections 2024-04-17T19:55:22.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.348+0000 7fcc1b200640 1 -- 172.21.15.29:0/346270571 wait complete. 2024-04-17T19:55:22.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.352+0000 7fcc1b200640 1 Processor -- start 2024-04-17T19:55:22.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.352+0000 7fcc1b200640 1 -- start start 2024-04-17T19:55:22.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.352+0000 7fcc1b200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbf8006a20 0x7fcc1415b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:22.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.352+0000 7fcc1b200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc1415baa0 0x7fcc141f3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:22.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.352+0000 7fcc1b200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc1413ddf0 0x7fcc14159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:22.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.352+0000 7fcc1b200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcc14158e70 con 0x7fcc1413ddf0 2024-04-17T19:55:22.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.352+0000 7fcc1b200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcc140c7580 con 0x7fcbf8006a20 2024-04-17T19:55:22.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.352+0000 7fcc1b200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcc14159aa0 con 0x7fcc1415baa0 2024-04-17T19:55:22.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.352+0000 7fcc19200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc1413ddf0 0x7fcc14159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:22.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.352+0000 7fcc19200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc1413ddf0 0x7fcc14159e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36150/0 (socket says 172.21.15.29:36150) 2024-04-17T19:55:22.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.352+0000 7fcc19200640 1 -- 172.21.15.29:0/1741737902 learned_addr learned my addr 172.21.15.29:0/1741737902 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:22.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.352+0000 7fcc13e00640 1 --2- 172.21.15.29:0/1741737902 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbf8006a20 0x7fcc1415b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:22.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.352+0000 7fcc0be00640 1 --2- 172.21.15.29:0/1741737902 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc1415baa0 0x7fcc141f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:22.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.356+0000 7fcc19200640 1 -- 172.21.15.29:0/1741737902 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc1415baa0 msgr2=0x7fcc141f3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:22.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.356+0000 7fcc19200640 1 --2- 172.21.15.29:0/1741737902 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc1415baa0 0x7fcc141f3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:22.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.356+0000 7fcc19200640 1 -- 172.21.15.29:0/1741737902 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbf8006a20 msgr2=0x7fcc1415b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:22.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.356+0000 7fcc19200640 1 --2- 172.21.15.29:0/1741737902 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbf8006a20 0x7fcc1415b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:22.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.356+0000 7fcc19200640 1 -- 172.21.15.29:0/1741737902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc140c4740 con 0x7fcc1413ddf0 2024-04-17T19:55:22.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.356+0000 7fcc19200640 1 --2- 172.21.15.29:0/1741737902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc1413ddf0 0x7fcc14159e20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fcc140cc800 tx=0x7fcc0c00f170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:22.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.356+0000 7fcc10c00640 1 -- 172.21.15.29:0/1741737902 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c010040 con 0x7fcc1413ddf0 2024-04-17T19:55:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.356+0000 7fcc10c00640 1 -- 172.21.15.29:0/1741737902 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc0c01a070 con 0x7fcc1413ddf0 2024-04-17T19:55:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.356+0000 7fcc10c00640 1 -- 172.21.15.29:0/1741737902 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c015410 con 0x7fcc1413ddf0 2024-04-17T19:55:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.356+0000 7fcc1b200640 1 -- 172.21.15.29:0/1741737902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc141f3540 con 0x7fcc1413ddf0 2024-04-17T19:55:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.356+0000 7fcc1b200640 1 -- 172.21.15.29:0/1741737902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc141f2800 con 0x7fcc1413ddf0 2024-04-17T19:55:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.356+0000 7fcc1b200640 1 -- 172.21.15.29:0/1741737902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc14069b90 con 0x7fcc1413ddf0 2024-04-17T19:55:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.356+0000 7fcc10c00640 1 -- 172.21.15.29:0/1741737902 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcc0c0155b0 con 0x7fcc1413ddf0 2024-04-17T19:55:22.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.356+0000 7fcc10c00640 1 --2- 172.21.15.29:0/1741737902 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbf0068380 0x7fcbf006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:22.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.360+0000 7fcc10c00640 1 -- 172.21.15.29:0/1741737902 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 7223+0+0 (secure 0 0 0) 0x7fcc0c08a400 con 0x7fcc1413ddf0 2024-04-17T19:55:22.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.360+0000 7fcc13e00640 1 --2- 172.21.15.29:0/1741737902 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbf0068380 0x7fcbf006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:22.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.360+0000 7fcc13e00640 1 --2- 172.21.15.29:0/1741737902 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbf0068380 0x7fcbf006a890 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcc04001060 tx=0x7fcc04000f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:22.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.364+0000 7fcc10c00640 1 -- 172.21.15.29:0/1741737902 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fcc0c059c50 con 0x7fcc1413ddf0 2024-04-17T19:55:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.560+0000 7fcc1b200640 1 -- 172.21.15.29:0/1741737902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc140cda80 con 0x7fcc1413ddf0 2024-04-17T19:55:22.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.560+0000 7fcc10c00640 1 -- 172.21.15.29:0/1741737902 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) ==== 94+0+6074 (secure 0 0 0) 0x7fcc0c05df50 con 0x7fcc1413ddf0 2024-04-17T19:55:22.556 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:22.556 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:18.882382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:replay","state_seq":1,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4302,5212,4305,5350]},"id":1}]} 2024-04-17T19:55:22.557 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 38 2024-04-17T19:55:22.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.564+0000 7fcc1b200640 1 -- 172.21.15.29:0/1741737902 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbf0068380 msgr2=0x7fcbf006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:22.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.564+0000 7fcc1b200640 1 --2- 172.21.15.29:0/1741737902 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbf0068380 0x7fcbf006a890 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcc04001060 tx=0x7fcc04000f90 comp rx=0 tx=0).stop 2024-04-17T19:55:22.558 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.564+0000 7fcc1b200640 1 -- 172.21.15.29:0/1741737902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc1413ddf0 msgr2=0x7fcc14159e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.564+0000 7fcc1b200640 1 --2- 172.21.15.29:0/1741737902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc1413ddf0 0x7fcc14159e20 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fcc140cc800 tx=0x7fcc0c00f170 comp rx=0 tx=0).stop 2024-04-17T19:55:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.564+0000 7fcc1b200640 1 -- 172.21.15.29:0/1741737902 shutdown_connections 2024-04-17T19:55:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.564+0000 7fcc1b200640 1 --2- 172.21.15.29:0/1741737902 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbf0068380 0x7fcbf006a890 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.564+0000 7fcc1b200640 1 --2- 172.21.15.29:0/1741737902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc1413ddf0 0x7fcc14159e20 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.564+0000 7fcc1b200640 1 --2- 172.21.15.29:0/1741737902 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc1415baa0 0x7fcc141f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.564+0000 7fcc1b200640 1 --2- 172.21.15.29:0/1741737902 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbf8006a20 0x7fcc1415b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.564+0000 7fcc1b200640 1 -- 172.21.15.29:0/1741737902 >> 172.21.15.29:0/1741737902 conn(0x7fcc1406fde0 msgr2=0x7fcc14141810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.564+0000 7fcc1b200640 1 -- 172.21.15.29:0/1741737902 shutdown_connections 2024-04-17T19:55:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:22.564+0000 7fcc1b200640 1 -- 172.21.15.29:0/1741737902 wait complete. 2024-04-17T19:55:22.579 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:55:24.580 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:24.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.748+0000 7fafdba00640 1 Processor -- start 2024-04-17T19:55:24.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.748+0000 7fafdba00640 1 -- start start 2024-04-17T19:55:24.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.752+0000 7fafdba00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafd40cc8c0 0x7fafd40cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.752+0000 7fafdba00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fafd41597a0 0x7fafd41635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.752+0000 7fafdba00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafd40c88b0 con 0x7fafd4159390 2024-04-17T19:55:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.752+0000 7fafdba00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fafd40c6ce0 con 0x7fafd41597a0 2024-04-17T19:55:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.752+0000 7fafdba00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fafd40c6fc0 con 0x7fafd40cc8c0 2024-04-17T19:55:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.752+0000 7fafd9a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fafd41597a0 0x7fafd41635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.752+0000 7fafd9a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fafd41597a0 0x7fafd41635e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36158/0 (socket says 172.21.15.29:36158) 2024-04-17T19:55:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.752+0000 7fafd9a00640 1 -- 172.21.15.29:0/768072834 learned_addr learned my addr 172.21.15.29:0/768072834 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.752+0000 7fafd9000640 1 --2- 172.21.15.29:0/768072834 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafd40cc8c0 0x7fafd40cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.752+0000 7fafd9a00640 1 -- 172.21.15.29:0/768072834 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafd40cc8c0 msgr2=0x7fafd40cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.752+0000 7fafd9a00640 1 --2- 172.21.15.29:0/768072834 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafd40cc8c0 0x7fafd40cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.752+0000 7fafd9a00640 1 -- 172.21.15.29:0/768072834 >> v1:172.21.15.37:6789/0 conn(0x7fafd4159390 legacy=0x7fafd40ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.752+0000 7fafd9a00640 1 -- 172.21.15.29:0/768072834 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd40c72a0 con 0x7fafd41597a0 2024-04-17T19:55:24.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.752+0000 7fafd9a00640 1 --2- 172.21.15.29:0/768072834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd41597a0 0x7fafd41635e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009e50 tx=0x7fafcc01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fed8c94899cad8a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:24.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.756+0000 7fafd3400640 1 -- 172.21.15.29:0/768072834 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafcc0a5700 con 0x7fafd41597a0 2024-04-17T19:55:24.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.756+0000 7fafd3400640 1 -- 172.21.15.29:0/768072834 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafcc0a58a0 con 0x7fafd41597a0 2024-04-17T19:55:24.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.756+0000 7fafd3400640 1 -- 172.21.15.29:0/768072834 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafcc0a5c70 con 0x7fafd41597a0 2024-04-17T19:55:24.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.756+0000 7fafdba00640 1 -- 172.21.15.29:0/768072834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd41597a0 msgr2=0x7fafd41635e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:24.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.756+0000 7fafdba00640 1 --2- 172.21.15.29:0/768072834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd41597a0 0x7fafd41635e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009e50 tx=0x7fafcc01ca30 comp rx=0 tx=0).stop 2024-04-17T19:55:24.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.756+0000 7fafdba00640 1 -- 172.21.15.29:0/768072834 shutdown_connections 2024-04-17T19:55:24.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.756+0000 7fafdba00640 1 --2- 172.21.15.29:0/768072834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd41597a0 0x7fafd41635e0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:24.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.756+0000 7fafdba00640 1 --2- 172.21.15.29:0/768072834 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafd40cc8c0 0x7fafd40cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:24.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.756+0000 7fafdba00640 1 -- 172.21.15.29:0/768072834 >> 172.21.15.29:0/768072834 conn(0x7fafd406fde0 msgr2=0x7fafd40701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:24.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.756+0000 7fafdba00640 1 -- 172.21.15.29:0/768072834 shutdown_connections 2024-04-17T19:55:24.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.756+0000 7fafdba00640 1 -- 172.21.15.29:0/768072834 wait complete. 2024-04-17T19:55:24.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.756+0000 7fafdba00640 1 Processor -- start 2024-04-17T19:55:24.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.756+0000 7fafdba00640 1 -- start start 2024-04-17T19:55:24.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafdba00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafd40cc8c0 0x7fafd41f5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:24.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafdba00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4159390 0x7fafd415e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:24.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafdba00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafd41597a0 0x7fafd4136e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:24.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafdba00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fafd40cabc0 con 0x7fafd4159390 2024-04-17T19:55:24.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafdba00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fafd40caa40 con 0x7fafd40cc8c0 2024-04-17T19:55:24.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafdba00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fafd40c6880 con 0x7fafd41597a0 2024-04-17T19:55:24.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafd3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4159390 0x7fafd415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:24.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafd3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4159390 0x7fafd415e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36166/0 (socket says 172.21.15.29:36166) 2024-04-17T19:55:24.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafd3e00640 1 -- 172.21.15.29:0/818090517 learned_addr learned my addr 172.21.15.29:0/818090517 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafd9000640 1 --2- 172.21.15.29:0/818090517 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafd40cc8c0 0x7fafd41f5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafd9a00640 1 --2- 172.21.15.29:0/818090517 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafd41597a0 0x7fafd4136e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafd3e00640 1 -- 172.21.15.29:0/818090517 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafd41597a0 msgr2=0x7fafd4136e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafd3e00640 1 --2- 172.21.15.29:0/818090517 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafd41597a0 0x7fafd4136e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafd3e00640 1 -- 172.21.15.29:0/818090517 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafd40cc8c0 msgr2=0x7fafd41f5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafd3e00640 1 --2- 172.21.15.29:0/818090517 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafd40cc8c0 0x7fafd41f5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafd3e00640 1 -- 172.21.15.29:0/818090517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd415ebc0 con 0x7fafd4159390 2024-04-17T19:55:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafd9000640 1 --2- 172.21.15.29:0/818090517 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafd40cc8c0 0x7fafd41f5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:55:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafd3e00640 1 --2- 172.21.15.29:0/818090517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4159390 0x7fafd415e680 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fafc000c340 tx=0x7fafc000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafd0c00640 1 -- 172.21.15.29:0/818090517 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc00052d0 con 0x7fafd4159390 2024-04-17T19:55:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafd0c00640 1 -- 172.21.15.29:0/818090517 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafc0005470 con 0x7fafd4159390 2024-04-17T19:55:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafdba00640 1 -- 172.21.15.29:0/818090517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd40cd030 con 0x7fafd4159390 2024-04-17T19:55:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafd0c00640 1 -- 172.21.15.29:0/818090517 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0007260 con 0x7fafd4159390 2024-04-17T19:55:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafdba00640 1 -- 172.21.15.29:0/818090517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd4140dc0 con 0x7fafd4159390 2024-04-17T19:55:24.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.760+0000 7fafdba00640 1 -- 172.21.15.29:0/818090517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd4069b90 con 0x7fafd4159390 2024-04-17T19:55:24.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.764+0000 7fafd0c00640 1 -- 172.21.15.29:0/818090517 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fafc0007400 con 0x7fafd4159390 2024-04-17T19:55:24.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.764+0000 7fafd0c00640 1 --2- 172.21.15.29:0/818090517 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fafb4068380 0x7fafb406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:24.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.764+0000 7fafd0c00640 1 -- 172.21.15.29:0/818090517 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 7223+0+0 (secure 0 0 0) 0x7fafc0086300 con 0x7fafd4159390 2024-04-17T19:55:24.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.764+0000 7fafd9000640 1 --2- 172.21.15.29:0/818090517 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fafb4068380 0x7fafb406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:24.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.764+0000 7fafd9000640 1 --2- 172.21.15.29:0/818090517 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fafb4068380 0x7fafb406a890 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fafc800b9f0 tx=0x7fafc8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:24.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.768+0000 7fafd0c00640 1 -- 172.21.15.29:0/818090517 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fafc0055c70 con 0x7fafd4159390 2024-04-17T19:55:24.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.972+0000 7fafdba00640 1 -- 172.21.15.29:0/818090517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafd4074780 con 0x7fafd4159390 2024-04-17T19:55:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.972+0000 7fafd0c00640 1 -- 172.21.15.29:0/818090517 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6076 (secure 0 0 0) 0x7fafc0059f70 con 0x7fafd4159390 2024-04-17T19:55:24.968 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:24.968 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:23.902424+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:resolve","state_seq":26,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,4305,5212,4302]},"id":1}]} 2024-04-17T19:55:24.969 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 39 2024-04-17T19:55:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.976+0000 7fafdba00640 1 -- 172.21.15.29:0/818090517 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fafb4068380 msgr2=0x7fafb406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.976+0000 7fafdba00640 1 --2- 172.21.15.29:0/818090517 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fafb4068380 0x7fafb406a890 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fafc800b9f0 tx=0x7fafc8008040 comp rx=0 tx=0).stop 2024-04-17T19:55:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.976+0000 7fafdba00640 1 -- 172.21.15.29:0/818090517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4159390 msgr2=0x7fafd415e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.976+0000 7fafdba00640 1 --2- 172.21.15.29:0/818090517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4159390 0x7fafd415e680 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fafc000c340 tx=0x7fafc000c800 comp rx=0 tx=0).stop 2024-04-17T19:55:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.976+0000 7fafdba00640 1 -- 172.21.15.29:0/818090517 shutdown_connections 2024-04-17T19:55:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.976+0000 7fafdba00640 1 --2- 172.21.15.29:0/818090517 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fafb4068380 0x7fafb406a890 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.976+0000 7fafdba00640 1 --2- 172.21.15.29:0/818090517 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fafd41597a0 0x7fafd4136e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.976+0000 7fafdba00640 1 --2- 172.21.15.29:0/818090517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4159390 0x7fafd415e680 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.976+0000 7fafdba00640 1 --2- 172.21.15.29:0/818090517 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fafd40cc8c0 0x7fafd41f5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.976+0000 7fafdba00640 1 -- 172.21.15.29:0/818090517 >> 172.21.15.29:0/818090517 conn(0x7fafd406fde0 msgr2=0x7fafd40cb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.976+0000 7fafdba00640 1 -- 172.21.15.29:0/818090517 shutdown_connections 2024-04-17T19:55:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:24.976+0000 7fafdba00640 1 -- 172.21.15.29:0/818090517 wait complete. 2024-04-17T19:55:24.991 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:55:26.991 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:27.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.160+0000 7f4f86c00640 1 Processor -- start 2024-04-17T19:55:27.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.160+0000 7f4f86c00640 1 -- start start 2024-04-17T19:55:27.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.164+0000 7f4f86c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f80159c90 0x7f4f800ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:27.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.164+0000 7f4f86c00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f4f800ca7f0 0x7f4f800c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:27.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.164+0000 7f4f86c00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f8015a0a0 con 0x7f4f800c74b0 2024-04-17T19:55:27.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.164+0000 7f4f86c00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f4f8015abe0 con 0x7f4f800ca7f0 2024-04-17T19:55:27.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.164+0000 7f4f86c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4f8015a340 con 0x7f4f80159c90 2024-04-17T19:55:27.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.164+0000 7f4f84c00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4f800c74b0 0x7f4f8006c660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54350/0 (socket says 172.21.15.29:54350) 2024-04-17T19:55:27.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.164+0000 7f4f84c00640 1 -- 172.21.15.29:0/3698537628 learned_addr learned my addr 172.21.15.29:0/3698537628 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:55:27.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.164+0000 7f4f77e00640 1 --2- 172.21.15.29:0/3698537628 >> v2:172.21.15.37:3300/0 conn(0x7f4f800ca7f0 0x7f4f800c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:27.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.164+0000 7f4f7fe00640 1 --2- 172.21.15.29:0/3698537628 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f80159c90 0x7f4f800ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:27.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.164+0000 7f4f7f400640 1 -- 172.21.15.29:0/3698537628 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2021053761 0 0) 0x7f4f8015a0a0 con 0x7f4f800c74b0 2024-04-17T19:55:27.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.164+0000 7f4f7f400640 1 -- 172.21.15.29:0/3698537628 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f64003530 con 0x7f4f800c74b0 2024-04-17T19:55:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.164+0000 7f4f7fe00640 1 -- 172.21.15.29:0/3698537628 >> v2:172.21.15.37:3300/0 conn(0x7f4f800ca7f0 msgr2=0x7f4f800c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.164+0000 7f4f7f400640 1 -- 172.21.15.29:0/3698537628 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3390257231 0 0) 0x7f4f64003530 con 0x7f4f800c74b0 2024-04-17T19:55:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.164+0000 7f4f7fe00640 1 --2- 172.21.15.29:0/3698537628 >> v2:172.21.15.37:3300/0 conn(0x7f4f800ca7f0 0x7f4f800c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.164+0000 7f4f7fe00640 1 -- 172.21.15.29:0/3698537628 >> v1:172.21.15.29:6789/0 conn(0x7f4f800c74b0 legacy=0x7f4f8006c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:27.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.164+0000 7f4f7fe00640 1 -- 172.21.15.29:0/3698537628 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8015a620 con 0x7f4f80159c90 2024-04-17T19:55:27.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.168+0000 7f4f7fe00640 1 --2- 172.21.15.29:0/3698537628 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f80159c90 0x7f4f800ca2b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4f700097c0 tx=0x7f4f7001c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6da9aab30566a979 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:27.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.168+0000 7f4f7f400640 1 -- 172.21.15.29:0/3698537628 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f700b0070 con 0x7f4f80159c90 2024-04-17T19:55:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.168+0000 7f4f7f400640 1 -- 172.21.15.29:0/3698537628 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f700ab400 con 0x7f4f80159c90 2024-04-17T19:55:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.168+0000 7f4f7f400640 1 -- 172.21.15.29:0/3698537628 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78001a20 con 0x7f4f80159c90 2024-04-17T19:55:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.168+0000 7f4f86c00640 1 -- 172.21.15.29:0/3698537628 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f80159c90 msgr2=0x7f4f800ca2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.168+0000 7f4f86c00640 1 --2- 172.21.15.29:0/3698537628 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f80159c90 0x7f4f800ca2b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4f700097c0 tx=0x7f4f7001c3c0 comp rx=0 tx=0).stop 2024-04-17T19:55:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.168+0000 7f4f86c00640 1 -- 172.21.15.29:0/3698537628 shutdown_connections 2024-04-17T19:55:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.168+0000 7f4f86c00640 1 --2- 172.21.15.29:0/3698537628 >> v2:172.21.15.37:3300/0 conn(0x7f4f800ca7f0 0x7f4f800c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:27.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.168+0000 7f4f86c00640 1 --2- 172.21.15.29:0/3698537628 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f80159c90 0x7f4f800ca2b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:27.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.168+0000 7f4f86c00640 1 -- 172.21.15.29:0/3698537628 >> 172.21.15.29:0/3698537628 conn(0x7f4f8006fde0 msgr2=0x7f4f800701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:27.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.168+0000 7f4f86c00640 1 -- 172.21.15.29:0/3698537628 shutdown_connections 2024-04-17T19:55:27.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.168+0000 7f4f86c00640 1 -- 172.21.15.29:0/3698537628 wait complete. 2024-04-17T19:55:27.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.168+0000 7f4f86c00640 1 Processor -- start 2024-04-17T19:55:27.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f86c00640 1 -- start start 2024-04-17T19:55:27.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f86c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f800c74b0 0x7f4f8014ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:27.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f86c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f800ca7f0 0x7f4f801f6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:27.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f86c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f80159c90 0x7f4f80147010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:27.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f86c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4f80166fb0 con 0x7f4f80159c90 2024-04-17T19:55:27.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f86c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4f80166e30 con 0x7f4f800ca7f0 2024-04-17T19:55:27.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f86c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4f80167130 con 0x7f4f800c74b0 2024-04-17T19:55:27.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f84c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f80159c90 0x7f4f80147010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:27.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f84c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f80159c90 0x7f4f80147010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36176/0 (socket says 172.21.15.29:36176) 2024-04-17T19:55:27.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f84c00640 1 -- 172.21.15.29:0/2450945334 learned_addr learned my addr 172.21.15.29:0/2450945334 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:27.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f7fe00640 1 --2- 172.21.15.29:0/2450945334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f800c74b0 0x7f4f8014ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:27.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f77e00640 1 --2- 172.21.15.29:0/2450945334 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f800ca7f0 0x7f4f801f6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:27.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f84c00640 1 -- 172.21.15.29:0/2450945334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f800c74b0 msgr2=0x7f4f8014ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:27.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f84c00640 1 --2- 172.21.15.29:0/2450945334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f800c74b0 0x7f4f8014ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:27.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f84c00640 1 -- 172.21.15.29:0/2450945334 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f800ca7f0 msgr2=0x7f4f801f6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:27.168 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f84c00640 1 --2- 172.21.15.29:0/2450945334 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f800ca7f0 0x7f4f801f6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:27.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f84c00640 1 -- 172.21.15.29:0/2450945334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f80150da0 con 0x7f4f80159c90 2024-04-17T19:55:27.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f84c00640 1 --2- 172.21.15.29:0/2450945334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f80159c90 0x7f4f80147010 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800c870 tx=0x7f4f7800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f7cc00640 1 -- 172.21.15.29:0/2450945334 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78018070 con 0x7f4f80159c90 2024-04-17T19:55:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f7cc00640 1 -- 172.21.15.29:0/2450945334 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f7800a5e0 con 0x7f4f80159c90 2024-04-17T19:55:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f7cc00640 1 -- 172.21.15.29:0/2450945334 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7800a9f0 con 0x7f4f80159c90 2024-04-17T19:55:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f86c00640 1 -- 172.21.15.29:0/2450945334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f801f6760 con 0x7f4f80159c90 2024-04-17T19:55:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f86c00640 1 -- 172.21.15.29:0/2450945334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f8014e7f0 con 0x7f4f80159c90 2024-04-17T19:55:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.172+0000 7f4f86c00640 1 -- 172.21.15.29:0/2450945334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f800c4210 con 0x7f4f80159c90 2024-04-17T19:55:27.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.176+0000 7f4f7cc00640 1 -- 172.21.15.29:0/2450945334 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4f7800ab90 con 0x7f4f80159c90 2024-04-17T19:55:27.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.176+0000 7f4f7cc00640 1 --2- 172.21.15.29:0/2450945334 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4f5c068380 0x7f4f5c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:27.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.176+0000 7f4f7cc00640 1 -- 172.21.15.29:0/2450945334 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 7223+0+0 (secure 0 0 0) 0x7f4f78086580 con 0x7f4f80159c90 2024-04-17T19:55:27.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.176+0000 7f4f7fe00640 1 --2- 172.21.15.29:0/2450945334 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4f5c068380 0x7f4f5c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:27.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.176+0000 7f4f7fe00640 1 --2- 172.21.15.29:0/2450945334 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4f5c068380 0x7f4f5c06a890 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4f700ad000 tx=0x7f4f700a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:27.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.180+0000 7f4f7cc00640 1 -- 172.21.15.29:0/2450945334 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4f78055e50 con 0x7f4f80159c90 2024-04-17T19:55:27.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.380+0000 7f4f86c00640 1 -- 172.21.15.29:0/2450945334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f80069b90 con 0x7f4f80159c90 2024-04-17T19:55:27.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.380+0000 7f4f7cc00640 1 -- 172.21.15.29:0/2450945334 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6076 (secure 0 0 0) 0x7f4f7805a150 con 0x7f4f80159c90 2024-04-17T19:55:27.377 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:27.378 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:23.902424+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:resolve","state_seq":26,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,4305,5212,4302]},"id":1}]} 2024-04-17T19:55:27.378 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 39 2024-04-17T19:55:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.384+0000 7f4f86c00640 1 -- 172.21.15.29:0/2450945334 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4f5c068380 msgr2=0x7f4f5c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.384+0000 7f4f86c00640 1 --2- 172.21.15.29:0/2450945334 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4f5c068380 0x7f4f5c06a890 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4f700ad000 tx=0x7f4f700a6000 comp rx=0 tx=0).stop 2024-04-17T19:55:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.384+0000 7f4f86c00640 1 -- 172.21.15.29:0/2450945334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f80159c90 msgr2=0x7f4f80147010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.384+0000 7f4f86c00640 1 --2- 172.21.15.29:0/2450945334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f80159c90 0x7f4f80147010 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800c870 tx=0x7f4f7800cd30 comp rx=0 tx=0).stop 2024-04-17T19:55:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.384+0000 7f4f86c00640 1 -- 172.21.15.29:0/2450945334 shutdown_connections 2024-04-17T19:55:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.384+0000 7f4f86c00640 1 --2- 172.21.15.29:0/2450945334 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4f5c068380 0x7f4f5c06a890 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.384+0000 7f4f86c00640 1 --2- 172.21.15.29:0/2450945334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f80159c90 0x7f4f80147010 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.384+0000 7f4f86c00640 1 --2- 172.21.15.29:0/2450945334 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f800ca7f0 0x7f4f801f6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.384+0000 7f4f86c00640 1 --2- 172.21.15.29:0/2450945334 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f800c74b0 0x7f4f8014ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.384+0000 7f4f86c00640 1 -- 172.21.15.29:0/2450945334 >> 172.21.15.29:0/2450945334 conn(0x7f4f8006fde0 msgr2=0x7f4f800701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.384+0000 7f4f86c00640 1 -- 172.21.15.29:0/2450945334 shutdown_connections 2024-04-17T19:55:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:27.384+0000 7f4f86c00640 1 -- 172.21.15.29:0/2450945334 wait complete. 2024-04-17T19:55:27.399 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:55:29.401 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:29.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.572+0000 7f0df7e00640 1 Processor -- start 2024-04-17T19:55:29.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.572+0000 7f0df7e00640 1 -- start start 2024-04-17T19:55:29.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0df7e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f0df0159c90 0x7f0df00ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:29.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0df7e00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0df015a0a0 con 0x7f0df006c660 2024-04-17T19:55:29.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0df7e00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0df015abe0 con 0x7f0df00ca7f0 2024-04-17T19:55:29.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0df7e00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f0df015a340 con 0x7f0df0159c90 2024-04-17T19:55:29.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0df5e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0df006c660 0x7f0df006ca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56492/0 (socket says 172.21.15.29:56492) 2024-04-17T19:55:29.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0df5e00640 1 -- 172.21.15.29:0/384231495 learned_addr learned my addr 172.21.15.29:0/384231495 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:55:29.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0df5400640 1 --2- 172.21.15.29:0/384231495 >> v2:172.21.15.37:3300/0 conn(0x7f0df0159c90 0x7f0df00ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0defe00640 1 -- 172.21.15.29:0/384231495 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2737479706 0 0) 0x7f0df015a0a0 con 0x7f0df006c660 2024-04-17T19:55:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0defe00640 1 -- 172.21.15.29:0/384231495 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dd8003530 con 0x7f0df006c660 2024-04-17T19:55:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0defe00640 1 -- 172.21.15.29:0/384231495 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2458485657 0 0) 0x7f0dd8003530 con 0x7f0df006c660 2024-04-17T19:55:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0defe00640 1 -- 172.21.15.29:0/384231495 >> v2:172.21.15.37:3300/0 conn(0x7f0df0159c90 msgr2=0x7f0df00ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0defe00640 1 --2- 172.21.15.29:0/384231495 >> v2:172.21.15.37:3300/0 conn(0x7f0df0159c90 0x7f0df00ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0defe00640 1 -- 172.21.15.29:0/384231495 >> v1:172.21.15.37:6789/0 conn(0x7f0df00ca7f0 legacy=0x7f0df00c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0defe00640 1 -- 172.21.15.29:0/384231495 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0df015a620 con 0x7f0df006c660 2024-04-17T19:55:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0defe00640 1 -- 172.21.15.29:0/384231495 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f0de8001a20 con 0x7f0df006c660 2024-04-17T19:55:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0defe00640 1 -- 172.21.15.29:0/384231495 >> v1:172.21.15.29:6789/0 conn(0x7f0df006c660 legacy=0x7f0df006ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0defe00640 1 --2- 172.21.15.29:0/384231495 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0dd8003a30 0x7f0dd8003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0defe00640 1 --2- 172.21.15.29:0/384231495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dd8004810 0x7f0dd8004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0defe00640 1 --2- 172.21.15.29:0/384231495 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0dd8005230 0x7f0dd800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0defe00640 1 -- 172.21.15.29:0/384231495 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0df015a340 con 0x7f0dd8004810 2024-04-17T19:55:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0defe00640 1 -- 172.21.15.29:0/384231495 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd800c1c0 con 0x7f0dd8003a30 2024-04-17T19:55:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0defe00640 1 -- 172.21.15.29:0/384231495 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0dd800c3c0 con 0x7f0dd8005230 2024-04-17T19:55:29.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0df4a00640 1 --2- 172.21.15.29:0/384231495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dd8004810 0x7f0dd8004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:29.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0df5400640 1 --2- 172.21.15.29:0/384231495 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0dd8003a30 0x7f0dd8003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:29.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0df5e00640 1 --2- 172.21.15.29:0/384231495 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0dd8005230 0x7f0dd800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:29.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0df4a00640 1 -- 172.21.15.29:0/384231495 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0dd8005230 msgr2=0x7f0dd800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:29.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0df4a00640 1 --2- 172.21.15.29:0/384231495 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0dd8005230 0x7f0dd800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:29.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0df4a00640 1 -- 172.21.15.29:0/384231495 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0dd8003a30 msgr2=0x7f0dd8003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:29.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0df4a00640 1 --2- 172.21.15.29:0/384231495 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0dd8003a30 0x7f0dd8003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:29.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0df4a00640 1 -- 172.21.15.29:0/384231495 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dd8003530 con 0x7f0dd8004810 2024-04-17T19:55:29.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.576+0000 7f0df5e00640 1 -- 172.21.15.29:0/384231495 reap_dead start 2024-04-17T19:55:29.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.580+0000 7f0df4a00640 1 --2- 172.21.15.29:0/384231495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dd8004810 0x7f0dd8004c20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc001750 tx=0x7f0ddc01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a2dd8f0e76fed0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:29.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.580+0000 7f0defe00640 1 -- 172.21.15.29:0/384231495 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc0013a0 con 0x7f0dd8004810 2024-04-17T19:55:29.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.580+0000 7f0defe00640 1 -- 172.21.15.29:0/384231495 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ddc0052f0 con 0x7f0dd8004810 2024-04-17T19:55:29.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.580+0000 7f0defe00640 1 -- 172.21.15.29:0/384231495 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc003840 con 0x7f0dd8004810 2024-04-17T19:55:29.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.580+0000 7f0df7e00640 1 -- 172.21.15.29:0/384231495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dd8004810 msgr2=0x7f0dd8004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:29.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.580+0000 7f0df7e00640 1 --2- 172.21.15.29:0/384231495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dd8004810 0x7f0dd8004c20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc001750 tx=0x7f0ddc01c3c0 comp rx=0 tx=0).stop 2024-04-17T19:55:29.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.580+0000 7f0df7e00640 1 -- 172.21.15.29:0/384231495 shutdown_connections 2024-04-17T19:55:29.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.580+0000 7f0df7e00640 1 --2- 172.21.15.29:0/384231495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0dd8004810 0x7f0dd8004c20 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:29.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.580+0000 7f0df7e00640 1 -- 172.21.15.29:0/384231495 >> 172.21.15.29:0/384231495 conn(0x7f0df006fde0 msgr2=0x7f0df00701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:29.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.580+0000 7f0df7e00640 1 -- 172.21.15.29:0/384231495 shutdown_connections 2024-04-17T19:55:29.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.580+0000 7f0df7e00640 1 -- 172.21.15.29:0/384231495 wait complete. 2024-04-17T19:55:29.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df7e00640 1 Processor -- start 2024-04-17T19:55:29.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df7e00640 1 -- start start 2024-04-17T19:55:29.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df7e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0dd8004810 0x7f0df014a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:29.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0df01e5330 0x7f0df00c7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:29.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df7e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0df0149dd0 0x7f0df015d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:29.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df7e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f0df006c140 con 0x7f0df01e5330 2024-04-17T19:55:29.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df7e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f0df006be60 con 0x7f0dd8004810 2024-04-17T19:55:29.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df7e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f0df00ca2b0 con 0x7f0df0149dd0 2024-04-17T19:55:29.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df4a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0df01e5330 0x7f0df00c7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:29.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df4a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0df01e5330 0x7f0df00c7130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35742/0 (socket says 172.21.15.29:35742) 2024-04-17T19:55:29.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df4a00640 1 -- 172.21.15.29:0/3122573347 learned_addr learned my addr 172.21.15.29:0/3122573347 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:29.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df5400640 1 --2- 172.21.15.29:0/3122573347 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0dd8004810 0x7f0df014a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:29.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df5e00640 1 --2- 172.21.15.29:0/3122573347 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0df0149dd0 0x7f0df015d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:29.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df4a00640 1 -- 172.21.15.29:0/3122573347 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0df0149dd0 msgr2=0x7f0df015d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:29.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df4a00640 1 --2- 172.21.15.29:0/3122573347 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0df0149dd0 0x7f0df015d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:29.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df4a00640 1 -- 172.21.15.29:0/3122573347 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0dd8004810 msgr2=0x7f0df014a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:29.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df4a00640 1 --2- 172.21.15.29:0/3122573347 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0dd8004810 0x7f0df014a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:29.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df4a00640 1 -- 172.21.15.29:0/3122573347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0df01f0b60 con 0x7f0df01e5330 2024-04-17T19:55:29.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df4a00640 1 --2- 172.21.15.29:0/3122573347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0df01e5330 0x7f0df00c7130 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc01c1f0 tx=0x7f0dd8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:29.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0ded600640 1 -- 172.21.15.29:0/3122573347 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc0013a0 con 0x7f0df01e5330 2024-04-17T19:55:29.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0ded600640 1 -- 172.21.15.29:0/3122573347 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ddc0ba070 con 0x7f0df01e5330 2024-04-17T19:55:29.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df7e00640 1 -- 172.21.15.29:0/3122573347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0df01f0e40 con 0x7f0df01e5330 2024-04-17T19:55:29.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0ded600640 1 -- 172.21.15.29:0/3122573347 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ddc007a40 con 0x7f0df01e5330 2024-04-17T19:55:29.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.584+0000 7f0df7e00640 1 -- 172.21.15.29:0/3122573347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0df014ac20 con 0x7f0df01e5330 2024-04-17T19:55:29.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.588+0000 7f0df7e00640 1 -- 172.21.15.29:0/3122573347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0df0069b90 con 0x7f0df01e5330 2024-04-17T19:55:29.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.588+0000 7f0ded600640 1 -- 172.21.15.29:0/3122573347 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f0ddc0052f0 con 0x7f0df01e5330 2024-04-17T19:55:29.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.588+0000 7f0ded600640 1 --2- 172.21.15.29:0/3122573347 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0dd0068380 0x7f0dd006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:29.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.588+0000 7f0ded600640 1 -- 172.21.15.29:0/3122573347 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 7135+0+0 (secure 0 0 0) 0x7f0ddc121370 con 0x7f0df01e5330 2024-04-17T19:55:29.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.588+0000 7f0df5400640 1 --2- 172.21.15.29:0/3122573347 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0dd0068380 0x7f0dd006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:29.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.592+0000 7f0df5400640 1 --2- 172.21.15.29:0/3122573347 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0dd0068380 0x7f0dd006a890 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0df00c8e90 tx=0x7f0de4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:29.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.592+0000 7f0ded600640 1 -- 172.21.15.29:0/3122573347 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0ddc0160d0 con 0x7f0df01e5330 2024-04-17T19:55:29.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.792+0000 7f0df7e00640 1 -- 172.21.15.29:0/3122573347 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0df0159280 con 0x7f0df01e5330 2024-04-17T19:55:29.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.792+0000 7f0ded600640 1 -- 172.21.15.29:0/3122573347 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+6076 (secure 0 0 0) 0x7f0ddc0f1300 con 0x7f0df01e5330 2024-04-17T19:55:29.789 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:29.789 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:23.902424+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:resolve","state_seq":26,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,4305,5212,4302]},"id":1}]} 2024-04-17T19:55:29.789 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 39 2024-04-17T19:55:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.796+0000 7f0df7e00640 1 -- 172.21.15.29:0/3122573347 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0dd0068380 msgr2=0x7f0dd006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.796+0000 7f0df7e00640 1 --2- 172.21.15.29:0/3122573347 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0dd0068380 0x7f0dd006a890 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0df00c8e90 tx=0x7f0de4009190 comp rx=0 tx=0).stop 2024-04-17T19:55:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.796+0000 7f0df7e00640 1 -- 172.21.15.29:0/3122573347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0df01e5330 msgr2=0x7f0df00c7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.796+0000 7f0df7e00640 1 --2- 172.21.15.29:0/3122573347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0df01e5330 0x7f0df00c7130 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0ddc01c1f0 tx=0x7f0dd8005840 comp rx=0 tx=0).stop 2024-04-17T19:55:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.796+0000 7f0df7e00640 1 -- 172.21.15.29:0/3122573347 shutdown_connections 2024-04-17T19:55:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.796+0000 7f0df7e00640 1 --2- 172.21.15.29:0/3122573347 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f0dd0068380 0x7f0dd006a890 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.796+0000 7f0df7e00640 1 --2- 172.21.15.29:0/3122573347 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f0df0149dd0 0x7f0df015d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.796+0000 7f0df7e00640 1 --2- 172.21.15.29:0/3122573347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0df01e5330 0x7f0df00c7130 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.796+0000 7f0df7e00640 1 --2- 172.21.15.29:0/3122573347 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f0dd8004810 0x7f0df014a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.796+0000 7f0df7e00640 1 -- 172.21.15.29:0/3122573347 >> 172.21.15.29:0/3122573347 conn(0x7f0df006fde0 msgr2=0x7f0df014d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.796+0000 7f0df7e00640 1 -- 172.21.15.29:0/3122573347 shutdown_connections 2024-04-17T19:55:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:29.796+0000 7f0df7e00640 1 -- 172.21.15.29:0/3122573347 wait complete. 2024-04-17T19:55:29.811 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:55:31.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:31.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.976+0000 7f1d97600640 1 Processor -- start 2024-04-17T19:55:31.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.976+0000 7f1d97600640 1 -- start start 2024-04-17T19:55:31.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d97600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d901625e0 0x7f1d90164a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:31.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d97600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d900cd0e0 con 0x7f1d900cc8c0 2024-04-17T19:55:31.974 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d97600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d900c88b0 con 0x7f1d900cccd0 2024-04-17T19:55:31.975 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d97600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1d900c6ce0 con 0x7f1d901625e0 2024-04-17T19:55:31.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d94c00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1d900cc8c0 0x7f1d90159390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56508/0 (socket says 172.21.15.29:56508) 2024-04-17T19:55:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d94c00640 1 -- 172.21.15.29:0/2838105110 learned_addr learned my addr 172.21.15.29:0/2838105110 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:55:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d95600640 1 --2- 172.21.15.29:0/2838105110 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d901625e0 0x7f1d90164a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d8fe00640 1 -- 172.21.15.29:0/2838105110 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 261573382 0 0) 0x7f1d900cd0e0 con 0x7f1d900cc8c0 2024-04-17T19:55:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d8fe00640 1 -- 172.21.15.29:0/2838105110 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d74003530 con 0x7f1d900cc8c0 2024-04-17T19:55:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d8fe00640 1 -- 172.21.15.29:0/2838105110 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3046403690 0 0) 0x7f1d74003530 con 0x7f1d900cc8c0 2024-04-17T19:55:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d8fe00640 1 -- 172.21.15.29:0/2838105110 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d901625e0 msgr2=0x7f1d90164a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d8fe00640 1 --2- 172.21.15.29:0/2838105110 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d901625e0 0x7f1d90164a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d8fe00640 1 -- 172.21.15.29:0/2838105110 >> v1:172.21.15.37:6789/0 conn(0x7f1d900cccd0 legacy=0x7f1d900ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d8fe00640 1 -- 172.21.15.29:0/2838105110 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d900c6fc0 con 0x7f1d900cc8c0 2024-04-17T19:55:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d8fe00640 1 -- 172.21.15.29:0/2838105110 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f1d80001680 con 0x7f1d900cc8c0 2024-04-17T19:55:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d8fe00640 1 -- 172.21.15.29:0/2838105110 >> v1:172.21.15.29:6789/0 conn(0x7f1d900cc8c0 legacy=0x7f1d90159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d8fe00640 1 --2- 172.21.15.29:0/2838105110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d74003a60 0x7f1d74003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d8fe00640 1 --2- 172.21.15.29:0/2838105110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d74004810 0x7f1d74004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d8fe00640 1 --2- 172.21.15.29:0/2838105110 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d901625e0 0x7f1d74005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d8fe00640 1 -- 172.21.15.29:0/2838105110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1d900c6ce0 con 0x7f1d74004810 2024-04-17T19:55:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d8fe00640 1 -- 172.21.15.29:0/2838105110 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7400bae0 con 0x7f1d74003a60 2024-04-17T19:55:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d8fe00640 1 -- 172.21.15.29:0/2838105110 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1d7400bce0 con 0x7f1d901625e0 2024-04-17T19:55:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d87e00640 1 --2- 172.21.15.29:0/2838105110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d74004810 0x7f1d74004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d94c00640 1 --2- 172.21.15.29:0/2838105110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d74003a60 0x7f1d74003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d95600640 1 --2- 172.21.15.29:0/2838105110 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d901625e0 0x7f1d74005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d87e00640 1 -- 172.21.15.29:0/2838105110 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d901625e0 msgr2=0x7f1d74005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d87e00640 1 --2- 172.21.15.29:0/2838105110 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d901625e0 0x7f1d74005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d87e00640 1 -- 172.21.15.29:0/2838105110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d74003a60 msgr2=0x7f1d74003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d87e00640 1 --2- 172.21.15.29:0/2838105110 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d74003a60 0x7f1d74003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.980+0000 7f1d87e00640 1 -- 172.21.15.29:0/2838105110 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d74003530 con 0x7f1d74004810 2024-04-17T19:55:31.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.984+0000 7f1d87e00640 1 --2- 172.21.15.29:0/2838105110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d74004810 0x7f1d74004c20 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f1d9006d610 tx=0x7f1d7801ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeeb1de4e04593cf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:31.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.984+0000 7f1d8fe00640 1 -- 172.21.15.29:0/2838105110 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d780b5070 con 0x7f1d74004810 2024-04-17T19:55:31.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.984+0000 7f1d8fe00640 1 -- 172.21.15.29:0/2838105110 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d780a8850 con 0x7f1d74004810 2024-04-17T19:55:31.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.984+0000 7f1d8fe00640 1 -- 172.21.15.29:0/2838105110 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d78001460 con 0x7f1d74004810 2024-04-17T19:55:31.980 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.984+0000 7f1d97600640 1 -- 172.21.15.29:0/2838105110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d74004810 msgr2=0x7f1d74004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:31.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.984+0000 7f1d97600640 1 --2- 172.21.15.29:0/2838105110 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d74004810 0x7f1d74004c20 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f1d9006d610 tx=0x7f1d7801ec30 comp rx=0 tx=0).stop 2024-04-17T19:55:31.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.984+0000 7f1d95600640 1 -- 172.21.15.29:0/2838105110 reap_dead start 2024-04-17T19:55:31.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.984+0000 7f1d97600640 1 -- 172.21.15.29:0/2838105110 shutdown_connections 2024-04-17T19:55:31.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.984+0000 7f1d97600640 1 -- 172.21.15.29:0/2838105110 >> 172.21.15.29:0/2838105110 conn(0x7f1d9006fde0 msgr2=0x7f1d900701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:31.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.984+0000 7f1d97600640 1 -- 172.21.15.29:0/2838105110 shutdown_connections 2024-04-17T19:55:31.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.984+0000 7f1d97600640 1 -- 172.21.15.29:0/2838105110 wait complete. 2024-04-17T19:55:31.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d97600640 1 Processor -- start 2024-04-17T19:55:31.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d97600640 1 -- start start 2024-04-17T19:55:31.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d97600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d74004810 0x7f1d9015dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:31.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d97600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d90158770 0x7f1d9015c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:31.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d97600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d90157c00 0x7f1d9013d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:31.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d97600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1d90158e70 con 0x7f1d90157c00 2024-04-17T19:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d97600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1d900c7580 con 0x7f1d74004810 2024-04-17T19:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d97600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1d90159aa0 con 0x7f1d90158770 2024-04-17T19:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d95600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d90157c00 0x7f1d9013d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d95600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d90157c00 0x7f1d9013d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35758/0 (socket says 172.21.15.29:35758) 2024-04-17T19:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d95600640 1 -- 172.21.15.29:0/3813250129 learned_addr learned my addr 172.21.15.29:0/3813250129 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d94c00640 1 --2- 172.21.15.29:0/3813250129 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d74004810 0x7f1d9015dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d87e00640 1 --2- 172.21.15.29:0/3813250129 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d90158770 0x7f1d9015c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d95600640 1 -- 172.21.15.29:0/3813250129 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d90158770 msgr2=0x7f1d9015c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d95600640 1 --2- 172.21.15.29:0/3813250129 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d90158770 0x7f1d9015c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d95600640 1 -- 172.21.15.29:0/3813250129 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d74004810 msgr2=0x7f1d9015dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d95600640 1 --2- 172.21.15.29:0/3813250129 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d74004810 0x7f1d9015dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d95600640 1 -- 172.21.15.29:0/3813250129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d901f0c80 con 0x7f1d90157c00 2024-04-17T19:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d95600640 1 --2- 172.21.15.29:0/3813250129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d90157c00 0x7f1d9013d630 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f1d900722f0 tx=0x7f1d8800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d8d600640 1 -- 172.21.15.29:0/3813250129 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d880039a0 con 0x7f1d90157c00 2024-04-17T19:55:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d8d600640 1 -- 172.21.15.29:0/3813250129 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d8801c070 con 0x7f1d90157c00 2024-04-17T19:55:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d97600640 1 -- 172.21.15.29:0/3813250129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d9015c7b0 con 0x7f1d90157c00 2024-04-17T19:55:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d8d600640 1 -- 172.21.15.29:0/3813250129 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d88003d60 con 0x7f1d90157c00 2024-04-17T19:55:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.988+0000 7f1d97600640 1 -- 172.21.15.29:0/3813250129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d901f03e0 con 0x7f1d90157c00 2024-04-17T19:55:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.992+0000 7f1d97600640 1 -- 172.21.15.29:0/3813250129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d90069b90 con 0x7f1d90157c00 2024-04-17T19:55:31.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.992+0000 7f1d8d600640 1 -- 172.21.15.29:0/3813250129 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1d88004030 con 0x7f1d90157c00 2024-04-17T19:55:31.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.992+0000 7f1d8d600640 1 --2- 172.21.15.29:0/3813250129 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1d6c068380 0x7f1d6c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:31.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.992+0000 7f1d8d600640 1 -- 172.21.15.29:0/3813250129 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 7135+0+0 (secure 0 0 0) 0x7f1d880879e0 con 0x7f1d90157c00 2024-04-17T19:55:31.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.992+0000 7f1d94c00640 1 --2- 172.21.15.29:0/3813250129 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1d6c068380 0x7f1d6c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:31.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.992+0000 7f1d94c00640 1 --2- 172.21.15.29:0/3813250129 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1d6c068380 0x7f1d6c06a890 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1d80000c10 tx=0x7f1d80009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:31.996+0000 7f1d8d600640 1 -- 172.21.15.29:0/3813250129 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1d88010040 con 0x7f1d90157c00 2024-04-17T19:55:32.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:32.196+0000 7f1d97600640 1 -- 172.21.15.29:0/3813250129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d901efe70 con 0x7f1d90157c00 2024-04-17T19:55:32.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:32.196+0000 7f1d8d600640 1 -- 172.21.15.29:0/3813250129 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6078 (secure 0 0 0) 0x7f1d88057970 con 0x7f1d90157c00 2024-04-17T19:55:32.192 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:32.192 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:29.896742+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:reconnect","state_seq":28,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4302,5212,4305,5350]},"id":1}]} 2024-04-17T19:55:32.192 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 40 2024-04-17T19:55:32.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:32.196+0000 7f1d97600640 1 -- 172.21.15.29:0/3813250129 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1d6c068380 msgr2=0x7f1d6c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:32.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:32.196+0000 7f1d97600640 1 --2- 172.21.15.29:0/3813250129 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1d6c068380 0x7f1d6c06a890 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1d80000c10 tx=0x7f1d80009040 comp rx=0 tx=0).stop 2024-04-17T19:55:32.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:32.196+0000 7f1d97600640 1 -- 172.21.15.29:0/3813250129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d90157c00 msgr2=0x7f1d9013d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:32.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:32.196+0000 7f1d97600640 1 --2- 172.21.15.29:0/3813250129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d90157c00 0x7f1d9013d630 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f1d900722f0 tx=0x7f1d8800f580 comp rx=0 tx=0).stop 2024-04-17T19:55:32.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:32.196+0000 7f1d97600640 1 -- 172.21.15.29:0/3813250129 shutdown_connections 2024-04-17T19:55:32.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:32.196+0000 7f1d97600640 1 --2- 172.21.15.29:0/3813250129 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1d6c068380 0x7f1d6c06a890 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:32.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:32.196+0000 7f1d97600640 1 --2- 172.21.15.29:0/3813250129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d90157c00 0x7f1d9013d630 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:32.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:32.196+0000 7f1d97600640 1 --2- 172.21.15.29:0/3813250129 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1d90158770 0x7f1d9015c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:32.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:32.196+0000 7f1d97600640 1 --2- 172.21.15.29:0/3813250129 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1d74004810 0x7f1d9015dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:32.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:32.196+0000 7f1d97600640 1 -- 172.21.15.29:0/3813250129 >> 172.21.15.29:0/3813250129 conn(0x7f1d9006fde0 msgr2=0x7f1d9006bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:32.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:32.196+0000 7f1d97600640 1 -- 172.21.15.29:0/3813250129 shutdown_connections 2024-04-17T19:55:32.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:32.196+0000 7f1d97600640 1 -- 172.21.15.29:0/3813250129 wait complete. 2024-04-17T19:55:32.215 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 40, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-17T19:47:49.315515+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 40, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4302": { "addr": "172.21.15.29:6835/270402476", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6834", "nonce": 270402476, "type": "v2" }, { "addr": "172.21.15.29:6835", "nonce": 270402476, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4302, "incarnation": 10, "join_fscid": -1, "name": "e", "rank": 1, "state": "up:active", "state_seq": 9 }, "gid_4305": { "addr": "172.21.15.29:6837/2691590158", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6833", "nonce": 2691590158, "type": "v2" }, { "addr": "172.21.15.29:6837", "nonce": 2691590158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4305, "incarnation": 5, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5144": { "addr": "172.21.15.37:6839/858161860", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6836", "nonce": 858161860, "type": "v2" }, { "addr": "172.21.15.37:6839", "nonce": 858161860, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5144, "incarnation": 38, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:reconnect", "state_seq": 28 }, "gid_5212": { "addr": "172.21.15.37:6838/767681664", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6835", "nonce": 767681664, "type": "v2" }, { "addr": "172.21.15.37:6838", "nonce": 767681664, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5212, "incarnation": 26, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 31 }, "gid_5350": { "addr": "172.21.15.37:6837/2495953425", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6834", "nonce": 2495953425, "type": "v2" }, { "addr": "172.21.15.37:6837", "nonce": 2495953425, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5350, "incarnation": 32, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-17T19:55:29.896742+0000", "qdb_cluster": [ 4302, 5212, 4305, 5350 ], "qdb_leader": 4305, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4305, "mds_1": 4302, "mds_2": 5144, "mds_3": 5350, "mds_4": 5212 } } } ], "standbys": [] } 2024-04-17T19:55:34.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:34.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.388+0000 7fa237a00640 1 Processor -- start 2024-04-17T19:55:34.383 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.388+0000 7fa237a00640 1 -- start start 2024-04-17T19:55:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.388+0000 7fa237a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa2300cc210 0x7fa2300cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.388+0000 7fa237a00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa23015f7e0 0x7fa23015fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.388+0000 7fa237a00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa23006e7b0 con 0x7fa23006e3a0 2024-04-17T19:55:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.388+0000 7fa237a00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fa2300cb610 con 0x7fa2300cc210 2024-04-17T19:55:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.388+0000 7fa237a00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa2300cb8d0 con 0x7fa23015f7e0 2024-04-17T19:55:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.388+0000 7fa22fe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa2300cc210 0x7fa2300cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.388+0000 7fa235000640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa23006e3a0 0x7fa23006ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56512/0 (socket says 172.21.15.29:56512) 2024-04-17T19:55:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.388+0000 7fa235000640 1 -- 172.21.15.29:0/829501834 learned_addr learned my addr 172.21.15.29:0/829501834 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:55:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.388+0000 7fa235a00640 1 --2- 172.21.15.29:0/829501834 >> v2:172.21.15.37:3300/0 conn(0x7fa23015f7e0 0x7fa23015fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.392+0000 7fa22fe00640 1 -- 172.21.15.29:0/829501834 >> v2:172.21.15.37:3300/0 conn(0x7fa23015f7e0 msgr2=0x7fa23015fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.392+0000 7fa22fe00640 1 --2- 172.21.15.29:0/829501834 >> v2:172.21.15.37:3300/0 conn(0x7fa23015f7e0 0x7fa23015fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.392+0000 7fa22fe00640 1 -- 172.21.15.29:0/829501834 >> v1:172.21.15.29:6789/0 conn(0x7fa23006e3a0 legacy=0x7fa23006ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:34.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.392+0000 7fa22f400640 1 -- 172.21.15.29:0/829501834 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2008109962 0 0) 0x7fa23006e7b0 con 0x7fa23006e3a0 2024-04-17T19:55:34.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.392+0000 7fa22fe00640 1 -- 172.21.15.29:0/829501834 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2300cbbb0 con 0x7fa2300cc210 2024-04-17T19:55:34.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.392+0000 7fa22fe00640 1 --2- 172.21.15.29:0/829501834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2300cc210 0x7fa2300cc620 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa230073e40 tx=0x7fa22001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e05a97db35ee948 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:34.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.392+0000 7fa22f400640 1 -- 172.21.15.29:0/829501834 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2200b5070 con 0x7fa2300cc210 2024-04-17T19:55:34.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.392+0000 7fa22f400640 1 -- 172.21.15.29:0/829501834 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2200a8970 con 0x7fa2300cc210 2024-04-17T19:55:34.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.392+0000 7fa22f400640 1 -- 172.21.15.29:0/829501834 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2200a8d40 con 0x7fa2300cc210 2024-04-17T19:55:34.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.392+0000 7fa237a00640 1 -- 172.21.15.29:0/829501834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2300cc210 msgr2=0x7fa2300cc620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:34.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.392+0000 7fa237a00640 1 --2- 172.21.15.29:0/829501834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2300cc210 0x7fa2300cc620 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fa230073e40 tx=0x7fa22001ec70 comp rx=0 tx=0).stop 2024-04-17T19:55:34.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.396+0000 7fa237a00640 1 -- 172.21.15.29:0/829501834 shutdown_connections 2024-04-17T19:55:34.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.396+0000 7fa237a00640 1 --2- 172.21.15.29:0/829501834 >> v2:172.21.15.37:3300/0 conn(0x7fa23015f7e0 0x7fa23015fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:34.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.396+0000 7fa237a00640 1 --2- 172.21.15.29:0/829501834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2300cc210 0x7fa2300cc620 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:34.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.396+0000 7fa237a00640 1 -- 172.21.15.29:0/829501834 >> 172.21.15.29:0/829501834 conn(0x7fa230071ca0 msgr2=0x7fa2300720e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:34.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.396+0000 7fa237a00640 1 -- 172.21.15.29:0/829501834 shutdown_connections 2024-04-17T19:55:34.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.396+0000 7fa237a00640 1 -- 172.21.15.29:0/829501834 wait complete. 2024-04-17T19:55:34.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.396+0000 7fa237a00640 1 Processor -- start 2024-04-17T19:55:34.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.396+0000 7fa237a00640 1 -- start start 2024-04-17T19:55:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.396+0000 7fa237a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa23006e3a0 0x7fa23015da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.396+0000 7fa237a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2300cc210 0x7fa230155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa237a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa23015f7e0 0x7fa23014bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa237a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa2300ca9b0 con 0x7fa23006e3a0 2024-04-17T19:55:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa237a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa2300ca6d0 con 0x7fa2300cc210 2024-04-17T19:55:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa237a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa230160130 con 0x7fa23015f7e0 2024-04-17T19:55:34.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa235000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa23006e3a0 0x7fa23015da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa235000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa23006e3a0 0x7fa23015da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35774/0 (socket says 172.21.15.29:35774) 2024-04-17T19:55:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa235000640 1 -- 172.21.15.29:0/63845392 learned_addr learned my addr 172.21.15.29:0/63845392 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa22fe00640 1 --2- 172.21.15.29:0/63845392 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2300cc210 0x7fa230155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa235a00640 1 --2- 172.21.15.29:0/63845392 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa23015f7e0 0x7fa23014bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa235000640 1 -- 172.21.15.29:0/63845392 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa23015f7e0 msgr2=0x7fa23014bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa235000640 1 --2- 172.21.15.29:0/63845392 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa23015f7e0 0x7fa23014bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa235000640 1 -- 172.21.15.29:0/63845392 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2300cc210 msgr2=0x7fa230155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa235000640 1 --2- 172.21.15.29:0/63845392 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2300cc210 0x7fa230155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa235000640 1 -- 172.21.15.29:0/63845392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa23015dfb0 con 0x7fa23006e3a0 2024-04-17T19:55:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa22fe00640 1 --2- 172.21.15.29:0/63845392 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2300cc210 0x7fa230155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:55:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa235000640 1 --2- 172.21.15.29:0/63845392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa23006e3a0 0x7fa23015da70 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa224005890 tx=0x7fa22400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa22cc00640 1 -- 172.21.15.29:0/63845392 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa224017070 con 0x7fa23006e3a0 2024-04-17T19:55:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa22cc00640 1 -- 172.21.15.29:0/63845392 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa224005d10 con 0x7fa23006e3a0 2024-04-17T19:55:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa22cc00640 1 -- 172.21.15.29:0/63845392 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa224007310 con 0x7fa23006e3a0 2024-04-17T19:55:34.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa237a00640 1 -- 172.21.15.29:0/63845392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa230154600 con 0x7fa23006e3a0 2024-04-17T19:55:34.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa237a00640 1 -- 172.21.15.29:0/63845392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa23015f080 con 0x7fa23006e3a0 2024-04-17T19:55:34.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa237a00640 1 -- 172.21.15.29:0/63845392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa230069b90 con 0x7fa23006e3a0 2024-04-17T19:55:34.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.400+0000 7fa22cc00640 1 -- 172.21.15.29:0/63845392 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa22401a070 con 0x7fa23006e3a0 2024-04-17T19:55:34.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.404+0000 7fa22cc00640 1 --2- 172.21.15.29:0/63845392 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa2140683b0 0x7fa21406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:34.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.404+0000 7fa22cc00640 1 -- 172.21.15.29:0/63845392 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 7135+0+0 (secure 0 0 0) 0x7fa224086370 con 0x7fa23006e3a0 2024-04-17T19:55:34.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.404+0000 7fa22fe00640 1 --2- 172.21.15.29:0/63845392 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa2140683b0 0x7fa21406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:34.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.404+0000 7fa22fe00640 1 --2- 172.21.15.29:0/63845392 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa2140683b0 0x7fa21406a8c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa2200b2000 tx=0x7fa220008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:34.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.408+0000 7fa22cc00640 1 -- 172.21.15.29:0/63845392 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa224056300 con 0x7fa23006e3a0 2024-04-17T19:55:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.604+0000 7fa237a00640 1 -- 172.21.15.29:0/63845392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2300cacd0 con 0x7fa23006e3a0 2024-04-17T19:55:34.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.608+0000 7fa22cc00640 1 -- 172.21.15.29:0/63845392 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+6078 (secure 0 0 0) 0x7fa224055c00 con 0x7fa23006e3a0 2024-04-17T19:55:34.602 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:34.602 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:29.896742+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:reconnect","state_seq":28,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4302,5212,4305,5350]},"id":1}]} 2024-04-17T19:55:34.602 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 40 2024-04-17T19:55:34.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.608+0000 7fa237a00640 1 -- 172.21.15.29:0/63845392 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa2140683b0 msgr2=0x7fa21406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:34.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.608+0000 7fa237a00640 1 --2- 172.21.15.29:0/63845392 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa2140683b0 0x7fa21406a8c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa2200b2000 tx=0x7fa220008040 comp rx=0 tx=0).stop 2024-04-17T19:55:34.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.608+0000 7fa237a00640 1 -- 172.21.15.29:0/63845392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa23006e3a0 msgr2=0x7fa23015da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:34.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.608+0000 7fa237a00640 1 --2- 172.21.15.29:0/63845392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa23006e3a0 0x7fa23015da70 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa224005890 tx=0x7fa22400c8e0 comp rx=0 tx=0).stop 2024-04-17T19:55:34.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.608+0000 7fa237a00640 1 -- 172.21.15.29:0/63845392 shutdown_connections 2024-04-17T19:55:34.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.608+0000 7fa237a00640 1 --2- 172.21.15.29:0/63845392 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa2140683b0 0x7fa21406a8c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:34.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.608+0000 7fa237a00640 1 --2- 172.21.15.29:0/63845392 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa23015f7e0 0x7fa23014bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:34.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.608+0000 7fa237a00640 1 --2- 172.21.15.29:0/63845392 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2300cc210 0x7fa230155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:34.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.608+0000 7fa237a00640 1 --2- 172.21.15.29:0/63845392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa23006e3a0 0x7fa23015da70 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:34.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.608+0000 7fa237a00640 1 -- 172.21.15.29:0/63845392 >> 172.21.15.29:0/63845392 conn(0x7fa230071ca0 msgr2=0x7fa2300c5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:34.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.608+0000 7fa237a00640 1 -- 172.21.15.29:0/63845392 shutdown_connections 2024-04-17T19:55:34.605 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:34.608+0000 7fa237a00640 1 -- 172.21.15.29:0/63845392 wait complete. 2024-04-17T19:55:34.625 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:55:36.626 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:36.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.784+0000 7feb0cc00640 1 Processor -- start 2024-04-17T19:55:36.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.784+0000 7feb0cc00640 1 -- start start 2024-04-17T19:55:36.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.784+0000 7feb0cc00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7feb080c6f70 0x7feb080c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:36.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.784+0000 7feb0cc00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb0815a0a0 con 0x7feb08159c90 2024-04-17T19:55:36.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.784+0000 7feb0cc00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb0815abe0 con 0x7feb0806c660 2024-04-17T19:55:36.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.784+0000 7feb0cc00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7feb0815a340 con 0x7feb080c6f70 2024-04-17T19:55:36.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.784+0000 7feb05e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7feb08159c90 0x7feb080ca2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56524/0 (socket says 172.21.15.29:56524) 2024-04-17T19:55:36.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.784+0000 7feb05e00640 1 -- 172.21.15.29:0/1392539578 learned_addr learned my addr 172.21.15.29:0/1392539578 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:55:36.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.784+0000 7feaffe00640 1 --2- 172.21.15.29:0/1392539578 >> v2:172.21.15.37:3300/0 conn(0x7feb080c6f70 0x7feb080c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:36.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.784+0000 7feb05400640 1 -- 172.21.15.29:0/1392539578 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2488740178 0 0) 0x7feb0815a0a0 con 0x7feb08159c90 2024-04-17T19:55:36.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.784+0000 7feb05400640 1 -- 172.21.15.29:0/1392539578 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feaec003530 con 0x7feb08159c90 2024-04-17T19:55:36.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.788+0000 7feaffe00640 1 -- 172.21.15.29:0/1392539578 >> v1:172.21.15.37:6789/0 conn(0x7feb0806c660 legacy=0x7feb0806ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:36.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.788+0000 7feaffe00640 1 -- 172.21.15.29:0/1392539578 >> v1:172.21.15.29:6789/0 conn(0x7feb08159c90 legacy=0x7feb080ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:36.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.788+0000 7feaffe00640 1 -- 172.21.15.29:0/1392539578 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb0815a620 con 0x7feb080c6f70 2024-04-17T19:55:36.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.788+0000 7feaffe00640 1 --2- 172.21.15.29:0/1392539578 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb080c6f70 0x7feb080c7380 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7feaf00097c0 tx=0x7feaf001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=247585472d2c8438 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:36.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.788+0000 7feb05400640 1 -- 172.21.15.29:0/1392539578 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf00b0070 con 0x7feb080c6f70 2024-04-17T19:55:36.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.788+0000 7feb05400640 1 -- 172.21.15.29:0/1392539578 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaf00ab600 con 0x7feb080c6f70 2024-04-17T19:55:36.785 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.788+0000 7feb05400640 1 -- 172.21.15.29:0/1392539578 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf00ab9b0 con 0x7feb080c6f70 2024-04-17T19:55:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.788+0000 7feb0cc00640 1 -- 172.21.15.29:0/1392539578 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb080c6f70 msgr2=0x7feb080c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.788+0000 7feb0cc00640 1 --2- 172.21.15.29:0/1392539578 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb080c6f70 0x7feb080c7380 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7feaf00097c0 tx=0x7feaf001c4a0 comp rx=0 tx=0).stop 2024-04-17T19:55:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.792+0000 7feb0cc00640 1 -- 172.21.15.29:0/1392539578 shutdown_connections 2024-04-17T19:55:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.792+0000 7feb0cc00640 1 --2- 172.21.15.29:0/1392539578 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb080c6f70 0x7feb080c7380 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.792+0000 7feb0cc00640 1 -- 172.21.15.29:0/1392539578 >> 172.21.15.29:0/1392539578 conn(0x7feb0806fde0 msgr2=0x7feb080701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.792+0000 7feb0cc00640 1 -- 172.21.15.29:0/1392539578 shutdown_connections 2024-04-17T19:55:36.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.792+0000 7feb0cc00640 1 -- 172.21.15.29:0/1392539578 wait complete. 2024-04-17T19:55:36.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.792+0000 7feb0cc00640 1 Processor -- start 2024-04-17T19:55:36.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.792+0000 7feb0cc00640 1 -- start start 2024-04-17T19:55:36.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.792+0000 7feb0cc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0806c660 0x7feb081f54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:36.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.792+0000 7feb0cc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb080c6f70 0x7feb080c5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:36.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.792+0000 7feb0cc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb08159c90 0x7feb08150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:36.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.792+0000 7feb0cc00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feb080cab40 con 0x7feb0806c660 2024-04-17T19:55:36.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.792+0000 7feb0cc00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7feb080ca9c0 con 0x7feb08159c90 2024-04-17T19:55:36.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.792+0000 7feb0cc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7feb08167e40 con 0x7feb080c6f70 2024-04-17T19:55:36.791 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.796+0000 7feb05e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0806c660 0x7feb081f54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:36.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.796+0000 7feb05e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0806c660 0x7feb081f54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35784/0 (socket says 172.21.15.29:35784) 2024-04-17T19:55:36.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.796+0000 7feb05e00640 1 -- 172.21.15.29:0/2075885220 learned_addr learned my addr 172.21.15.29:0/2075885220 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:36.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.796+0000 7feb06800640 1 --2- 172.21.15.29:0/2075885220 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb08159c90 0x7feb08150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:36.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.796+0000 7feaffe00640 1 --2- 172.21.15.29:0/2075885220 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb080c6f70 0x7feb080c5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:36.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.796+0000 7feb05e00640 1 -- 172.21.15.29:0/2075885220 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb080c6f70 msgr2=0x7feb080c5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:36.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.796+0000 7feb05e00640 1 --2- 172.21.15.29:0/2075885220 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb080c6f70 0x7feb080c5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:36.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.796+0000 7feb05e00640 1 -- 172.21.15.29:0/2075885220 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb08159c90 msgr2=0x7feb08150630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:36.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.796+0000 7feb05e00640 1 --2- 172.21.15.29:0/2075885220 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb08159c90 0x7feb08150630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:36.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.796+0000 7feb05e00640 1 -- 172.21.15.29:0/2075885220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb080c5aa0 con 0x7feb0806c660 2024-04-17T19:55:36.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.796+0000 7feb05e00640 1 --2- 172.21.15.29:0/2075885220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0806c660 0x7feb081f54c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7feaf800c460 tx=0x7feaf800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:36.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.796+0000 7feafe000640 1 -- 172.21.15.29:0/2075885220 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf8017070 con 0x7feb0806c660 2024-04-17T19:55:36.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.796+0000 7feafe000640 1 -- 172.21.15.29:0/2075885220 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaf8005cf0 con 0x7feb0806c660 2024-04-17T19:55:36.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.796+0000 7feb0cc00640 1 -- 172.21.15.29:0/2075885220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb081582d0 con 0x7feb0806c660 2024-04-17T19:55:36.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.796+0000 7feafe000640 1 -- 172.21.15.29:0/2075885220 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf80072f0 con 0x7feb0806c660 2024-04-17T19:55:36.792 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.796+0000 7feb0cc00640 1 -- 172.21.15.29:0/2075885220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb080c6120 con 0x7feb0806c660 2024-04-17T19:55:36.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.796+0000 7feb0cc00640 1 -- 172.21.15.29:0/2075885220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb08069b90 con 0x7feb0806c660 2024-04-17T19:55:36.793 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.796+0000 7feafe000640 1 -- 172.21.15.29:0/2075885220 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7feaf8007490 con 0x7feb0806c660 2024-04-17T19:55:36.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.800+0000 7feafe000640 1 --2- 172.21.15.29:0/2075885220 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7feae4068380 0x7feae406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:36.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.800+0000 7feafe000640 1 -- 172.21.15.29:0/2075885220 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 7135+0+0 (secure 0 0 0) 0x7feaf8086260 con 0x7feb0806c660 2024-04-17T19:55:36.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.800+0000 7feaffe00640 1 --2- 172.21.15.29:0/2075885220 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7feae4068380 0x7feae406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:36.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.800+0000 7feaffe00640 1 --2- 172.21.15.29:0/2075885220 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7feae4068380 0x7feae406a890 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7feaf00ad000 tx=0x7feaf00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:36.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:36.804+0000 7feafe000640 1 -- 172.21.15.29:0/2075885220 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7feaf8056280 con 0x7feb0806c660 2024-04-17T19:55:37.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:37.012+0000 7feb0cc00640 1 -- 172.21.15.29:0/2075885220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb0806ce80 con 0x7feb0806c660 2024-04-17T19:55:37.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:37.012+0000 7feafe000640 1 -- 172.21.15.29:0/2075885220 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6075 (secure 0 0 0) 0x7feaf8055b80 con 0x7feb0806c660 2024-04-17T19:55:37.009 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:37.010 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:35.895605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:rejoin","state_seq":30,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,4305,5212,4302]},"id":1}]} 2024-04-17T19:55:37.010 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 41 2024-04-17T19:55:37.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:37.016+0000 7feb0cc00640 1 -- 172.21.15.29:0/2075885220 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7feae4068380 msgr2=0x7feae406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:37.016+0000 7feb0cc00640 1 --2- 172.21.15.29:0/2075885220 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7feae4068380 0x7feae406a890 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7feaf00ad000 tx=0x7feaf00a6000 comp rx=0 tx=0).stop 2024-04-17T19:55:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:37.016+0000 7feb0cc00640 1 -- 172.21.15.29:0/2075885220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0806c660 msgr2=0x7feb081f54c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:37.016+0000 7feb0cc00640 1 --2- 172.21.15.29:0/2075885220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0806c660 0x7feb081f54c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7feaf800c460 tx=0x7feaf800c920 comp rx=0 tx=0).stop 2024-04-17T19:55:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:37.016+0000 7feb0cc00640 1 -- 172.21.15.29:0/2075885220 shutdown_connections 2024-04-17T19:55:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:37.016+0000 7feb0cc00640 1 --2- 172.21.15.29:0/2075885220 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7feae4068380 0x7feae406a890 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:37.016+0000 7feb0cc00640 1 --2- 172.21.15.29:0/2075885220 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb08159c90 0x7feb08150630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:37.016+0000 7feb0cc00640 1 --2- 172.21.15.29:0/2075885220 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb080c6f70 0x7feb080c5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:37.016+0000 7feb0cc00640 1 --2- 172.21.15.29:0/2075885220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb0806c660 0x7feb081f54c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:37.016+0000 7feb0cc00640 1 -- 172.21.15.29:0/2075885220 >> 172.21.15.29:0/2075885220 conn(0x7feb0806fde0 msgr2=0x7feb08151530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:37.016+0000 7feb0cc00640 1 -- 172.21.15.29:0/2075885220 shutdown_connections 2024-04-17T19:55:37.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:37.016+0000 7feb0cc00640 1 -- 172.21.15.29:0/2075885220 wait complete. 2024-04-17T19:55:37.031 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:55:39.033 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:39.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.200+0000 7f61c8800640 1 Processor -- start 2024-04-17T19:55:39.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.200+0000 7f61c8800640 1 -- start start 2024-04-17T19:55:39.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.204+0000 7f61c8800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f61c00cc8c0 0x7f61c00cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:39.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.204+0000 7f61c8800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61c00c88b0 con 0x7f61c0159390 2024-04-17T19:55:39.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.204+0000 7f61c8800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61c00c6ce0 con 0x7f61c01597a0 2024-04-17T19:55:39.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.204+0000 7f61c8800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f61c00c6fc0 con 0x7f61c00cc8c0 2024-04-17T19:55:39.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.204+0000 7f61c5400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f61c0159390 0x7f61c00ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48638/0 (socket says 172.21.15.29:48638) 2024-04-17T19:55:39.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.204+0000 7f61c5400640 1 -- 172.21.15.29:0/1769667525 learned_addr learned my addr 172.21.15.29:0/1769667525 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:55:39.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.204+0000 7f61c5e00640 1 --2- 172.21.15.29:0/1769667525 >> v2:172.21.15.37:3300/0 conn(0x7f61c00cc8c0 0x7f61c00cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:39.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.204+0000 7f61c4a00640 1 -- 172.21.15.29:0/1769667525 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2205154184 0 0) 0x7f61c00c88b0 con 0x7f61c0159390 2024-04-17T19:55:39.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.204+0000 7f61c4a00640 1 -- 172.21.15.29:0/1769667525 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61a8003530 con 0x7f61c0159390 2024-04-17T19:55:39.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.204+0000 7f61c5e00640 1 -- 172.21.15.29:0/1769667525 >> v1:172.21.15.37:6789/0 conn(0x7f61c01597a0 legacy=0x7f61c01635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:39.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.204+0000 7f61c5e00640 1 -- 172.21.15.29:0/1769667525 >> v1:172.21.15.29:6789/0 conn(0x7f61c0159390 legacy=0x7f61c00ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:39.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.204+0000 7f61c5e00640 1 -- 172.21.15.29:0/1769667525 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c00c72a0 con 0x7f61c00cc8c0 2024-04-17T19:55:39.202 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.208+0000 7f61c5e00640 1 --2- 172.21.15.29:0/1769667525 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61c00cc8c0 0x7f61c00cccd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f61b40097c0 tx=0x7f61b401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b71c411c6395803 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:39.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.208+0000 7f61c4a00640 1 -- 172.21.15.29:0/1769667525 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b40b0070 con 0x7f61c00cc8c0 2024-04-17T19:55:39.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.208+0000 7f61c4a00640 1 -- 172.21.15.29:0/1769667525 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61b40ab600 con 0x7f61c00cc8c0 2024-04-17T19:55:39.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.208+0000 7f61c4a00640 1 -- 172.21.15.29:0/1769667525 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b40ab9b0 con 0x7f61c00cc8c0 2024-04-17T19:55:39.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.208+0000 7f61c8800640 1 -- 172.21.15.29:0/1769667525 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61c00cc8c0 msgr2=0x7f61c00cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:39.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.208+0000 7f61c8800640 1 --2- 172.21.15.29:0/1769667525 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61c00cc8c0 0x7f61c00cccd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f61b40097c0 tx=0x7f61b401c4a0 comp rx=0 tx=0).stop 2024-04-17T19:55:39.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.208+0000 7f61c8800640 1 -- 172.21.15.29:0/1769667525 shutdown_connections 2024-04-17T19:55:39.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.208+0000 7f61c8800640 1 --2- 172.21.15.29:0/1769667525 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61c00cc8c0 0x7f61c00cccd0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:39.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.208+0000 7f61c8800640 1 -- 172.21.15.29:0/1769667525 >> 172.21.15.29:0/1769667525 conn(0x7f61c006fde0 msgr2=0x7f61c00701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:39.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.208+0000 7f61c8800640 1 -- 172.21.15.29:0/1769667525 shutdown_connections 2024-04-17T19:55:39.204 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.208+0000 7f61c8800640 1 -- 172.21.15.29:0/1769667525 wait complete. 2024-04-17T19:55:39.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c8800640 1 Processor -- start 2024-04-17T19:55:39.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c8800640 1 -- start start 2024-04-17T19:55:39.207 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c8800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00cc8c0 0x7f61c01f5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:39.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c8800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61c0159390 0x7f61c015ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:39.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c8800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61c01597a0 0x7f61c0140a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:39.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c8800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f61c00cabc0 con 0x7f61c0159390 2024-04-17T19:55:39.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c8800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f61c00caa40 con 0x7f61c01597a0 2024-04-17T19:55:39.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c8800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f61c00c6880 con 0x7f61c00cc8c0 2024-04-17T19:55:39.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c5400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61c0159390 0x7f61c015ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:39.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c5400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61c0159390 0x7f61c015ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40398/0 (socket says 172.21.15.29:40398) 2024-04-17T19:55:39.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c5400640 1 -- 172.21.15.29:0/298277010 learned_addr learned my addr 172.21.15.29:0/298277010 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:39.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c6800640 1 --2- 172.21.15.29:0/298277010 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61c01597a0 0x7f61c0140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:39.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c5e00640 1 --2- 172.21.15.29:0/298277010 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00cc8c0 0x7f61c01f5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:39.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c5400640 1 -- 172.21.15.29:0/298277010 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00cc8c0 msgr2=0x7f61c01f5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:39.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c5400640 1 --2- 172.21.15.29:0/298277010 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00cc8c0 0x7f61c01f5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:39.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c5400640 1 -- 172.21.15.29:0/298277010 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61c01597a0 msgr2=0x7f61c0140a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:39.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c5400640 1 --2- 172.21.15.29:0/298277010 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61c01597a0 0x7f61c0140a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:39.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c5400640 1 -- 172.21.15.29:0/298277010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c0141b80 con 0x7f61c0159390 2024-04-17T19:55:39.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c5400640 1 --2- 172.21.15.29:0/298277010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61c0159390 0x7f61c015ba90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f61b000c430 tx=0x7f61b000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:39.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61ba000640 1 -- 172.21.15.29:0/298277010 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b0017070 con 0x7f61c0159390 2024-04-17T19:55:39.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61ba000640 1 -- 172.21.15.29:0/298277010 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61b0005d10 con 0x7f61c0159390 2024-04-17T19:55:39.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c8800640 1 -- 172.21.15.29:0/298277010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61c01f8b40 con 0x7f61c0159390 2024-04-17T19:55:39.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61ba000640 1 -- 172.21.15.29:0/298277010 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b0007310 con 0x7f61c0159390 2024-04-17T19:55:39.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.212+0000 7f61c8800640 1 -- 172.21.15.29:0/298277010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61c01f9100 con 0x7f61c0159390 2024-04-17T19:55:39.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.216+0000 7f61c8800640 1 -- 172.21.15.29:0/298277010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61c0069b90 con 0x7f61c0159390 2024-04-17T19:55:39.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.216+0000 7f61ba000640 1 -- 172.21.15.29:0/298277010 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f61b00074b0 con 0x7f61c0159390 2024-04-17T19:55:39.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.216+0000 7f61ba000640 1 --2- 172.21.15.29:0/298277010 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f61a40683b0 0x7f61a406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:39.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.216+0000 7f61ba000640 1 -- 172.21.15.29:0/298277010 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 7135+0+0 (secure 0 0 0) 0x7f61b00862e0 con 0x7f61c0159390 2024-04-17T19:55:39.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.216+0000 7f61c5e00640 1 --2- 172.21.15.29:0/298277010 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f61a40683b0 0x7f61a406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:39.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.216+0000 7f61c5e00640 1 --2- 172.21.15.29:0/298277010 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f61a40683b0 0x7f61a406a8c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f61b40ad000 tx=0x7f61b40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:39.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.220+0000 7f61ba000640 1 -- 172.21.15.29:0/298277010 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f61b0055b10 con 0x7f61c0159390 2024-04-17T19:55:39.415 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.420+0000 7f61c8800640 1 -- 172.21.15.29:0/298277010 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61c013c500 con 0x7f61c0159390 2024-04-17T19:55:39.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.420+0000 7f61ba000640 1 -- 172.21.15.29:0/298277010 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6075 (secure 0 0 0) 0x7f61b0059e10 con 0x7f61c0159390 2024-04-17T19:55:39.416 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:39.416 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:35.895605+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:rejoin","state_seq":30,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,4305,5212,4302]},"id":1}]} 2024-04-17T19:55:39.417 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 41 2024-04-17T19:55:39.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.424+0000 7f61c8800640 1 -- 172.21.15.29:0/298277010 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f61a40683b0 msgr2=0x7f61a406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:39.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.424+0000 7f61c8800640 1 --2- 172.21.15.29:0/298277010 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f61a40683b0 0x7f61a406a8c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f61b40ad000 tx=0x7f61b40a6000 comp rx=0 tx=0).stop 2024-04-17T19:55:39.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.424+0000 7f61c8800640 1 -- 172.21.15.29:0/298277010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61c0159390 msgr2=0x7f61c015ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:39.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.424+0000 7f61c8800640 1 --2- 172.21.15.29:0/298277010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61c0159390 0x7f61c015ba90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f61b000c430 tx=0x7f61b000c8f0 comp rx=0 tx=0).stop 2024-04-17T19:55:39.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.424+0000 7f61c8800640 1 -- 172.21.15.29:0/298277010 shutdown_connections 2024-04-17T19:55:39.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.424+0000 7f61c8800640 1 --2- 172.21.15.29:0/298277010 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f61a40683b0 0x7f61a406a8c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:39.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.424+0000 7f61c8800640 1 --2- 172.21.15.29:0/298277010 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61c01597a0 0x7f61c0140a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:39.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.424+0000 7f61c8800640 1 --2- 172.21.15.29:0/298277010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61c0159390 0x7f61c015ba90 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:39.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.424+0000 7f61c8800640 1 --2- 172.21.15.29:0/298277010 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61c00cc8c0 0x7f61c01f5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:39.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.424+0000 7f61c8800640 1 -- 172.21.15.29:0/298277010 >> 172.21.15.29:0/298277010 conn(0x7f61c006fde0 msgr2=0x7f61c00c4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:39.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.424+0000 7f61c8800640 1 -- 172.21.15.29:0/298277010 shutdown_connections 2024-04-17T19:55:39.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:39.424+0000 7f61c8800640 1 -- 172.21.15.29:0/298277010 wait complete. 2024-04-17T19:55:39.438 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:55:40.937 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:40.950 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:40.953 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:40.954 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:41.439 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:41.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.596+0000 7f695a200640 1 Processor -- start 2024-04-17T19:55:41.591 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.596+0000 7f695a200640 1 -- start start 2024-04-17T19:55:41.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.596+0000 7f695a200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6954159c90 0x7f69540ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:41.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.596+0000 7f695a200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f69540ca7f0 0x7f69540c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:41.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.596+0000 7f695a200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69540c74b0 0x7f695406c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:41.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.596+0000 7f695a200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f695415a0a0 con 0x7f69540ca7f0 2024-04-17T19:55:41.593 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.596+0000 7f695a200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f695415abe0 con 0x7f6954159c90 2024-04-17T19:55:41.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.596+0000 7f695a200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f695415a340 con 0x7f69540c74b0 2024-04-17T19:55:41.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.596+0000 7f694be00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f69540ca7f0 0x7f69540c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:41.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.596+0000 7f694be00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f69540ca7f0 0x7f69540c6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40412/0 (socket says 172.21.15.29:40412) 2024-04-17T19:55:41.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.596+0000 7f694be00640 1 -- 172.21.15.29:0/2026826784 learned_addr learned my addr 172.21.15.29:0/2026826784 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:41.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.596+0000 7f6953400640 1 --2- 172.21.15.29:0/2026826784 >> v2:172.21.15.37:3300/0 conn(0x7f6954159c90 0x7f69540ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:41.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.596+0000 7f6953e00640 1 --2- 172.21.15.29:0/2026826784 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69540c74b0 0x7f695406c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:41.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.596+0000 7f694be00640 1 -- 172.21.15.29:0/2026826784 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69540c74b0 msgr2=0x7f695406c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:41.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.596+0000 7f694be00640 1 --2- 172.21.15.29:0/2026826784 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69540c74b0 0x7f695406c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:41.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.596+0000 7f694be00640 1 -- 172.21.15.29:0/2026826784 >> v2:172.21.15.37:3300/0 conn(0x7f6954159c90 msgr2=0x7f69540ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:41.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.596+0000 7f694be00640 1 --2- 172.21.15.29:0/2026826784 >> v2:172.21.15.37:3300/0 conn(0x7f6954159c90 0x7f69540ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:41.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.596+0000 7f694be00640 1 -- 172.21.15.29:0/2026826784 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f695415a620 con 0x7f69540ca7f0 2024-04-17T19:55:41.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.600+0000 7f694be00640 1 --2- 172.21.15.29:0/2026826784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69540ca7f0 0x7f69540c6f70 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6954071f40 tx=0x7f693c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87b2bcd5b04e2cf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:41.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.600+0000 7f6952a00640 1 -- 172.21.15.29:0/2026826784 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c0ab5f0 con 0x7f69540ca7f0 2024-04-17T19:55:41.596 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.600+0000 7f6952a00640 1 -- 172.21.15.29:0/2026826784 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f693c0b0070 con 0x7f69540ca7f0 2024-04-17T19:55:41.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.600+0000 7f6952a00640 1 -- 172.21.15.29:0/2026826784 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c0ab9b0 con 0x7f69540ca7f0 2024-04-17T19:55:41.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.600+0000 7f695a200640 1 -- 172.21.15.29:0/2026826784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69540ca7f0 msgr2=0x7f69540c6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:41.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.600+0000 7f695a200640 1 --2- 172.21.15.29:0/2026826784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69540ca7f0 0x7f69540c6f70 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6954071f40 tx=0x7f693c01c520 comp rx=0 tx=0).stop 2024-04-17T19:55:41.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.600+0000 7f695a200640 1 -- 172.21.15.29:0/2026826784 shutdown_connections 2024-04-17T19:55:41.597 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.600+0000 7f695a200640 1 --2- 172.21.15.29:0/2026826784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69540ca7f0 0x7f69540c6f70 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:41.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.600+0000 7f695a200640 1 --2- 172.21.15.29:0/2026826784 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69540c74b0 0x7f695406c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:41.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.600+0000 7f695a200640 1 --2- 172.21.15.29:0/2026826784 >> v2:172.21.15.37:3300/0 conn(0x7f6954159c90 0x7f69540ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:41.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.600+0000 7f695a200640 1 -- 172.21.15.29:0/2026826784 >> 172.21.15.29:0/2026826784 conn(0x7f695406fde0 msgr2=0x7f69540701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:41.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.600+0000 7f695a200640 1 -- 172.21.15.29:0/2026826784 shutdown_connections 2024-04-17T19:55:41.598 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.600+0000 7f695a200640 1 -- 172.21.15.29:0/2026826784 wait complete. 2024-04-17T19:55:41.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.604+0000 7f695a200640 1 Processor -- start 2024-04-17T19:55:41.599 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.604+0000 7f695a200640 1 -- start start 2024-04-17T19:55:41.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.604+0000 7f695a200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69540c74b0 0x7f69541f60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:41.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.604+0000 7f695a200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69540ca7f0 0x7f6954153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:41.602 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.604+0000 7f695a200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6954159c90 0x7f69540c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:41.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.604+0000 7f695a200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f695415da90 con 0x7f69540ca7f0 2024-04-17T19:55:41.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.604+0000 7f695a200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f695406cfb0 con 0x7f6954159c90 2024-04-17T19:55:41.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.604+0000 7f695a200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f695415dc10 con 0x7f69540c74b0 2024-04-17T19:55:41.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.604+0000 7f694be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69540ca7f0 0x7f6954153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:41.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.604+0000 7f694be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69540ca7f0 0x7f6954153760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40420/0 (socket says 172.21.15.29:40420) 2024-04-17T19:55:41.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.604+0000 7f694be00640 1 -- 172.21.15.29:0/3487966757 learned_addr learned my addr 172.21.15.29:0/3487966757 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:41.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.604+0000 7f6953400640 1 --2- 172.21.15.29:0/3487966757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69540c74b0 0x7f69541f60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:41.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.604+0000 7f6953e00640 1 --2- 172.21.15.29:0/3487966757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6954159c90 0x7f69540c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:41.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.608+0000 7f694be00640 1 -- 172.21.15.29:0/3487966757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69540c74b0 msgr2=0x7f69541f60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:41.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.608+0000 7f694be00640 1 --2- 172.21.15.29:0/3487966757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69540c74b0 0x7f69541f60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:41.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.608+0000 7f694be00640 1 -- 172.21.15.29:0/3487966757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6954159c90 msgr2=0x7f69540c4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:41.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.608+0000 7f694be00640 1 --2- 172.21.15.29:0/3487966757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6954159c90 0x7f69540c4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:41.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.608+0000 7f694be00640 1 -- 172.21.15.29:0/3487966757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69541f6610 con 0x7f69540ca7f0 2024-04-17T19:55:41.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.608+0000 7f6953400640 1 --2- 172.21.15.29:0/3487966757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69540c74b0 0x7f69541f60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:55:41.603 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.608+0000 7f694be00640 1 --2- 172.21.15.29:0/3487966757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69540ca7f0 0x7f6954153760 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f693c01abd0 tx=0x7f693c01ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:41.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.608+0000 7f694b400640 1 -- 172.21.15.29:0/3487966757 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c0be3f0 con 0x7f69540ca7f0 2024-04-17T19:55:41.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.608+0000 7f694b400640 1 -- 172.21.15.29:0/3487966757 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f693c0be590 con 0x7f69540ca7f0 2024-04-17T19:55:41.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.608+0000 7f694b400640 1 -- 172.21.15.29:0/3487966757 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c0be980 con 0x7f69540ca7f0 2024-04-17T19:55:41.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.608+0000 7f695a200640 1 -- 172.21.15.29:0/3487966757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f695415cf40 con 0x7f69540ca7f0 2024-04-17T19:55:41.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.608+0000 7f695a200640 1 -- 172.21.15.29:0/3487966757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6954150510 con 0x7f69540ca7f0 2024-04-17T19:55:41.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.608+0000 7f695a200640 1 -- 172.21.15.29:0/3487966757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6954069b90 con 0x7f69540ca7f0 2024-04-17T19:55:41.604 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.608+0000 7f694b400640 1 -- 172.21.15.29:0/3487966757 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f693c0b43f0 con 0x7f69540ca7f0 2024-04-17T19:55:41.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.612+0000 7f694b400640 1 --2- 172.21.15.29:0/3487966757 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6930068380 0x7f693006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:41.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.612+0000 7f694b400640 1 -- 172.21.15.29:0/3487966757 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 7135+0+0 (secure 0 0 0) 0x7f693c120f40 con 0x7f69540ca7f0 2024-04-17T19:55:41.606 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.612+0000 7f6953400640 1 --2- 172.21.15.29:0/3487966757 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6930068380 0x7f693006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:41.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.612+0000 7f6953400640 1 --2- 172.21.15.29:0/3487966757 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6930068380 0x7f693006a890 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f69540c8e90 tx=0x7f6944008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:41.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.616+0000 7f694b400640 1 -- 172.21.15.29:0/3487966757 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f693c016110 con 0x7f69540ca7f0 2024-04-17T19:55:41.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.812+0000 7f695a200640 1 -- 172.21.15.29:0/3487966757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69540ca5b0 con 0x7f69540ca7f0 2024-04-17T19:55:41.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.812+0000 7f694b400640 1 -- 172.21.15.29:0/3487966757 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6080 (secure 0 0 0) 0x7f693c0f0910 con 0x7f69540ca7f0 2024-04-17T19:55:41.809 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:41.809 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:40.944059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4302,5212,4305,5144,5350]},"id":1}]} 2024-04-17T19:55:41.809 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 42 2024-04-17T19:55:41.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.816+0000 7f695a200640 1 -- 172.21.15.29:0/3487966757 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6930068380 msgr2=0x7f693006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:41.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.816+0000 7f695a200640 1 --2- 172.21.15.29:0/3487966757 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6930068380 0x7f693006a890 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f69540c8e90 tx=0x7f6944008040 comp rx=0 tx=0).stop 2024-04-17T19:55:41.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.816+0000 7f695a200640 1 -- 172.21.15.29:0/3487966757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69540ca7f0 msgr2=0x7f6954153760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:41.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.816+0000 7f695a200640 1 --2- 172.21.15.29:0/3487966757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69540ca7f0 0x7f6954153760 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f693c01abd0 tx=0x7f693c01ad60 comp rx=0 tx=0).stop 2024-04-17T19:55:41.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.816+0000 7f695a200640 1 -- 172.21.15.29:0/3487966757 shutdown_connections 2024-04-17T19:55:41.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.816+0000 7f695a200640 1 --2- 172.21.15.29:0/3487966757 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6930068380 0x7f693006a890 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:41.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.816+0000 7f695a200640 1 --2- 172.21.15.29:0/3487966757 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6954159c90 0x7f69540c4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:41.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.816+0000 7f695a200640 1 --2- 172.21.15.29:0/3487966757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69540ca7f0 0x7f6954153760 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:41.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.816+0000 7f695a200640 1 --2- 172.21.15.29:0/3487966757 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f69540c74b0 0x7f69541f60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:41.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.816+0000 7f695a200640 1 -- 172.21.15.29:0/3487966757 >> 172.21.15.29:0/3487966757 conn(0x7f695406fde0 msgr2=0x7f69540c40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:41.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.816+0000 7f695a200640 1 -- 172.21.15.29:0/3487966757 shutdown_connections 2024-04-17T19:55:41.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:41.816+0000 7f695a200640 1 -- 172.21.15.29:0/3487966757 wait complete. 2024-04-17T19:55:41.831 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=2, replacing gid=4259 2024-04-17T19:55:41.831 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.a 2024-04-17T19:55:44.837 INFO:tasks.ceph.mds.c.smithi029.stderr:2024-04-17T19:55:44.840+0000 7ff3b0200640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-17T19:55:45.954 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:45.963 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:45.964 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:46.410 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:46.424 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:46.425 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:46.425 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:46.425 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:01 sshd 2024-04-17T19:55:46.425 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:46.425 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:46.425 INFO:tasks.workunit.client.0.smithi029.stdout: 39375 ? 00:00:00 ps 2024-04-17T19:55:46.425 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:46.946 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:46.950 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:46.961 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:46.963 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:46.965 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:46.984 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:46.995 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:46.996 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:47.091 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:47.107 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:47.107 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:47.107 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:47.107 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:01 sshd 2024-04-17T19:55:47.107 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:47.107 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:47.107 INFO:tasks.workunit.client.0.smithi029.stdout: 39385 ? 00:00:00 ps 2024-04-17T19:55:47.108 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:47.128 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:47.132 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:47.143 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:47.146 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:47.147 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:47.164 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:47.174 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:47.176 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:47.275 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:47.290 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:47.290 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:47.290 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:47.290 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:01 sshd 2024-04-17T19:55:47.290 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:47.290 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:47.290 INFO:tasks.workunit.client.0.smithi029.stdout: 39395 ? 00:00:00 ps 2024-04-17T19:55:47.291 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:47.309 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:47.312 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:47.925 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:47.928 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:47.929 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:48.422 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:48.435 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:48.436 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:48.576 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:48.593 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:48.593 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:48.593 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:48.593 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:01 sshd 2024-04-17T19:55:48.594 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:48.594 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:48.594 INFO:tasks.workunit.client.0.smithi029.stdout: 39405 ? 00:00:00 ps 2024-04-17T19:55:48.594 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:48.663 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:48.665 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:48.673 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:48.676 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:48.677 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:48.692 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:48.702 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:48.703 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:48.809 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:48.824 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:48.824 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:48.825 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:48.825 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:01 sshd 2024-04-17T19:55:48.825 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:48.825 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:48.825 INFO:tasks.workunit.client.0.smithi029.stdout: 39415 ? 00:00:00 ps 2024-04-17T19:55:48.825 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:48.832 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2024-04-17T19:55:48.833 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T19:55:48.833 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T19:55:48.836 INFO:tasks.ceph.mds.a:Started 2024-04-17T19:55:48.836 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:48.944 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2024-04-17T19:55:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.000+0000 7fbb8a000640 1 Processor -- start 2024-04-17T19:55:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.000+0000 7fbb8a000640 1 -- start start 2024-04-17T19:55:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.004+0000 7fbb8a000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbb84152d90 0x7fbb840c2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.004+0000 7fbb8a000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb840c3360 0x7fbb8415c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.004+0000 7fbb8a000640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb840c95e0 con 0x7fbb8415c660 2024-04-17T19:55:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.004+0000 7fbb8a000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fbb84071e30 con 0x7fbb84152d90 2024-04-17T19:55:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.004+0000 7fbb8a000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbb8415ce00 con 0x7fbb840c3360 2024-04-17T19:55:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.004+0000 7fbb88c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbb84152d90 0x7fbb840c2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.004+0000 7fbb89600640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbb8415c660 0x7fbb84158b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59362/0 (socket says 172.21.15.29:59362) 2024-04-17T19:55:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.004+0000 7fbb89600640 1 -- 172.21.15.29:0/3512877574 learned_addr learned my addr 172.21.15.29:0/3512877574 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:55:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.004+0000 7fbb83e00640 1 --2- 172.21.15.29:0/3512877574 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb840c3360 0x7fbb8415c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.004+0000 7fbb88c00640 1 -- 172.21.15.29:0/3512877574 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb840c3360 msgr2=0x7fbb8415c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:55:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.004+0000 7fbb88c00640 1 --2- 172.21.15.29:0/3512877574 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb840c3360 0x7fbb8415c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.004+0000 7fbb88c00640 1 -- 172.21.15.29:0/3512877574 >> v1:172.21.15.29:6789/0 conn(0x7fbb8415c660 legacy=0x7fbb84158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.004+0000 7fbb88c00640 1 -- 172.21.15.29:0/3512877574 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb84073eb0 con 0x7fbb84152d90 2024-04-17T19:55:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.004+0000 7fbb83400640 1 -- 172.21.15.29:0/3512877574 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2749832528 0 0) 0x7fbb840c95e0 con 0x7fbb8415c660 2024-04-17T19:55:49.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.008+0000 7fbb88c00640 1 --2- 172.21.15.29:0/3512877574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb84152d90 0x7fbb840c2e20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fbb780097c0 tx=0x7fbb7801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e80f6b64f871532 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:49.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.008+0000 7fbb83400640 1 -- 172.21.15.29:0/3512877574 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb780ab650 con 0x7fbb84152d90 2024-04-17T19:55:49.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.008+0000 7fbb83400640 1 -- 172.21.15.29:0/3512877574 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb780b0070 con 0x7fbb84152d90 2024-04-17T19:55:49.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.008+0000 7fbb83400640 1 -- 172.21.15.29:0/3512877574 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb780aba10 con 0x7fbb84152d90 2024-04-17T19:55:49.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.008+0000 7fbb8a000640 1 -- 172.21.15.29:0/3512877574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb84152d90 msgr2=0x7fbb840c2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:49.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.008+0000 7fbb8a000640 1 --2- 172.21.15.29:0/3512877574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb84152d90 0x7fbb840c2e20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fbb780097c0 tx=0x7fbb7801c470 comp rx=0 tx=0).stop 2024-04-17T19:55:49.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.008+0000 7fbb8a000640 1 -- 172.21.15.29:0/3512877574 shutdown_connections 2024-04-17T19:55:49.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.008+0000 7fbb8a000640 1 --2- 172.21.15.29:0/3512877574 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb840c3360 0x7fbb8415c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:49.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.008+0000 7fbb8a000640 1 --2- 172.21.15.29:0/3512877574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb84152d90 0x7fbb840c2e20 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:49.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.008+0000 7fbb8a000640 1 -- 172.21.15.29:0/3512877574 >> 172.21.15.29:0/3512877574 conn(0x7fbb840ccf80 msgr2=0x7fbb840cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:49.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.008+0000 7fbb8a000640 1 -- 172.21.15.29:0/3512877574 shutdown_connections 2024-04-17T19:55:49.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.008+0000 7fbb8a000640 1 -- 172.21.15.29:0/3512877574 wait complete. 2024-04-17T19:55:49.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.008+0000 7fbb8a000640 1 Processor -- start 2024-04-17T19:55:49.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.008+0000 7fbb8a000640 1 -- start start 2024-04-17T19:55:49.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb8a000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb840c3360 0x7fbb841f4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:49.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb8a000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb84152d90 0x7fbb841582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:49.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb8a000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb8415c660 0x7fbb840b1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:49.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb8a000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8406d210 con 0x7fbb84152d90 2024-04-17T19:55:49.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb8a000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fbb8415ca70 con 0x7fbb8415c660 2024-04-17T19:55:49.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb8a000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fbb8406d390 con 0x7fbb840c3360 2024-04-17T19:55:49.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb83e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb84152d90 0x7fbb841582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:49.007 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb83e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb84152d90 0x7fbb841582b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56072/0 (socket says 172.21.15.29:56072) 2024-04-17T19:55:49.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb83e00640 1 -- 172.21.15.29:0/2387047027 learned_addr learned my addr 172.21.15.29:0/2387047027 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:49.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb88c00640 1 --2- 172.21.15.29:0/2387047027 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb840c3360 0x7fbb841f4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:49.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb89600640 1 --2- 172.21.15.29:0/2387047027 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb8415c660 0x7fbb840b1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:49.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb83e00640 1 -- 172.21.15.29:0/2387047027 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb840c3360 msgr2=0x7fbb841f4d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:49.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb83e00640 1 --2- 172.21.15.29:0/2387047027 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb840c3360 0x7fbb841f4d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:49.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb83e00640 1 -- 172.21.15.29:0/2387047027 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb8415c660 msgr2=0x7fbb840b1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:49.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb83e00640 1 --2- 172.21.15.29:0/2387047027 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb8415c660 0x7fbb840b1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:49.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb83e00640 1 -- 172.21.15.29:0/2387047027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb841f5280 con 0x7fbb84152d90 2024-04-17T19:55:49.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb88c00640 1 --2- 172.21.15.29:0/2387047027 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb840c3360 0x7fbb841f4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:55:49.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb83e00640 1 --2- 172.21.15.29:0/2387047027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb84152d90 0x7fbb841582b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbb7000e840 tx=0x7fbb7000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:49.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb80c00640 1 -- 172.21.15.29:0/2387047027 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb70017070 con 0x7fbb84152d90 2024-04-17T19:55:49.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb80c00640 1 -- 172.21.15.29:0/2387047027 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb7000ee90 con 0x7fbb84152d90 2024-04-17T19:55:49.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb8a000640 1 -- 172.21.15.29:0/2387047027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb841f8320 con 0x7fbb84152d90 2024-04-17T19:55:49.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb80c00640 1 -- 172.21.15.29:0/2387047027 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb7000cb60 con 0x7fbb84152d90 2024-04-17T19:55:49.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb8a000640 1 -- 172.21.15.29:0/2387047027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb841f88e0 con 0x7fbb84152d90 2024-04-17T19:55:49.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.012+0000 7fbb8a000640 1 -- 172.21.15.29:0/2387047027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb841f8090 con 0x7fbb84152d90 2024-04-17T19:55:49.010 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.016+0000 7fbb80c00640 1 -- 172.21.15.29:0/2387047027 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fbb7001a070 con 0x7fbb84152d90 2024-04-17T19:55:49.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.016+0000 7fbb80c00640 1 --2- 172.21.15.29:0/2387047027 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbb640683b0 0x7fbb6406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:49.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.016+0000 7fbb80c00640 1 -- 172.21.15.29:0/2387047027 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 7135+0+0 (secure 0 0 0) 0x7fbb70086d30 con 0x7fbb84152d90 2024-04-17T19:55:49.011 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.016+0000 7fbb88c00640 1 --2- 172.21.15.29:0/2387047027 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbb640683b0 0x7fbb6406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:49.012 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.016+0000 7fbb88c00640 1 --2- 172.21.15.29:0/2387047027 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbb640683b0 0x7fbb6406a8c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbb780af000 tx=0x7fbb780a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:49.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.020+0000 7fbb80c00640 1 -- 172.21.15.29:0/2387047027 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fbb70056670 con 0x7fbb84152d90 2024-04-17T19:55:49.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.220+0000 7fbb8a000640 1 -- 172.21.15.29:0/2387047027 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb8404ff10 con 0x7fbb84152d90 2024-04-17T19:55:49.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.220+0000 7fbb80c00640 1 -- 172.21.15.29:0/2387047027 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+6080 (secure 0 0 0) 0x7fbb7005a970 con 0x7fbb84152d90 2024-04-17T19:55:49.216 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:49.216 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:40.944059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4302,5212,4305,5144,5350]},"id":1}]} 2024-04-17T19:55:49.217 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 42 2024-04-17T19:55:49.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.220+0000 7fbb8a000640 1 -- 172.21.15.29:0/2387047027 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbb640683b0 msgr2=0x7fbb6406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:49.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.220+0000 7fbb8a000640 1 --2- 172.21.15.29:0/2387047027 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbb640683b0 0x7fbb6406a8c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbb780af000 tx=0x7fbb780a7000 comp rx=0 tx=0).stop 2024-04-17T19:55:49.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.220+0000 7fbb8a000640 1 -- 172.21.15.29:0/2387047027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb84152d90 msgr2=0x7fbb841582b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:49.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.220+0000 7fbb8a000640 1 --2- 172.21.15.29:0/2387047027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb84152d90 0x7fbb841582b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbb7000e840 tx=0x7fbb7000ec40 comp rx=0 tx=0).stop 2024-04-17T19:55:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.220+0000 7fbb8a000640 1 -- 172.21.15.29:0/2387047027 shutdown_connections 2024-04-17T19:55:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.220+0000 7fbb8a000640 1 --2- 172.21.15.29:0/2387047027 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fbb640683b0 0x7fbb6406a8c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.220+0000 7fbb8a000640 1 --2- 172.21.15.29:0/2387047027 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fbb8415c660 0x7fbb840b1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.220+0000 7fbb8a000640 1 --2- 172.21.15.29:0/2387047027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb84152d90 0x7fbb841582b0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:49.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.220+0000 7fbb8a000640 1 --2- 172.21.15.29:0/2387047027 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fbb840c3360 0x7fbb841f4d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.220+0000 7fbb8a000640 1 -- 172.21.15.29:0/2387047027 >> 172.21.15.29:0/2387047027 conn(0x7fbb840ccf80 msgr2=0x7fbb840c2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.220+0000 7fbb8a000640 1 -- 172.21.15.29:0/2387047027 shutdown_connections 2024-04-17T19:55:49.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:49.220+0000 7fbb8a000640 1 -- 172.21.15.29:0/2387047027 wait complete. 2024-04-17T19:55:49.237 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-04-17T19:55:49.791 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:50.181 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:50.189 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:50.192 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:50.193 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:50.207 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:50.216 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:50.217 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:50.463 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:50.480 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:50.480 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:50.480 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:50.480 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:01 sshd 2024-04-17T19:55:50.481 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:50.481 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:50.481 INFO:tasks.workunit.client.0.smithi029.stdout: 39474 ? 00:00:00 ps 2024-04-17T19:55:50.481 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:50.498 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:50.500 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:50.508 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:50.510 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:50.512 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:50.524 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:50.534 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:50.535 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:50.653 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:50.668 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:50.668 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:50.668 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:50.668 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:01 sshd 2024-04-17T19:55:50.668 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:50.668 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:50.668 INFO:tasks.workunit.client.0.smithi029.stdout: 39484 ? 00:00:00 ps 2024-04-17T19:55:50.669 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:50.688 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:50.690 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:50.698 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:50.700 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:50.702 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:50.716 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:50.726 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:50.727 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:50.837 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:50.852 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:50.852 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:50.852 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:50.852 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:01 sshd 2024-04-17T19:55:50.852 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:50.852 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:50.852 INFO:tasks.workunit.client.0.smithi029.stdout: 39494 ? 00:00:00 ps 2024-04-17T19:55:50.853 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:50.871 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:50.873 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:50.881 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:50.883 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:50.884 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:50.898 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:50.907 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:50.908 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:51.021 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:51.039 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:51.039 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:51.039 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:51.040 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:01 sshd 2024-04-17T19:55:51.040 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:51.040 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:51.040 INFO:tasks.workunit.client.0.smithi029.stdout: 39504 ? 00:00:00 ps 2024-04-17T19:55:51.041 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:51.095 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:51.100 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:51.112 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:51.117 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:51.119 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:51.136 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:51.147 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:51.150 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:51.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:55:51.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.396+0000 7f8e27e00640 1 Processor -- start 2024-04-17T19:55:51.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.396+0000 7f8e27e00640 1 -- start start 2024-04-17T19:55:51.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.396+0000 7f8e27e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8e280cc020 0x7f8e280cc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:51.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.396+0000 7f8e27e00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e280ca7c0 con 0x7f8e2815f680 2024-04-17T19:55:51.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.396+0000 7f8e27e00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e2806c7e0 con 0x7f8e28156140 2024-04-17T19:55:51.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.396+0000 7f8e27e00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f8e2806cac0 con 0x7f8e280cc020 2024-04-17T19:55:51.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.396+0000 7f8e1fe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8e280cc020 0x7f8e280cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:51.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.396+0000 7f8e27400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8e2815f680 0x7f8e2815fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59368/0 (socket says 172.21.15.29:59368) 2024-04-17T19:55:51.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.396+0000 7f8e27400640 1 -- 172.21.15.29:0/1158315289 learned_addr learned my addr 172.21.15.29:0/1158315289 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:55:51.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.400+0000 7f8e26000640 1 -- 172.21.15.29:0/1158315289 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 954796661 0 0) 0x7f8e280ca7c0 con 0x7f8e2815f680 2024-04-17T19:55:51.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.400+0000 7f8e26000640 1 -- 172.21.15.29:0/1158315289 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e0c003530 con 0x7f8e2815f680 2024-04-17T19:55:51.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.400+0000 7f8e1fe00640 1 -- 172.21.15.29:0/1158315289 >> v1:172.21.15.37:6789/0 conn(0x7f8e28156140 legacy=0x7f8e280c9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:51.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.400+0000 7f8e1fe00640 1 -- 172.21.15.29:0/1158315289 >> v1:172.21.15.29:6789/0 conn(0x7f8e2815f680 legacy=0x7f8e2815fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:51.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.400+0000 7f8e1fe00640 1 -- 172.21.15.29:0/1158315289 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e2806cda0 con 0x7f8e280cc020 2024-04-17T19:55:51.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.400+0000 7f8e1fe00640 1 --2- 172.21.15.29:0/1158315289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e280cc020 0x7f8e280cc430 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8e100097c0 tx=0x7f8e1001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=273534bd0f5ea126 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.400+0000 7f8e26000640 1 -- 172.21.15.29:0/1158315289 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e100b5070 con 0x7f8e280cc020 2024-04-17T19:55:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.400+0000 7f8e26000640 1 -- 172.21.15.29:0/1158315289 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e100a8990 con 0x7f8e280cc020 2024-04-17T19:55:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.400+0000 7f8e26000640 1 -- 172.21.15.29:0/1158315289 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e100a8d60 con 0x7f8e280cc020 2024-04-17T19:55:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.404+0000 7f8e27e00640 1 -- 172.21.15.29:0/1158315289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e280cc020 msgr2=0x7f8e280cc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.404+0000 7f8e27e00640 1 --2- 172.21.15.29:0/1158315289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e280cc020 0x7f8e280cc430 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8e100097c0 tx=0x7f8e1001ec90 comp rx=0 tx=0).stop 2024-04-17T19:55:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.404+0000 7f8e27e00640 1 -- 172.21.15.29:0/1158315289 shutdown_connections 2024-04-17T19:55:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.404+0000 7f8e27e00640 1 --2- 172.21.15.29:0/1158315289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e280cc020 0x7f8e280cc430 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.404+0000 7f8e27e00640 1 -- 172.21.15.29:0/1158315289 >> 172.21.15.29:0/1158315289 conn(0x7f8e2806e1a0 msgr2=0x7f8e2806e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.404+0000 7f8e27e00640 1 -- 172.21.15.29:0/1158315289 shutdown_connections 2024-04-17T19:55:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.404+0000 7f8e27e00640 1 -- 172.21.15.29:0/1158315289 wait complete. 2024-04-17T19:55:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.404+0000 7f8e27e00640 1 Processor -- start 2024-04-17T19:55:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.404+0000 7f8e27e00640 1 -- start start 2024-04-17T19:55:51.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e27e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e280cc020 0x7f8e281e6b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:51.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e27e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e28156140 0x7f8e28150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:51.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e27e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e2815f680 0x7f8e28146a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:51.403 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e27e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f8e28156550 con 0x7f8e280cc020 2024-04-17T19:55:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e27e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8e280cb900 con 0x7f8e28156140 2024-04-17T19:55:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e27e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f8e281566d0 con 0x7f8e2815f680 2024-04-17T19:55:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e26a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e280cc020 0x7f8e281e6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e26a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e280cc020 0x7f8e281e6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56102/0 (socket says 172.21.15.29:56102) 2024-04-17T19:55:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e26a00640 1 -- 172.21.15.29:0/3371127395 learned_addr learned my addr 172.21.15.29:0/3371127395 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:55:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e1fe00640 1 --2- 172.21.15.29:0/3371127395 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e28156140 0x7f8e28150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e27400640 1 --2- 172.21.15.29:0/3371127395 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e2815f680 0x7f8e28146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e26a00640 1 -- 172.21.15.29:0/3371127395 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e2815f680 msgr2=0x7f8e28146a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e26a00640 1 --2- 172.21.15.29:0/3371127395 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e2815f680 0x7f8e28146a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e26a00640 1 -- 172.21.15.29:0/3371127395 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e28156140 msgr2=0x7f8e28150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e26a00640 1 --2- 172.21.15.29:0/3371127395 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e28156140 0x7f8e28150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e26a00640 1 -- 172.21.15.29:0/3371127395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e281e7060 con 0x7f8e280cc020 2024-04-17T19:55:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e1fe00640 1 --2- 172.21.15.29:0/3371127395 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e28156140 0x7f8e28150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:55:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e26a00640 1 --2- 172.21.15.29:0/3371127395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e280cc020 0x7f8e281e6b20 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f8e1800c3a0 tx=0x7f8e1800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e1ea00640 1 -- 172.21.15.29:0/3371127395 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e180052d0 con 0x7f8e280cc020 2024-04-17T19:55:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e1ea00640 1 -- 172.21.15.29:0/3371127395 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e18005470 con 0x7f8e280cc020 2024-04-17T19:55:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e27e00640 1 -- 172.21.15.29:0/3371127395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e281e5980 con 0x7f8e280cc020 2024-04-17T19:55:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e1ea00640 1 -- 172.21.15.29:0/3371127395 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e18007260 con 0x7f8e280cc020 2024-04-17T19:55:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e27e00640 1 -- 172.21.15.29:0/3371127395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e281e5f40 con 0x7f8e280cc020 2024-04-17T19:55:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.408+0000 7f8e27e00640 1 -- 172.21.15.29:0/3371127395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e2804ff10 con 0x7f8e280cc020 2024-04-17T19:55:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.412+0000 7f8e1ea00640 1 -- 172.21.15.29:0/3371127395 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f8e18007400 con 0x7f8e280cc020 2024-04-17T19:55:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.412+0000 7f8e1ea00640 1 --2- 172.21.15.29:0/3371127395 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8e040683b0 0x7f8e0406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:55:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.412+0000 7f8e1ea00640 1 -- 172.21.15.29:0/3371127395 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 7255+0+0 (secure 0 0 0) 0x7f8e18086500 con 0x7f8e280cc020 2024-04-17T19:55:51.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.412+0000 7f8e1fe00640 1 --2- 172.21.15.29:0/3371127395 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8e040683b0 0x7f8e0406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:55:51.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.412+0000 7f8e1fe00640 1 --2- 172.21.15.29:0/3371127395 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8e040683b0 0x7f8e0406a8c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8e100b2000 tx=0x7f8e10008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:55:51.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.416+0000 7f8e1ea00640 1 -- 172.21.15.29:0/3371127395 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f8e18055d30 con 0x7f8e280cc020 2024-04-17T19:55:51.550 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:51.564 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:51.564 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:51.565 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:51.565 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:01 sshd 2024-04-17T19:55:51.565 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:51.565 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:51.565 INFO:tasks.workunit.client.0.smithi029.stdout: 39538 ? 00:00:00 ps 2024-04-17T19:55:51.565 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:51.618 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:51.620 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:51.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.628+0000 7f8e27e00640 1 -- 172.21.15.29:0/3371127395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e2806dd80 con 0x7f8e280cc020 2024-04-17T19:55:51.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.628+0000 7f8e1ea00640 1 -- 172.21.15.29:0/3371127395 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6904 (secure 0 0 0) 0x7f8e1805a030 con 0x7f8e280cc020 2024-04-17T19:55:51.625 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:55:51.625 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:40.944059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5144,4305,5212,4302]},"id":1}]} 2024-04-17T19:55:51.626 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 43 2024-04-17T19:55:51.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.632+0000 7f8e27e00640 1 -- 172.21.15.29:0/3371127395 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8e040683b0 msgr2=0x7f8e0406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:51.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.632+0000 7f8e27e00640 1 --2- 172.21.15.29:0/3371127395 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8e040683b0 0x7f8e0406a8c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8e100b2000 tx=0x7f8e10008040 comp rx=0 tx=0).stop 2024-04-17T19:55:51.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.632+0000 7f8e27e00640 1 -- 172.21.15.29:0/3371127395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e280cc020 msgr2=0x7f8e281e6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:55:51.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.632+0000 7f8e27e00640 1 --2- 172.21.15.29:0/3371127395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e280cc020 0x7f8e281e6b20 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f8e1800c3a0 tx=0x7f8e1800c860 comp rx=0 tx=0).stop 2024-04-17T19:55:51.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.632+0000 7f8e27e00640 1 -- 172.21.15.29:0/3371127395 shutdown_connections 2024-04-17T19:55:51.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.632+0000 7f8e27e00640 1 --2- 172.21.15.29:0/3371127395 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f8e040683b0 0x7f8e0406a8c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:51.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.632+0000 7f8e27e00640 1 --2- 172.21.15.29:0/3371127395 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f8e2815f680 0x7f8e28146a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:51.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.632+0000 7f8e27e00640 1 --2- 172.21.15.29:0/3371127395 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f8e28156140 0x7f8e28150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:51.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.632+0000 7f8e27e00640 1 --2- 172.21.15.29:0/3371127395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e280cc020 0x7f8e281e6b20 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:55:51.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.632+0000 7f8e27e00640 1 -- 172.21.15.29:0/3371127395 >> 172.21.15.29:0/3371127395 conn(0x7f8e2806e1a0 msgr2=0x7f8e28152e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:55:51.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.632+0000 7f8e27e00640 1 -- 172.21.15.29:0/3371127395 shutdown_connections 2024-04-17T19:55:51.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:55:51.632+0000 7f8e27e00640 1 -- 172.21.15.29:0/3371127395 wait complete. 2024-04-17T19:55:51.629 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:51.632 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:51.633 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:51.648 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:51.649 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2024-04-17T19:55:51.649 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 116 secs before thrashing 2024-04-17T19:55:51.657 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:51.658 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:51.871 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:51.899 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:51.899 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:51.900 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:51.900 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:01 sshd 2024-04-17T19:55:51.900 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:51.900 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:51.900 INFO:tasks.workunit.client.0.smithi029.stdout: 39550 ? 00:00:00 ps 2024-04-17T19:55:51.901 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:52.012 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:55:52.015 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:55:52.137 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:52.142 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:52.157 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:52.160 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:52.161 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:52.177 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:52.184 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:52.185 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:52.761 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:52.777 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:52.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:52.777 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:52.777 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:01 sshd 2024-04-17T19:55:52.777 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:52.777 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:52.777 INFO:tasks.workunit.client.0.smithi029.stdout: 39562 ? 00:00:00 ps 2024-04-17T19:55:52.777 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:52.794 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:52.797 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:52.805 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:52.808 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:52.810 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:52.822 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:52.832 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:52.833 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:53.152 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:53.169 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:53.169 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:53.169 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:53.169 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:01 sshd 2024-04-17T19:55:53.169 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:53.169 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:53.169 INFO:tasks.workunit.client.0.smithi029.stdout: 39572 ? 00:00:00 ps 2024-04-17T19:55:53.170 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:53.196 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:53.198 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:53.205 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:53.208 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:53.209 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:53.221 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:53.890 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:53.890 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:54.236 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:54.251 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:54.251 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:54.251 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:54.251 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:02 sshd 2024-04-17T19:55:54.251 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:54.251 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:54.251 INFO:tasks.workunit.client.0.smithi029.stdout: 39582 ? 00:00:00 ps 2024-04-17T19:55:54.252 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:54.271 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:54.273 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:54.280 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:54.283 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:54.284 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:54.297 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:54.306 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:54.307 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:54.570 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:54.584 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:54.585 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:54.585 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:54.585 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:02 sshd 2024-04-17T19:55:54.585 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:54.585 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:54.585 INFO:tasks.workunit.client.0.smithi029.stdout: 39592 ? 00:00:00 ps 2024-04-17T19:55:54.585 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:55.041 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:55.045 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:55.078 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a 2024-04-17T19:55:55.081 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/one 2024-04-17T19:55:55.083 INFO:tasks.workunit.client.0.smithi029.stderr:+ cat 2024-04-17T19:55:55.095 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir a/.snap/two 2024-04-17T19:55:55.103 INFO:tasks.workunit.client.0.smithi029.stderr:+ chmod 777 a/foo 2024-04-17T19:55:55.104 INFO:tasks.workunit.client.0.smithi029.stderr:+ sync 2024-04-17T19:55:55.421 INFO:tasks.workunit.client.0.smithi029.stderr:+ ps 2024-04-17T19:55:55.435 INFO:tasks.workunit.client.0.smithi029.stdout: PID TTY TIME CMD 2024-04-17T19:55:55.435 INFO:tasks.workunit.client.0.smithi029.stdout: 1242 ? 00:00:00 systemd 2024-04-17T19:55:55.435 INFO:tasks.workunit.client.0.smithi029.stdout: 1243 ? 00:00:00 (sd-pam) 2024-04-17T19:55:55.435 INFO:tasks.workunit.client.0.smithi029.stdout: 17123 ? 00:00:02 sshd 2024-04-17T19:55:55.436 INFO:tasks.workunit.client.0.smithi029.stdout: 38824 ? 00:00:00 timeout 2024-04-17T19:55:55.436 INFO:tasks.workunit.client.0.smithi029.stdout: 38827 ? 00:00:00 snaptest-double 2024-04-17T19:55:55.436 INFO:tasks.workunit.client.0.smithi029.stdout: 39602 ? 00:00:00 ps 2024-04-17T19:55:55.436 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a/.snap/one a/.snap/two 2024-04-17T19:55:55.453 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm a/foo 2024-04-17T19:55:55.456 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir a 2024-04-17T19:55:55.462 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo OK 2024-04-17T19:55:55.463 INFO:tasks.workunit.client.0.smithi029.stdout:OK 2024-04-17T19:55:55.463 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-17T19:55:55.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-17T19:55:55.494 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2024-04-17T19:55:55.494 DEBUG:teuthology.orchestra.run.smithi029:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f3a98d3f110319c567ec86bf70955c141495ccf6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2024-04-17T19:55:59.967 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/foo 2024-04-17T19:56:00.175 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo We want ENOENT, not ESTALE, here. 2024-04-17T19:56:00.175 INFO:tasks.workunit.client.0.smithi029.stdout:We want ENOENT, not ESTALE, here. 2024-04-17T19:56:00.176 INFO:tasks.workunit.client.0.smithi029.stderr:+ seq 1 100 2024-04-17T19:56:00.177 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/1 2024-04-17T19:56:00.177 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.179 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2024-04-17T19:56:00.180 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/2 2024-04-17T19:56:00.180 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.182 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2024-04-17T19:56:00.183 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/3 2024-04-17T19:56:00.183 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.184 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2024-04-17T19:56:00.185 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/4 2024-04-17T19:56:00.185 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.186 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2024-04-17T19:56:00.187 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/5 2024-04-17T19:56:00.187 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.189 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2024-04-17T19:56:00.189 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/6 2024-04-17T19:56:00.189 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.191 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2024-04-17T19:56:00.192 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/7 2024-04-17T19:56:00.192 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.193 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2024-04-17T19:56:00.194 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/8 2024-04-17T19:56:00.194 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.195 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2024-04-17T19:56:00.196 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/9 2024-04-17T19:56:00.196 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.197 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2024-04-17T19:56:00.198 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/10 2024-04-17T19:56:00.198 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.200 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2024-04-17T19:56:00.201 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/11 2024-04-17T19:56:00.201 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.202 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2024-04-17T19:56:00.203 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/12 2024-04-17T19:56:00.203 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.204 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2024-04-17T19:56:00.205 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/13 2024-04-17T19:56:00.205 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.207 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2024-04-17T19:56:00.207 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/14 2024-04-17T19:56:00.207 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.209 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2024-04-17T19:56:00.210 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/15 2024-04-17T19:56:00.210 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.211 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2024-04-17T19:56:00.212 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/16 2024-04-17T19:56:00.212 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.213 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2024-04-17T19:56:00.214 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/17 2024-04-17T19:56:00.214 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.215 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2024-04-17T19:56:00.216 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/18 2024-04-17T19:56:00.216 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.218 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2024-04-17T19:56:00.218 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/19 2024-04-17T19:56:00.219 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.220 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2024-04-17T19:56:00.221 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/20 2024-04-17T19:56:00.221 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.222 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2024-04-17T19:56:00.223 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/21 2024-04-17T19:56:00.223 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.224 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2024-04-17T19:56:00.225 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/22 2024-04-17T19:56:00.225 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.227 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2024-04-17T19:56:00.227 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/23 2024-04-17T19:56:00.227 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.229 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2024-04-17T19:56:00.230 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/24 2024-04-17T19:56:00.230 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.231 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2024-04-17T19:56:00.232 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/25 2024-04-17T19:56:00.232 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.233 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2024-04-17T19:56:00.234 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/26 2024-04-17T19:56:00.234 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.235 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2024-04-17T19:56:00.236 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/27 2024-04-17T19:56:00.236 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.238 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2024-04-17T19:56:00.239 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/28 2024-04-17T19:56:00.239 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.240 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2024-04-17T19:56:00.241 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/29 2024-04-17T19:56:00.241 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.242 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2024-04-17T19:56:00.243 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/30 2024-04-17T19:56:00.243 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.244 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2024-04-17T19:56:00.245 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/31 2024-04-17T19:56:00.245 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.247 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2024-04-17T19:56:00.248 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/32 2024-04-17T19:56:00.248 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.249 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2024-04-17T19:56:00.250 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/33 2024-04-17T19:56:00.250 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.251 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2024-04-17T19:56:00.252 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/34 2024-04-17T19:56:00.252 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.253 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2024-04-17T19:56:00.254 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/35 2024-04-17T19:56:00.254 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.256 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2024-04-17T19:56:00.256 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/36 2024-04-17T19:56:00.256 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.258 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2024-04-17T19:56:00.259 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/37 2024-04-17T19:56:00.259 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.260 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2024-04-17T19:56:00.261 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/38 2024-04-17T19:56:00.261 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.262 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2024-04-17T19:56:00.263 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/39 2024-04-17T19:56:00.263 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.264 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2024-04-17T19:56:00.265 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/40 2024-04-17T19:56:00.265 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.694 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2024-04-17T19:56:00.695 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/41 2024-04-17T19:56:00.696 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.974 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2024-04-17T19:56:00.975 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/42 2024-04-17T19:56:00.975 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.977 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2024-04-17T19:56:00.978 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/43 2024-04-17T19:56:00.978 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.979 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2024-04-17T19:56:00.980 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/44 2024-04-17T19:56:00.980 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.982 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2024-04-17T19:56:00.983 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/45 2024-04-17T19:56:00.983 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.984 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2024-04-17T19:56:00.985 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/46 2024-04-17T19:56:00.985 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.987 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2024-04-17T19:56:00.987 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/47 2024-04-17T19:56:00.988 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.989 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2024-04-17T19:56:00.990 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/48 2024-04-17T19:56:00.990 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.991 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2024-04-17T19:56:00.992 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/49 2024-04-17T19:56:00.992 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.994 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2024-04-17T19:56:00.995 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/50 2024-04-17T19:56:00.995 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.996 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2024-04-17T19:56:00.997 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/51 2024-04-17T19:56:00.997 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:00.999 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2024-04-17T19:56:01.000 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/52 2024-04-17T19:56:01.000 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.001 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2024-04-17T19:56:01.002 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/53 2024-04-17T19:56:01.002 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.004 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2024-04-17T19:56:01.005 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/54 2024-04-17T19:56:01.005 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.006 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2024-04-17T19:56:01.007 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/55 2024-04-17T19:56:01.007 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.009 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2024-04-17T19:56:01.010 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/56 2024-04-17T19:56:01.010 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.011 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2024-04-17T19:56:01.012 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/57 2024-04-17T19:56:01.012 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.014 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2024-04-17T19:56:01.015 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/58 2024-04-17T19:56:01.015 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.016 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2024-04-17T19:56:01.017 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/59 2024-04-17T19:56:01.017 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.019 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2024-04-17T19:56:01.020 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/60 2024-04-17T19:56:01.020 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.021 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2024-04-17T19:56:01.022 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/61 2024-04-17T19:56:01.022 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.023 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2024-04-17T19:56:01.024 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/62 2024-04-17T19:56:01.024 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.026 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2024-04-17T19:56:01.027 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/63 2024-04-17T19:56:01.027 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.028 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2024-04-17T19:56:01.029 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/64 2024-04-17T19:56:01.029 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.031 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2024-04-17T19:56:01.031 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/65 2024-04-17T19:56:01.032 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.033 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2024-04-17T19:56:01.034 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/66 2024-04-17T19:56:01.034 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.035 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2024-04-17T19:56:01.037 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/67 2024-04-17T19:56:01.037 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.038 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2024-04-17T19:56:01.038 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/68 2024-04-17T19:56:01.039 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.040 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2024-04-17T19:56:01.041 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/69 2024-04-17T19:56:01.041 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.042 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2024-04-17T19:56:01.043 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/70 2024-04-17T19:56:01.043 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.045 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2024-04-17T19:56:01.046 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/71 2024-04-17T19:56:01.046 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.047 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2024-04-17T19:56:01.048 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/72 2024-04-17T19:56:01.048 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.049 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2024-04-17T19:56:01.050 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/73 2024-04-17T19:56:01.050 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.052 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2024-04-17T19:56:01.052 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/74 2024-04-17T19:56:01.053 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.054 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2024-04-17T19:56:01.055 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/75 2024-04-17T19:56:01.055 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.056 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2024-04-17T19:56:01.057 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/76 2024-04-17T19:56:01.057 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.058 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2024-04-17T19:56:01.059 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/77 2024-04-17T19:56:01.059 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.061 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2024-04-17T19:56:01.061 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/78 2024-04-17T19:56:01.062 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.063 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2024-04-17T19:56:01.064 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/79 2024-04-17T19:56:01.064 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.065 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2024-04-17T19:56:01.066 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/80 2024-04-17T19:56:01.067 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.068 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2024-04-17T19:56:01.069 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/81 2024-04-17T19:56:01.069 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.070 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2024-04-17T19:56:01.071 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/82 2024-04-17T19:56:01.071 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.072 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2024-04-17T19:56:01.073 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/83 2024-04-17T19:56:01.073 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.075 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2024-04-17T19:56:01.076 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/84 2024-04-17T19:56:01.076 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.077 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2024-04-17T19:56:01.078 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/85 2024-04-17T19:56:01.078 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.079 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2024-04-17T19:56:01.080 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/86 2024-04-17T19:56:01.080 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.081 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2024-04-17T19:56:01.082 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/87 2024-04-17T19:56:01.082 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.084 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2024-04-17T19:56:01.085 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/88 2024-04-17T19:56:01.085 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.086 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2024-04-17T19:56:01.087 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/89 2024-04-17T19:56:01.087 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.088 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2024-04-17T19:56:01.089 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/90 2024-04-17T19:56:01.089 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.091 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2024-04-17T19:56:01.091 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/91 2024-04-17T19:56:01.091 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.093 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2024-04-17T19:56:01.094 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/92 2024-04-17T19:56:01.094 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.095 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2024-04-17T19:56:01.096 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/93 2024-04-17T19:56:01.096 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.097 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2024-04-17T19:56:01.098 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/94 2024-04-17T19:56:01.098 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.100 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2024-04-17T19:56:01.101 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/95 2024-04-17T19:56:01.101 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.102 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2024-04-17T19:56:01.103 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/96 2024-04-17T19:56:01.103 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.104 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2024-04-17T19:56:01.105 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/97 2024-04-17T19:56:01.105 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.106 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2024-04-17T19:56:01.107 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/98 2024-04-17T19:56:01.107 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.109 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2024-04-17T19:56:01.109 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/99 2024-04-17T19:56:01.109 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.111 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2024-04-17T19:56:01.112 INFO:tasks.workunit.client.0.smithi029.stderr:+ stat .snap/foo/100 2024-04-17T19:56:01.112 INFO:tasks.workunit.client.0.smithi029.stderr:+ grep No such file 2024-04-17T19:56:01.113 INFO:tasks.workunit.client.0.smithi029.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2024-04-17T19:56:01.113 INFO:tasks.workunit.client.0.smithi029.stderr:+ rmdir .snap/foo 2024-04-17T19:56:01.126 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo OK 2024-04-17T19:56:01.127 INFO:tasks.workunit.client.0.smithi029.stdout:OK 2024-04-17T19:56:01.127 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-17T19:56:01.127 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-17T19:56:01.153 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2024-04-17T19:56:01.153 DEBUG:teuthology.orchestra.run.smithi029:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f3a98d3f110319c567ec86bf70955c141495ccf6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2024-04-17T19:56:01.216 INFO:tasks.workunit.client.0.smithi029.stderr:+ set -e 2024-04-17T19:56:01.216 INFO:tasks.workunit.client.0.smithi029.stderr:+ sudo git config --global http.sslVerify false 2024-04-17T19:56:01.232 INFO:tasks.workunit.client.0.smithi029.stderr:+ sudo git config --global http.postBuffer 1048576000 2024-04-17T19:56:01.245 INFO:tasks.workunit.client.0.smithi029.stderr:+ retried=false 2024-04-17T19:56:01.245 INFO:tasks.workunit.client.0.smithi029.stderr:+ trap -- retry EXIT 2024-04-17T19:56:01.246 INFO:tasks.workunit.client.0.smithi029.stderr:+ rm -rf ceph 2024-04-17T19:56:01.246 INFO:tasks.workunit.client.0.smithi029.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2024-04-17T19:56:01.250 INFO:tasks.workunit.client.0.smithi029.stderr:Cloning into 'ceph'... 2024-04-17T19:56:22.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:56:22.080 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:56:52.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:56:52.136 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:57:22.189 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:57:22.193 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:57:47.651 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:57:47.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.806+0000 7f34c2000640 1 Processor -- start 2024-04-17T19:57:47.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.806+0000 7f34c2000640 1 -- start start 2024-04-17T19:57:47.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.806+0000 7f34c2000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f34bc159c90 0x7f34bc0ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:47.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.806+0000 7f34c2000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34bc06c660 0x7f34bc06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:47.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.806+0000 7f34c2000640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34bc15a0a0 con 0x7f34bc0ca7f0 2024-04-17T19:57:47.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.806+0000 7f34c2000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f34bc15abe0 con 0x7f34bc06c660 2024-04-17T19:57:47.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.806+0000 7f34c2000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f34bc15a340 con 0x7f34bc159c90 2024-04-17T19:57:47.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.810+0000 7f34bbe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34bc06c660 0x7f34bc06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:47.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.810+0000 7f34bbe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34bc06c660 0x7f34bc06ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60866/0 (socket says 172.21.15.29:60866) 2024-04-17T19:57:47.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.810+0000 7f34bbe00640 1 -- 172.21.15.29:0/2104835630 learned_addr learned my addr 172.21.15.29:0/2104835630 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:57:47.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.810+0000 7f34bb400640 1 --2- 172.21.15.29:0/2104835630 >> v2:172.21.15.37:3300/0 conn(0x7f34bc159c90 0x7f34bc0ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:47.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.810+0000 7f34bbe00640 1 -- 172.21.15.29:0/2104835630 >> v2:172.21.15.37:3300/0 conn(0x7f34bc159c90 msgr2=0x7f34bc0ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:57:47.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.810+0000 7f34bbe00640 1 --2- 172.21.15.29:0/2104835630 >> v2:172.21.15.37:3300/0 conn(0x7f34bc159c90 0x7f34bc0ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:47.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.810+0000 7f34bbe00640 1 -- 172.21.15.29:0/2104835630 >> v1:172.21.15.29:6789/0 conn(0x7f34bc0ca7f0 legacy=0x7f34bc0c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:47.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.810+0000 7f34bbe00640 1 -- 172.21.15.29:0/2104835630 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34bc15a620 con 0x7f34bc06c660 2024-04-17T19:57:47.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.810+0000 7f34baa00640 1 -- 172.21.15.29:0/2104835630 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4154019376 0 0) 0x7f34bc15a0a0 con 0x7f34bc0ca7f0 2024-04-17T19:57:47.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.810+0000 7f34bbe00640 1 --2- 172.21.15.29:0/2104835630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34bc06c660 0x7f34bc06ca70 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f34bc071f40 tx=0x7f34b401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1e6292ad3a15fd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:57:47.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.810+0000 7f34baa00640 1 -- 172.21.15.29:0/2104835630 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b40a57d0 con 0x7f34bc06c660 2024-04-17T19:57:47.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.810+0000 7f34baa00640 1 -- 172.21.15.29:0/2104835630 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34b40a5970 con 0x7f34bc06c660 2024-04-17T19:57:47.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.810+0000 7f34baa00640 1 -- 172.21.15.29:0/2104835630 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b40a5d40 con 0x7f34bc06c660 2024-04-17T19:57:47.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.814+0000 7f34c2000640 1 -- 172.21.15.29:0/2104835630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34bc06c660 msgr2=0x7f34bc06ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:47.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.814+0000 7f34c2000640 1 --2- 172.21.15.29:0/2104835630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34bc06c660 0x7f34bc06ca70 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f34bc071f40 tx=0x7f34b401c9f0 comp rx=0 tx=0).stop 2024-04-17T19:57:47.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.814+0000 7f34c2000640 1 -- 172.21.15.29:0/2104835630 shutdown_connections 2024-04-17T19:57:47.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.814+0000 7f34c2000640 1 --2- 172.21.15.29:0/2104835630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34bc06c660 0x7f34bc06ca70 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:47.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.814+0000 7f34c2000640 1 --2- 172.21.15.29:0/2104835630 >> v2:172.21.15.37:3300/0 conn(0x7f34bc159c90 0x7f34bc0ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:47.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.814+0000 7f34c2000640 1 -- 172.21.15.29:0/2104835630 >> 172.21.15.29:0/2104835630 conn(0x7f34bc06fde0 msgr2=0x7f34bc0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:57:47.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.814+0000 7f34c2000640 1 -- 172.21.15.29:0/2104835630 shutdown_connections 2024-04-17T19:57:47.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.814+0000 7f34c2000640 1 -- 172.21.15.29:0/2104835630 wait complete. 2024-04-17T19:57:47.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.814+0000 7f34c2000640 1 Processor -- start 2024-04-17T19:57:47.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.814+0000 7f34c2000640 1 -- start start 2024-04-17T19:57:47.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34c2000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34bc06c660 0x7f34bc14f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:47.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34c2000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34bc0ca7f0 0x7f34bc1f7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:47.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34c2000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34bc159c90 0x7f34bc147e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:47.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34c2000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f34bc0c7680 con 0x7f34bc06c660 2024-04-17T19:57:47.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34c2000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f34bc06cfb0 con 0x7f34bc159c90 2024-04-17T19:57:47.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34c2000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f34bc0c7800 con 0x7f34bc0ca7f0 2024-04-17T19:57:47.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34bb400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34bc06c660 0x7f34bc14f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:47.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34bb400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34bc06c660 0x7f34bc14f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60870/0 (socket says 172.21.15.29:60870) 2024-04-17T19:57:47.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34bb400640 1 -- 172.21.15.29:0/1804109201 learned_addr learned my addr 172.21.15.29:0/1804109201 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:57:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34bbe00640 1 --2- 172.21.15.29:0/1804109201 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34bc159c90 0x7f34bc147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34b3e00640 1 --2- 172.21.15.29:0/1804109201 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34bc0ca7f0 0x7f34bc1f7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34bb400640 1 -- 172.21.15.29:0/1804109201 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34bc0ca7f0 msgr2=0x7f34bc1f7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34bb400640 1 --2- 172.21.15.29:0/1804109201 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34bc0ca7f0 0x7f34bc1f7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34bb400640 1 -- 172.21.15.29:0/1804109201 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34bc159c90 msgr2=0x7f34bc147e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34bb400640 1 --2- 172.21.15.29:0/1804109201 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34bc159c90 0x7f34bc147e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34bb400640 1 -- 172.21.15.29:0/1804109201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34bc1f79c0 con 0x7f34bc06c660 2024-04-17T19:57:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34b3e00640 1 --2- 172.21.15.29:0/1804109201 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34bc0ca7f0 0x7f34bc1f7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T19:57:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34bb400640 1 --2- 172.21.15.29:0/1804109201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34bc06c660 0x7f34bc14f3c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00e870 tx=0x7f34ac00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:57:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34b3400640 1 -- 172.21.15.29:0/1804109201 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac017070 con 0x7f34bc06c660 2024-04-17T19:57:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34b3400640 1 -- 172.21.15.29:0/1804109201 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34ac00c8e0 con 0x7f34bc06c660 2024-04-17T19:57:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34b3400640 1 -- 172.21.15.29:0/1804109201 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac00ccb0 con 0x7f34bc06c660 2024-04-17T19:57:47.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34c2000640 1 -- 172.21.15.29:0/1804109201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34bc1f44b0 con 0x7f34bc06c660 2024-04-17T19:57:47.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34c2000640 1 -- 172.21.15.29:0/1804109201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34bc1f39d0 con 0x7f34bc06c660 2024-04-17T19:57:47.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34c2000640 1 -- 172.21.15.29:0/1804109201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34bc069b90 con 0x7f34bc06c660 2024-04-17T19:57:47.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.818+0000 7f34b3400640 1 -- 172.21.15.29:0/1804109201 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f34ac01a070 con 0x7f34bc06c660 2024-04-17T19:57:47.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.822+0000 7f34b3400640 1 --2- 172.21.15.29:0/1804109201 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f34980683b0 0x7f349806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:47.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.822+0000 7f34b3400640 1 -- 172.21.15.29:0/1804109201 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7135+0+0 (secure 0 0 0) 0x7f34ac086e80 con 0x7f34bc06c660 2024-04-17T19:57:47.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.822+0000 7f34b3e00640 1 --2- 172.21.15.29:0/1804109201 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f34980683b0 0x7f349806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:47.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.822+0000 7f34b3e00640 1 --2- 172.21.15.29:0/1804109201 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f34980683b0 0x7f349806a8c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f34a4000f70 tx=0x7f34a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:57:47.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:47.826+0000 7f34b3400640 1 -- 172.21.15.29:0/1804109201 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f34ac056d80 con 0x7f34bc06c660 2024-04-17T19:57:48.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.034+0000 7f34c2000640 1 -- 172.21.15.29:0/1804109201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34bc14a800 con 0x7f34bc06c660 2024-04-17T19:57:48.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.038+0000 7f34b3400640 1 -- 172.21.15.29:0/1804109201 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6904 (secure 0 0 0) 0x7f34ac056680 con 0x7f34bc06c660 2024-04-17T19:57:48.035 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:57:48.035 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:40.944059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5144,4305,5212,4302]},"id":1}]} 2024-04-17T19:57:48.036 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 43 2024-04-17T19:57:48.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.038+0000 7f34c2000640 1 -- 172.21.15.29:0/1804109201 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f34980683b0 msgr2=0x7f349806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.038+0000 7f34c2000640 1 --2- 172.21.15.29:0/1804109201 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f34980683b0 0x7f349806a8c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f34a4000f70 tx=0x7f34a4009040 comp rx=0 tx=0).stop 2024-04-17T19:57:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.038+0000 7f34c2000640 1 -- 172.21.15.29:0/1804109201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34bc06c660 msgr2=0x7f34bc14f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.038+0000 7f34c2000640 1 --2- 172.21.15.29:0/1804109201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34bc06c660 0x7f34bc14f3c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00e870 tx=0x7f34ac00ec70 comp rx=0 tx=0).stop 2024-04-17T19:57:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.038+0000 7f34c2000640 1 -- 172.21.15.29:0/1804109201 shutdown_connections 2024-04-17T19:57:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.038+0000 7f34c2000640 1 --2- 172.21.15.29:0/1804109201 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f34980683b0 0x7f349806a8c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.038+0000 7f34c2000640 1 --2- 172.21.15.29:0/1804109201 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f34bc159c90 0x7f34bc147e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.038+0000 7f34c2000640 1 --2- 172.21.15.29:0/1804109201 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f34bc0ca7f0 0x7f34bc1f7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.038+0000 7f34c2000640 1 --2- 172.21.15.29:0/1804109201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34bc06c660 0x7f34bc14f3c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.038+0000 7f34c2000640 1 -- 172.21.15.29:0/1804109201 >> 172.21.15.29:0/1804109201 conn(0x7f34bc06fde0 msgr2=0x7f34bc0cb530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:57:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.038+0000 7f34c2000640 1 -- 172.21.15.29:0/1804109201 shutdown_connections 2024-04-17T19:57:48.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.038+0000 7f34c2000640 1 -- 172.21.15.29:0/1804109201 wait complete. 2024-04-17T19:57:48.066 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=1) 2024-04-17T19:57:48.067 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-04-17T19:57:48.067 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T19:57:48.083 INFO:tasks.ceph.mds.e:Stopped 2024-04-17T19:57:48.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:57:48.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.274+0000 7f59b9400640 1 Processor -- start 2024-04-17T19:57:48.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.274+0000 7f59b9400640 1 -- start start 2024-04-17T19:57:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.278+0000 7f59b9400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f59b40c9b20 0x7f59b41560d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.278+0000 7f59b9400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f59b415c540 0x7f59b406e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:48.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.278+0000 7f59b9400640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59b40c55e0 con 0x7f59b406e300 2024-04-17T19:57:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.278+0000 7f59b9400640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f59b40c8fa0 con 0x7f59b40c9b20 2024-04-17T19:57:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.278+0000 7f59b9400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f59b40c9f30 con 0x7f59b415c540 2024-04-17T19:57:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.278+0000 7f59b2800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f59b40c9b20 0x7f59b41560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.278+0000 7f59b2800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f59b40c9b20 0x7f59b41560d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60878/0 (socket says 172.21.15.29:60878) 2024-04-17T19:57:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.278+0000 7f59b2800640 1 -- 172.21.15.29:0/1302539321 learned_addr learned my addr 172.21.15.29:0/1302539321 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:57:48.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.278+0000 7f59b3200640 1 --2- 172.21.15.29:0/1302539321 >> v2:172.21.15.37:3300/0 conn(0x7f59b415c540 0x7f59b406e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:48.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.278+0000 7f59b2800640 1 -- 172.21.15.29:0/1302539321 >> v2:172.21.15.37:3300/0 conn(0x7f59b415c540 msgr2=0x7f59b406e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:57:48.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.278+0000 7f59b2800640 1 --2- 172.21.15.29:0/1302539321 >> v2:172.21.15.37:3300/0 conn(0x7f59b415c540 0x7f59b406e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:48.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.278+0000 7f59b2800640 1 -- 172.21.15.29:0/1302539321 >> v1:172.21.15.37:6789/0 conn(0x7f59b406e300 legacy=0x7f59b415be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:48.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.278+0000 7f59b2800640 1 -- 172.21.15.29:0/1302539321 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b40ca1d0 con 0x7f59b40c9b20 2024-04-17T19:57:48.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.278+0000 7f59b3200640 1 --2- 172.21.15.29:0/1302539321 >> v2:172.21.15.37:3300/0 conn(0x7f59b415c540 0x7f59b406e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:57:48.278 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.278+0000 7f59b2800640 1 --2- 172.21.15.29:0/1302539321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59b40c9b20 0x7f59b41560d0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f59b4072040 tx=0x7f59a401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2523ed8c9ef23982 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:57:48.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.282+0000 7f59b1e00640 1 -- 172.21.15.29:0/1302539321 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a40b5070 con 0x7f59b40c9b20 2024-04-17T19:57:48.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.282+0000 7f59b1e00640 1 -- 172.21.15.29:0/1302539321 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59a40a8a10 con 0x7f59b40c9b20 2024-04-17T19:57:48.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.282+0000 7f59b1e00640 1 -- 172.21.15.29:0/1302539321 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a40a8de0 con 0x7f59b40c9b20 2024-04-17T19:57:48.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.282+0000 7f59b9400640 1 -- 172.21.15.29:0/1302539321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59b40c9b20 msgr2=0x7f59b41560d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:48.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.282+0000 7f59b9400640 1 --2- 172.21.15.29:0/1302539321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59b40c9b20 0x7f59b41560d0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f59b4072040 tx=0x7f59a401ed10 comp rx=0 tx=0).stop 2024-04-17T19:57:48.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.282+0000 7f59b9400640 1 -- 172.21.15.29:0/1302539321 shutdown_connections 2024-04-17T19:57:48.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.282+0000 7f59b9400640 1 --2- 172.21.15.29:0/1302539321 >> v2:172.21.15.37:3300/0 conn(0x7f59b415c540 0x7f59b406e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:48.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.282+0000 7f59b9400640 1 --2- 172.21.15.29:0/1302539321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59b40c9b20 0x7f59b41560d0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:48.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.282+0000 7f59b9400640 1 -- 172.21.15.29:0/1302539321 >> 172.21.15.29:0/1302539321 conn(0x7f59b406c3c0 msgr2=0x7f59b40ca3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:57:48.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.282+0000 7f59b9400640 1 -- 172.21.15.29:0/1302539321 shutdown_connections 2024-04-17T19:57:48.281 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.282+0000 7f59b9400640 1 -- 172.21.15.29:0/1302539321 wait complete. 2024-04-17T19:57:48.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.282+0000 7f59b9400640 1 Processor -- start 2024-04-17T19:57:48.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.282+0000 7f59b9400640 1 -- start start 2024-04-17T19:57:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59b9400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59b406e300 0x7f59b414d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59b9400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59b40c9b20 0x7f59b41f6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59b9400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f59b415c540 0x7f59b41e6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59b9400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f59b40c8670 con 0x7f59b40c9b20 2024-04-17T19:57:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59b9400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f59b40c8390 con 0x7f59b406e300 2024-04-17T19:57:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59b9400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f59b40c8950 con 0x7f59b415c540 2024-04-17T19:57:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59a9e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59b40c9b20 0x7f59b41f6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:48.285 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59a9e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59b40c9b20 0x7f59b41f6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60892/0 (socket says 172.21.15.29:60892) 2024-04-17T19:57:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59a9e00640 1 -- 172.21.15.29:0/3731103165 learned_addr learned my addr 172.21.15.29:0/3731103165 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:57:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59b2800640 1 --2- 172.21.15.29:0/3731103165 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59b406e300 0x7f59b414d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59b3200640 1 --2- 172.21.15.29:0/3731103165 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f59b415c540 0x7f59b41e6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59a9e00640 1 -- 172.21.15.29:0/3731103165 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f59b415c540 msgr2=0x7f59b41e6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59a9e00640 1 --2- 172.21.15.29:0/3731103165 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f59b415c540 0x7f59b41e6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59a9e00640 1 -- 172.21.15.29:0/3731103165 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59b406e300 msgr2=0x7f59b414d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59a9e00640 1 --2- 172.21.15.29:0/3731103165 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59b406e300 0x7f59b414d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59a9e00640 1 -- 172.21.15.29:0/3731103165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59b414feb0 con 0x7f59b40c9b20 2024-04-17T19:57:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59a9e00640 1 --2- 172.21.15.29:0/3731103165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59b40c9b20 0x7f59b41f6760 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f599c00c370 tx=0x7f599c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:57:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59ab400640 1 -- 172.21.15.29:0/3731103165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599c0052d0 con 0x7f59b40c9b20 2024-04-17T19:57:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59ab400640 1 -- 172.21.15.29:0/3731103165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f599c005470 con 0x7f59b40c9b20 2024-04-17T19:57:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59ab400640 1 -- 172.21.15.29:0/3731103165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599c007260 con 0x7f59b40c9b20 2024-04-17T19:57:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59b9400640 1 -- 172.21.15.29:0/3731103165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59b41f3740 con 0x7f59b40c9b20 2024-04-17T19:57:48.286 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59b9400640 1 -- 172.21.15.29:0/3731103165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59b41f2c60 con 0x7f59b40c9b20 2024-04-17T19:57:48.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.286+0000 7f59b9400640 1 -- 172.21.15.29:0/3731103165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59b414fc20 con 0x7f59b40c9b20 2024-04-17T19:57:48.287 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.290+0000 7f59ab400640 1 -- 172.21.15.29:0/3731103165 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f599c007400 con 0x7f59b40c9b20 2024-04-17T19:57:48.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.290+0000 7f59ab400640 1 --2- 172.21.15.29:0/3731103165 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f59900683b0 0x7f599006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:48.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.290+0000 7f59ab400640 1 -- 172.21.15.29:0/3731103165 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7135+0+0 (secure 0 0 0) 0x7f599c0862e0 con 0x7f59b40c9b20 2024-04-17T19:57:48.288 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.290+0000 7f59b2800640 1 --2- 172.21.15.29:0/3731103165 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f59900683b0 0x7f599006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:48.289 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.290+0000 7f59b2800640 1 --2- 172.21.15.29:0/3731103165 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f59900683b0 0x7f599006a8c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f59a40b2000 tx=0x7f59a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:57:48.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.294+0000 7f59ab400640 1 -- 172.21.15.29:0/3731103165 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f59b40c55e0 con 0x7f59b40c9b20 2024-04-17T19:57:48.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.490+0000 7f59b9400640 1 -- 172.21.15.29:0/3731103165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59b4069b90 con 0x7f59b40c9b20 2024-04-17T19:57:48.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.494+0000 7f59ab400640 1 -- 172.21.15.29:0/3731103165 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6904 (secure 0 0 0) 0x7f599c055cc0 con 0x7f59b40c9b20 2024-04-17T19:57:48.491 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:57:48.491 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:40.944059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5144,4305,5212,4302]},"id":1}]} 2024-04-17T19:57:48.492 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 43 2024-04-17T19:57:48.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.494+0000 7f59b9400640 1 -- 172.21.15.29:0/3731103165 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f59900683b0 msgr2=0x7f599006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:48.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.494+0000 7f59b9400640 1 --2- 172.21.15.29:0/3731103165 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f59900683b0 0x7f599006a8c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f59a40b2000 tx=0x7f59a4009040 comp rx=0 tx=0).stop 2024-04-17T19:57:48.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.494+0000 7f59b9400640 1 -- 172.21.15.29:0/3731103165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59b40c9b20 msgr2=0x7f59b41f6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.494+0000 7f59b9400640 1 --2- 172.21.15.29:0/3731103165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59b40c9b20 0x7f59b41f6760 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f599c00c370 tx=0x7f599c00c830 comp rx=0 tx=0).stop 2024-04-17T19:57:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.494+0000 7f59b9400640 1 -- 172.21.15.29:0/3731103165 shutdown_connections 2024-04-17T19:57:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.494+0000 7f59b9400640 1 --2- 172.21.15.29:0/3731103165 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f59900683b0 0x7f599006a8c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.494+0000 7f59b9400640 1 --2- 172.21.15.29:0/3731103165 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f59b415c540 0x7f59b41e6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.494+0000 7f59b9400640 1 --2- 172.21.15.29:0/3731103165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59b40c9b20 0x7f59b41f6760 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.494+0000 7f59b9400640 1 --2- 172.21.15.29:0/3731103165 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59b406e300 0x7f59b414d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.494+0000 7f59b9400640 1 -- 172.21.15.29:0/3731103165 >> 172.21.15.29:0/3731103165 conn(0x7f59b406c3c0 msgr2=0x7f59b41e7280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:57:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.494+0000 7f59b9400640 1 -- 172.21.15.29:0/3731103165 shutdown_connections 2024-04-17T19:57:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:48.494+0000 7f59b9400640 1 -- 172.21.15.29:0/3731103165 wait complete. 2024-04-17T19:57:48.513 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-17T19:57:50.514 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:57:50.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.670+0000 7efc95200640 1 Processor -- start 2024-04-17T19:57:50.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.670+0000 7efc95200640 1 -- start start 2024-04-17T19:57:50.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.674+0000 7efc95200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7efc900c6f70 0x7efc900c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:50.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.674+0000 7efc95200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efc9006c660 0x7efc9006ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:50.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.674+0000 7efc95200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc9015a0a0 con 0x7efc90159c90 2024-04-17T19:57:50.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.674+0000 7efc95200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7efc9015abe0 con 0x7efc9006c660 2024-04-17T19:57:50.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.674+0000 7efc95200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7efc9015a340 con 0x7efc900c6f70 2024-04-17T19:57:50.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.674+0000 7efc8de00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7efc90159c90 0x7efc900ca2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33990/0 (socket says 172.21.15.29:33990) 2024-04-17T19:57:50.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.674+0000 7efc8de00640 1 -- 172.21.15.29:0/932039128 learned_addr learned my addr 172.21.15.29:0/932039128 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:57:50.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.674+0000 7efc8e800640 1 --2- 172.21.15.29:0/932039128 >> v2:172.21.15.29:3300/0 conn(0x7efc9006c660 0x7efc9006ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:50.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.674+0000 7efc87e00640 1 --2- 172.21.15.29:0/932039128 >> v2:172.21.15.37:3300/0 conn(0x7efc900c6f70 0x7efc900c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:50.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.674+0000 7efc8e800640 1 -- 172.21.15.29:0/932039128 >> v2:172.21.15.37:3300/0 conn(0x7efc900c6f70 msgr2=0x7efc900c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:57:50.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.674+0000 7efc8e800640 1 --2- 172.21.15.29:0/932039128 >> v2:172.21.15.37:3300/0 conn(0x7efc900c6f70 0x7efc900c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:50.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.674+0000 7efc8e800640 1 -- 172.21.15.29:0/932039128 >> v1:172.21.15.29:6789/0 conn(0x7efc90159c90 legacy=0x7efc900ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:50.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.674+0000 7efc8d400640 1 -- 172.21.15.29:0/932039128 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1483548039 0 0) 0x7efc9015a0a0 con 0x7efc90159c90 2024-04-17T19:57:50.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.674+0000 7efc8e800640 1 -- 172.21.15.29:0/932039128 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc9015a620 con 0x7efc9006c660 2024-04-17T19:57:50.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.678+0000 7efc8e800640 1 --2- 172.21.15.29:0/932039128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc9006c660 0x7efc9006ca70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7efc900cd310 tx=0x7efc8801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=335f1cfe0555136a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:57:50.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.678+0000 7efc8d400640 1 -- 172.21.15.29:0/932039128 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc880b2070 con 0x7efc9006c660 2024-04-17T19:57:50.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.678+0000 7efc8d400640 1 -- 172.21.15.29:0/932039128 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc880a57f0 con 0x7efc9006c660 2024-04-17T19:57:50.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.678+0000 7efc8d400640 1 -- 172.21.15.29:0/932039128 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc880a5bc0 con 0x7efc9006c660 2024-04-17T19:57:50.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.678+0000 7efc95200640 1 -- 172.21.15.29:0/932039128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc9006c660 msgr2=0x7efc9006ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:50.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.678+0000 7efc95200640 1 --2- 172.21.15.29:0/932039128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc9006c660 0x7efc9006ca70 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7efc900cd310 tx=0x7efc8801c930 comp rx=0 tx=0).stop 2024-04-17T19:57:50.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.678+0000 7efc95200640 1 -- 172.21.15.29:0/932039128 shutdown_connections 2024-04-17T19:57:50.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.678+0000 7efc95200640 1 --2- 172.21.15.29:0/932039128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc9006c660 0x7efc9006ca70 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:50.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.678+0000 7efc95200640 1 --2- 172.21.15.29:0/932039128 >> v2:172.21.15.37:3300/0 conn(0x7efc900c6f70 0x7efc900c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:50.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.678+0000 7efc95200640 1 -- 172.21.15.29:0/932039128 >> 172.21.15.29:0/932039128 conn(0x7efc9006fde0 msgr2=0x7efc900701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:57:50.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.678+0000 7efc95200640 1 -- 172.21.15.29:0/932039128 shutdown_connections 2024-04-17T19:57:50.677 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.678+0000 7efc95200640 1 -- 172.21.15.29:0/932039128 wait complete. 2024-04-17T19:57:50.678 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.678+0000 7efc95200640 1 Processor -- start 2024-04-17T19:57:50.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.678+0000 7efc95200640 1 -- start start 2024-04-17T19:57:50.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc95200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc9006c660 0x7efc901f89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:50.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc95200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc900c6f70 0x7efc901517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:50.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc95200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc90159c90 0x7efc901639a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:50.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc95200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efc900ca9c0 con 0x7efc9006c660 2024-04-17T19:57:50.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc95200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7efc9006cfb0 con 0x7efc90159c90 2024-04-17T19:57:50.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc95200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7efc900cab40 con 0x7efc900c6f70 2024-04-17T19:57:50.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc8de00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc9006c660 0x7efc901f89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:50.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc8de00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc9006c660 0x7efc901f89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46092/0 (socket says 172.21.15.29:46092) 2024-04-17T19:57:50.681 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc8de00640 1 -- 172.21.15.29:0/2265089717 learned_addr learned my addr 172.21.15.29:0/2265089717 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:57:50.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc8e800640 1 --2- 172.21.15.29:0/2265089717 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc90159c90 0x7efc901639a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:50.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc87e00640 1 --2- 172.21.15.29:0/2265089717 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc900c6f70 0x7efc901517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:50.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc8de00640 1 -- 172.21.15.29:0/2265089717 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc900c6f70 msgr2=0x7efc901517d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:50.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc8de00640 1 --2- 172.21.15.29:0/2265089717 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc900c6f70 0x7efc901517d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:50.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc8de00640 1 -- 172.21.15.29:0/2265089717 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc90159c90 msgr2=0x7efc901639a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:50.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc8de00640 1 --2- 172.21.15.29:0/2265089717 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc90159c90 0x7efc901639a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:50.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc8de00640 1 -- 172.21.15.29:0/2265089717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc901f8f30 con 0x7efc9006c660 2024-04-17T19:57:50.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc8e800640 1 --2- 172.21.15.29:0/2265089717 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc90159c90 0x7efc901639a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:57:50.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc87e00640 1 --2- 172.21.15.29:0/2265089717 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc900c6f70 0x7efc901517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:57:50.682 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc8de00640 1 --2- 172.21.15.29:0/2265089717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc9006c660 0x7efc901f89f0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7efc8000c410 tx=0x7efc8000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:57:50.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc86000640 1 -- 172.21.15.29:0/2265089717 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc80017070 con 0x7efc9006c660 2024-04-17T19:57:50.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc86000640 1 -- 172.21.15.29:0/2265089717 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc80005cf0 con 0x7efc9006c660 2024-04-17T19:57:50.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc95200640 1 -- 172.21.15.29:0/2265089717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc9014e4c0 con 0x7efc9006c660 2024-04-17T19:57:50.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc86000640 1 -- 172.21.15.29:0/2265089717 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc800072f0 con 0x7efc9006c660 2024-04-17T19:57:50.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc95200640 1 -- 172.21.15.29:0/2265089717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc9014ea80 con 0x7efc9006c660 2024-04-17T19:57:50.683 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.682+0000 7efc95200640 1 -- 172.21.15.29:0/2265089717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc90069b90 con 0x7efc9006c660 2024-04-17T19:57:50.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.686+0000 7efc86000640 1 -- 172.21.15.29:0/2265089717 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efc8001a070 con 0x7efc9006c660 2024-04-17T19:57:50.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.686+0000 7efc86000640 1 --2- 172.21.15.29:0/2265089717 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efc6c068380 0x7efc6c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:50.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.686+0000 7efc86000640 1 -- 172.21.15.29:0/2265089717 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7135+0+0 (secure 0 0 0) 0x7efc800863b0 con 0x7efc9006c660 2024-04-17T19:57:50.684 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.686+0000 7efc87e00640 1 --2- 172.21.15.29:0/2265089717 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efc6c068380 0x7efc6c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:50.685 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.686+0000 7efc87e00640 1 --2- 172.21.15.29:0/2265089717 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efc6c068380 0x7efc6c06a890 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7efc90153cf0 tx=0x7efc78008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:57:50.689 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.690+0000 7efc86000640 1 -- 172.21.15.29:0/2265089717 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7efc80056340 con 0x7efc9006c660 2024-04-17T19:57:50.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.894+0000 7efc95200640 1 -- 172.21.15.29:0/2265089717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efc90151d10 con 0x7efc9006c660 2024-04-17T19:57:50.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.898+0000 7efc86000640 1 -- 172.21.15.29:0/2265089717 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6904 (secure 0 0 0) 0x7efc80055c40 con 0x7efc9006c660 2024-04-17T19:57:50.896 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:57:50.896 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:40.944059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5144,4305,5212,4302]},"id":1}]} 2024-04-17T19:57:50.897 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 43 2024-04-17T19:57:50.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.898+0000 7efc95200640 1 -- 172.21.15.29:0/2265089717 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efc6c068380 msgr2=0x7efc6c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:50.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.898+0000 7efc95200640 1 --2- 172.21.15.29:0/2265089717 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efc6c068380 0x7efc6c06a890 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7efc90153cf0 tx=0x7efc78008040 comp rx=0 tx=0).stop 2024-04-17T19:57:50.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.898+0000 7efc95200640 1 -- 172.21.15.29:0/2265089717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc9006c660 msgr2=0x7efc901f89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:50.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.898+0000 7efc95200640 1 --2- 172.21.15.29:0/2265089717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc9006c660 0x7efc901f89f0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7efc8000c410 tx=0x7efc8000c8d0 comp rx=0 tx=0).stop 2024-04-17T19:57:50.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.898+0000 7efc95200640 1 -- 172.21.15.29:0/2265089717 shutdown_connections 2024-04-17T19:57:50.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.898+0000 7efc95200640 1 --2- 172.21.15.29:0/2265089717 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efc6c068380 0x7efc6c06a890 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:50.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.898+0000 7efc95200640 1 --2- 172.21.15.29:0/2265089717 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efc90159c90 0x7efc901639a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:50.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.898+0000 7efc95200640 1 --2- 172.21.15.29:0/2265089717 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efc900c6f70 0x7efc901517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:50.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.898+0000 7efc95200640 1 --2- 172.21.15.29:0/2265089717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc9006c660 0x7efc901f89f0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:50.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.898+0000 7efc95200640 1 -- 172.21.15.29:0/2265089717 >> 172.21.15.29:0/2265089717 conn(0x7efc9006fde0 msgr2=0x7efc900720e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:57:50.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.898+0000 7efc95200640 1 -- 172.21.15.29:0/2265089717 shutdown_connections 2024-04-17T19:57:50.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:50.898+0000 7efc95200640 1 -- 172.21.15.29:0/2265089717 wait complete. 2024-04-17T19:57:50.917 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-17T19:57:52.752 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:57:52.756 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:57:52.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:57:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.074+0000 7f4b1d400640 1 Processor -- start 2024-04-17T19:57:53.072 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.074+0000 7f4b1d400640 1 -- start start 2024-04-17T19:57:53.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.074+0000 7f4b1d400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4b18155ff0 0x7f4b1815c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:53.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.074+0000 7f4b1d400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b180ce490 0x7f4b180ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:53.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.074+0000 7f4b1d400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f4b180c98c0 0x7f4b180c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:53.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.074+0000 7f4b1d400640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f4b180c5770 con 0x7f4b18155ff0 2024-04-17T19:57:53.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.074+0000 7f4b1d400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f4b180cede0 con 0x7f4b180c98c0 2024-04-17T19:57:53.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.074+0000 7f4b1d400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4b180c9150 con 0x7f4b180ce490 2024-04-17T19:57:53.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.074+0000 7f4b17e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4b18155ff0 0x7f4b1815c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:53.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.074+0000 7f4b17e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4b18155ff0 0x7f4b1815c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46102/0 (socket says 172.21.15.29:46102) 2024-04-17T19:57:53.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.074+0000 7f4b17e00640 1 -- 172.21.15.29:0/4001470120 learned_addr learned my addr 172.21.15.29:0/4001470120 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:57:53.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.074+0000 7f4b1ca00640 1 --2- 172.21.15.29:0/4001470120 >> v2:172.21.15.37:3300/0 conn(0x7f4b180c98c0 0x7f4b180c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:53.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.074+0000 7f4b17400640 1 --2- 172.21.15.29:0/4001470120 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b180ce490 0x7f4b180ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:53.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.074+0000 7f4b17e00640 1 -- 172.21.15.29:0/4001470120 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b180ce490 msgr2=0x7f4b180ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:57:53.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.074+0000 7f4b17e00640 1 --2- 172.21.15.29:0/4001470120 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b180ce490 0x7f4b180ce8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:53.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.074+0000 7f4b17e00640 1 -- 172.21.15.29:0/4001470120 >> v2:172.21.15.37:3300/0 conn(0x7f4b180c98c0 msgr2=0x7f4b180c9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:57:53.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.074+0000 7f4b17e00640 1 --2- 172.21.15.29:0/4001470120 >> v2:172.21.15.37:3300/0 conn(0x7f4b180c98c0 0x7f4b180c9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:53.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.074+0000 7f4b17e00640 1 -- 172.21.15.29:0/4001470120 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b180ca210 con 0x7f4b18155ff0 2024-04-17T19:57:53.076 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.078+0000 7f4b17e00640 1 --2- 172.21.15.29:0/4001470120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b18155ff0 0x7f4b1815c870 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4b18071df0 tx=0x7f4b0801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c92b230f396c2e2c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:57:53.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.078+0000 7f4b16a00640 1 -- 172.21.15.29:0/4001470120 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b080b0070 con 0x7f4b18155ff0 2024-04-17T19:57:53.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.078+0000 7f4b16a00640 1 -- 172.21.15.29:0/4001470120 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b080ab600 con 0x7f4b18155ff0 2024-04-17T19:57:53.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.078+0000 7f4b16a00640 1 -- 172.21.15.29:0/4001470120 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b080ab9d0 con 0x7f4b18155ff0 2024-04-17T19:57:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.078+0000 7f4b1d400640 1 -- 172.21.15.29:0/4001470120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b18155ff0 msgr2=0x7f4b1815c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.078+0000 7f4b1d400640 1 --2- 172.21.15.29:0/4001470120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b18155ff0 0x7f4b1815c870 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4b18071df0 tx=0x7f4b0801c490 comp rx=0 tx=0).stop 2024-04-17T19:57:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.078+0000 7f4b1d400640 1 -- 172.21.15.29:0/4001470120 shutdown_connections 2024-04-17T19:57:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.078+0000 7f4b1d400640 1 --2- 172.21.15.29:0/4001470120 >> v2:172.21.15.37:3300/0 conn(0x7f4b180c98c0 0x7f4b180c9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.078+0000 7f4b1d400640 1 --2- 172.21.15.29:0/4001470120 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b180ce490 0x7f4b180ce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.078+0000 7f4b1d400640 1 --2- 172.21.15.29:0/4001470120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b18155ff0 0x7f4b1815c870 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.078+0000 7f4b1d400640 1 -- 172.21.15.29:0/4001470120 >> 172.21.15.29:0/4001470120 conn(0x7f4b18072db0 msgr2=0x7f4b180731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:57:53.078 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.078+0000 7f4b1d400640 1 -- 172.21.15.29:0/4001470120 shutdown_connections 2024-04-17T19:57:53.079 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.078+0000 7f4b1d400640 1 -- 172.21.15.29:0/4001470120 wait complete. 2024-04-17T19:57:53.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b1d400640 1 Processor -- start 2024-04-17T19:57:53.080 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b1d400640 1 -- start start 2024-04-17T19:57:53.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b1d400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b180c98c0 0x7f4b181457f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:53.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b1d400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b180ce490 0x7f4b1815ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:53.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b1d400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b18155ff0 0x7f4b180cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b1d400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4b180c93a0 con 0x7f4b180c98c0 2024-04-17T19:57:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b1d400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4b1806d4c0 con 0x7f4b180ce490 2024-04-17T19:57:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b1d400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4b1806c320 con 0x7f4b18155ff0 2024-04-17T19:57:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b17e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b180c98c0 0x7f4b181457f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b17e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b180c98c0 0x7f4b181457f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46110/0 (socket says 172.21.15.29:46110) 2024-04-17T19:57:53.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b17e00640 1 -- 172.21.15.29:0/2345200077 learned_addr learned my addr 172.21.15.29:0/2345200077 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:57:53.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b1ca00640 1 --2- 172.21.15.29:0/2345200077 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b18155ff0 0x7f4b180cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:53.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b17400640 1 --2- 172.21.15.29:0/2345200077 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b180ce490 0x7f4b1815ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:53.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b17e00640 1 -- 172.21.15.29:0/2345200077 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b18155ff0 msgr2=0x7f4b180cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:53.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b17e00640 1 --2- 172.21.15.29:0/2345200077 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b18155ff0 0x7f4b180cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:53.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b17e00640 1 -- 172.21.15.29:0/2345200077 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b180ce490 msgr2=0x7f4b1815ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:53.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b17e00640 1 --2- 172.21.15.29:0/2345200077 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b180ce490 0x7f4b1815ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:53.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b17e00640 1 -- 172.21.15.29:0/2345200077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b1815a690 con 0x7f4b180c98c0 2024-04-17T19:57:53.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b1ca00640 1 --2- 172.21.15.29:0/2345200077 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b18155ff0 0x7f4b180cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:57:53.086 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b17e00640 1 --2- 172.21.15.29:0/2345200077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b180c98c0 0x7f4b181457f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4b0801ab40 tx=0x7f4b08019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:57:53.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b0fe00640 1 -- 172.21.15.29:0/2345200077 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b080b9070 con 0x7f4b180c98c0 2024-04-17T19:57:53.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b0fe00640 1 -- 172.21.15.29:0/2345200077 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b080b4420 con 0x7f4b180c98c0 2024-04-17T19:57:53.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b0fe00640 1 -- 172.21.15.29:0/2345200077 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b080b0040 con 0x7f4b180c98c0 2024-04-17T19:57:53.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b1d400640 1 -- 172.21.15.29:0/2345200077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b1814a1e0 con 0x7f4b180c98c0 2024-04-17T19:57:53.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.082+0000 7f4b1d400640 1 -- 172.21.15.29:0/2345200077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b1815b390 con 0x7f4b180c98c0 2024-04-17T19:57:53.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.086+0000 7f4b1d400640 1 -- 172.21.15.29:0/2345200077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b1804ff10 con 0x7f4b180c98c0 2024-04-17T19:57:53.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.086+0000 7f4b0fe00640 1 -- 172.21.15.29:0/2345200077 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4b080bc070 con 0x7f4b180c98c0 2024-04-17T19:57:53.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.086+0000 7f4b0fe00640 1 --2- 172.21.15.29:0/2345200077 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4af80683b0 0x7f4af806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:53.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.086+0000 7f4b0fe00640 1 -- 172.21.15.29:0/2345200077 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7135+0+0 (secure 0 0 0) 0x7f4b081201c0 con 0x7f4b180c98c0 2024-04-17T19:57:53.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.086+0000 7f4b17400640 1 --2- 172.21.15.29:0/2345200077 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4af80683b0 0x7f4af806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:53.088 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.090+0000 7f4b17400640 1 --2- 172.21.15.29:0/2345200077 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4af80683b0 0x7f4af806a8c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4b18151ad0 tx=0x7f4b04008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:57:53.089 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.090+0000 7f4b0fe00640 1 -- 172.21.15.29:0/2345200077 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4b180ca210 con 0x7f4b180c98c0 2024-04-17T19:57:53.290 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.290+0000 7f4b1d400640 1 -- 172.21.15.29:0/2345200077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b18152050 con 0x7f4b180c98c0 2024-04-17T19:57:53.291 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.290+0000 7f4b0fe00640 1 -- 172.21.15.29:0/2345200077 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6904 (secure 0 0 0) 0x7f4b08016110 con 0x7f4b180c98c0 2024-04-17T19:57:53.291 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:57:53.291 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:40.944059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5144,4305,5212,4302]},"id":1}]} 2024-04-17T19:57:53.291 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 43 2024-04-17T19:57:53.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.294+0000 7f4b1d400640 1 -- 172.21.15.29:0/2345200077 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4af80683b0 msgr2=0x7f4af806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:53.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.294+0000 7f4b1d400640 1 --2- 172.21.15.29:0/2345200077 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4af80683b0 0x7f4af806a8c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4b18151ad0 tx=0x7f4b04008040 comp rx=0 tx=0).stop 2024-04-17T19:57:53.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.294+0000 7f4b1d400640 1 -- 172.21.15.29:0/2345200077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b180c98c0 msgr2=0x7f4b181457f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:53.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.294+0000 7f4b1d400640 1 --2- 172.21.15.29:0/2345200077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b180c98c0 0x7f4b181457f0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4b0801ab40 tx=0x7f4b08019230 comp rx=0 tx=0).stop 2024-04-17T19:57:53.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.294+0000 7f4b1d400640 1 -- 172.21.15.29:0/2345200077 shutdown_connections 2024-04-17T19:57:53.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.294+0000 7f4b1d400640 1 --2- 172.21.15.29:0/2345200077 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4af80683b0 0x7f4af806a8c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:53.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.294+0000 7f4b1d400640 1 --2- 172.21.15.29:0/2345200077 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4b18155ff0 0x7f4b180cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:53.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.294+0000 7f4b1d400640 1 --2- 172.21.15.29:0/2345200077 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4b180ce490 0x7f4b1815ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:53.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.294+0000 7f4b1d400640 1 --2- 172.21.15.29:0/2345200077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4b180c98c0 0x7f4b181457f0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:53.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.294+0000 7f4b1d400640 1 -- 172.21.15.29:0/2345200077 >> 172.21.15.29:0/2345200077 conn(0x7f4b18072db0 msgr2=0x7f4b180ca790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:57:53.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.294+0000 7f4b1d400640 1 -- 172.21.15.29:0/2345200077 shutdown_connections 2024-04-17T19:57:53.294 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:53.294+0000 7f4b1d400640 1 -- 172.21.15.29:0/2345200077 wait complete. 2024-04-17T19:57:53.313 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-17T19:57:55.315 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:57:55.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.474+0000 7f99bee00640 1 Processor -- start 2024-04-17T19:57:55.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.474+0000 7f99bee00640 1 -- start start 2024-04-17T19:57:55.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.478+0000 7f99bee00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f99b8141ee0 0x7f99b806cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:55.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.478+0000 7f99bee00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f99b806d140 0x7f99b80c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:55.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.478+0000 7f99bee00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99b8158af0 con 0x7f99b8141ad0 2024-04-17T19:57:55.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.478+0000 7f99bee00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f99b8158dd0 con 0x7f99b806d140 2024-04-17T19:57:55.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.478+0000 7f99bee00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f99b81590b0 con 0x7f99b8141ee0 2024-04-17T19:57:55.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.478+0000 7f99be400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f99b806d140 0x7f99b80c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:55.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.478+0000 7f99be400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f99b806d140 0x7f99b80c77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46120/0 (socket says 172.21.15.29:46120) 2024-04-17T19:57:55.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.478+0000 7f99be400640 1 -- 172.21.15.29:0/2776708372 learned_addr learned my addr 172.21.15.29:0/2776708372 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:57:55.473 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.478+0000 7f99bd000640 1 --2- 172.21.15.29:0/2776708372 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f99b8141ee0 0x7f99b806cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:55.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.478+0000 7f99be400640 1 -- 172.21.15.29:0/2776708372 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f99b8141ee0 msgr2=0x7f99b806cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:57:55.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.478+0000 7f99be400640 1 --2- 172.21.15.29:0/2776708372 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f99b8141ee0 0x7f99b806cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:55.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.478+0000 7f99be400640 1 -- 172.21.15.29:0/2776708372 >> v1:172.21.15.37:6789/0 conn(0x7f99b8141ad0 legacy=0x7f99b80c54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:55.474 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.478+0000 7f99be400640 1 -- 172.21.15.29:0/2776708372 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b8159390 con 0x7f99b806d140 2024-04-17T19:57:55.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.482+0000 7f99be400640 1 --2- 172.21.15.29:0/2776708372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b806d140 0x7f99b80c77d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009ed0 tx=0x7f99b00216c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faca1ca2eb178380 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:57:55.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.482+0000 7f99b7e00640 1 -- 172.21.15.29:0/2776708372 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b00b0470 con 0x7f99b806d140 2024-04-17T19:57:55.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.482+0000 7f99b7e00640 1 -- 172.21.15.29:0/2776708372 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99b00b5070 con 0x7f99b806d140 2024-04-17T19:57:55.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.482+0000 7f99b7e00640 1 -- 172.21.15.29:0/2776708372 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b00b0830 con 0x7f99b806d140 2024-04-17T19:57:55.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.482+0000 7f99bee00640 1 -- 172.21.15.29:0/2776708372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b806d140 msgr2=0x7f99b80c77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:55.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.482+0000 7f99bee00640 1 --2- 172.21.15.29:0/2776708372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b806d140 0x7f99b80c77d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f99b0009ed0 tx=0x7f99b00216c0 comp rx=0 tx=0).stop 2024-04-17T19:57:55.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.482+0000 7f99bee00640 1 -- 172.21.15.29:0/2776708372 shutdown_connections 2024-04-17T19:57:55.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.482+0000 7f99bee00640 1 --2- 172.21.15.29:0/2776708372 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b806d140 0x7f99b80c77d0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:55.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.482+0000 7f99bee00640 1 --2- 172.21.15.29:0/2776708372 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f99b8141ee0 0x7f99b806cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:55.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.482+0000 7f99bee00640 1 -- 172.21.15.29:0/2776708372 >> 172.21.15.29:0/2776708372 conn(0x7f99b8072db0 msgr2=0x7f99b80731c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:57:55.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.482+0000 7f99bee00640 1 -- 172.21.15.29:0/2776708372 shutdown_connections 2024-04-17T19:57:55.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.482+0000 7f99bee00640 1 -- 172.21.15.29:0/2776708372 wait complete. 2024-04-17T19:57:55.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bee00640 1 Processor -- start 2024-04-17T19:57:55.480 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bee00640 1 -- start start 2024-04-17T19:57:55.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bee00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f99b806d140 0x7f99b81e33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:55.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bee00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b8141ad0 0x7f99b813efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:55.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bee00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99b8140940 0x7f99b8159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:55.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bee00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f99b80c5be0 con 0x7f99b8141ad0 2024-04-17T19:57:55.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bee00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f99b806d550 con 0x7f99b8140940 2024-04-17T19:57:55.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bee00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f99b80c5d60 con 0x7f99b806d140 2024-04-17T19:57:55.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bd000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b8141ad0 0x7f99b813efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:55.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bd000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b8141ad0 0x7f99b813efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46130/0 (socket says 172.21.15.29:46130) 2024-04-17T19:57:55.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bd000640 1 -- 172.21.15.29:0/1096618880 learned_addr learned my addr 172.21.15.29:0/1096618880 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:57:55.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99be400640 1 --2- 172.21.15.29:0/1096618880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99b8140940 0x7f99b8159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:55.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bda00640 1 --2- 172.21.15.29:0/1096618880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f99b806d140 0x7f99b81e33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:55.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bd000640 1 -- 172.21.15.29:0/1096618880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f99b806d140 msgr2=0x7f99b81e33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:55.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bd000640 1 --2- 172.21.15.29:0/1096618880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f99b806d140 0x7f99b81e33d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:55.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bd000640 1 -- 172.21.15.29:0/1096618880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99b8140940 msgr2=0x7f99b8159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:55.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bd000640 1 --2- 172.21.15.29:0/1096618880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99b8140940 0x7f99b8159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:55.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bd000640 1 -- 172.21.15.29:0/1096618880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b815a090 con 0x7f99b8141ad0 2024-04-17T19:57:55.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bda00640 1 --2- 172.21.15.29:0/1096618880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f99b806d140 0x7f99b81e33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:57:55.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bd000640 1 --2- 172.21.15.29:0/1096618880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b8141ad0 0x7f99b813efc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f99a400e8a0 tx=0x7f99a400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:57:55.484 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99b5600640 1 -- 172.21.15.29:0/1096618880 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a4017070 con 0x7f99b8141ad0 2024-04-17T19:57:55.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99b5600640 1 -- 172.21.15.29:0/1096618880 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99a400c940 con 0x7f99b8141ad0 2024-04-17T19:57:55.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.486+0000 7f99bee00640 1 -- 172.21.15.29:0/1096618880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99b81ed8a0 con 0x7f99b8141ad0 2024-04-17T19:57:55.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.490+0000 7f99b5600640 1 -- 172.21.15.29:0/1096618880 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a400ccf0 con 0x7f99b8141ad0 2024-04-17T19:57:55.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.490+0000 7f99bee00640 1 -- 172.21.15.29:0/1096618880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99b81ede60 con 0x7f99b8141ad0 2024-04-17T19:57:55.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.490+0000 7f99bee00640 1 -- 172.21.15.29:0/1096618880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99b804ff10 con 0x7f99b8141ad0 2024-04-17T19:57:55.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.490+0000 7f99b5600640 1 -- 172.21.15.29:0/1096618880 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f99a401a070 con 0x7f99b8141ad0 2024-04-17T19:57:55.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.490+0000 7f99b5600640 1 --2- 172.21.15.29:0/1096618880 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9998068380 0x7f999806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:55.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.490+0000 7f99b5600640 1 -- 172.21.15.29:0/1096618880 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7135+0+0 (secure 0 0 0) 0x7f99a4087120 con 0x7f99b8141ad0 2024-04-17T19:57:55.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.494+0000 7f99bda00640 1 --2- 172.21.15.29:0/1096618880 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9998068380 0x7f999806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:55.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.494+0000 7f99bda00640 1 --2- 172.21.15.29:0/1096618880 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9998068380 0x7f999806a890 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f99ac001670 tx=0x7f99ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:57:55.491 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.498+0000 7f99b5600640 1 -- 172.21.15.29:0/1096618880 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f99a4056b00 con 0x7f99b8141ad0 2024-04-17T19:57:55.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.706+0000 7f99bee00640 1 -- 172.21.15.29:0/1096618880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99b8071490 con 0x7f99b8141ad0 2024-04-17T19:57:55.702 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.706+0000 7f99b5600640 1 -- 172.21.15.29:0/1096618880 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6904 (secure 0 0 0) 0x7f99a405ae00 con 0x7f99b8141ad0 2024-04-17T19:57:55.703 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:57:55.703 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:40.944059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5144,4305,5212,4302]},"id":1}]} 2024-04-17T19:57:55.703 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 43 2024-04-17T19:57:55.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.710+0000 7f99bee00640 1 -- 172.21.15.29:0/1096618880 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9998068380 msgr2=0x7f999806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:55.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.710+0000 7f99bee00640 1 --2- 172.21.15.29:0/1096618880 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9998068380 0x7f999806a890 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f99ac001670 tx=0x7f99ac009040 comp rx=0 tx=0).stop 2024-04-17T19:57:55.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.710+0000 7f99bee00640 1 -- 172.21.15.29:0/1096618880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b8141ad0 msgr2=0x7f99b813efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:55.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.710+0000 7f99bee00640 1 --2- 172.21.15.29:0/1096618880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b8141ad0 0x7f99b813efc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f99a400e8a0 tx=0x7f99a400eca0 comp rx=0 tx=0).stop 2024-04-17T19:57:55.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.710+0000 7f99bee00640 1 -- 172.21.15.29:0/1096618880 shutdown_connections 2024-04-17T19:57:55.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.710+0000 7f99bee00640 1 --2- 172.21.15.29:0/1096618880 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9998068380 0x7f999806a890 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:55.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.710+0000 7f99bee00640 1 --2- 172.21.15.29:0/1096618880 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f99b8140940 0x7f99b8159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:55.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.710+0000 7f99bee00640 1 --2- 172.21.15.29:0/1096618880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99b8141ad0 0x7f99b813efc0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:55.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.710+0000 7f99bee00640 1 --2- 172.21.15.29:0/1096618880 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f99b806d140 0x7f99b81e33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:55.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.710+0000 7f99bee00640 1 -- 172.21.15.29:0/1096618880 >> 172.21.15.29:0/1096618880 conn(0x7f99b8072db0 msgr2=0x7f99b80cad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:57:55.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.710+0000 7f99bee00640 1 -- 172.21.15.29:0/1096618880 shutdown_connections 2024-04-17T19:57:55.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:55.710+0000 7f99bee00640 1 -- 172.21.15.29:0/1096618880 wait complete. 2024-04-17T19:57:55.725 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-17T19:57:57.726 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:57:57.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.914+0000 7ff8c6200640 1 Processor -- start 2024-04-17T19:57:57.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.914+0000 7ff8c6200640 1 -- start start 2024-04-17T19:57:57.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.914+0000 7ff8c6200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8c0152d90 0x7ff8c00c2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:57.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.914+0000 7ff8c6200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff8c00c3360 0x7ff8c015c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:57.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.914+0000 7ff8c6200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8c015c660 0x7ff8c0158b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:57.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.914+0000 7ff8c6200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7ff8c0071e30 con 0x7ff8c015c660 2024-04-17T19:57:57.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.914+0000 7ff8c6200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7ff8c00c95e0 con 0x7ff8c00c3360 2024-04-17T19:57:57.911 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.914+0000 7ff8c6200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff8c015ce00 con 0x7ff8c0152d90 2024-04-17T19:57:57.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.918+0000 7ff8c5800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8c015c660 0x7ff8c0158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:57.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.918+0000 7ff8c5800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8c015c660 0x7ff8c0158b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46146/0 (socket says 172.21.15.29:46146) 2024-04-17T19:57:57.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.918+0000 7ff8bfe00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7ff8c00c3360 0x7ff8c015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:57.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.918+0000 7ff8c5800640 1 -- 172.21.15.29:0/2322605651 learned_addr learned my addr 172.21.15.29:0/2322605651 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:57:57.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.918+0000 7ff8c4e00640 1 --2- 172.21.15.29:0/2322605651 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8c0152d90 0x7ff8c00c2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:57.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.918+0000 7ff8c5800640 1 -- 172.21.15.29:0/2322605651 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8c0152d90 msgr2=0x7ff8c00c2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:57:57.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.918+0000 7ff8c5800640 1 --2- 172.21.15.29:0/2322605651 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8c0152d90 0x7ff8c00c2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:57.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.918+0000 7ff8c5800640 1 -- 172.21.15.29:0/2322605651 >> v2:172.21.15.37:3300/0 conn(0x7ff8c00c3360 msgr2=0x7ff8c015c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:57:57.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.918+0000 7ff8c5800640 1 --2- 172.21.15.29:0/2322605651 >> v2:172.21.15.37:3300/0 conn(0x7ff8c00c3360 0x7ff8c015c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:57.913 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.918+0000 7ff8c5800640 1 -- 172.21.15.29:0/2322605651 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8c0073eb0 con 0x7ff8c015c660 2024-04-17T19:57:57.914 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.918+0000 7ff8c5800640 1 --2- 172.21.15.29:0/2322605651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8c015c660 0x7ff8c0158b30 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff8c006efc0 tx=0x7ff8b801c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f39c9b77919e40 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:57:57.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.918+0000 7ff8bf400640 1 -- 172.21.15.29:0/2322605651 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b80a5820 con 0x7ff8c015c660 2024-04-17T19:57:57.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.918+0000 7ff8bf400640 1 -- 172.21.15.29:0/2322605651 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8b80a59c0 con 0x7ff8c015c660 2024-04-17T19:57:57.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.918+0000 7ff8bf400640 1 -- 172.21.15.29:0/2322605651 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b80a5d90 con 0x7ff8c015c660 2024-04-17T19:57:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.922+0000 7ff8c6200640 1 -- 172.21.15.29:0/2322605651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8c015c660 msgr2=0x7ff8c0158b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.922+0000 7ff8c6200640 1 --2- 172.21.15.29:0/2322605651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8c015c660 0x7ff8c0158b30 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff8c006efc0 tx=0x7ff8b801c9c0 comp rx=0 tx=0).stop 2024-04-17T19:57:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.922+0000 7ff8c6200640 1 -- 172.21.15.29:0/2322605651 shutdown_connections 2024-04-17T19:57:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.922+0000 7ff8c6200640 1 --2- 172.21.15.29:0/2322605651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8c015c660 0x7ff8c0158b30 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.922+0000 7ff8c6200640 1 --2- 172.21.15.29:0/2322605651 >> v2:172.21.15.37:3300/0 conn(0x7ff8c00c3360 0x7ff8c015c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.922+0000 7ff8c6200640 1 --2- 172.21.15.29:0/2322605651 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8c0152d90 0x7ff8c00c2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.922+0000 7ff8c6200640 1 -- 172.21.15.29:0/2322605651 >> 172.21.15.29:0/2322605651 conn(0x7ff8c00ccf80 msgr2=0x7ff8c00cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:57:57.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.922+0000 7ff8c6200640 1 -- 172.21.15.29:0/2322605651 shutdown_connections 2024-04-17T19:57:57.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.922+0000 7ff8c6200640 1 -- 172.21.15.29:0/2322605651 wait complete. 2024-04-17T19:57:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.922+0000 7ff8c6200640 1 Processor -- start 2024-04-17T19:57:57.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.922+0000 7ff8c6200640 1 -- start start 2024-04-17T19:57:57.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c6200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8c00c3360 0x7ff8c00c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:57.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c6200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8c0152d90 0x7ff8c01442c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:57.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c6200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8c015c660 0x7ff8c0144800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:57.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c6200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c006d210 con 0x7ff8c015c660 2024-04-17T19:57:57.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c6200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c015ca70 con 0x7ff8c00c3360 2024-04-17T19:57:57.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c6200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff8c006d390 con 0x7ff8c0152d90 2024-04-17T19:57:57.920 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c5800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8c015c660 0x7ff8c0144800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:57.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c5800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8c015c660 0x7ff8c0144800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46152/0 (socket says 172.21.15.29:46152) 2024-04-17T19:57:57.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c5800640 1 -- 172.21.15.29:0/181589993 learned_addr learned my addr 172.21.15.29:0/181589993 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:57:57.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c4e00640 1 --2- 172.21.15.29:0/181589993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8c00c3360 0x7ff8c00c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:57.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8bfe00640 1 --2- 172.21.15.29:0/181589993 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8c0152d90 0x7ff8c01442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:57.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c5800640 1 -- 172.21.15.29:0/181589993 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8c0152d90 msgr2=0x7ff8c01442c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:57.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c5800640 1 --2- 172.21.15.29:0/181589993 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8c0152d90 0x7ff8c01442c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:57.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c5800640 1 -- 172.21.15.29:0/181589993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8c00c3360 msgr2=0x7ff8c00c7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:57.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c5800640 1 --2- 172.21.15.29:0/181589993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8c00c3360 0x7ff8c00c7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:57.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c5800640 1 -- 172.21.15.29:0/181589993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8c0148a10 con 0x7ff8c015c660 2024-04-17T19:57:57.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c5800640 1 --2- 172.21.15.29:0/181589993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8c015c660 0x7ff8c0144800 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff8b801b070 tx=0x7ff8b8019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:57:57.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8bcc00640 1 -- 172.21.15.29:0/181589993 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b80b8670 con 0x7ff8c015c660 2024-04-17T19:57:57.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8bcc00640 1 -- 172.21.15.29:0/181589993 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8b80be070 con 0x7ff8c015c660 2024-04-17T19:57:57.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c6200640 1 -- 172.21.15.29:0/181589993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8c0148cf0 con 0x7ff8c015c660 2024-04-17T19:57:57.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8bcc00640 1 -- 172.21.15.29:0/181589993 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b80b8a90 con 0x7ff8c015c660 2024-04-17T19:57:57.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c6200640 1 -- 172.21.15.29:0/181589993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8c01625c0 con 0x7ff8c015c660 2024-04-17T19:57:57.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.926+0000 7ff8c6200640 1 -- 172.21.15.29:0/181589993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8c014cab0 con 0x7ff8c015c660 2024-04-17T19:57:57.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.930+0000 7ff8bcc00640 1 -- 172.21.15.29:0/181589993 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff8b80b8c30 con 0x7ff8c015c660 2024-04-17T19:57:57.924 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.930+0000 7ff8bcc00640 1 --2- 172.21.15.29:0/181589993 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff8a0068380 0x7ff8a006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:57:57.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.930+0000 7ff8bcc00640 1 -- 172.21.15.29:0/181589993 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7135+0+0 (secure 0 0 0) 0x7ff8b81221e0 con 0x7ff8c015c660 2024-04-17T19:57:57.925 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.930+0000 7ff8c4e00640 1 --2- 172.21.15.29:0/181589993 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff8a0068380 0x7ff8a006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:57:57.926 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.930+0000 7ff8c4e00640 1 --2- 172.21.15.29:0/181589993 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff8a0068380 0x7ff8a006a890 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4009ac0 tx=0x7ff8b4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:57:57.929 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:57.934+0000 7ff8bcc00640 1 -- 172.21.15.29:0/181589993 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff8b80165e0 con 0x7ff8c015c660 2024-04-17T19:57:58.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:58.134+0000 7ff8c6200640 1 -- 172.21.15.29:0/181589993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8c004ff10 con 0x7ff8c015c660 2024-04-17T19:57:58.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:58.138+0000 7ff8bcc00640 1 -- 172.21.15.29:0/181589993 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6904 (secure 0 0 0) 0x7ff8b80f2290 con 0x7ff8c015c660 2024-04-17T19:57:58.132 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:57:58.132 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:40.944059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5144,4305,5212,4302]},"id":1}]} 2024-04-17T19:57:58.133 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 43 2024-04-17T19:57:58.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:58.138+0000 7ff8c6200640 1 -- 172.21.15.29:0/181589993 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff8a0068380 msgr2=0x7ff8a006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:58.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:58.138+0000 7ff8c6200640 1 --2- 172.21.15.29:0/181589993 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff8a0068380 0x7ff8a006a890 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4009ac0 tx=0x7ff8b4009190 comp rx=0 tx=0).stop 2024-04-17T19:57:58.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:58.138+0000 7ff8c6200640 1 -- 172.21.15.29:0/181589993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8c015c660 msgr2=0x7ff8c0144800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:57:58.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:58.138+0000 7ff8c6200640 1 --2- 172.21.15.29:0/181589993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8c015c660 0x7ff8c0144800 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff8b801b070 tx=0x7ff8b8019760 comp rx=0 tx=0).stop 2024-04-17T19:57:58.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:58.138+0000 7ff8c6200640 1 -- 172.21.15.29:0/181589993 shutdown_connections 2024-04-17T19:57:58.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:58.138+0000 7ff8c6200640 1 --2- 172.21.15.29:0/181589993 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff8a0068380 0x7ff8a006a890 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:58.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:58.138+0000 7ff8c6200640 1 --2- 172.21.15.29:0/181589993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8c015c660 0x7ff8c0144800 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:58.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:58.138+0000 7ff8c6200640 1 --2- 172.21.15.29:0/181589993 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff8c0152d90 0x7ff8c01442c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:58.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:58.138+0000 7ff8c6200640 1 --2- 172.21.15.29:0/181589993 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff8c00c3360 0x7ff8c00c7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:57:58.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:58.138+0000 7ff8c6200640 1 -- 172.21.15.29:0/181589993 >> 172.21.15.29:0/181589993 conn(0x7ff8c00ccf80 msgr2=0x7ff8c00cbe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:57:58.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:58.138+0000 7ff8c6200640 1 -- 172.21.15.29:0/181589993 shutdown_connections 2024-04-17T19:57:58.135 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:57:58.138+0000 7ff8c6200640 1 -- 172.21.15.29:0/181589993 wait complete. 2024-04-17T19:57:58.153 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-17T19:58:00.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:58:00.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.314+0000 7f5c12600640 1 Processor -- start 2024-04-17T19:58:00.310 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.314+0000 7f5c12600640 1 -- start start 2024-04-17T19:58:00.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c12600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5c0c155ff0 0x7f5c0c0c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:00.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c12600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c0c15ce40 con 0x7f5c0c0cb4e0 2024-04-17T19:58:00.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c12600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c0c0cb8f0 con 0x7f5c0c0cb0d0 2024-04-17T19:58:00.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c12600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5c0c15c9b0 con 0x7f5c0c155ff0 2024-04-17T19:58:00.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0be00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5c0c0cb4e0 0x7f5c0c1667f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45262/0 (socket says 172.21.15.29:45262) 2024-04-17T19:58:00.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0be00640 1 -- 172.21.15.29:0/1689874583 learned_addr learned my addr 172.21.15.29:0/1689874583 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:58:00.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0b400640 1 --2- 172.21.15.29:0/1689874583 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5c0c155ff0 0x7f5c0c0c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:00.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0a000640 1 -- 172.21.15.29:0/1689874583 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 847598802 0 0) 0x7f5c0c15ce40 con 0x7f5c0c0cb4e0 2024-04-17T19:58:00.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0a000640 1 -- 172.21.15.29:0/1689874583 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5bf0003530 con 0x7f5c0c0cb4e0 2024-04-17T19:58:00.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0a000640 1 -- 172.21.15.29:0/1689874583 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2363177483 0 0) 0x7f5bf0003530 con 0x7f5c0c0cb4e0 2024-04-17T19:58:00.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0a000640 1 -- 172.21.15.29:0/1689874583 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5c0c155ff0 msgr2=0x7f5c0c0c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:00.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0a000640 1 --2- 172.21.15.29:0/1689874583 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5c0c155ff0 0x7f5c0c0c9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:00.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0a000640 1 -- 172.21.15.29:0/1689874583 >> v1:172.21.15.37:6789/0 conn(0x7f5c0c0cb0d0 legacy=0x7f5c0c0cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:00.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0a000640 1 -- 172.21.15.29:0/1689874583 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0c070770 con 0x7f5c0c0cb4e0 2024-04-17T19:58:00.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0a000640 1 -- 172.21.15.29:0/1689874583 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f5c04001a00 con 0x7f5c0c0cb4e0 2024-04-17T19:58:00.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0a000640 1 -- 172.21.15.29:0/1689874583 >> v1:172.21.15.29:6789/0 conn(0x7f5c0c0cb4e0 legacy=0x7f5c0c1667f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:00.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0a000640 1 --2- 172.21.15.29:0/1689874583 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5bf0003a60 0x7f5bf0003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:00.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0a000640 1 --2- 172.21.15.29:0/1689874583 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5c0c155ff0 0x7f5bf0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:00.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0a000640 1 --2- 172.21.15.29:0/1689874583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bf0004d50 0x7f5bf00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:00.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0a000640 1 -- 172.21.15.29:0/1689874583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c15c9b0 con 0x7f5bf0004d50 2024-04-17T19:58:00.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0a000640 1 -- 172.21.15.29:0/1689874583 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5bf000bb60 con 0x7f5bf0003a60 2024-04-17T19:58:00.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0a000640 1 -- 172.21.15.29:0/1689874583 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5bf000bd60 con 0x7f5c0c155ff0 2024-04-17T19:58:00.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0be00640 1 --2- 172.21.15.29:0/1689874583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bf0004d50 0x7f5bf00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:00.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0b400640 1 --2- 172.21.15.29:0/1689874583 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5bf0003a60 0x7f5bf0003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:00.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0aa00640 1 --2- 172.21.15.29:0/1689874583 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5c0c155ff0 0x7f5bf0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:00.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0be00640 1 -- 172.21.15.29:0/1689874583 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5c0c155ff0 msgr2=0x7f5bf0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:00.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0be00640 1 --2- 172.21.15.29:0/1689874583 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5c0c155ff0 0x7f5bf0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:00.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0be00640 1 -- 172.21.15.29:0/1689874583 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5bf0003a60 msgr2=0x7f5bf0003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:00.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0be00640 1 --2- 172.21.15.29:0/1689874583 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5bf0003a60 0x7f5bf0003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:00.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.318+0000 7f5c0be00640 1 -- 172.21.15.29:0/1689874583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bf0003530 con 0x7f5bf0004d50 2024-04-17T19:58:00.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.322+0000 7f5c0be00640 1 --2- 172.21.15.29:0/1689874583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bf0004d50 0x7f5bf00051c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5c0400a760 tx=0x7f5c0401d280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3609988e87d7bf6b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:00.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.322+0000 7f5c0a000640 1 -- 172.21.15.29:0/1689874583 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0400a3b0 con 0x7f5bf0004d50 2024-04-17T19:58:00.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.322+0000 7f5c0a000640 1 -- 172.21.15.29:0/1689874583 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c040ad630 con 0x7f5bf0004d50 2024-04-17T19:58:00.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.322+0000 7f5c0a000640 1 -- 172.21.15.29:0/1689874583 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c040a5e60 con 0x7f5bf0004d50 2024-04-17T19:58:00.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.326+0000 7f5c12600640 1 -- 172.21.15.29:0/1689874583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bf0004d50 msgr2=0x7f5bf00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:00.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.326+0000 7f5c12600640 1 --2- 172.21.15.29:0/1689874583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bf0004d50 0x7f5bf00051c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5c0400a760 tx=0x7f5c0401d280 comp rx=0 tx=0).stop 2024-04-17T19:58:00.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.326+0000 7f5c0be00640 1 -- 172.21.15.29:0/1689874583 reap_dead start 2024-04-17T19:58:00.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.326+0000 7f5c12600640 1 -- 172.21.15.29:0/1689874583 shutdown_connections 2024-04-17T19:58:00.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.326+0000 7f5c12600640 1 -- 172.21.15.29:0/1689874583 >> 172.21.15.29:0/1689874583 conn(0x7f5c0c06e3d0 msgr2=0x7f5c0c06e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:00.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.326+0000 7f5c12600640 1 -- 172.21.15.29:0/1689874583 shutdown_connections 2024-04-17T19:58:00.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.326+0000 7f5c12600640 1 -- 172.21.15.29:0/1689874583 wait complete. 2024-04-17T19:58:00.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.326+0000 7f5c12600640 1 Processor -- start 2024-04-17T19:58:00.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.326+0000 7f5c12600640 1 -- start start 2024-04-17T19:58:00.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c12600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5bf0004d50 0x7f5c0c0c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:00.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c12600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5c0c0c7780 0x7f5c0c0ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:00.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c12600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c0c0c7b90 0x7f5c0c1e6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:00.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c12600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c15c360 con 0x7f5c0c0c7b90 2024-04-17T19:58:00.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c12600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c06cbf0 con 0x7f5bf0004d50 2024-04-17T19:58:00.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c12600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5c0c15c640 con 0x7f5c0c0c7780 2024-04-17T19:58:00.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c0be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c0c0c7b90 0x7f5c0c1e6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:00.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c0be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c0c0c7b90 0x7f5c0c1e6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34270/0 (socket says 172.21.15.29:34270) 2024-04-17T19:58:00.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c0be00640 1 -- 172.21.15.29:0/2064479193 learned_addr learned my addr 172.21.15.29:0/2064479193 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:00.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c0b400640 1 --2- 172.21.15.29:0/2064479193 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5bf0004d50 0x7f5c0c0c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:00.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c0aa00640 1 --2- 172.21.15.29:0/2064479193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5c0c0c7780 0x7f5c0c0ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:00.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c0be00640 1 -- 172.21.15.29:0/2064479193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5c0c0c7780 msgr2=0x7f5c0c0ce3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:00.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c0be00640 1 --2- 172.21.15.29:0/2064479193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5c0c0c7780 0x7f5c0c0ce3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:00.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c0be00640 1 -- 172.21.15.29:0/2064479193 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5bf0004d50 msgr2=0x7f5c0c0c67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:00.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c0be00640 1 --2- 172.21.15.29:0/2064479193 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5bf0004d50 0x7f5c0c0c67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:00.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c0be00640 1 -- 172.21.15.29:0/2064479193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0c0c6d20 con 0x7f5c0c0c7b90 2024-04-17T19:58:00.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c0aa00640 1 --2- 172.21.15.29:0/2064479193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5c0c0c7780 0x7f5c0c0ce3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:58:00.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c0b400640 1 --2- 172.21.15.29:0/2064479193 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5bf0004d50 0x7f5c0c0c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:58:00.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c0be00640 1 --2- 172.21.15.29:0/2064479193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c0c0c7b90 0x7f5c0c1e6610 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5c04000f90 tx=0x7f5bf0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:00.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c03400640 1 -- 172.21.15.29:0/2064479193 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c040a5e60 con 0x7f5c0c0c7b90 2024-04-17T19:58:00.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c03400640 1 -- 172.21.15.29:0/2064479193 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c040ad430 con 0x7f5c0c0c7b90 2024-04-17T19:58:00.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c03400640 1 -- 172.21.15.29:0/2064479193 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c04001210 con 0x7f5c0c0c7b90 2024-04-17T19:58:00.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c12600640 1 -- 172.21.15.29:0/2064479193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c0c14c2c0 con 0x7f5c0c0c7b90 2024-04-17T19:58:00.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c12600640 1 -- 172.21.15.29:0/2064479193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c0c152e50 con 0x7f5c0c0c7b90 2024-04-17T19:58:00.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.330+0000 7f5c12600640 1 -- 172.21.15.29:0/2064479193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c0c1e7d10 con 0x7f5c0c0c7b90 2024-04-17T19:58:00.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.334+0000 7f5c03400640 1 -- 172.21.15.29:0/2064479193 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5c040bc030 con 0x7f5c0c0c7b90 2024-04-17T19:58:00.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.334+0000 7f5c03400640 1 --2- 172.21.15.29:0/2064479193 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5bec0683b0 0x7f5bec06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:00.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.334+0000 7f5c03400640 1 -- 172.21.15.29:0/2064479193 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7135+0+0 (secure 0 0 0) 0x7f5c04120f60 con 0x7f5c0c0c7b90 2024-04-17T19:58:00.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.334+0000 7f5c0b400640 1 --2- 172.21.15.29:0/2064479193 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5bec0683b0 0x7f5bec06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:00.331 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.338+0000 7f5c0b400640 1 --2- 172.21.15.29:0/2064479193 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5bec0683b0 0x7f5bec06a8c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc009be0 tx=0x7f5bfc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:00.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.342+0000 7f5c03400640 1 -- 172.21.15.29:0/2064479193 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5c040c3020 con 0x7f5c0c0c7b90 2024-04-17T19:58:00.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.562+0000 7f5c12600640 1 -- 172.21.15.29:0/2064479193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c0c069b90 con 0x7f5c0c0c7b90 2024-04-17T19:58:00.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.562+0000 7f5c03400640 1 -- 172.21.15.29:0/2064479193 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6904 (secure 0 0 0) 0x7f5c04016f90 con 0x7f5c0c0c7b90 2024-04-17T19:58:00.556 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:58:00.557 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:40.944059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5144,4305,5212,4302]},"id":1}]} 2024-04-17T19:58:00.557 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 43 2024-04-17T19:58:00.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.562+0000 7f5c12600640 1 -- 172.21.15.29:0/2064479193 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5bec0683b0 msgr2=0x7f5bec06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:00.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.562+0000 7f5c12600640 1 --2- 172.21.15.29:0/2064479193 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5bec0683b0 0x7f5bec06a8c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc009be0 tx=0x7f5bfc008040 comp rx=0 tx=0).stop 2024-04-17T19:58:00.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.562+0000 7f5c12600640 1 -- 172.21.15.29:0/2064479193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c0c0c7b90 msgr2=0x7f5c0c1e6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:00.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.562+0000 7f5c12600640 1 --2- 172.21.15.29:0/2064479193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c0c0c7b90 0x7f5c0c1e6610 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f5c04000f90 tx=0x7f5bf0005840 comp rx=0 tx=0).stop 2024-04-17T19:58:00.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.562+0000 7f5c12600640 1 -- 172.21.15.29:0/2064479193 shutdown_connections 2024-04-17T19:58:00.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.562+0000 7f5c12600640 1 --2- 172.21.15.29:0/2064479193 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5bec0683b0 0x7f5bec06a8c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:00.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.562+0000 7f5c12600640 1 --2- 172.21.15.29:0/2064479193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c0c0c7b90 0x7f5c0c1e6610 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:00.559 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.562+0000 7f5c12600640 1 --2- 172.21.15.29:0/2064479193 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5c0c0c7780 0x7f5c0c0ce3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:00.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.562+0000 7f5c12600640 1 --2- 172.21.15.29:0/2064479193 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5bf0004d50 0x7f5c0c0c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:00.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.562+0000 7f5c12600640 1 -- 172.21.15.29:0/2064479193 >> 172.21.15.29:0/2064479193 conn(0x7f5c0c06e3d0 msgr2=0x7f5c0c0cf6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:00.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.562+0000 7f5c12600640 1 -- 172.21.15.29:0/2064479193 shutdown_connections 2024-04-17T19:58:00.560 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:00.562+0000 7f5c12600640 1 -- 172.21.15.29:0/2064479193 wait complete. 2024-04-17T19:58:00.579 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-17T19:58:02.580 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:58:02.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.742+0000 7f7df7200640 1 Processor -- start 2024-04-17T19:58:02.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.742+0000 7f7df7200640 1 -- start start 2024-04-17T19:58:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.742+0000 7f7df7200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7df8156370 0x7f7df80ca110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.742+0000 7f7df7200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7df815c970 0x7f7df80caff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.742+0000 7f7df7200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7df806c730 con 0x7f7df80cabe0 2024-04-17T19:58:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.742+0000 7f7df7200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f7df806c9f0 con 0x7f7df815c970 2024-04-17T19:58:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.742+0000 7f7df7200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f7df806ccd0 con 0x7f7df8156370 2024-04-17T19:58:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.742+0000 7f7df6800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7df815c970 0x7f7df80caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.742+0000 7f7df6800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7df815c970 0x7f7df80caff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34280/0 (socket says 172.21.15.29:34280) 2024-04-17T19:58:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.742+0000 7f7df6800640 1 -- 172.21.15.29:0/833049196 learned_addr learned my addr 172.21.15.29:0/833049196 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.742+0000 7f7df5e00640 1 --2- 172.21.15.29:0/833049196 >> v2:172.21.15.37:3300/0 conn(0x7f7df8156370 0x7f7df80ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.742+0000 7f7df6800640 1 -- 172.21.15.29:0/833049196 >> v2:172.21.15.37:3300/0 conn(0x7f7df8156370 msgr2=0x7f7df80ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.742+0000 7f7df6800640 1 --2- 172.21.15.29:0/833049196 >> v2:172.21.15.37:3300/0 conn(0x7f7df8156370 0x7f7df80ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:02.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.742+0000 7f7df6800640 1 -- 172.21.15.29:0/833049196 >> v1:172.21.15.37:6789/0 conn(0x7f7df80cabe0 legacy=0x7f7df815c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:02.739 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.742+0000 7f7df6800640 1 -- 172.21.15.29:0/833049196 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df806cfb0 con 0x7f7df815c970 2024-04-17T19:58:02.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.746+0000 7f7df6800640 1 --2- 172.21.15.29:0/833049196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7df815c970 0x7f7df80caff0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7df80cfff0 tx=0x7f7df001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f18687bda930d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:02.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.746+0000 7f7df4a00640 1 -- 172.21.15.29:0/833049196 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df00b4070 con 0x7f7df815c970 2024-04-17T19:58:02.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.746+0000 7f7df4a00640 1 -- 172.21.15.29:0/833049196 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7df00af400 con 0x7f7df815c970 2024-04-17T19:58:02.741 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.746+0000 7f7df4a00640 1 -- 172.21.15.29:0/833049196 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df00af7d0 con 0x7f7df815c970 2024-04-17T19:58:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.746+0000 7f7df7200640 1 -- 172.21.15.29:0/833049196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7df815c970 msgr2=0x7f7df80caff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:02.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.746+0000 7f7df7200640 1 --2- 172.21.15.29:0/833049196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7df815c970 0x7f7df80caff0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7df80cfff0 tx=0x7f7df001f1e0 comp rx=0 tx=0).stop 2024-04-17T19:58:02.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.746+0000 7f7df7200640 1 -- 172.21.15.29:0/833049196 shutdown_connections 2024-04-17T19:58:02.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.746+0000 7f7df7200640 1 --2- 172.21.15.29:0/833049196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7df815c970 0x7f7df80caff0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:02.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.746+0000 7f7df7200640 1 --2- 172.21.15.29:0/833049196 >> v2:172.21.15.37:3300/0 conn(0x7f7df8156370 0x7f7df80ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:02.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.746+0000 7f7df7200640 1 -- 172.21.15.29:0/833049196 >> 172.21.15.29:0/833049196 conn(0x7f7df80680e0 msgr2=0x7f7df806dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:02.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.746+0000 7f7df7200640 1 -- 172.21.15.29:0/833049196 shutdown_connections 2024-04-17T19:58:02.744 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.746+0000 7f7df7200640 1 -- 172.21.15.29:0/833049196 wait complete. 2024-04-17T19:58:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df7200640 1 Processor -- start 2024-04-17T19:58:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df7200640 1 -- start start 2024-04-17T19:58:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df7200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7df80cabe0 0x7f7df8156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df7200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7df8156370 0x7f7df80c8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df7200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7df815c970 0x7f7df8159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df7200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7df80ca650 con 0x7f7df80cabe0 2024-04-17T19:58:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df7200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7df806d570 con 0x7f7df8156370 2024-04-17T19:58:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df7200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7df806e7c0 con 0x7f7df815c970 2024-04-17T19:58:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df5e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7df80cabe0 0x7f7df8156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df5e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7df80cabe0 0x7f7df8156940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34292/0 (socket says 172.21.15.29:34292) 2024-04-17T19:58:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df5e00640 1 -- 172.21.15.29:0/692794057 learned_addr learned my addr 172.21.15.29:0/692794057 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df6800640 1 --2- 172.21.15.29:0/692794057 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7df815c970 0x7f7df8159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df5400640 1 --2- 172.21.15.29:0/692794057 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7df8156370 0x7f7df80c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df5e00640 1 -- 172.21.15.29:0/692794057 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7df815c970 msgr2=0x7f7df8159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df5e00640 1 --2- 172.21.15.29:0/692794057 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7df815c970 0x7f7df8159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df5e00640 1 -- 172.21.15.29:0/692794057 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7df8156370 msgr2=0x7f7df80c8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df5e00640 1 --2- 172.21.15.29:0/692794057 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7df8156370 0x7f7df80c8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df5e00640 1 -- 172.21.15.29:0/692794057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df8156e80 con 0x7f7df80cabe0 2024-04-17T19:58:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df5e00640 1 --2- 172.21.15.29:0/692794057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7df80cabe0 0x7f7df8156940 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7de800d620 tx=0x7f7de800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7dee000640 1 -- 172.21.15.29:0/692794057 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de800b620 con 0x7f7df80cabe0 2024-04-17T19:58:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7dee000640 1 -- 172.21.15.29:0/692794057 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7de8019070 con 0x7f7df80cabe0 2024-04-17T19:58:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7dee000640 1 -- 172.21.15.29:0/692794057 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de800b9e0 con 0x7f7df80cabe0 2024-04-17T19:58:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.750+0000 7f7df7200640 1 -- 172.21.15.29:0/692794057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7df81f73d0 con 0x7f7df80cabe0 2024-04-17T19:58:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.754+0000 7f7df7200640 1 -- 172.21.15.29:0/692794057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7df81f7950 con 0x7f7df80cabe0 2024-04-17T19:58:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.754+0000 7f7df7200640 1 -- 172.21.15.29:0/692794057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7df804ff10 con 0x7f7df80cabe0 2024-04-17T19:58:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.754+0000 7f7dee000640 1 -- 172.21.15.29:0/692794057 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7de800bb80 con 0x7f7df80cabe0 2024-04-17T19:58:02.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.754+0000 7f7dee000640 1 --2- 172.21.15.29:0/692794057 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7dd8068380 0x7f7dd806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:02.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.754+0000 7f7dee000640 1 -- 172.21.15.29:0/692794057 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7135+0+0 (secure 0 0 0) 0x7f7de8087460 con 0x7f7df80cabe0 2024-04-17T19:58:02.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.754+0000 7f7df5400640 1 --2- 172.21.15.29:0/692794057 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7dd8068380 0x7f7dd806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:02.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.758+0000 7f7df5400640 1 --2- 172.21.15.29:0/692794057 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7dd8068380 0x7f7dd806a890 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7df80c8bf0 tx=0x7f7de4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:02.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.758+0000 7f7dee000640 1 -- 172.21.15.29:0/692794057 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7de8057360 con 0x7f7df80cabe0 2024-04-17T19:58:02.959 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.962+0000 7f7df7200640 1 -- 172.21.15.29:0/692794057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7df806dd80 con 0x7f7df80cabe0 2024-04-17T19:58:02.960 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:58:02.960 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:55:40.944059+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":4302,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4302":{"gid":4302,"name":"e","rank":1,"incarnation":10,"state":"up:active","state_seq":9,"addr":"172.21.15.29:6835/270402476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":270402476},{"type":"v1","addr":"172.21.15.29:6835","nonce":270402476}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5144,4305,5212,4302]},"id":1}]} 2024-04-17T19:58:02.960 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 43 2024-04-17T19:58:03.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.966+0000 7f7dee000640 1 -- 172.21.15.29:0/692794057 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) ==== 94+0+6904 (secure 0 0 0) 0x7f7de8056c60 con 0x7f7df80cabe0 2024-04-17T19:58:03.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.966+0000 7f7df7200640 1 -- 172.21.15.29:0/692794057 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7dd8068380 msgr2=0x7f7dd806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:03.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.966+0000 7f7df7200640 1 --2- 172.21.15.29:0/692794057 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7dd8068380 0x7f7dd806a890 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7df80c8bf0 tx=0x7f7de4009040 comp rx=0 tx=0).stop 2024-04-17T19:58:03.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.966+0000 7f7df7200640 1 -- 172.21.15.29:0/692794057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7df80cabe0 msgr2=0x7f7df8156940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:03.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.966+0000 7f7df7200640 1 --2- 172.21.15.29:0/692794057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7df80cabe0 0x7f7df8156940 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7de800d620 tx=0x7f7de800dae0 comp rx=0 tx=0).stop 2024-04-17T19:58:03.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.966+0000 7f7df7200640 1 -- 172.21.15.29:0/692794057 shutdown_connections 2024-04-17T19:58:03.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.966+0000 7f7df7200640 1 --2- 172.21.15.29:0/692794057 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7dd8068380 0x7f7dd806a890 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:03.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.966+0000 7f7df7200640 1 --2- 172.21.15.29:0/692794057 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7df815c970 0x7f7df8159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:03.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.966+0000 7f7df7200640 1 --2- 172.21.15.29:0/692794057 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7df8156370 0x7f7df80c8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:03.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.966+0000 7f7df7200640 1 --2- 172.21.15.29:0/692794057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7df80cabe0 0x7f7df8156940 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:03.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.966+0000 7f7df7200640 1 -- 172.21.15.29:0/692794057 >> 172.21.15.29:0/692794057 conn(0x7f7df80680e0 msgr2=0x7f7df8154510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:03.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.966+0000 7f7df7200640 1 -- 172.21.15.29:0/692794057 shutdown_connections 2024-04-17T19:58:03.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:02.966+0000 7f7df7200640 1 -- 172.21.15.29:0/692794057 wait complete. 2024-04-17T19:58:03.487 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2024-04-17T19:58:05.488 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:58:05.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.658+0000 7f4a7f200640 1 Processor -- start 2024-04-17T19:58:05.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.658+0000 7f4a7f200640 1 -- start start 2024-04-17T19:58:05.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.662+0000 7f4a7f200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4a7806e960 0x7f4a78070d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:05.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.662+0000 7f4a7f200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a7815cae0 0x7f4a781561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:05.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.662+0000 7f4a7f200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a7815bf10 con 0x7f4a78155da0 2024-04-17T19:58:05.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.662+0000 7f4a7f200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f4a781566f0 con 0x7f4a7806e960 2024-04-17T19:58:05.656 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.662+0000 7f4a7f200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4a780c98e0 con 0x7f4a7815cae0 2024-04-17T19:58:05.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.662+0000 7f4a77e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4a7806e960 0x7f4a78070d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:05.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.662+0000 7f4a77e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4a7806e960 0x7f4a78070d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34306/0 (socket says 172.21.15.29:34306) 2024-04-17T19:58:05.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.662+0000 7f4a77e00640 1 -- 172.21.15.29:0/1537364330 learned_addr learned my addr 172.21.15.29:0/1537364330 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:05.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.662+0000 7f4a7d200640 1 --2- 172.21.15.29:0/1537364330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a7815cae0 0x7f4a781561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:05.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.662+0000 7f4a77e00640 1 -- 172.21.15.29:0/1537364330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a7815cae0 msgr2=0x7f4a781561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:05.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.662+0000 7f4a77e00640 1 --2- 172.21.15.29:0/1537364330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a7815cae0 0x7f4a781561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:05.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.662+0000 7f4a77e00640 1 -- 172.21.15.29:0/1537364330 >> v1:172.21.15.37:6789/0 conn(0x7f4a78155da0 legacy=0x7f4a7815c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:05.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.662+0000 7f4a77e00640 1 -- 172.21.15.29:0/1537364330 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a780c9bc0 con 0x7f4a7806e960 2024-04-17T19:58:05.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.666+0000 7f4a77e00640 1 --2- 172.21.15.29:0/1537364330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a7806e960 0x7f4a78070d70 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009870 tx=0x7f4a6801ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e29ab657ac0b365e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:05.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.666+0000 7f4a77400640 1 -- 172.21.15.29:0/1537364330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a680b5070 con 0x7f4a7806e960 2024-04-17T19:58:05.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.666+0000 7f4a77400640 1 -- 172.21.15.29:0/1537364330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a680a8a50 con 0x7f4a7806e960 2024-04-17T19:58:05.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.666+0000 7f4a77400640 1 -- 172.21.15.29:0/1537364330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a680a8e20 con 0x7f4a7806e960 2024-04-17T19:58:05.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.666+0000 7f4a7f200640 1 -- 172.21.15.29:0/1537364330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a7806e960 msgr2=0x7f4a78070d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:05.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.666+0000 7f4a7f200640 1 --2- 172.21.15.29:0/1537364330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a7806e960 0x7f4a78070d70 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f4a68009870 tx=0x7f4a6801ed50 comp rx=0 tx=0).stop 2024-04-17T19:58:05.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.666+0000 7f4a7f200640 1 -- 172.21.15.29:0/1537364330 shutdown_connections 2024-04-17T19:58:05.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.666+0000 7f4a7f200640 1 --2- 172.21.15.29:0/1537364330 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a7815cae0 0x7f4a781561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:05.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.666+0000 7f4a7f200640 1 --2- 172.21.15.29:0/1537364330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a7806e960 0x7f4a78070d70 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:05.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.666+0000 7f4a7f200640 1 -- 172.21.15.29:0/1537364330 >> 172.21.15.29:0/1537364330 conn(0x7f4a780ce5e0 msgr2=0x7f4a780ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:05.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.666+0000 7f4a7f200640 1 -- 172.21.15.29:0/1537364330 shutdown_connections 2024-04-17T19:58:05.662 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.666+0000 7f4a7f200640 1 -- 172.21.15.29:0/1537364330 wait complete. 2024-04-17T19:58:05.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a7f200640 1 Processor -- start 2024-04-17T19:58:05.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a7f200640 1 -- start start 2024-04-17T19:58:05.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a7f200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a7806e960 0x7f4a7815b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:05.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a7f200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a78155da0 0x7f4a7813fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:05.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a7f200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a7815cae0 0x7f4a78142ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:05.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a7f200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4a780ca460 con 0x7f4a78155da0 2024-04-17T19:58:05.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a7f200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4a780ca180 con 0x7f4a7815cae0 2024-04-17T19:58:05.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a7f200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4a7806f7b0 con 0x7f4a7806e960 2024-04-17T19:58:05.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a6fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a78155da0 0x7f4a7813fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:05.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a6fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a78155da0 0x7f4a7813fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34314/0 (socket says 172.21.15.29:34314) 2024-04-17T19:58:05.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a6fe00640 1 -- 172.21.15.29:0/692191708 learned_addr learned my addr 172.21.15.29:0/692191708 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:05.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a7d200640 1 --2- 172.21.15.29:0/692191708 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a7815cae0 0x7f4a78142ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:05.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a77e00640 1 --2- 172.21.15.29:0/692191708 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a7806e960 0x7f4a7815b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:05.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a6fe00640 1 -- 172.21.15.29:0/692191708 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a7806e960 msgr2=0x7f4a7815b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:05.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a6fe00640 1 --2- 172.21.15.29:0/692191708 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a7806e960 0x7f4a7815b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:05.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a6fe00640 1 -- 172.21.15.29:0/692191708 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a7815cae0 msgr2=0x7f4a78142ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:05.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a6fe00640 1 --2- 172.21.15.29:0/692191708 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a7815cae0 0x7f4a78142ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:05.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a6fe00640 1 -- 172.21.15.29:0/692191708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a7814bee0 con 0x7f4a78155da0 2024-04-17T19:58:05.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a7d200640 1 --2- 172.21.15.29:0/692191708 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a7815cae0 0x7f4a78142ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:58:05.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a77e00640 1 --2- 172.21.15.29:0/692191708 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a7806e960 0x7f4a7815b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:58:05.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a6fe00640 1 --2- 172.21.15.29:0/692191708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a78155da0 0x7f4a7813fe30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000c390 tx=0x7f4a6000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:05.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a74c00640 1 -- 172.21.15.29:0/692191708 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a60017070 con 0x7f4a78155da0 2024-04-17T19:58:05.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a74c00640 1 -- 172.21.15.29:0/692191708 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a600052d0 con 0x7f4a78155da0 2024-04-17T19:58:05.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a74c00640 1 -- 172.21.15.29:0/692191708 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a60003840 con 0x7f4a78155da0 2024-04-17T19:58:05.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a7f200640 1 -- 172.21.15.29:0/692191708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a7814b920 con 0x7f4a78155da0 2024-04-17T19:58:05.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.670+0000 7f4a7f200640 1 -- 172.21.15.29:0/692191708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a78159e90 con 0x7f4a78155da0 2024-04-17T19:58:05.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.674+0000 7f4a7f200640 1 -- 172.21.15.29:0/692191708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a78069b90 con 0x7f4a78155da0 2024-04-17T19:58:05.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.674+0000 7f4a74c00640 1 -- 172.21.15.29:0/692191708 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4a6001a070 con 0x7f4a78155da0 2024-04-17T19:58:05.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.674+0000 7f4a74c00640 1 --2- 172.21.15.29:0/692191708 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4a54068380 0x7f4a5406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:05.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.674+0000 7f4a74c00640 1 -- 172.21.15.29:0/692191708 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7f4a60086250 con 0x7f4a78155da0 2024-04-17T19:58:05.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.674+0000 7f4a77e00640 1 --2- 172.21.15.29:0/692191708 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4a54068380 0x7f4a5406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:05.672 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.678+0000 7f4a77e00640 1 --2- 172.21.15.29:0/692191708 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4a54068380 0x7f4a5406a890 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4a680b2000 tx=0x7f4a68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:05.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.682+0000 7f4a74c00640 1 -- 172.21.15.29:0/692191708 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4a7815bf10 con 0x7f4a78155da0 2024-04-17T19:58:05.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.918+0000 7f4a7f200640 1 -- 172.21.15.29:0/692191708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a78069580 con 0x7f4a78155da0 2024-04-17T19:58:05.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.918+0000 7f4a74c00640 1 -- 172.21.15.29:0/692191708 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+6078 (secure 0 0 0) 0x7f4a60005480 con 0x7f4a78155da0 2024-04-17T19:58:05.913 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:58:05.913 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:03.917169+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:replay","state_seq":1,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5144,5212,5350]},"id":1}]} 2024-04-17T19:58:05.913 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 44 2024-04-17T19:58:05.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.918+0000 7f4a7f200640 1 -- 172.21.15.29:0/692191708 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4a54068380 msgr2=0x7f4a5406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:05.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.918+0000 7f4a7f200640 1 --2- 172.21.15.29:0/692191708 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4a54068380 0x7f4a5406a890 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4a680b2000 tx=0x7f4a68008040 comp rx=0 tx=0).stop 2024-04-17T19:58:05.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.918+0000 7f4a7f200640 1 -- 172.21.15.29:0/692191708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a78155da0 msgr2=0x7f4a7813fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:05.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.918+0000 7f4a7f200640 1 --2- 172.21.15.29:0/692191708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a78155da0 0x7f4a7813fe30 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f4a6000c390 tx=0x7f4a6000c850 comp rx=0 tx=0).stop 2024-04-17T19:58:05.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.918+0000 7f4a7f200640 1 -- 172.21.15.29:0/692191708 shutdown_connections 2024-04-17T19:58:05.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.918+0000 7f4a7f200640 1 --2- 172.21.15.29:0/692191708 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4a54068380 0x7f4a5406a890 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:05.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.918+0000 7f4a7f200640 1 --2- 172.21.15.29:0/692191708 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4a7815cae0 0x7f4a78142ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:05.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.918+0000 7f4a7f200640 1 --2- 172.21.15.29:0/692191708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a78155da0 0x7f4a7813fe30 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:05.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.918+0000 7f4a7f200640 1 --2- 172.21.15.29:0/692191708 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4a7806e960 0x7f4a7815b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:05.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.918+0000 7f4a7f200640 1 -- 172.21.15.29:0/692191708 >> 172.21.15.29:0/692191708 conn(0x7f4a780ce5e0 msgr2=0x7f4a78070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:05.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.918+0000 7f4a7f200640 1 -- 172.21.15.29:0/692191708 shutdown_connections 2024-04-17T19:58:05.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:05.918+0000 7f4a7f200640 1 -- 172.21.15.29:0/692191708 wait complete. 2024-04-17T19:58:05.937 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2024-04-17T19:58:05.937 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-17T19:58:05.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:58:06.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.098+0000 7f66bf000640 1 Processor -- start 2024-04-17T19:58:06.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.098+0000 7f66bf000640 1 -- start start 2024-04-17T19:58:06.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.098+0000 7f66bf000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f66b80cc8c0 0x7f66b80cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:06.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.098+0000 7f66bf000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f66b81597a0 0x7f66b81635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:06.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.098+0000 7f66bf000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66b80c88b0 con 0x7f66b8159390 2024-04-17T19:58:06.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.098+0000 7f66bf000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f66b80c6ce0 con 0x7f66b81597a0 2024-04-17T19:58:06.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.098+0000 7f66bf000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f66b80c6fc0 con 0x7f66b80cc8c0 2024-04-17T19:58:06.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.098+0000 7f66bd000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f66b81597a0 0x7f66b81635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:06.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.098+0000 7f66b7e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f66b80cc8c0 0x7f66b80cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:06.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.102+0000 7f66af400640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f66b8159390 0x7f66b80ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:46848/0 (socket says 172.21.15.29:46848) 2024-04-17T19:58:06.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.102+0000 7f66b7e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f66b80cc8c0 0x7f66b80cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:40520/0 (socket says 172.21.15.29:40520) 2024-04-17T19:58:06.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.102+0000 7f66af400640 1 -- 172.21.15.29:0/1826981592 learned_addr learned my addr 172.21.15.29:0/1826981592 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:58:06.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.102+0000 7f66bd000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f66b81597a0 0x7f66b81635e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:33636/0 (socket says 172.21.15.29:33636) 2024-04-17T19:58:06.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.102+0000 7f66b7e00640 1 -- 172.21.15.29:0/1826981592 >> v2:172.21.15.37:3300/0 conn(0x7f66b81597a0 msgr2=0x7f66b81635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:06.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.102+0000 7f66b7e00640 1 --2- 172.21.15.29:0/1826981592 >> v2:172.21.15.37:3300/0 conn(0x7f66b81597a0 0x7f66b81635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:06.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.102+0000 7f66b7e00640 1 -- 172.21.15.29:0/1826981592 >> v1:172.21.15.37:6789/0 conn(0x7f66b8159390 legacy=0x7f66b80ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:06.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.102+0000 7f66b7e00640 1 -- 172.21.15.29:0/1826981592 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b80c72a0 con 0x7f66b80cc8c0 2024-04-17T19:58:06.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.102+0000 7f66bd000640 1 --2- 172.21.15.29:0/1826981592 >> v2:172.21.15.37:3300/0 conn(0x7f66b81597a0 0x7f66b81635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T19:58:06.098 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.102+0000 7f66b7e00640 1 --2- 172.21.15.29:0/1826981592 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f66b80cc8c0 0x7f66b80cccd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f66a80097c0 tx=0x7f66a801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e77a44ce7a935c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.102+0000 7f66b7400640 1 -- 172.21.15.29:0/1826981592 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a80b0070 con 0x7f66b80cc8c0 2024-04-17T19:58:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.106+0000 7f66b7400640 1 -- 172.21.15.29:0/1826981592 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66a80ab600 con 0x7f66b80cc8c0 2024-04-17T19:58:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.106+0000 7f66b7400640 1 -- 172.21.15.29:0/1826981592 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a80ab9d0 con 0x7f66b80cc8c0 2024-04-17T19:58:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.106+0000 7f66bf000640 1 -- 172.21.15.29:0/1826981592 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f66b80cc8c0 msgr2=0x7f66b80cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:06.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.106+0000 7f66bf000640 1 --2- 172.21.15.29:0/1826981592 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f66b80cc8c0 0x7f66b80cccd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f66a80097c0 tx=0x7f66a801c4a0 comp rx=0 tx=0).stop 2024-04-17T19:58:06.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.106+0000 7f66bf000640 1 -- 172.21.15.29:0/1826981592 shutdown_connections 2024-04-17T19:58:06.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.106+0000 7f66bf000640 1 --2- 172.21.15.29:0/1826981592 >> v2:172.21.15.37:3300/0 conn(0x7f66b81597a0 0x7f66b81635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:06.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.106+0000 7f66bf000640 1 --2- 172.21.15.29:0/1826981592 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f66b80cc8c0 0x7f66b80cccd0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:06.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.106+0000 7f66bf000640 1 -- 172.21.15.29:0/1826981592 >> 172.21.15.29:0/1826981592 conn(0x7f66b806fde0 msgr2=0x7f66b80701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:06.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.106+0000 7f66bf000640 1 -- 172.21.15.29:0/1826981592 shutdown_connections 2024-04-17T19:58:06.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.106+0000 7f66bf000640 1 -- 172.21.15.29:0/1826981592 wait complete. 2024-04-17T19:58:06.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.106+0000 7f66bf000640 1 Processor -- start 2024-04-17T19:58:06.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.106+0000 7f66bf000640 1 -- start start 2024-04-17T19:58:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66bf000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66b80cc8c0 0x7f66b815b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66bf000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f66b8159390 0x7f66b813ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66bf000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66b81597a0 0x7f66b813d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66bf000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f66b80cabc0 con 0x7f66b81597a0 2024-04-17T19:58:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66bf000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f66b80caa40 con 0x7f66b80cc8c0 2024-04-17T19:58:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66bf000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f66b80c6880 con 0x7f66b8159390 2024-04-17T19:58:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66bd000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66b81597a0 0x7f66b813d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:06.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66bd000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66b81597a0 0x7f66b813d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34322/0 (socket says 172.21.15.29:34322) 2024-04-17T19:58:06.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66bd000640 1 -- 172.21.15.29:0/3676383390 learned_addr learned my addr 172.21.15.29:0/3676383390 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:06.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66b7e00640 1 --2- 172.21.15.29:0/3676383390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66b80cc8c0 0x7f66b815b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:06.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66af400640 1 --2- 172.21.15.29:0/3676383390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f66b8159390 0x7f66b813ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:06.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66bd000640 1 -- 172.21.15.29:0/3676383390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f66b8159390 msgr2=0x7f66b813ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:06.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66bd000640 1 --2- 172.21.15.29:0/3676383390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f66b8159390 0x7f66b813ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:06.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66bd000640 1 -- 172.21.15.29:0/3676383390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66b80cc8c0 msgr2=0x7f66b815b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:06.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66bd000640 1 --2- 172.21.15.29:0/3676383390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66b80cc8c0 0x7f66b815b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:06.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66bd000640 1 -- 172.21.15.29:0/3676383390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b815a070 con 0x7f66b81597a0 2024-04-17T19:58:06.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66bd000640 1 --2- 172.21.15.29:0/3676383390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66b81597a0 0x7f66b813d370 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f66b000efc0 tx=0x7f66b000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:06.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66b4c00640 1 -- 172.21.15.29:0/3676383390 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0018030 con 0x7f66b81597a0 2024-04-17T19:58:06.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66b4c00640 1 -- 172.21.15.29:0/3676383390 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b000cdf0 con 0x7f66b81597a0 2024-04-17T19:58:06.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66b4c00640 1 -- 172.21.15.29:0/3676383390 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b0003bc0 con 0x7f66b81597a0 2024-04-17T19:58:06.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66bf000640 1 -- 172.21.15.29:0/3676383390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b815a350 con 0x7f66b81597a0 2024-04-17T19:58:06.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66bf000640 1 -- 172.21.15.29:0/3676383390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b815a910 con 0x7f66b81597a0 2024-04-17T19:58:06.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.110+0000 7f66bf000640 1 -- 172.21.15.29:0/3676383390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66b8069b90 con 0x7f66b81597a0 2024-04-17T19:58:06.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.114+0000 7f66b4c00640 1 -- 172.21.15.29:0/3676383390 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f66b001b070 con 0x7f66b81597a0 2024-04-17T19:58:06.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.114+0000 7f66b4c00640 1 --2- 172.21.15.29:0/3676383390 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6694068380 0x7f669406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:06.108 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.114+0000 7f66b4c00640 1 -- 172.21.15.29:0/3676383390 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7f66b0087860 con 0x7f66b81597a0 2024-04-17T19:58:06.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.114+0000 7f66b7e00640 1 --2- 172.21.15.29:0/3676383390 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6694068380 0x7f669406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:06.110 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.114+0000 7f66b7e00640 1 --2- 172.21.15.29:0/3676383390 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6694068380 0x7f669406a890 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f66a80ad000 tx=0x7f66a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:06.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.118+0000 7f66b4c00640 1 -- 172.21.15.29:0/3676383390 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f66b0010040 con 0x7f66b81597a0 2024-04-17T19:58:06.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.318+0000 7f66bf000640 1 -- 172.21.15.29:0/3676383390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66b8166940 con 0x7f66b81597a0 2024-04-17T19:58:06.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.318+0000 7f66b4c00640 1 -- 172.21.15.29:0/3676383390 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+6078 (secure 0 0 0) 0x7f66b0057180 con 0x7f66b81597a0 2024-04-17T19:58:06.315 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:58:06.315 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:03.917169+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:replay","state_seq":1,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5144,5212,5350]},"id":1}]} 2024-04-17T19:58:06.315 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 44 2024-04-17T19:58:06.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.322+0000 7f66bf000640 1 -- 172.21.15.29:0/3676383390 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6694068380 msgr2=0x7f669406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:06.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.322+0000 7f66bf000640 1 --2- 172.21.15.29:0/3676383390 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6694068380 0x7f669406a890 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f66a80ad000 tx=0x7f66a8008040 comp rx=0 tx=0).stop 2024-04-17T19:58:06.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.322+0000 7f66bf000640 1 -- 172.21.15.29:0/3676383390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66b81597a0 msgr2=0x7f66b813d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:06.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.322+0000 7f66bf000640 1 --2- 172.21.15.29:0/3676383390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66b81597a0 0x7f66b813d370 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f66b000efc0 tx=0x7f66b000c490 comp rx=0 tx=0).stop 2024-04-17T19:58:06.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.322+0000 7f66bf000640 1 -- 172.21.15.29:0/3676383390 shutdown_connections 2024-04-17T19:58:06.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.322+0000 7f66bf000640 1 --2- 172.21.15.29:0/3676383390 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6694068380 0x7f669406a890 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:06.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.322+0000 7f66bf000640 1 --2- 172.21.15.29:0/3676383390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66b81597a0 0x7f66b813d370 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:06.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.322+0000 7f66bf000640 1 --2- 172.21.15.29:0/3676383390 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f66b8159390 0x7f66b813ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:06.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.322+0000 7f66bf000640 1 --2- 172.21.15.29:0/3676383390 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f66b80cc8c0 0x7f66b815b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:06.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.322+0000 7f66bf000640 1 -- 172.21.15.29:0/3676383390 >> 172.21.15.29:0/3676383390 conn(0x7f66b806fde0 msgr2=0x7f66b80cbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:06.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.322+0000 7f66bf000640 1 -- 172.21.15.29:0/3676383390 shutdown_connections 2024-04-17T19:58:06.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:06.322+0000 7f66bf000640 1 -- 172.21.15.29:0/3676383390 wait complete. 2024-04-17T19:58:06.340 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 44, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-17T19:47:49.315515+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 44, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4305": { "addr": "172.21.15.29:6837/2691590158", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6833", "nonce": 2691590158, "type": "v2" }, { "addr": "172.21.15.29:6837", "nonce": 2691590158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4305, "incarnation": 5, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5144": { "addr": "172.21.15.37:6839/858161860", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6836", "nonce": 858161860, "type": "v2" }, { "addr": "172.21.15.37:6839", "nonce": 858161860, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5144, "incarnation": 38, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_5212": { "addr": "172.21.15.37:6838/767681664", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6835", "nonce": 767681664, "type": "v2" }, { "addr": "172.21.15.37:6838", "nonce": 767681664, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5212, "incarnation": 26, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 31 }, "gid_5350": { "addr": "172.21.15.37:6837/2495953425", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6834", "nonce": 2495953425, "type": "v2" }, { "addr": "172.21.15.37:6837", "nonce": 2495953425, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5350, "incarnation": 32, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 18 }, "gid_5631": { "addr": "172.21.15.29:6836/2376553580", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6832", "nonce": 2376553580, "type": "v2" }, { "addr": "172.21.15.29:6836", "nonce": 2376553580, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5631, "incarnation": 44, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 73, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-17T19:58:03.917169+0000", "qdb_cluster": [ 4305, 5144, 5212, 5350 ], "qdb_leader": 4305, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4305, "mds_1": 5631, "mds_2": 5144, "mds_3": 5350, "mds_4": 5212 } } } ], "standbys": [] } 2024-04-17T19:58:08.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:58:08.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe9a6200640 1 Processor -- start 2024-04-17T19:58:08.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe9a6200640 1 -- start start 2024-04-17T19:58:08.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe9a6200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe9a00c9370 0x7fe9a0167940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:08.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe9a6200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9a006c1e0 con 0x7fe9a015ca30 2024-04-17T19:58:08.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe9a6200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9a015f720 con 0x7fe9a00c8f60 2024-04-17T19:58:08.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe9a6200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fe9a01604c0 con 0x7fe9a00c9370 2024-04-17T19:58:08.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe99f400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe9a015ca30 0x7fe9a0156170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45264/0 (socket says 172.21.15.29:45264) 2024-04-17T19:58:08.502 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe99f400640 1 -- 172.21.15.29:0/3089005489 learned_addr learned my addr 172.21.15.29:0/3089005489 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:58:08.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe99fe00640 1 --2- 172.21.15.29:0/3089005489 >> v2:172.21.15.37:3300/0 conn(0x7fe9a00c9370 0x7fe9a0167940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:08.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe99e000640 1 -- 172.21.15.29:0/3089005489 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3945459999 0 0) 0x7fe9a006c1e0 con 0x7fe9a015ca30 2024-04-17T19:58:08.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe99e000640 1 -- 172.21.15.29:0/3089005489 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe984003530 con 0x7fe9a015ca30 2024-04-17T19:58:08.504 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe99e000640 1 -- 172.21.15.29:0/3089005489 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4169830223 0 0) 0x7fe9a015f720 con 0x7fe9a00c8f60 2024-04-17T19:58:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe99e000640 1 -- 172.21.15.29:0/3089005489 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9a006c1e0 con 0x7fe9a00c8f60 2024-04-17T19:58:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe99e000640 1 -- 172.21.15.29:0/3089005489 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4229752074 0 0) 0x7fe984003530 con 0x7fe9a015ca30 2024-04-17T19:58:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe99e000640 1 -- 172.21.15.29:0/3089005489 >> v2:172.21.15.37:3300/0 conn(0x7fe9a00c9370 msgr2=0x7fe9a0167940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe99e000640 1 --2- 172.21.15.29:0/3089005489 >> v2:172.21.15.37:3300/0 conn(0x7fe9a00c9370 0x7fe9a0167940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe99e000640 1 -- 172.21.15.29:0/3089005489 >> v1:172.21.15.37:6789/0 conn(0x7fe9a00c8f60 legacy=0x7fe9a0067ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe99e000640 1 -- 172.21.15.29:0/3089005489 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a006c450 con 0x7fe9a015ca30 2024-04-17T19:58:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.506+0000 7fe99e000640 1 -- 172.21.15.29:0/3089005489 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fe990001680 con 0x7fe9a015ca30 2024-04-17T19:58:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.510+0000 7fe99e000640 1 -- 172.21.15.29:0/3089005489 >> v1:172.21.15.29:6789/0 conn(0x7fe9a015ca30 legacy=0x7fe9a0156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.510+0000 7fe99e000640 1 --2- 172.21.15.29:0/3089005489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe984003bc0 0x7fe984004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.510+0000 7fe99e000640 1 --2- 172.21.15.29:0/3089005489 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe984004810 0x7fe984004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.510+0000 7fe99e000640 1 --2- 172.21.15.29:0/3089005489 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe984005220 0x7fe98400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.510+0000 7fe99e000640 1 -- 172.21.15.29:0/3089005489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a01604c0 con 0x7fe984003bc0 2024-04-17T19:58:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.510+0000 7fe99e000640 1 -- 172.21.15.29:0/3089005489 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe98400c1c0 con 0x7fe984005220 2024-04-17T19:58:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.510+0000 7fe99e000640 1 -- 172.21.15.29:0/3089005489 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe98400c3c0 con 0x7fe984004810 2024-04-17T19:58:08.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.510+0000 7fe99f400640 1 --2- 172.21.15.29:0/3089005489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe984003bc0 0x7fe984004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:08.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.510+0000 7fe99fe00640 1 --2- 172.21.15.29:0/3089005489 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe984005220 0x7fe98400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:08.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.510+0000 7fe99ea00640 1 --2- 172.21.15.29:0/3089005489 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe984004810 0x7fe984004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:08.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.510+0000 7fe99f400640 1 -- 172.21.15.29:0/3089005489 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe984004810 msgr2=0x7fe984004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:08.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.510+0000 7fe99f400640 1 --2- 172.21.15.29:0/3089005489 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe984004810 0x7fe984004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:08.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.510+0000 7fe99f400640 1 -- 172.21.15.29:0/3089005489 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe984005220 msgr2=0x7fe98400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:08.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.510+0000 7fe99f400640 1 --2- 172.21.15.29:0/3089005489 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe984005220 0x7fe98400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:08.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.510+0000 7fe99f400640 1 -- 172.21.15.29:0/3089005489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe984003530 con 0x7fe984003bc0 2024-04-17T19:58:08.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.510+0000 7fe99fe00640 1 -- 172.21.15.29:0/3089005489 reap_dead start 2024-04-17T19:58:08.506 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.510+0000 7fe99ea00640 1 --2- 172.21.15.29:0/3089005489 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe984004810 0x7fe984004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:58:08.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.514+0000 7fe99f400640 1 --2- 172.21.15.29:0/3089005489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe984003bc0 0x7fe984004010 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe99000c190 tx=0x7fe99001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1cc42ac8b2ce2a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:08.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.514+0000 7fe99e000640 1 -- 172.21.15.29:0/3089005489 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe990007cd0 con 0x7fe984003bc0 2024-04-17T19:58:08.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.514+0000 7fe99e000640 1 -- 172.21.15.29:0/3089005489 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9900a9020 con 0x7fe984003bc0 2024-04-17T19:58:08.509 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.514+0000 7fe99e000640 1 -- 172.21.15.29:0/3089005489 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe990005ae0 con 0x7fe984003bc0 2024-04-17T19:58:08.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.514+0000 7fe9a6200640 1 -- 172.21.15.29:0/3089005489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe984003bc0 msgr2=0x7fe984004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:08.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.514+0000 7fe9a6200640 1 --2- 172.21.15.29:0/3089005489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe984003bc0 0x7fe984004010 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe99000c190 tx=0x7fe99001cc20 comp rx=0 tx=0).stop 2024-04-17T19:58:08.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.514+0000 7fe9a6200640 1 -- 172.21.15.29:0/3089005489 shutdown_connections 2024-04-17T19:58:08.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.514+0000 7fe9a6200640 1 --2- 172.21.15.29:0/3089005489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe984003bc0 0x7fe984004010 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:08.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.514+0000 7fe9a6200640 1 -- 172.21.15.29:0/3089005489 >> 172.21.15.29:0/3089005489 conn(0x7fe9a00713d0 msgr2=0x7fe9a00ce350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:08.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.514+0000 7fe9a6200640 1 -- 172.21.15.29:0/3089005489 shutdown_connections 2024-04-17T19:58:08.510 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.514+0000 7fe9a6200640 1 -- 172.21.15.29:0/3089005489 wait complete. 2024-04-17T19:58:08.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe9a6200640 1 Processor -- start 2024-04-17T19:58:08.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe9a6200640 1 -- start start 2024-04-17T19:58:08.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe9a6200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe984003bc0 0x7fe9a0140d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:08.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe9a6200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9a01490f0 0x7fe9a01eaca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:08.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe9a6200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe9a013dc40 0x7fe9a013e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:08.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe9a6200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a006d680 con 0x7fe9a01490f0 2024-04-17T19:58:08.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe9a6200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe9a015ce40 con 0x7fe9a013dc40 2024-04-17T19:58:08.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe9a6200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe9a0156580 con 0x7fe984003bc0 2024-04-17T19:58:08.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe99ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9a01490f0 0x7fe9a01eaca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:08.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe99ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9a01490f0 0x7fe9a01eaca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34354/0 (socket says 172.21.15.29:34354) 2024-04-17T19:58:08.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe99ea00640 1 -- 172.21.15.29:0/3197246493 learned_addr learned my addr 172.21.15.29:0/3197246493 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:08.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe99f400640 1 --2- 172.21.15.29:0/3197246493 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe984003bc0 0x7fe9a0140d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:08.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe99fe00640 1 --2- 172.21.15.29:0/3197246493 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe9a013dc40 0x7fe9a013e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:08.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe99ea00640 1 -- 172.21.15.29:0/3197246493 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe984003bc0 msgr2=0x7fe9a0140d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:08.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe99ea00640 1 --2- 172.21.15.29:0/3197246493 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe984003bc0 0x7fe9a0140d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:08.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe99ea00640 1 -- 172.21.15.29:0/3197246493 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe9a013dc40 msgr2=0x7fe9a013e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:08.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe99ea00640 1 --2- 172.21.15.29:0/3197246493 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe9a013dc40 0x7fe9a013e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:08.516 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe99ea00640 1 -- 172.21.15.29:0/3197246493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9a0141240 con 0x7fe9a01490f0 2024-04-17T19:58:08.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.518+0000 7fe99ea00640 1 --2- 172.21.15.29:0/3197246493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9a01490f0 0x7fe9a01eaca0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe98c007860 tx=0x7fe98c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:08.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.522+0000 7fe997400640 1 -- 172.21.15.29:0/3197246493 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c001cf0 con 0x7fe9a01490f0 2024-04-17T19:58:08.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.522+0000 7fe997400640 1 -- 172.21.15.29:0/3197246493 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe98c0050d0 con 0x7fe9a01490f0 2024-04-17T19:58:08.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.522+0000 7fe997400640 1 -- 172.21.15.29:0/3197246493 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe98c0052c0 con 0x7fe9a01490f0 2024-04-17T19:58:08.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.522+0000 7fe9a6200640 1 -- 172.21.15.29:0/3197246493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9a00cd670 con 0x7fe9a01490f0 2024-04-17T19:58:08.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.522+0000 7fe9a6200640 1 -- 172.21.15.29:0/3197246493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9a01477b0 con 0x7fe9a01490f0 2024-04-17T19:58:08.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.522+0000 7fe9a6200640 1 -- 172.21.15.29:0/3197246493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9a0069b90 con 0x7fe9a01490f0 2024-04-17T19:58:08.517 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.522+0000 7fe997400640 1 -- 172.21.15.29:0/3197246493 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe98c01a020 con 0x7fe9a01490f0 2024-04-17T19:58:08.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.522+0000 7fe997400640 1 --2- 172.21.15.29:0/3197246493 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe980068380 0x7fe98006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:08.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.522+0000 7fe997400640 1 -- 172.21.15.29:0/3197246493 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7fe98c0864e0 con 0x7fe9a01490f0 2024-04-17T19:58:08.519 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.526+0000 7fe99f400640 1 --2- 172.21.15.29:0/3197246493 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe980068380 0x7fe98006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:08.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.526+0000 7fe99f400640 1 --2- 172.21.15.29:0/3197246493 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe980068380 0x7fe98006a890 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe9900aa000 tx=0x7fe9900019d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:08.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.530+0000 7fe997400640 1 -- 172.21.15.29:0/3197246493 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe98c055e10 con 0x7fe9a01490f0 2024-04-17T19:58:08.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.734+0000 7fe9a6200640 1 -- 172.21.15.29:0/3197246493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9a00cd9a0 con 0x7fe9a01490f0 2024-04-17T19:58:08.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.734+0000 7fe997400640 1 -- 172.21.15.29:0/3197246493 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+6078 (secure 0 0 0) 0x7fe98c05a110 con 0x7fe9a01490f0 2024-04-17T19:58:08.730 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:58:08.730 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:03.917169+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:replay","state_seq":1,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5144,5212,5350]},"id":1}]} 2024-04-17T19:58:08.730 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 44 2024-04-17T19:58:08.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.738+0000 7fe9a6200640 1 -- 172.21.15.29:0/3197246493 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe980068380 msgr2=0x7fe98006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:08.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.738+0000 7fe9a6200640 1 --2- 172.21.15.29:0/3197246493 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe980068380 0x7fe98006a890 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe9900aa000 tx=0x7fe9900019d0 comp rx=0 tx=0).stop 2024-04-17T19:58:08.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.738+0000 7fe9a6200640 1 -- 172.21.15.29:0/3197246493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9a01490f0 msgr2=0x7fe9a01eaca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:08.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.738+0000 7fe9a6200640 1 --2- 172.21.15.29:0/3197246493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9a01490f0 0x7fe9a01eaca0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe98c007860 tx=0x7fe98c00ec10 comp rx=0 tx=0).stop 2024-04-17T19:58:08.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.738+0000 7fe9a6200640 1 -- 172.21.15.29:0/3197246493 shutdown_connections 2024-04-17T19:58:08.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.738+0000 7fe9a6200640 1 --2- 172.21.15.29:0/3197246493 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe980068380 0x7fe98006a890 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:08.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.738+0000 7fe9a6200640 1 --2- 172.21.15.29:0/3197246493 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe9a013dc40 0x7fe9a013e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:08.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.738+0000 7fe9a6200640 1 --2- 172.21.15.29:0/3197246493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9a01490f0 0x7fe9a01eaca0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:08.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.738+0000 7fe9a6200640 1 --2- 172.21.15.29:0/3197246493 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe984003bc0 0x7fe9a0140d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:08.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.738+0000 7fe9a6200640 1 -- 172.21.15.29:0/3197246493 >> 172.21.15.29:0/3197246493 conn(0x7fe9a00713d0 msgr2=0x7fe9a0143550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:08.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.738+0000 7fe9a6200640 1 -- 172.21.15.29:0/3197246493 shutdown_connections 2024-04-17T19:58:08.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:08.738+0000 7fe9a6200640 1 -- 172.21.15.29:0/3197246493 wait complete. 2024-04-17T19:58:08.752 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:58:10.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:58:10.954 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.958+0000 7fb06ac00640 1 Processor -- start 2024-04-17T19:58:10.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.958+0000 7fb06ac00640 1 -- start start 2024-04-17T19:58:10.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.962+0000 7fb06ac00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb064159c90 0x7fb0640ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:10.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.962+0000 7fb06ac00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0640ca7f0 0x7fb0640c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:10.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.962+0000 7fb06ac00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0640c74b0 0x7fb06406c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.962+0000 7fb06ac00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fb06415a0a0 con 0x7fb0640ca7f0 2024-04-17T19:58:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.962+0000 7fb06ac00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb06415abe0 con 0x7fb064159c90 2024-04-17T19:58:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.962+0000 7fb06ac00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb06415a340 con 0x7fb0640c74b0 2024-04-17T19:58:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.962+0000 7fb063400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0640ca7f0 0x7fb0640c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.962+0000 7fb063400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0640ca7f0 0x7fb0640c6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40106/0 (socket says 172.21.15.29:40106) 2024-04-17T19:58:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.962+0000 7fb063400640 1 -- 172.21.15.29:0/2905029098 learned_addr learned my addr 172.21.15.29:0/2905029098 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.962+0000 7fb063e00640 1 --2- 172.21.15.29:0/2905029098 >> v2:172.21.15.37:3300/0 conn(0x7fb064159c90 0x7fb0640ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.962+0000 7fb068c00640 1 --2- 172.21.15.29:0/2905029098 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0640c74b0 0x7fb06406c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.962+0000 7fb063400640 1 -- 172.21.15.29:0/2905029098 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0640c74b0 msgr2=0x7fb06406c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.962+0000 7fb063400640 1 --2- 172.21.15.29:0/2905029098 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0640c74b0 0x7fb06406c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.962+0000 7fb063400640 1 -- 172.21.15.29:0/2905029098 >> v2:172.21.15.37:3300/0 conn(0x7fb064159c90 msgr2=0x7fb0640ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.962+0000 7fb063400640 1 --2- 172.21.15.29:0/2905029098 >> v2:172.21.15.37:3300/0 conn(0x7fb064159c90 0x7fb0640ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.962+0000 7fb063400640 1 -- 172.21.15.29:0/2905029098 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb06415a620 con 0x7fb0640ca7f0 2024-04-17T19:58:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.962+0000 7fb063e00640 1 --2- 172.21.15.29:0/2905029098 >> v2:172.21.15.37:3300/0 conn(0x7fb064159c90 0x7fb0640ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:58:10.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.966+0000 7fb063400640 1 --2- 172.21.15.29:0/2905029098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0640ca7f0 0x7fb0640c6f70 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb064071f40 tx=0x7fb05001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1502ef9613735a90 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:10.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.966+0000 7fb062a00640 1 -- 172.21.15.29:0/2905029098 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0500ab5f0 con 0x7fb0640ca7f0 2024-04-17T19:58:10.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.966+0000 7fb062a00640 1 -- 172.21.15.29:0/2905029098 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0500b0070 con 0x7fb0640ca7f0 2024-04-17T19:58:10.960 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.966+0000 7fb062a00640 1 -- 172.21.15.29:0/2905029098 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0500ab9b0 con 0x7fb0640ca7f0 2024-04-17T19:58:10.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.966+0000 7fb06ac00640 1 -- 172.21.15.29:0/2905029098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0640ca7f0 msgr2=0x7fb0640c6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:10.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.966+0000 7fb06ac00640 1 --2- 172.21.15.29:0/2905029098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0640ca7f0 0x7fb0640c6f70 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb064071f40 tx=0x7fb05001c520 comp rx=0 tx=0).stop 2024-04-17T19:58:10.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.966+0000 7fb06ac00640 1 -- 172.21.15.29:0/2905029098 shutdown_connections 2024-04-17T19:58:10.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.966+0000 7fb06ac00640 1 --2- 172.21.15.29:0/2905029098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0640ca7f0 0x7fb0640c6f70 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:10.961 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.966+0000 7fb06ac00640 1 --2- 172.21.15.29:0/2905029098 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb0640c74b0 0x7fb06406c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:10.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.966+0000 7fb06ac00640 1 --2- 172.21.15.29:0/2905029098 >> v2:172.21.15.37:3300/0 conn(0x7fb064159c90 0x7fb0640ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:10.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.966+0000 7fb06ac00640 1 -- 172.21.15.29:0/2905029098 >> 172.21.15.29:0/2905029098 conn(0x7fb06406fde0 msgr2=0x7fb0640701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:10.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.966+0000 7fb06ac00640 1 -- 172.21.15.29:0/2905029098 shutdown_connections 2024-04-17T19:58:10.962 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.966+0000 7fb06ac00640 1 -- 172.21.15.29:0/2905029098 wait complete. 2024-04-17T19:58:10.963 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb06ac00640 1 Processor -- start 2024-04-17T19:58:10.964 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb06ac00640 1 -- start start 2024-04-17T19:58:10.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb06ac00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0640c74b0 0x7fb0641f8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:10.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb06ac00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0640ca7f0 0x7fb064163d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:10.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb06ac00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb064159c90 0x7fb06414ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:10.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb06ac00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb06415da90 con 0x7fb0640ca7f0 2024-04-17T19:58:10.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb06ac00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb06406cfb0 con 0x7fb0640c74b0 2024-04-17T19:58:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb06ac00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb06415dc10 con 0x7fb064159c90 2024-04-17T19:58:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb063400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0640ca7f0 0x7fb064163d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb063400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0640ca7f0 0x7fb064163d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40116/0 (socket says 172.21.15.29:40116) 2024-04-17T19:58:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb063e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0640c74b0 0x7fb0641f8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb063400640 1 -- 172.21.15.29:0/2820549681 learned_addr learned my addr 172.21.15.29:0/2820549681 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb068c00640 1 --2- 172.21.15.29:0/2820549681 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb064159c90 0x7fb06414ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb063400640 1 -- 172.21.15.29:0/2820549681 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb064159c90 msgr2=0x7fb06414ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb063400640 1 --2- 172.21.15.29:0/2820549681 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb064159c90 0x7fb06414ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb063400640 1 -- 172.21.15.29:0/2820549681 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0640c74b0 msgr2=0x7fb0641f8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb063400640 1 --2- 172.21.15.29:0/2820549681 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0640c74b0 0x7fb0641f8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb063400640 1 -- 172.21.15.29:0/2820549681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0641642d0 con 0x7fb0640ca7f0 2024-04-17T19:58:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb063400640 1 --2- 172.21.15.29:0/2820549681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0640ca7f0 0x7fb064163d90 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fb05001abd0 tx=0x7fb05001ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb05be00640 1 -- 172.21.15.29:0/2820549681 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0500be5f0 con 0x7fb0640ca7f0 2024-04-17T19:58:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb05be00640 1 -- 172.21.15.29:0/2820549681 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0500be790 con 0x7fb0640ca7f0 2024-04-17T19:58:10.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb05be00640 1 -- 172.21.15.29:0/2820549681 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0500beb80 con 0x7fb0640ca7f0 2024-04-17T19:58:10.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.970+0000 7fb06ac00640 1 -- 172.21.15.29:0/2820549681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb06414bb70 con 0x7fb0640ca7f0 2024-04-17T19:58:10.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.974+0000 7fb06ac00640 1 -- 172.21.15.29:0/2820549681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0640c4840 con 0x7fb0640ca7f0 2024-04-17T19:58:10.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.974+0000 7fb06ac00640 1 -- 172.21.15.29:0/2820549681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb064069b90 con 0x7fb0640ca7f0 2024-04-17T19:58:10.969 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.974+0000 7fb05be00640 1 -- 172.21.15.29:0/2820549681 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb0500b43f0 con 0x7fb0640ca7f0 2024-04-17T19:58:10.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.974+0000 7fb05be00640 1 --2- 172.21.15.29:0/2820549681 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb044068380 0x7fb04406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:10.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.974+0000 7fb05be00640 1 -- 172.21.15.29:0/2820549681 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7fb050121650 con 0x7fb0640ca7f0 2024-04-17T19:58:10.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.974+0000 7fb063e00640 1 --2- 172.21.15.29:0/2820549681 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb044068380 0x7fb04406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:10.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.978+0000 7fb063e00640 1 --2- 172.21.15.29:0/2820549681 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb044068380 0x7fb04406a890 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb0640c8e90 tx=0x7fb054008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:10.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:10.982+0000 7fb05be00640 1 -- 172.21.15.29:0/2820549681 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb050016110 con 0x7fb0640ca7f0 2024-04-17T19:58:11.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:11.202+0000 7fb06ac00640 1 -- 172.21.15.29:0/2820549681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb06414b560 con 0x7fb0640ca7f0 2024-04-17T19:58:11.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:11.202+0000 7fb05be00640 1 -- 172.21.15.29:0/2820549681 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+6080 (secure 0 0 0) 0x7fb0500f1000 con 0x7fb0640ca7f0 2024-04-17T19:58:11.198 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:58:11.198 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:08.931195+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:resolve","state_seq":36,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5212,5144,4305]},"id":1}]} 2024-04-17T19:58:11.199 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 45 2024-04-17T19:58:11.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:11.206+0000 7fb06ac00640 1 -- 172.21.15.29:0/2820549681 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb044068380 msgr2=0x7fb04406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:11.200 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:11.206+0000 7fb06ac00640 1 --2- 172.21.15.29:0/2820549681 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb044068380 0x7fb04406a890 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb0640c8e90 tx=0x7fb054008040 comp rx=0 tx=0).stop 2024-04-17T19:58:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:11.206+0000 7fb06ac00640 1 -- 172.21.15.29:0/2820549681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0640ca7f0 msgr2=0x7fb064163d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:11.206+0000 7fb06ac00640 1 --2- 172.21.15.29:0/2820549681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0640ca7f0 0x7fb064163d90 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fb05001abd0 tx=0x7fb05001ad60 comp rx=0 tx=0).stop 2024-04-17T19:58:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:11.206+0000 7fb06ac00640 1 -- 172.21.15.29:0/2820549681 shutdown_connections 2024-04-17T19:58:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:11.206+0000 7fb06ac00640 1 --2- 172.21.15.29:0/2820549681 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb044068380 0x7fb04406a890 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:11.206+0000 7fb06ac00640 1 --2- 172.21.15.29:0/2820549681 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb064159c90 0x7fb06414ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:11.206+0000 7fb06ac00640 1 --2- 172.21.15.29:0/2820549681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0640ca7f0 0x7fb064163d90 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:11.206+0000 7fb06ac00640 1 --2- 172.21.15.29:0/2820549681 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb0640c74b0 0x7fb0641f8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:11.206+0000 7fb06ac00640 1 -- 172.21.15.29:0/2820549681 >> 172.21.15.29:0/2820549681 conn(0x7fb06406fde0 msgr2=0x7fb0640c40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:11.206+0000 7fb06ac00640 1 -- 172.21.15.29:0/2820549681 shutdown_connections 2024-04-17T19:58:11.201 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:11.206+0000 7fb06ac00640 1 -- 172.21.15.29:0/2820549681 wait complete. 2024-04-17T19:58:11.221 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:58:13.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:58:13.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.518+0000 7f1de7e00640 1 Processor -- start 2024-04-17T19:58:13.512 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.518+0000 7f1de7e00640 1 -- start start 2024-04-17T19:58:13.513 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.518+0000 7f1de7e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f1de01521e0 0x7f1de0158910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:13.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.518+0000 7f1de7e00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1de00cacc0 con 0x7f1de00c8560 2024-04-17T19:58:13.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.518+0000 7f1de7e00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1de00ca750 con 0x7f1de00c8150 2024-04-17T19:58:13.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.518+0000 7f1de7e00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f1de0158e50 con 0x7f1de01521e0 2024-04-17T19:58:13.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.518+0000 7f1de5e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1de00c8560 0x7f1de0162840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33698/0 (socket says 172.21.15.29:33698) 2024-04-17T19:58:13.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.518+0000 7f1de5e00640 1 -- 172.21.15.29:0/3307042687 learned_addr learned my addr 172.21.15.29:0/3307042687 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:58:13.514 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.518+0000 7f1de5400640 1 --2- 172.21.15.29:0/3307042687 >> v2:172.21.15.37:3300/0 conn(0x7f1de01521e0 0x7f1de0158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:13.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.518+0000 7f1ddfe00640 1 -- 172.21.15.29:0/3307042687 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4198535971 0 0) 0x7f1de00cacc0 con 0x7f1de00c8560 2024-04-17T19:58:13.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.518+0000 7f1ddfe00640 1 -- 172.21.15.29:0/3307042687 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1dc8003530 con 0x7f1de00c8560 2024-04-17T19:58:13.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.518+0000 7f1de5400640 1 -- 172.21.15.29:0/3307042687 >> v1:172.21.15.37:6789/0 conn(0x7f1de00c8150 legacy=0x7f1de00cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:13.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.518+0000 7f1de5400640 1 -- 172.21.15.29:0/3307042687 >> v1:172.21.15.29:6789/0 conn(0x7f1de00c8560 legacy=0x7f1de0162840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:13.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.518+0000 7f1de5400640 1 -- 172.21.15.29:0/3307042687 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1de015b5e0 con 0x7f1de01521e0 2024-04-17T19:58:13.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.522+0000 7f1de5400640 1 --2- 172.21.15.29:0/3307042687 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1de01521e0 0x7f1de0158910 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc00a960 tx=0x7f1dcc01d870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f705af6c506d0a62 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:13.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.522+0000 7f1ddfe00640 1 -- 172.21.15.29:0/3307042687 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc0b3070 con 0x7f1de01521e0 2024-04-17T19:58:13.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.522+0000 7f1ddfe00640 1 -- 172.21.15.29:0/3307042687 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dcc0a6740 con 0x7f1de01521e0 2024-04-17T19:58:13.518 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.522+0000 7f1ddfe00640 1 -- 172.21.15.29:0/3307042687 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc00b040 con 0x7f1de01521e0 2024-04-17T19:58:13.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.526+0000 7f1de7e00640 1 -- 172.21.15.29:0/3307042687 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1de01521e0 msgr2=0x7f1de0158910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:13.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.526+0000 7f1de7e00640 1 --2- 172.21.15.29:0/3307042687 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1de01521e0 0x7f1de0158910 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc00a960 tx=0x7f1dcc01d870 comp rx=0 tx=0).stop 2024-04-17T19:58:13.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.526+0000 7f1de7e00640 1 -- 172.21.15.29:0/3307042687 shutdown_connections 2024-04-17T19:58:13.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.526+0000 7f1de7e00640 1 --2- 172.21.15.29:0/3307042687 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1de01521e0 0x7f1de0158910 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:13.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.526+0000 7f1de7e00640 1 -- 172.21.15.29:0/3307042687 >> 172.21.15.29:0/3307042687 conn(0x7f1de006fde0 msgr2=0x7f1de00701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:13.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.526+0000 7f1de7e00640 1 -- 172.21.15.29:0/3307042687 shutdown_connections 2024-04-17T19:58:13.520 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.526+0000 7f1de7e00640 1 -- 172.21.15.29:0/3307042687 wait complete. 2024-04-17T19:58:13.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.526+0000 7f1de7e00640 1 Processor -- start 2024-04-17T19:58:13.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.526+0000 7f1de7e00640 1 -- start start 2024-04-17T19:58:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de7e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1de00c8150 0x7f1de0159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de7e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1de00c8560 0x7f1de01f9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1de01521e0 0x7f1de0073630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de7e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f1de015c360 con 0x7f1de01521e0 2024-04-17T19:58:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de7e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f1de00cb1b0 con 0x7f1de00c8560 2024-04-17T19:58:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de7e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f1de00cc310 con 0x7f1de00c8150 2024-04-17T19:58:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de5e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1de01521e0 0x7f1de0073630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de5e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1de01521e0 0x7f1de0073630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40122/0 (socket says 172.21.15.29:40122) 2024-04-17T19:58:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de5e00640 1 -- 172.21.15.29:0/3627708890 learned_addr learned my addr 172.21.15.29:0/3627708890 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:13.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de5400640 1 --2- 172.21.15.29:0/3627708890 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1de00c8150 0x7f1de0159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:13.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de4a00640 1 --2- 172.21.15.29:0/3627708890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1de00c8560 0x7f1de01f9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:13.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de5e00640 1 -- 172.21.15.29:0/3627708890 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1de00c8150 msgr2=0x7f1de0159f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:13.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de5e00640 1 --2- 172.21.15.29:0/3627708890 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1de00c8150 0x7f1de0159f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:13.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de5e00640 1 -- 172.21.15.29:0/3627708890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1de00c8560 msgr2=0x7f1de01f9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:13.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de5e00640 1 --2- 172.21.15.29:0/3627708890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1de00c8560 0x7f1de01f9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:13.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de5e00640 1 -- 172.21.15.29:0/3627708890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1de01f9c40 con 0x7f1de01521e0 2024-04-17T19:58:13.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de5e00640 1 --2- 172.21.15.29:0/3627708890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1de01521e0 0x7f1de0073630 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800c660 tx=0x7f1dd800cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:13.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1ddd600640 1 -- 172.21.15.29:0/3627708890 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8007960 con 0x7f1de01521e0 2024-04-17T19:58:13.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1ddd600640 1 -- 172.21.15.29:0/3627708890 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dd8008040 con 0x7f1de01521e0 2024-04-17T19:58:13.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de7e00640 1 -- 172.21.15.29:0/3627708890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1de015ad10 con 0x7f1de01521e0 2024-04-17T19:58:13.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1ddd600640 1 -- 172.21.15.29:0/3627708890 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8007dc0 con 0x7f1de01521e0 2024-04-17T19:58:13.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de7e00640 1 -- 172.21.15.29:0/3627708890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1de015b2d0 con 0x7f1de01521e0 2024-04-17T19:58:13.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.530+0000 7f1de7e00640 1 -- 172.21.15.29:0/3627708890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1de01f9e10 con 0x7f1de01521e0 2024-04-17T19:58:13.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.534+0000 7f1ddd600640 1 -- 172.21.15.29:0/3627708890 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f1dd800f410 con 0x7f1de01521e0 2024-04-17T19:58:13.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.534+0000 7f1ddd600640 1 --2- 172.21.15.29:0/3627708890 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1dc0068380 0x7f1dc006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:13.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.534+0000 7f1ddd600640 1 -- 172.21.15.29:0/3627708890 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7f1dd8087150 con 0x7f1de01521e0 2024-04-17T19:58:13.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.534+0000 7f1de5400640 1 --2- 172.21.15.29:0/3627708890 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1dc0068380 0x7f1dc006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:13.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.534+0000 7f1de5400640 1 --2- 172.21.15.29:0/3627708890 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1dc0068380 0x7f1dc006a890 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc0b0000 tx=0x7f1dcc0a8070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:13.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.542+0000 7f1ddd600640 1 -- 172.21.15.29:0/3627708890 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1de01f9e10 con 0x7f1de01521e0 2024-04-17T19:58:13.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.754+0000 7f1de7e00640 1 -- 172.21.15.29:0/3627708890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1de0069b90 con 0x7f1de01521e0 2024-04-17T19:58:13.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.754+0000 7f1ddd600640 1 -- 172.21.15.29:0/3627708890 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+6080 (secure 0 0 0) 0x7f1dd8014030 con 0x7f1de01521e0 2024-04-17T19:58:13.750 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:58:13.750 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:08.931195+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:resolve","state_seq":36,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5212,5144,4305]},"id":1}]} 2024-04-17T19:58:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 45 2024-04-17T19:58:13.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.758+0000 7f1de7e00640 1 -- 172.21.15.29:0/3627708890 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1dc0068380 msgr2=0x7f1dc006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:13.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.758+0000 7f1de7e00640 1 --2- 172.21.15.29:0/3627708890 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1dc0068380 0x7f1dc006a890 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc0b0000 tx=0x7f1dcc0a8070 comp rx=0 tx=0).stop 2024-04-17T19:58:13.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.758+0000 7f1de7e00640 1 -- 172.21.15.29:0/3627708890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1de01521e0 msgr2=0x7f1de0073630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:13.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.758+0000 7f1de7e00640 1 --2- 172.21.15.29:0/3627708890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1de01521e0 0x7f1de0073630 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800c660 tx=0x7f1dd800cb20 comp rx=0 tx=0).stop 2024-04-17T19:58:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.758+0000 7f1de7e00640 1 -- 172.21.15.29:0/3627708890 shutdown_connections 2024-04-17T19:58:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.758+0000 7f1de7e00640 1 --2- 172.21.15.29:0/3627708890 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1dc0068380 0x7f1dc006a890 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.758+0000 7f1de7e00640 1 --2- 172.21.15.29:0/3627708890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1de01521e0 0x7f1de0073630 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.758+0000 7f1de7e00640 1 --2- 172.21.15.29:0/3627708890 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f1de00c8560 0x7f1de01f9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.758+0000 7f1de7e00640 1 --2- 172.21.15.29:0/3627708890 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1de00c8150 0x7f1de0159f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.758+0000 7f1de7e00640 1 -- 172.21.15.29:0/3627708890 >> 172.21.15.29:0/3627708890 conn(0x7f1de006fde0 msgr2=0x7f1de00c5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.758+0000 7f1de7e00640 1 -- 172.21.15.29:0/3627708890 shutdown_connections 2024-04-17T19:58:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:13.758+0000 7f1de7e00640 1 -- 172.21.15.29:0/3627708890 wait complete. 2024-04-17T19:58:13.773 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:58:15.774 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:58:15.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.990+0000 7f402ac00640 1 Processor -- start 2024-04-17T19:58:15.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.990+0000 7f402ac00640 1 -- start start 2024-04-17T19:58:15.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.994+0000 7f402ac00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f40240caf70 0x7f40240cb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:15.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.994+0000 7f402ac00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4024071230 con 0x7f4024155d80 2024-04-17T19:58:15.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.994+0000 7f402ac00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f402415bf70 con 0x7f4024156190 2024-04-17T19:58:15.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.994+0000 7f402ac00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f402415c230 con 0x7f40240caf70 2024-04-17T19:58:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.994+0000 7f4029800640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4024155d80 0x7f402406d280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33700/0 (socket says 172.21.15.29:33700) 2024-04-17T19:58:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.994+0000 7f4029800640 1 -- 172.21.15.29:0/295600969 learned_addr learned my addr 172.21.15.29:0/295600969 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:58:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.994+0000 7f402a200640 1 --2- 172.21.15.29:0/295600969 >> v2:172.21.15.37:3300/0 conn(0x7f40240caf70 0x7f40240cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.994+0000 7f4023e00640 1 -- 172.21.15.29:0/295600969 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 725178880 0 0) 0x7f4024071230 con 0x7f4024155d80 2024-04-17T19:58:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.994+0000 7f4023e00640 1 -- 172.21.15.29:0/295600969 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f400c003530 con 0x7f4024155d80 2024-04-17T19:58:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.994+0000 7f4023e00640 1 -- 172.21.15.29:0/295600969 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2884189572 0 0) 0x7f402415bf70 con 0x7f4024156190 2024-04-17T19:58:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.994+0000 7f4023e00640 1 -- 172.21.15.29:0/295600969 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4024071230 con 0x7f4024156190 2024-04-17T19:58:15.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.994+0000 7f402a200640 1 -- 172.21.15.29:0/295600969 >> v1:172.21.15.37:6789/0 conn(0x7f4024156190 legacy=0x7f40240cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:15.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.994+0000 7f4023e00640 1 -- 172.21.15.29:0/295600969 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 755576682 0 0) 0x7f400c003530 con 0x7f4024155d80 2024-04-17T19:58:15.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.994+0000 7f402a200640 1 -- 172.21.15.29:0/295600969 >> v1:172.21.15.29:6789/0 conn(0x7f4024155d80 legacy=0x7f402406d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:15.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.994+0000 7f402a200640 1 -- 172.21.15.29:0/295600969 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f402415c510 con 0x7f40240caf70 2024-04-17T19:58:15.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.998+0000 7f402a200640 1 --2- 172.21.15.29:0/295600969 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40240caf70 0x7f40240cb380 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f401c009d90 tx=0x7f401c01c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaf738766d5eb8e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:15.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.998+0000 7f4023e00640 1 -- 172.21.15.29:0/295600969 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f401c0b2070 con 0x7f40240caf70 2024-04-17T19:58:15.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.998+0000 7f4023e00640 1 -- 172.21.15.29:0/295600969 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f401c0a5530 con 0x7f40240caf70 2024-04-17T19:58:15.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.998+0000 7f4023e00640 1 -- 172.21.15.29:0/295600969 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4018001680 con 0x7f40240caf70 2024-04-17T19:58:15.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.998+0000 7f402ac00640 1 -- 172.21.15.29:0/295600969 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40240caf70 msgr2=0x7f40240cb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:15.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.998+0000 7f402ac00640 1 --2- 172.21.15.29:0/295600969 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40240caf70 0x7f40240cb380 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f401c009d90 tx=0x7f401c01c800 comp rx=0 tx=0).stop 2024-04-17T19:58:15.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.998+0000 7f402ac00640 1 -- 172.21.15.29:0/295600969 shutdown_connections 2024-04-17T19:58:15.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.998+0000 7f402ac00640 1 --2- 172.21.15.29:0/295600969 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f40240caf70 0x7f40240cb380 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:15.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.998+0000 7f402ac00640 1 -- 172.21.15.29:0/295600969 >> 172.21.15.29:0/295600969 conn(0x7f402406e5d0 msgr2=0x7f4024071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:15.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.998+0000 7f402ac00640 1 -- 172.21.15.29:0/295600969 shutdown_connections 2024-04-17T19:58:15.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:15.998+0000 7f402ac00640 1 -- 172.21.15.29:0/295600969 wait complete. 2024-04-17T19:58:15.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f402ac00640 1 Processor -- start 2024-04-17T19:58:15.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f402ac00640 1 -- start start 2024-04-17T19:58:15.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f402ac00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40240caf70 0x7f40241f5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:15.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f402ac00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4024155d80 0x7f40240c8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:15.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f402ac00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4024156190 0x7f4024146be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:15.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f402ac00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f40240c9e10 con 0x7f40240caf70 2024-04-17T19:58:15.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f402ac00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f40240cb8c0 con 0x7f4024156190 2024-04-17T19:58:15.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f402ac00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f40240cc5b0 con 0x7f4024155d80 2024-04-17T19:58:15.998 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f4029800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40240caf70 0x7f40241f5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:15.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f4029800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40240caf70 0x7f40241f5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40132/0 (socket says 172.21.15.29:40132) 2024-04-17T19:58:15.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f4029800640 1 -- 172.21.15.29:0/1956783645 learned_addr learned my addr 172.21.15.29:0/1956783645 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:15.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f402a200640 1 --2- 172.21.15.29:0/1956783645 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4024156190 0x7f4024146be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:15.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f4028e00640 1 --2- 172.21.15.29:0/1956783645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4024155d80 0x7f40240c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f4029800640 1 -- 172.21.15.29:0/1956783645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4024155d80 msgr2=0x7f40240c8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f4029800640 1 --2- 172.21.15.29:0/1956783645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4024155d80 0x7f40240c8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f4029800640 1 -- 172.21.15.29:0/1956783645 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4024156190 msgr2=0x7f4024146be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f4029800640 1 --2- 172.21.15.29:0/1956783645 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4024156190 0x7f4024146be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f4029800640 1 -- 172.21.15.29:0/1956783645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f402406d7c0 con 0x7f40240caf70 2024-04-17T19:58:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.002+0000 7f4029800640 1 --2- 172.21.15.29:0/1956783645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40240caf70 0x7f40241f5cf0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f401800c580 tx=0x7f401800ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.006+0000 7f4021600640 1 -- 172.21.15.29:0/1956783645 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4018017070 con 0x7f40240caf70 2024-04-17T19:58:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.006+0000 7f4021600640 1 -- 172.21.15.29:0/1956783645 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4018005d10 con 0x7f40240caf70 2024-04-17T19:58:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.006+0000 7f4021600640 1 -- 172.21.15.29:0/1956783645 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4018007330 con 0x7f40240caf70 2024-04-17T19:58:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.006+0000 7f402ac00640 1 -- 172.21.15.29:0/1956783645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40241f6230 con 0x7f40240caf70 2024-04-17T19:58:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.006+0000 7f402ac00640 1 -- 172.21.15.29:0/1956783645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40241f9520 con 0x7f40240caf70 2024-04-17T19:58:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.006+0000 7f402ac00640 1 -- 172.21.15.29:0/1956783645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f402404ff10 con 0x7f40240caf70 2024-04-17T19:58:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.006+0000 7f4021600640 1 -- 172.21.15.29:0/1956783645 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f401801a070 con 0x7f40240caf70 2024-04-17T19:58:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.010+0000 7f4021600640 1 --2- 172.21.15.29:0/1956783645 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f40040683b0 0x7f400406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.010+0000 7f4021600640 1 -- 172.21.15.29:0/1956783645 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7f4018086490 con 0x7f40240caf70 2024-04-17T19:58:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.010+0000 7f4028e00640 1 --2- 172.21.15.29:0/1956783645 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f40040683b0 0x7f400406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:16.005 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.010+0000 7f4028e00640 1 --2- 172.21.15.29:0/1956783645 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f40040683b0 0x7f400406a8c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4024159b30 tx=0x7f4010009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:16.009 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.014+0000 7f4021600640 1 -- 172.21.15.29:0/1956783645 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4018055db0 con 0x7f40240caf70 2024-04-17T19:58:16.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.230+0000 7f402ac00640 1 -- 172.21.15.29:0/1956783645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f402406cd40 con 0x7f40240caf70 2024-04-17T19:58:16.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.230+0000 7f4021600640 1 -- 172.21.15.29:0/1956783645 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6082 (secure 0 0 0) 0x7f401805a0b0 con 0x7f40240caf70 2024-04-17T19:58:16.225 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:58:16.225 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:14.802287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:reconnect","state_seq":38,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5144,5212,5350]},"id":1}]} 2024-04-17T19:58:16.226 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 46 2024-04-17T19:58:16.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.234+0000 7f402ac00640 1 -- 172.21.15.29:0/1956783645 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f40040683b0 msgr2=0x7f400406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:16.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.234+0000 7f402ac00640 1 --2- 172.21.15.29:0/1956783645 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f40040683b0 0x7f400406a8c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4024159b30 tx=0x7f4010009040 comp rx=0 tx=0).stop 2024-04-17T19:58:16.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.234+0000 7f402ac00640 1 -- 172.21.15.29:0/1956783645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40240caf70 msgr2=0x7f40241f5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:16.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.234+0000 7f402ac00640 1 --2- 172.21.15.29:0/1956783645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40240caf70 0x7f40241f5cf0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f401800c580 tx=0x7f401800ca40 comp rx=0 tx=0).stop 2024-04-17T19:58:16.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.234+0000 7f402ac00640 1 -- 172.21.15.29:0/1956783645 shutdown_connections 2024-04-17T19:58:16.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.234+0000 7f402ac00640 1 --2- 172.21.15.29:0/1956783645 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f40040683b0 0x7f400406a8c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:16.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.234+0000 7f402ac00640 1 --2- 172.21.15.29:0/1956783645 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4024156190 0x7f4024146be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:16.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.234+0000 7f402ac00640 1 --2- 172.21.15.29:0/1956783645 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4024155d80 0x7f40240c8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:16.228 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.234+0000 7f402ac00640 1 --2- 172.21.15.29:0/1956783645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40240caf70 0x7f40241f5cf0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:16.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.234+0000 7f402ac00640 1 -- 172.21.15.29:0/1956783645 >> 172.21.15.29:0/1956783645 conn(0x7f402406e5d0 msgr2=0x7f402415b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:16.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.234+0000 7f402ac00640 1 -- 172.21.15.29:0/1956783645 shutdown_connections 2024-04-17T19:58:16.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:16.234+0000 7f402ac00640 1 -- 172.21.15.29:0/1956783645 wait complete. 2024-04-17T19:58:16.248 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:58:18.250 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:58:18.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.414+0000 7f70b9000640 1 Processor -- start 2024-04-17T19:58:18.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.414+0000 7f70b9000640 1 -- start start 2024-04-17T19:58:18.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.418+0000 7f70b9000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f70b4159c90 0x7f70b40ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:18.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.418+0000 7f70b9000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f70b406c660 0x7f70b406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:18.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.418+0000 7f70b9000640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70b415a0a0 con 0x7f70b40ca7f0 2024-04-17T19:58:18.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.418+0000 7f70b9000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f70b415abe0 con 0x7f70b406c660 2024-04-17T19:58:18.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.418+0000 7f70b9000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f70b415a340 con 0x7f70b4159c90 2024-04-17T19:58:18.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.418+0000 7f70b2800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f70b406c660 0x7f70b406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:18.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.418+0000 7f70abe00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f70b40ca7f0 0x7f70b40c6f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33708/0 (socket says 172.21.15.29:33708) 2024-04-17T19:58:18.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.418+0000 7f70abe00640 1 -- 172.21.15.29:0/1360526397 learned_addr learned my addr 172.21.15.29:0/1360526397 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:58:18.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.418+0000 7f70b1e00640 1 --2- 172.21.15.29:0/1360526397 >> v2:172.21.15.37:3300/0 conn(0x7f70b4159c90 0x7f70b40ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:18.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.418+0000 7f70b2800640 1 -- 172.21.15.29:0/1360526397 >> v2:172.21.15.37:3300/0 conn(0x7f70b4159c90 msgr2=0x7f70b40ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:18.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.418+0000 7f70b2800640 1 --2- 172.21.15.29:0/1360526397 >> v2:172.21.15.37:3300/0 conn(0x7f70b4159c90 0x7f70b40ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:18.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.418+0000 7f70b2800640 1 -- 172.21.15.29:0/1360526397 >> v1:172.21.15.29:6789/0 conn(0x7f70b40ca7f0 legacy=0x7f70b40c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:18.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.418+0000 7f70b2800640 1 -- 172.21.15.29:0/1360526397 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b415a620 con 0x7f70b406c660 2024-04-17T19:58:18.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.422+0000 7f70b2800640 1 --2- 172.21.15.29:0/1360526397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b406c660 0x7f70b406ca70 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f70b4071f40 tx=0x7f70ac01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b81f4dc43ddad25b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:18.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.422+0000 7f70b1400640 1 -- 172.21.15.29:0/1360526397 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ac0b2070 con 0x7f70b406c660 2024-04-17T19:58:18.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.422+0000 7f70b1400640 1 -- 172.21.15.29:0/1360526397 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70ac0a5900 con 0x7f70b406c660 2024-04-17T19:58:18.416 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.422+0000 7f70b1400640 1 -- 172.21.15.29:0/1360526397 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ac0a5cd0 con 0x7f70b406c660 2024-04-17T19:58:18.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.422+0000 7f70b9000640 1 -- 172.21.15.29:0/1360526397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b406c660 msgr2=0x7f70b406ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:18.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.422+0000 7f70b9000640 1 --2- 172.21.15.29:0/1360526397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b406c660 0x7f70b406ca70 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f70b4071f40 tx=0x7f70ac01c930 comp rx=0 tx=0).stop 2024-04-17T19:58:18.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.422+0000 7f70b9000640 1 -- 172.21.15.29:0/1360526397 shutdown_connections 2024-04-17T19:58:18.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.422+0000 7f70b9000640 1 --2- 172.21.15.29:0/1360526397 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b406c660 0x7f70b406ca70 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:18.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.422+0000 7f70b9000640 1 --2- 172.21.15.29:0/1360526397 >> v2:172.21.15.37:3300/0 conn(0x7f70b4159c90 0x7f70b40ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:18.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.422+0000 7f70b9000640 1 -- 172.21.15.29:0/1360526397 >> 172.21.15.29:0/1360526397 conn(0x7f70b406fde0 msgr2=0x7f70b40701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:18.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.422+0000 7f70b9000640 1 -- 172.21.15.29:0/1360526397 shutdown_connections 2024-04-17T19:58:18.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.422+0000 7f70b9000640 1 -- 172.21.15.29:0/1360526397 wait complete. 2024-04-17T19:58:18.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.422+0000 7f70b9000640 1 Processor -- start 2024-04-17T19:58:18.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b9000640 1 -- start start 2024-04-17T19:58:18.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b9000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b406c660 0x7f70b41f61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:18.422 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b9000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f70b40ca7f0 0x7f70b4145e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:18.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b9000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f70b4159c90 0x7f70b41468d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:18.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b9000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f70b40c7680 con 0x7f70b406c660 2024-04-17T19:58:18.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b9000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f70b406cfb0 con 0x7f70b4159c90 2024-04-17T19:58:18.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b9000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f70b40c7800 con 0x7f70b40ca7f0 2024-04-17T19:58:18.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b1e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b406c660 0x7f70b41f61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:18.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b1e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b406c660 0x7f70b41f61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40146/0 (socket says 172.21.15.29:40146) 2024-04-17T19:58:18.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b1e00640 1 -- 172.21.15.29:0/4255363793 learned_addr learned my addr 172.21.15.29:0/4255363793 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:18.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b2800640 1 --2- 172.21.15.29:0/4255363793 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f70b4159c90 0x7f70b41468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:18.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70abe00640 1 --2- 172.21.15.29:0/4255363793 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f70b40ca7f0 0x7f70b4145e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:18.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b1e00640 1 -- 172.21.15.29:0/4255363793 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f70b40ca7f0 msgr2=0x7f70b4145e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:18.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b1e00640 1 --2- 172.21.15.29:0/4255363793 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f70b40ca7f0 0x7f70b4145e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:18.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b1e00640 1 -- 172.21.15.29:0/4255363793 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f70b4159c90 msgr2=0x7f70b41468d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:18.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b1e00640 1 --2- 172.21.15.29:0/4255363793 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f70b4159c90 0x7f70b41468d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:18.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b1e00640 1 -- 172.21.15.29:0/4255363793 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70b41f8b80 con 0x7f70b406c660 2024-04-17T19:58:18.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b2800640 1 --2- 172.21.15.29:0/4255363793 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f70b4159c90 0x7f70b41468d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:58:18.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b1e00640 1 --2- 172.21.15.29:0/4255363793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b406c660 0x7f70b41f61d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f70a400d600 tx=0x7f70a400dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:18.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70aa000640 1 -- 172.21.15.29:0/4255363793 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a4004240 con 0x7f70b406c660 2024-04-17T19:58:18.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70aa000640 1 -- 172.21.15.29:0/4255363793 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70a40043e0 con 0x7f70b406c660 2024-04-17T19:58:18.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70aa000640 1 -- 172.21.15.29:0/4255363793 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70a4003840 con 0x7f70b406c660 2024-04-17T19:58:18.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b9000640 1 -- 172.21.15.29:0/4255363793 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70b4147ed0 con 0x7f70b406c660 2024-04-17T19:58:18.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b9000640 1 -- 172.21.15.29:0/4255363793 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70b41f56f0 con 0x7f70b406c660 2024-04-17T19:58:18.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.426+0000 7f70b9000640 1 -- 172.21.15.29:0/4255363793 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70b4069b90 con 0x7f70b406c660 2024-04-17T19:58:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.430+0000 7f70aa000640 1 -- 172.21.15.29:0/4255363793 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f70a40039e0 con 0x7f70b406c660 2024-04-17T19:58:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.430+0000 7f70aa000640 1 --2- 172.21.15.29:0/4255363793 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7090068380 0x7f709006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.430+0000 7f70aa000640 1 -- 172.21.15.29:0/4255363793 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7f70a4087330 con 0x7f70b406c660 2024-04-17T19:58:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.430+0000 7f70abe00640 1 --2- 172.21.15.29:0/4255363793 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7090068380 0x7f709006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.430+0000 7f70abe00640 1 --2- 172.21.15.29:0/4255363793 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7090068380 0x7f709006a890 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f70b415e260 tx=0x7f709c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:18.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.434+0000 7f70aa000640 1 -- 172.21.15.29:0/4255363793 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f70a401c360 con 0x7f70b406c660 2024-04-17T19:58:18.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.634+0000 7f70b9000640 1 -- 172.21.15.29:0/4255363793 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70b40c8510 con 0x7f70b406c660 2024-04-17T19:58:18.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.634+0000 7f70aa000640 1 -- 172.21.15.29:0/4255363793 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6082 (secure 0 0 0) 0x7f70a40572a0 con 0x7f70b406c660 2024-04-17T19:58:18.629 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:58:18.629 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:14.802287+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:reconnect","state_seq":38,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5144,5212,5350]},"id":1}]} 2024-04-17T19:58:18.630 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 46 2024-04-17T19:58:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.634+0000 7f70b9000640 1 -- 172.21.15.29:0/4255363793 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7090068380 msgr2=0x7f709006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.634+0000 7f70b9000640 1 --2- 172.21.15.29:0/4255363793 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7090068380 0x7f709006a890 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f70b415e260 tx=0x7f709c009040 comp rx=0 tx=0).stop 2024-04-17T19:58:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.634+0000 7f70b9000640 1 -- 172.21.15.29:0/4255363793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b406c660 msgr2=0x7f70b41f61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.634+0000 7f70b9000640 1 --2- 172.21.15.29:0/4255363793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b406c660 0x7f70b41f61d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f70a400d600 tx=0x7f70a400dac0 comp rx=0 tx=0).stop 2024-04-17T19:58:18.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.634+0000 7f70b9000640 1 -- 172.21.15.29:0/4255363793 shutdown_connections 2024-04-17T19:58:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.634+0000 7f70b9000640 1 --2- 172.21.15.29:0/4255363793 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7090068380 0x7f709006a890 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.634+0000 7f70b9000640 1 --2- 172.21.15.29:0/4255363793 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f70b4159c90 0x7f70b41468d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.634+0000 7f70b9000640 1 --2- 172.21.15.29:0/4255363793 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f70b40ca7f0 0x7f70b4145e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.634+0000 7f70b9000640 1 --2- 172.21.15.29:0/4255363793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70b406c660 0x7f70b41f61d0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.634+0000 7f70b9000640 1 -- 172.21.15.29:0/4255363793 >> 172.21.15.29:0/4255363793 conn(0x7f70b406fde0 msgr2=0x7f70b414d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.638+0000 7f70b9000640 1 -- 172.21.15.29:0/4255363793 shutdown_connections 2024-04-17T19:58:18.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:18.638+0000 7f70b9000640 1 -- 172.21.15.29:0/4255363793 wait complete. 2024-04-17T19:58:18.652 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 46, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-17T19:47:49.315515+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 46, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_4305": { "addr": "172.21.15.29:6837/2691590158", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6833", "nonce": 2691590158, "type": "v2" }, { "addr": "172.21.15.29:6837", "nonce": 2691590158, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4305, "incarnation": 5, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_5144": { "addr": "172.21.15.37:6839/858161860", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6836", "nonce": 858161860, "type": "v2" }, { "addr": "172.21.15.37:6839", "nonce": 858161860, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5144, "incarnation": 38, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_5212": { "addr": "172.21.15.37:6838/767681664", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6835", "nonce": 767681664, "type": "v2" }, { "addr": "172.21.15.37:6838", "nonce": 767681664, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5212, "incarnation": 26, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 31 }, "gid_5350": { "addr": "172.21.15.37:6837/2495953425", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6834", "nonce": 2495953425, "type": "v2" }, { "addr": "172.21.15.37:6837", "nonce": 2495953425, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5350, "incarnation": 32, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 18 }, "gid_5631": { "addr": "172.21.15.29:6836/2376553580", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6832", "nonce": 2376553580, "type": "v2" }, { "addr": "172.21.15.29:6836", "nonce": 2376553580, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5631, "incarnation": 44, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:reconnect", "state_seq": 38 } }, "last_failure": 0, "last_failure_osd_epoch": 73, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-17T19:58:14.802287+0000", "qdb_cluster": [ 4305, 5144, 5212, 5350 ], "qdb_leader": 4305, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4305, "mds_1": 5631, "mds_2": 5144, "mds_3": 5350, "mds_4": 5212 } } } ], "standbys": [] } 2024-04-17T19:58:20.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:58:20.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.814+0000 7f7f9e800640 1 Processor -- start 2024-04-17T19:58:20.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.814+0000 7f7f9e800640 1 -- start start 2024-04-17T19:58:20.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.814+0000 7f7f9e800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7f980b1e60 0x7f7f981596e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:20.810 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.814+0000 7f7f9e800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7f9815c510 0x7f7f9815c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:20.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.814+0000 7f7f9e800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f9815c050 con 0x7f7f9815ff80 2024-04-17T19:58:20.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.814+0000 7f7f9e800640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f7f980cb550 con 0x7f7f9815c510 2024-04-17T19:58:20.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.814+0000 7f7f9e800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f7f980cafb0 con 0x7f7f980b1e60 2024-04-17T19:58:20.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.814+0000 7f7f8fe00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7f9815ff80 0x7f7f980cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36112/0 (socket says 172.21.15.29:36112) 2024-04-17T19:58:20.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.814+0000 7f7f8fe00640 1 -- 172.21.15.29:0/470840223 learned_addr learned my addr 172.21.15.29:0/470840223 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:58:20.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.814+0000 7f7f97e00640 1 --2- 172.21.15.29:0/470840223 >> v2:172.21.15.29:3300/0 conn(0x7f7f9815c510 0x7f7f9815c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:20.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.814+0000 7f7f97400640 1 --2- 172.21.15.29:0/470840223 >> v2:172.21.15.37:3300/0 conn(0x7f7f980b1e60 0x7f7f981596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:20.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.818+0000 7f7f96a00640 1 -- 172.21.15.29:0/470840223 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3080991405 0 0) 0x7f7f80001120 con 0x7f7f9815ff80 2024-04-17T19:58:20.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.818+0000 7f7f96a00640 1 -- 172.21.15.29:0/470840223 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f7c003530 con 0x7f7f9815ff80 2024-04-17T19:58:20.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.818+0000 7f7f97e00640 1 -- 172.21.15.29:0/470840223 >> v2:172.21.15.37:3300/0 conn(0x7f7f980b1e60 msgr2=0x7f7f981596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:20.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.818+0000 7f7f97e00640 1 --2- 172.21.15.29:0/470840223 >> v2:172.21.15.37:3300/0 conn(0x7f7f980b1e60 0x7f7f981596e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:20.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.818+0000 7f7f97e00640 1 -- 172.21.15.29:0/470840223 >> v1:172.21.15.29:6789/0 conn(0x7f7f9815ff80 legacy=0x7f7f980cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:20.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.818+0000 7f7f97e00640 1 -- 172.21.15.29:0/470840223 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f98160390 con 0x7f7f9815c510 2024-04-17T19:58:20.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.822+0000 7f7f97e00640 1 --2- 172.21.15.29:0/470840223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9815c510 0x7f7f9815c920 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7f98073450 tx=0x7f7f9001c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caadc7598341ccc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:20.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.822+0000 7f7f96a00640 1 -- 172.21.15.29:0/470840223 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f900b2070 con 0x7f7f9815c510 2024-04-17T19:58:20.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.822+0000 7f7f96a00640 1 -- 172.21.15.29:0/470840223 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f900a5850 con 0x7f7f9815c510 2024-04-17T19:58:20.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.822+0000 7f7f96a00640 1 -- 172.21.15.29:0/470840223 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f900a5c00 con 0x7f7f9815c510 2024-04-17T19:58:20.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.822+0000 7f7f9e800640 1 -- 172.21.15.29:0/470840223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9815c510 msgr2=0x7f7f9815c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:20.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.822+0000 7f7f9e800640 1 --2- 172.21.15.29:0/470840223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9815c510 0x7f7f9815c920 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7f98073450 tx=0x7f7f9001c8a0 comp rx=0 tx=0).stop 2024-04-17T19:58:20.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.822+0000 7f7f9e800640 1 -- 172.21.15.29:0/470840223 shutdown_connections 2024-04-17T19:58:20.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.822+0000 7f7f9e800640 1 --2- 172.21.15.29:0/470840223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9815c510 0x7f7f9815c920 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:20.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.822+0000 7f7f9e800640 1 --2- 172.21.15.29:0/470840223 >> v2:172.21.15.37:3300/0 conn(0x7f7f980b1e60 0x7f7f981596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:20.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.822+0000 7f7f9e800640 1 -- 172.21.15.29:0/470840223 >> 172.21.15.29:0/470840223 conn(0x7f7f9806e3d0 msgr2=0x7f7f9806e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:20.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.822+0000 7f7f9e800640 1 -- 172.21.15.29:0/470840223 shutdown_connections 2024-04-17T19:58:20.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.822+0000 7f7f9e800640 1 -- 172.21.15.29:0/470840223 wait complete. 2024-04-17T19:58:20.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f9e800640 1 Processor -- start 2024-04-17T19:58:20.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f9e800640 1 -- start start 2024-04-17T19:58:20.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f9e800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f980b1e60 0x7f7f980b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:20.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f9e800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f9815c510 0x7f7f980b0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:20.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f9e800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9815ff80 0x7f7f9815e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:20.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f9e800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7f980ccbc0 con 0x7f7f9815ff80 2024-04-17T19:58:20.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f9e800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9815ce60 con 0x7f7f9815c510 2024-04-17T19:58:20.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f9e800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7f980ccd40 con 0x7f7f980b1e60 2024-04-17T19:58:20.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f97e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9815ff80 0x7f7f9815e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:20.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f97e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9815ff80 0x7f7f9815e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57060/0 (socket says 172.21.15.29:57060) 2024-04-17T19:58:20.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f97e00640 1 -- 172.21.15.29:0/3964776809 learned_addr learned my addr 172.21.15.29:0/3964776809 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:20.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f8fe00640 1 --2- 172.21.15.29:0/3964776809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f9815c510 0x7f7f980b0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:20.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f97400640 1 --2- 172.21.15.29:0/3964776809 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f980b1e60 0x7f7f980b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:20.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f97e00640 1 -- 172.21.15.29:0/3964776809 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f980b1e60 msgr2=0x7f7f980b4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:20.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f97e00640 1 --2- 172.21.15.29:0/3964776809 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f980b1e60 0x7f7f980b4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:20.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f97e00640 1 -- 172.21.15.29:0/3964776809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f9815c510 msgr2=0x7f7f980b0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:20.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f97e00640 1 --2- 172.21.15.29:0/3964776809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f9815c510 0x7f7f980b0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:20.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f97e00640 1 -- 172.21.15.29:0/3964776809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f980b1260 con 0x7f7f9815ff80 2024-04-17T19:58:20.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f97e00640 1 --2- 172.21.15.29:0/3964776809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9815ff80 0x7f7f9815e5e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7f900194b0 tx=0x7f7f90019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:20.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f8f400640 1 -- 172.21.15.29:0/3964776809 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f900b2050 con 0x7f7f9815ff80 2024-04-17T19:58:20.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f8f400640 1 -- 172.21.15.29:0/3964776809 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f900be070 con 0x7f7f9815ff80 2024-04-17T19:58:20.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f8f400640 1 -- 172.21.15.29:0/3964776809 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f900ad870 con 0x7f7f9815ff80 2024-04-17T19:58:20.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f9e800640 1 -- 172.21.15.29:0/3964776809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f980b1050 con 0x7f7f9815ff80 2024-04-17T19:58:20.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.826+0000 7f7f9e800640 1 -- 172.21.15.29:0/3964776809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f980afd50 con 0x7f7f9815ff80 2024-04-17T19:58:20.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.830+0000 7f7f9e800640 1 -- 172.21.15.29:0/3964776809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f980b2440 con 0x7f7f9815ff80 2024-04-17T19:58:20.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.830+0000 7f7f8f400640 1 -- 172.21.15.29:0/3964776809 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7f900ada10 con 0x7f7f9815ff80 2024-04-17T19:58:20.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.830+0000 7f7f8f400640 1 --2- 172.21.15.29:0/3964776809 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7f74068380 0x7f7f7406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:20.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.830+0000 7f7f8f400640 1 -- 172.21.15.29:0/3964776809 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7f7f90121f90 con 0x7f7f9815ff80 2024-04-17T19:58:20.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.830+0000 7f7f97400640 1 --2- 172.21.15.29:0/3964776809 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7f74068380 0x7f7f7406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:20.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.834+0000 7f7f97400640 1 --2- 172.21.15.29:0/3964776809 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7f74068380 0x7f7f7406a890 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7f88009ac0 tx=0x7f7f88009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:20.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:20.838+0000 7f7f8f400640 1 -- 172.21.15.29:0/3964776809 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7f90016520 con 0x7f7f9815ff80 2024-04-17T19:58:21.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:21.042+0000 7f7f9e800640 1 -- 172.21.15.29:0/3964776809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f980af7f0 con 0x7f7f9815ff80 2024-04-17T19:58:21.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:21.042+0000 7f7f8f400640 1 -- 172.21.15.29:0/3964776809 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6079 (secure 0 0 0) 0x7f7f980af7f0 con 0x7f7f9815ff80 2024-04-17T19:58:21.039 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:58:21.039 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:20.063899+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5212,5144,4305]},"id":1}]} 2024-04-17T19:58:21.039 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 47 2024-04-17T19:58:21.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:21.046+0000 7f7f9e800640 1 -- 172.21.15.29:0/3964776809 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7f74068380 msgr2=0x7f7f7406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:21.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:21.046+0000 7f7f9e800640 1 --2- 172.21.15.29:0/3964776809 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7f74068380 0x7f7f7406a890 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7f88009ac0 tx=0x7f7f88009190 comp rx=0 tx=0).stop 2024-04-17T19:58:21.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:21.046+0000 7f7f9e800640 1 -- 172.21.15.29:0/3964776809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9815ff80 msgr2=0x7f7f9815e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:21.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:21.046+0000 7f7f9e800640 1 --2- 172.21.15.29:0/3964776809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9815ff80 0x7f7f9815e5e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7f900194b0 tx=0x7f7f90019640 comp rx=0 tx=0).stop 2024-04-17T19:58:21.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:21.046+0000 7f7f9e800640 1 -- 172.21.15.29:0/3964776809 shutdown_connections 2024-04-17T19:58:21.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:21.046+0000 7f7f9e800640 1 --2- 172.21.15.29:0/3964776809 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7f74068380 0x7f7f7406a890 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:21.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:21.046+0000 7f7f9e800640 1 --2- 172.21.15.29:0/3964776809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f9815ff80 0x7f7f9815e5e0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:21.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:21.046+0000 7f7f9e800640 1 --2- 172.21.15.29:0/3964776809 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7f9815c510 0x7f7f980b0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:21.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:21.046+0000 7f7f9e800640 1 --2- 172.21.15.29:0/3964776809 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7f980b1e60 0x7f7f980b4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:21.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:21.046+0000 7f7f9e800640 1 -- 172.21.15.29:0/3964776809 >> 172.21.15.29:0/3964776809 conn(0x7f7f9806e3d0 msgr2=0x7f7f9806c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:21.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:21.046+0000 7f7f9e800640 1 -- 172.21.15.29:0/3964776809 shutdown_connections 2024-04-17T19:58:21.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:21.046+0000 7f7f9e800640 1 -- 172.21.15.29:0/3964776809 wait complete. 2024-04-17T19:58:21.062 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:58:22.819 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:58:22.823 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:58:23.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:58:23.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.234+0000 7f9a8a800640 1 Processor -- start 2024-04-17T19:58:23.230 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.234+0000 7f9a8a800640 1 -- start start 2024-04-17T19:58:23.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.238+0000 7f9a8a800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a841560d0 0x7f9a8415be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:23.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.238+0000 7f9a8a800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9a8415c370 0x7f9a841667b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:23.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.238+0000 7f9a8a800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a840c55e0 con 0x7f9a840c9b20 2024-04-17T19:58:23.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.238+0000 7f9a8a800640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f9a840c8fa0 con 0x7f9a8415c370 2024-04-17T19:58:23.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.238+0000 7f9a8a800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9a840c9f30 con 0x7f9a841560d0 2024-04-17T19:58:23.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.238+0000 7f9a83400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9a840c9b20 0x7f9a8406e300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36122/0 (socket says 172.21.15.29:36122) 2024-04-17T19:58:23.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.238+0000 7f9a83400640 1 -- 172.21.15.29:0/4112470584 learned_addr learned my addr 172.21.15.29:0/4112470584 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:58:23.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.238+0000 7f9a83e00640 1 --2- 172.21.15.29:0/4112470584 >> v2:172.21.15.29:3300/0 conn(0x7f9a8415c370 0x7f9a841667b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:23.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.238+0000 7f9a7aa00640 1 --2- 172.21.15.29:0/4112470584 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a841560d0 0x7f9a8415be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:23.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.238+0000 7f9a82a00640 1 -- 172.21.15.29:0/4112470584 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4269337056 0 0) 0x7f9a74001120 con 0x7f9a840c9b20 2024-04-17T19:58:23.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.238+0000 7f9a82a00640 1 -- 172.21.15.29:0/4112470584 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a68003530 con 0x7f9a840c9b20 2024-04-17T19:58:23.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.238+0000 7f9a83e00640 1 -- 172.21.15.29:0/4112470584 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a841560d0 msgr2=0x7f9a8415be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:23.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.238+0000 7f9a83e00640 1 --2- 172.21.15.29:0/4112470584 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a841560d0 0x7f9a8415be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:23.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.238+0000 7f9a83e00640 1 -- 172.21.15.29:0/4112470584 >> v1:172.21.15.29:6789/0 conn(0x7f9a840c9b20 legacy=0x7f9a8406e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:23.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.238+0000 7f9a83e00640 1 -- 172.21.15.29:0/4112470584 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a840ca1d0 con 0x7f9a8415c370 2024-04-17T19:58:23.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.242+0000 7f9a83e00640 1 --2- 172.21.15.29:0/4112470584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a8415c370 0x7f9a841667b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c009d90 tx=0x7f9a7c01f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=995508b8d60d994d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:23.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.242+0000 7f9a82a00640 1 -- 172.21.15.29:0/4112470584 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c0b4070 con 0x7f9a8415c370 2024-04-17T19:58:23.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.242+0000 7f9a82a00640 1 -- 172.21.15.29:0/4112470584 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a7c0af600 con 0x7f9a8415c370 2024-04-17T19:58:23.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.242+0000 7f9a82a00640 1 -- 172.21.15.29:0/4112470584 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c0af9b0 con 0x7f9a8415c370 2024-04-17T19:58:23.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.242+0000 7f9a8a800640 1 -- 172.21.15.29:0/4112470584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a8415c370 msgr2=0x7f9a841667b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:23.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.242+0000 7f9a8a800640 1 --2- 172.21.15.29:0/4112470584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a8415c370 0x7f9a841667b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c009d90 tx=0x7f9a7c01f150 comp rx=0 tx=0).stop 2024-04-17T19:58:23.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.242+0000 7f9a8a800640 1 -- 172.21.15.29:0/4112470584 shutdown_connections 2024-04-17T19:58:23.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.242+0000 7f9a8a800640 1 --2- 172.21.15.29:0/4112470584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a8415c370 0x7f9a841667b0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:23.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.242+0000 7f9a8a800640 1 --2- 172.21.15.29:0/4112470584 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a841560d0 0x7f9a8415be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:23.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.242+0000 7f9a8a800640 1 -- 172.21.15.29:0/4112470584 >> 172.21.15.29:0/4112470584 conn(0x7f9a8406c3c0 msgr2=0x7f9a840ca3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:23.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.242+0000 7f9a8a800640 1 -- 172.21.15.29:0/4112470584 shutdown_connections 2024-04-17T19:58:23.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.242+0000 7f9a8a800640 1 -- 172.21.15.29:0/4112470584 wait complete. 2024-04-17T19:58:23.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.246+0000 7f9a8a800640 1 Processor -- start 2024-04-17T19:58:23.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.246+0000 7f9a8a800640 1 -- start start 2024-04-17T19:58:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.246+0000 7f9a8a800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a840c9b20 0x7f9a841e74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.246+0000 7f9a8a800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a841560d0 0x7f9a840cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.246+0000 7f9a8a800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a8415c370 0x7f9a841f5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.246+0000 7f9a8a800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9a840c8670 con 0x7f9a841560d0 2024-04-17T19:58:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.246+0000 7f9a8a800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9a840c8390 con 0x7f9a8415c370 2024-04-17T19:58:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.246+0000 7f9a8a800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9a840c8950 con 0x7f9a840c9b20 2024-04-17T19:58:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.246+0000 7f9a7aa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a841560d0 0x7f9a840cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.246+0000 7f9a7aa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a841560d0 0x7f9a840cc910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57090/0 (socket says 172.21.15.29:57090) 2024-04-17T19:58:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.246+0000 7f9a7aa00640 1 -- 172.21.15.29:0/1183760223 learned_addr learned my addr 172.21.15.29:0/1183760223 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:23.246 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.246+0000 7f9a83400640 1 --2- 172.21.15.29:0/1183760223 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a840c9b20 0x7f9a841e74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.246+0000 7f9a83e00640 1 --2- 172.21.15.29:0/1183760223 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a8415c370 0x7f9a841f5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.246+0000 7f9a7aa00640 1 -- 172.21.15.29:0/1183760223 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a840c9b20 msgr2=0x7f9a841e74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.246+0000 7f9a7aa00640 1 --2- 172.21.15.29:0/1183760223 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a840c9b20 0x7f9a841e74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.246+0000 7f9a7aa00640 1 -- 172.21.15.29:0/1183760223 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a8415c370 msgr2=0x7f9a841f5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:23.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.246+0000 7f9a7aa00640 1 --2- 172.21.15.29:0/1183760223 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a8415c370 0x7f9a841f5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.246+0000 7f9a7aa00640 1 -- 172.21.15.29:0/1183760223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a841e6d30 con 0x7f9a841560d0 2024-04-17T19:58:23.249 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.250+0000 7f9a83e00640 1 --2- 172.21.15.29:0/1183760223 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a8415c370 0x7f9a841f5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T19:58:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.250+0000 7f9a7aa00640 1 --2- 172.21.15.29:0/1183760223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a841560d0 0x7f9a840cc910 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00e870 tx=0x7f9a6c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.250+0000 7f9a7be00640 1 -- 172.21.15.29:0/1183760223 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c017070 con 0x7f9a841560d0 2024-04-17T19:58:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.250+0000 7f9a7be00640 1 -- 172.21.15.29:0/1183760223 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a6c00c940 con 0x7f9a841560d0 2024-04-17T19:58:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.250+0000 7f9a8a800640 1 -- 172.21.15.29:0/1183760223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a841e51d0 con 0x7f9a841560d0 2024-04-17T19:58:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.250+0000 7f9a7be00640 1 -- 172.21.15.29:0/1183760223 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6c00cd10 con 0x7f9a841560d0 2024-04-17T19:58:23.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.250+0000 7f9a8a800640 1 -- 172.21.15.29:0/1183760223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a841e5a30 con 0x7f9a841560d0 2024-04-17T19:58:23.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.250+0000 7f9a8a800640 1 -- 172.21.15.29:0/1183760223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a84069b90 con 0x7f9a841560d0 2024-04-17T19:58:23.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.250+0000 7f9a7be00640 1 -- 172.21.15.29:0/1183760223 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9a6c01a070 con 0x7f9a841560d0 2024-04-17T19:58:23.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.254+0000 7f9a7be00640 1 --2- 172.21.15.29:0/1183760223 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9a600683b0 0x7f9a6006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:23.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.254+0000 7f9a7be00640 1 -- 172.21.15.29:0/1183760223 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7f9a6c087220 con 0x7f9a841560d0 2024-04-17T19:58:23.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.254+0000 7f9a83400640 1 --2- 172.21.15.29:0/1183760223 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9a600683b0 0x7f9a6006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:23.251 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.254+0000 7f9a83400640 1 --2- 172.21.15.29:0/1183760223 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9a600683b0 0x7f9a6006a8c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9a74005890 tx=0x7f9a74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:23.252 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.258+0000 7f9a7be00640 1 -- 172.21.15.29:0/1183760223 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9a6c057190 con 0x7f9a841560d0 2024-04-17T19:58:23.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.458+0000 7f9a8a800640 1 -- 172.21.15.29:0/1183760223 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a840ca580 con 0x7f9a841560d0 2024-04-17T19:58:23.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.462+0000 7f9a7be00640 1 -- 172.21.15.29:0/1183760223 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6079 (secure 0 0 0) 0x7f9a6c056a90 con 0x7f9a841560d0 2024-04-17T19:58:23.456 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:58:23.456 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:20.063899+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5212,5144,4305]},"id":1}]} 2024-04-17T19:58:23.456 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 47 2024-04-17T19:58:23.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.462+0000 7f9a8a800640 1 -- 172.21.15.29:0/1183760223 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9a600683b0 msgr2=0x7f9a6006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:23.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.462+0000 7f9a8a800640 1 --2- 172.21.15.29:0/1183760223 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9a600683b0 0x7f9a6006a8c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9a74005890 tx=0x7f9a74009040 comp rx=0 tx=0).stop 2024-04-17T19:58:23.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.462+0000 7f9a8a800640 1 -- 172.21.15.29:0/1183760223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a841560d0 msgr2=0x7f9a840cc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:23.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.462+0000 7f9a8a800640 1 --2- 172.21.15.29:0/1183760223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a841560d0 0x7f9a840cc910 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c00e870 tx=0x7f9a6c00ec70 comp rx=0 tx=0).stop 2024-04-17T19:58:23.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.462+0000 7f9a8a800640 1 -- 172.21.15.29:0/1183760223 shutdown_connections 2024-04-17T19:58:23.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.462+0000 7f9a8a800640 1 --2- 172.21.15.29:0/1183760223 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9a600683b0 0x7f9a6006a8c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:23.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.462+0000 7f9a8a800640 1 --2- 172.21.15.29:0/1183760223 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9a8415c370 0x7f9a841f5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:23.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.462+0000 7f9a8a800640 1 --2- 172.21.15.29:0/1183760223 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a841560d0 0x7f9a840cc910 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:23.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.462+0000 7f9a8a800640 1 --2- 172.21.15.29:0/1183760223 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9a840c9b20 0x7f9a841e74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:23.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.462+0000 7f9a8a800640 1 -- 172.21.15.29:0/1183760223 >> 172.21.15.29:0/1183760223 conn(0x7f9a8406c3c0 msgr2=0x7f9a8406e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:23.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.462+0000 7f9a8a800640 1 -- 172.21.15.29:0/1183760223 shutdown_connections 2024-04-17T19:58:23.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:23.462+0000 7f9a8a800640 1 -- 172.21.15.29:0/1183760223 wait complete. 2024-04-17T19:58:23.484 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:58:25.485 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:58:25.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.646+0000 7ff162e00640 1 Processor -- start 2024-04-17T19:58:25.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.646+0000 7ff162e00640 1 -- start start 2024-04-17T19:58:25.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.646+0000 7ff162e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff15c0c9ea0 0x7ff15c0ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:25.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.646+0000 7ff162e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff15c159ff0 0x7ff15c15a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:25.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.646+0000 7ff162e00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff15c0c4c10 con 0x7ff15c157250 2024-04-17T19:58:25.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.646+0000 7ff162e00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7ff15c06ef50 con 0x7ff15c0c9ea0 2024-04-17T19:58:25.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.646+0000 7ff162e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff15c0c5a70 con 0x7ff15c159ff0 2024-04-17T19:58:25.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.646+0000 7ff161000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff15c0c9ea0 0x7ff15c0ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:25.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.646+0000 7ff161000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff15c0c9ea0 0x7ff15c0ca2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57094/0 (socket says 172.21.15.29:57094) 2024-04-17T19:58:25.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.646+0000 7ff161000640 1 -- 172.21.15.29:0/3654538814 learned_addr learned my addr 172.21.15.29:0/3654538814 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:25.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.646+0000 7ff162400640 1 --2- 172.21.15.29:0/3654538814 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff15c159ff0 0x7ff15c15a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:25.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.646+0000 7ff161000640 1 -- 172.21.15.29:0/3654538814 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff15c159ff0 msgr2=0x7ff15c15a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:25.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.646+0000 7ff161000640 1 --2- 172.21.15.29:0/3654538814 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff15c159ff0 0x7ff15c15a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:25.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.646+0000 7ff161000640 1 -- 172.21.15.29:0/3654538814 >> v1:172.21.15.37:6789/0 conn(0x7ff15c157250 legacy=0x7ff15c0cb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:25.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.646+0000 7ff161000640 1 -- 172.21.15.29:0/3654538814 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15c0c4ef0 con 0x7ff15c0c9ea0 2024-04-17T19:58:25.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.650+0000 7ff161000640 1 --2- 172.21.15.29:0/3654538814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15c0c9ea0 0x7ff15c0ca2b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff148009870 tx=0x7ff14801ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ed243ff2193d809 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.650+0000 7ff15be00640 1 -- 172.21.15.29:0/3654538814 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1480b5070 con 0x7ff15c0c9ea0 2024-04-17T19:58:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.650+0000 7ff15be00640 1 -- 172.21.15.29:0/3654538814 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1480a8a50 con 0x7ff15c0c9ea0 2024-04-17T19:58:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.650+0000 7ff15be00640 1 -- 172.21.15.29:0/3654538814 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1480a8e20 con 0x7ff15c0c9ea0 2024-04-17T19:58:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.650+0000 7ff162e00640 1 -- 172.21.15.29:0/3654538814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15c0c9ea0 msgr2=0x7ff15c0ca2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.650+0000 7ff162e00640 1 --2- 172.21.15.29:0/3654538814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15c0c9ea0 0x7ff15c0ca2b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff148009870 tx=0x7ff14801ed50 comp rx=0 tx=0).stop 2024-04-17T19:58:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.650+0000 7ff162e00640 1 -- 172.21.15.29:0/3654538814 shutdown_connections 2024-04-17T19:58:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.650+0000 7ff162e00640 1 --2- 172.21.15.29:0/3654538814 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff15c159ff0 0x7ff15c15a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.650+0000 7ff162e00640 1 --2- 172.21.15.29:0/3654538814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15c0c9ea0 0x7ff15c0ca2b0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.650+0000 7ff162e00640 1 -- 172.21.15.29:0/3654538814 >> 172.21.15.29:0/3654538814 conn(0x7ff15c0680e0 msgr2=0x7ff15c06dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.650+0000 7ff162e00640 1 -- 172.21.15.29:0/3654538814 shutdown_connections 2024-04-17T19:58:25.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.650+0000 7ff162e00640 1 -- 172.21.15.29:0/3654538814 wait complete. 2024-04-17T19:58:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff162e00640 1 Processor -- start 2024-04-17T19:58:25.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff162e00640 1 -- start start 2024-04-17T19:58:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff162e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff15c0c9ea0 0x7ff15c1f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff162e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15c157250 0x7ff15c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:25.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff162e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff15c159ff0 0x7ff15c0b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff162e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c0c54b0 con 0x7ff15c157250 2024-04-17T19:58:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff162e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff15c157660 con 0x7ff15c159ff0 2024-04-17T19:58:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff162e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff15c0c5790 con 0x7ff15c0c9ea0 2024-04-17T19:58:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff161000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15c157250 0x7ff15c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff161000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15c157250 0x7ff15c0b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57098/0 (socket says 172.21.15.29:57098) 2024-04-17T19:58:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff161000640 1 -- 172.21.15.29:0/1631891379 learned_addr learned my addr 172.21.15.29:0/1631891379 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff161a00640 1 --2- 172.21.15.29:0/1631891379 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff15c0c9ea0 0x7ff15c1f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff162400640 1 --2- 172.21.15.29:0/1631891379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff15c159ff0 0x7ff15c0b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff161000640 1 -- 172.21.15.29:0/1631891379 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff15c0c9ea0 msgr2=0x7ff15c1f2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff161000640 1 --2- 172.21.15.29:0/1631891379 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff15c0c9ea0 0x7ff15c1f2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff161000640 1 -- 172.21.15.29:0/1631891379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff15c159ff0 msgr2=0x7ff15c0b2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:25.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff161000640 1 --2- 172.21.15.29:0/1631891379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff15c159ff0 0x7ff15c0b2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:25.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff161000640 1 -- 172.21.15.29:0/1631891379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15c06d9a0 con 0x7ff15c157250 2024-04-17T19:58:25.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff161000640 1 --2- 172.21.15.29:0/1631891379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15c157250 0x7ff15c0b3f30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff1480b3040 tx=0x7ff14801baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:25.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff159600640 1 -- 172.21.15.29:0/1631891379 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1480c0070 con 0x7ff15c157250 2024-04-17T19:58:25.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff159600640 1 -- 172.21.15.29:0/1631891379 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1480ae550 con 0x7ff15c157250 2024-04-17T19:58:25.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff159600640 1 -- 172.21.15.29:0/1631891379 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1480b5070 con 0x7ff15c157250 2024-04-17T19:58:25.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.654+0000 7ff162e00640 1 -- 172.21.15.29:0/1631891379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff15c1f3220 con 0x7ff15c157250 2024-04-17T19:58:25.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.658+0000 7ff162e00640 1 -- 172.21.15.29:0/1631891379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff15c158180 con 0x7ff15c157250 2024-04-17T19:58:25.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.658+0000 7ff162e00640 1 -- 172.21.15.29:0/1631891379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff15c04ff10 con 0x7ff15c157250 2024-04-17T19:58:25.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.658+0000 7ff159600640 1 -- 172.21.15.29:0/1631891379 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff1480b9070 con 0x7ff15c157250 2024-04-17T19:58:25.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.658+0000 7ff159600640 1 --2- 172.21.15.29:0/1631891379 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff13c068380 0x7ff13c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:25.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.658+0000 7ff159600640 1 -- 172.21.15.29:0/1631891379 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7ff148124640 con 0x7ff15c157250 2024-04-17T19:58:25.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.658+0000 7ff161a00640 1 --2- 172.21.15.29:0/1631891379 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff13c068380 0x7ff13c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:25.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.662+0000 7ff161a00640 1 --2- 172.21.15.29:0/1631891379 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff13c068380 0x7ff13c06a890 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff150000f70 tx=0x7ff150009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:25.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.662+0000 7ff159600640 1 -- 172.21.15.29:0/1631891379 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff15c06d9a0 con 0x7ff15c157250 2024-04-17T19:58:25.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.866+0000 7ff162e00640 1 -- 172.21.15.29:0/1631891379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff15c0ca9c0 con 0x7ff15c157250 2024-04-17T19:58:25.862 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.866+0000 7ff159600640 1 -- 172.21.15.29:0/1631891379 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+6079 (secure 0 0 0) 0x7ff1480188b0 con 0x7ff15c157250 2024-04-17T19:58:25.862 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:58:25.862 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:20.063899+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5212,5144,4305]},"id":1}]} 2024-04-17T19:58:25.863 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 47 2024-04-17T19:58:25.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.870+0000 7ff162e00640 1 -- 172.21.15.29:0/1631891379 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff13c068380 msgr2=0x7ff13c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:25.865 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.870+0000 7ff162e00640 1 --2- 172.21.15.29:0/1631891379 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff13c068380 0x7ff13c06a890 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff150000f70 tx=0x7ff150009040 comp rx=0 tx=0).stop 2024-04-17T19:58:25.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.870+0000 7ff162e00640 1 -- 172.21.15.29:0/1631891379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15c157250 msgr2=0x7ff15c0b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:25.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.870+0000 7ff162e00640 1 --2- 172.21.15.29:0/1631891379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15c157250 0x7ff15c0b3f30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff1480b3040 tx=0x7ff14801baf0 comp rx=0 tx=0).stop 2024-04-17T19:58:25.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.870+0000 7ff162e00640 1 -- 172.21.15.29:0/1631891379 shutdown_connections 2024-04-17T19:58:25.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.870+0000 7ff162e00640 1 --2- 172.21.15.29:0/1631891379 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff13c068380 0x7ff13c06a890 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:25.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.870+0000 7ff162e00640 1 --2- 172.21.15.29:0/1631891379 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff15c159ff0 0x7ff15c0b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:25.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.870+0000 7ff162e00640 1 --2- 172.21.15.29:0/1631891379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15c157250 0x7ff15c0b3f30 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:25.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.870+0000 7ff162e00640 1 --2- 172.21.15.29:0/1631891379 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff15c0c9ea0 0x7ff15c1f2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:25.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.870+0000 7ff162e00640 1 -- 172.21.15.29:0/1631891379 >> 172.21.15.29:0/1631891379 conn(0x7ff15c0680e0 msgr2=0x7ff15c152fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:25.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.870+0000 7ff162e00640 1 -- 172.21.15.29:0/1631891379 shutdown_connections 2024-04-17T19:58:25.866 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:25.870+0000 7ff162e00640 1 -- 172.21.15.29:0/1631891379 wait complete. 2024-04-17T19:58:25.891 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T19:58:27.892 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:58:28.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.094+0000 7f6f3f400640 1 Processor -- start 2024-04-17T19:58:28.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.094+0000 7f6f3f400640 1 -- start start 2024-04-17T19:58:28.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.098+0000 7f6f3f400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6f38162cd0 0x7f6f38073180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:28.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.098+0000 7f6f3f400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f6f380ca5c0 0x7f6f380cd360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:28.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.098+0000 7f6f3f400640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f380c9720 con 0x7f6f380ccf50 2024-04-17T19:58:28.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.098+0000 7f6f3f400640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f6f381630e0 con 0x7f6f38162cd0 2024-04-17T19:58:28.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.098+0000 7f6f3f400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f6f38168e20 con 0x7f6f380ca5c0 2024-04-17T19:58:28.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.102+0000 7f6f3ca00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6f38162cd0 0x7f6f38073180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:28.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.102+0000 7f6f3ca00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6f38162cd0 0x7f6f38073180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57102/0 (socket says 172.21.15.29:57102) 2024-04-17T19:58:28.711 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.102+0000 7f6f3ca00640 1 -- 172.21.15.29:0/1055427356 learned_addr learned my addr 172.21.15.29:0/1055427356 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:28.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.106+0000 7f6f37400640 1 -- 172.21.15.29:0/1055427356 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3440350204 0 0) 0x7f6f380c9720 con 0x7f6f380ccf50 2024-04-17T19:58:28.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.102+0000 7f6f3d400640 1 --2- 172.21.15.29:0/1055427356 >> v2:172.21.15.37:3300/0 conn(0x7f6f380ca5c0 0x7f6f380cd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:28.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.714+0000 7f6f37400640 1 -- 172.21.15.29:0/1055427356 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f20003980 con 0x7f6f380ccf50 2024-04-17T19:58:28.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37400640 1 -- 172.21.15.29:0/1055427356 <== mon.1 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4187551076 0 0) 0x7f6f20003980 con 0x7f6f380ccf50 2024-04-17T19:58:28.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37400640 1 -- 172.21.15.29:0/1055427356 >> v2:172.21.15.37:3300/0 conn(0x7f6f380ca5c0 msgr2=0x7f6f380cd360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:28.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37400640 1 --2- 172.21.15.29:0/1055427356 >> v2:172.21.15.37:3300/0 conn(0x7f6f380ca5c0 0x7f6f380cd360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:28.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37400640 1 -- 172.21.15.29:0/1055427356 >> v2:172.21.15.29:3300/0 conn(0x7f6f38162cd0 msgr2=0x7f6f38073180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:28.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37400640 1 --2- 172.21.15.29:0/1055427356 >> v2:172.21.15.29:3300/0 conn(0x7f6f38162cd0 0x7f6f38073180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37400640 1 -- 172.21.15.29:0/1055427356 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f38169100 con 0x7f6f380ccf50 2024-04-17T19:58:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37400640 1 -- 172.21.15.29:0/1055427356 <== mon.1 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f6f2c001680 con 0x7f6f380ccf50 2024-04-17T19:58:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37400640 1 -- 172.21.15.29:0/1055427356 >> v1:172.21.15.37:6789/0 conn(0x7f6f380ccf50 legacy=0x7f6f380c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37400640 1 --2- 172.21.15.29:0/1055427356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f20003c00 0x7f6f20004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37400640 1 --2- 172.21.15.29:0/1055427356 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6f20004550 0x7f6f20004c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37400640 1 --2- 172.21.15.29:0/1055427356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f200051a0 0x7f6f200055f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37400640 1 -- 172.21.15.29:0/1055427356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6f381630e0 con 0x7f6f200051a0 2024-04-17T19:58:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37400640 1 -- 172.21.15.29:0/1055427356 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6f38168e20 con 0x7f6f20003c00 2024-04-17T19:58:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37400640 1 -- 172.21.15.29:0/1055427356 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6f2000c0f0 con 0x7f6f20004550 2024-04-17T19:58:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37e00640 1 --2- 172.21.15.29:0/1055427356 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6f20004550 0x7f6f20004c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f3ca00640 1 --2- 172.21.15.29:0/1055427356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f20003c00 0x7f6f20004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37e00640 1 -- 172.21.15.29:0/1055427356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f20003c00 msgr2=0x7f6f20004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:28.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37e00640 1 --2- 172.21.15.29:0/1055427356 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f20003c00 0x7f6f20004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:28.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37e00640 1 -- 172.21.15.29:0/1055427356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f200051a0 msgr2=0x7f6f200055f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:28.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37e00640 1 --2- 172.21.15.29:0/1055427356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f200051a0 0x7f6f200055f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:28.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f37e00640 1 -- 172.21.15.29:0/1055427356 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f38169100 con 0x7f6f20004550 2024-04-17T19:58:28.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.718+0000 7f6f3d400640 1 -- 172.21.15.29:0/1055427356 reap_dead start 2024-04-17T19:58:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.726+0000 7f6f37e00640 1 --2- 172.21.15.29:0/1055427356 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6f20004550 0x7f6f20004c60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c00e4a0 tx=0x7f6f2c025900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ba9bea0fb9e2da0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.726+0000 7f6f37400640 1 -- 172.21.15.29:0/1055427356 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c0bb070 con 0x7f6f20004550 2024-04-17T19:58:28.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.726+0000 7f6f37400640 1 -- 172.21.15.29:0/1055427356 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f2c0ae910 con 0x7f6f20004550 2024-04-17T19:58:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.726+0000 7f6f3f400640 1 -- 172.21.15.29:0/1055427356 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6f20004550 msgr2=0x7f6f20004c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:28.722 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.726+0000 7f6f3f400640 1 --2- 172.21.15.29:0/1055427356 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6f20004550 0x7f6f20004c60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c00e4a0 tx=0x7f6f2c025900 comp rx=0 tx=0).stop 2024-04-17T19:58:28.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.730+0000 7f6f3f400640 1 -- 172.21.15.29:0/1055427356 shutdown_connections 2024-04-17T19:58:28.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.730+0000 7f6f3f400640 1 --2- 172.21.15.29:0/1055427356 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6f20004550 0x7f6f20004c60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:28.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.730+0000 7f6f3f400640 1 -- 172.21.15.29:0/1055427356 >> 172.21.15.29:0/1055427356 conn(0x7f6f3806de80 msgr2=0x7f6f3806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:28.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.730+0000 7f6f3f400640 1 -- 172.21.15.29:0/1055427356 shutdown_connections 2024-04-17T19:58:28.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.742+0000 7f6f3f400640 1 -- 172.21.15.29:0/1055427356 wait complete. 2024-04-17T19:58:28.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.762+0000 7f6f3f400640 1 Processor -- start 2024-04-17T19:58:28.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.762+0000 7f6f3f400640 1 -- start start 2024-04-17T19:58:28.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.766+0000 7f6f3f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f20004550 0x7f6f38163930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:28.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.770+0000 7f6f3f400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6f3816cb40 0x7f6f3816bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:28.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.770+0000 7f6f3f400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f3816c4a0 0x7f6f382027c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.770+0000 7f6f3f400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f6f380c9990 con 0x7f6f20004550 2024-04-17T19:58:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.770+0000 7f6f3f400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f6f381696c0 con 0x7f6f3816c4a0 2024-04-17T19:58:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.770+0000 7f6f3f400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f6f38168990 con 0x7f6f3816cb40 2024-04-17T19:58:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.770+0000 7f6f3d400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f3816c4a0 0x7f6f382027c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.770+0000 7f6f37e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6f3816cb40 0x7f6f3816bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.770+0000 7f6f37e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6f3816cb40 0x7f6f3816bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:46540/0 (socket says 172.21.15.29:46540) 2024-04-17T19:58:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.770+0000 7f6f37e00640 1 -- 172.21.15.29:0/3925125116 learned_addr learned my addr 172.21.15.29:0/3925125116 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.770+0000 7f6f3ca00640 1 --2- 172.21.15.29:0/3925125116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f20004550 0x7f6f38163930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.770+0000 7f6f37e00640 1 -- 172.21.15.29:0/3925125116 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f3816c4a0 msgr2=0x7f6f382027c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.770+0000 7f6f37e00640 1 --2- 172.21.15.29:0/3925125116 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f3816c4a0 0x7f6f382027c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.770+0000 7f6f37e00640 1 -- 172.21.15.29:0/3925125116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f20004550 msgr2=0x7f6f38163930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.770+0000 7f6f37e00640 1 --2- 172.21.15.29:0/3925125116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f20004550 0x7f6f38163930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:28.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.770+0000 7f6f37e00640 1 -- 172.21.15.29:0/3925125116 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f38163e70 con 0x7f6f3816cb40 2024-04-17T19:58:28.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.770+0000 7f6f37e00640 1 --2- 172.21.15.29:0/3925125116 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6f3816cb40 0x7f6f3816bf60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c025730 tx=0x7f6f20005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:28.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.774+0000 7f6f34c00640 1 -- 172.21.15.29:0/3925125116 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c0c6070 con 0x7f6f3816cb40 2024-04-17T19:58:28.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.774+0000 7f6f34c00640 1 -- 172.21.15.29:0/3925125116 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f2c0bc040 con 0x7f6f3816cb40 2024-04-17T19:58:28.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.774+0000 7f6f34c00640 1 -- 172.21.15.29:0/3925125116 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c0bb070 con 0x7f6f3816cb40 2024-04-17T19:58:28.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.774+0000 7f6f3f400640 1 -- 172.21.15.29:0/3925125116 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f381657f0 con 0x7f6f3816cb40 2024-04-17T19:58:28.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.774+0000 7f6f3f400640 1 -- 172.21.15.29:0/3925125116 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f38163410 con 0x7f6f3816cb40 2024-04-17T19:58:28.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.778+0000 7f6f3f400640 1 -- 172.21.15.29:0/3925125116 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f38074b20 con 0x7f6f3816cb40 2024-04-17T19:58:28.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.782+0000 7f6f34c00640 1 -- 172.21.15.29:0/3925125116 <== mon.2 v2:172.21.15.37:3301/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f6f2c0c9070 con 0x7f6f3816cb40 2024-04-17T19:58:28.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.786+0000 7f6f34c00640 1 --2- 172.21.15.29:0/3925125116 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6f1c068480 0x7f6f1c06a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:28.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.786+0000 7f6f34c00640 1 -- 172.21.15.29:0/3925125116 <== mon.2 v2:172.21.15.37:3301/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7f6f2c129eb0 con 0x7f6f3816cb40 2024-04-17T19:58:28.782 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.786+0000 7f6f3ca00640 1 --2- 172.21.15.29:0/3925125116 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6f1c068480 0x7f6f1c06a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:28.807 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.802+0000 7f6f3ca00640 1 --2- 172.21.15.29:0/3925125116 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6f1c068480 0x7f6f1c06a990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6f38051840 tx=0x7f6f28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:28.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:28.810+0000 7f6f34c00640 1 -- 172.21.15.29:0/3925125116 <== mon.2 v2:172.21.15.37:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f6f2c01f610 con 0x7f6f3816cb40 2024-04-17T19:58:29.450 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:29.454+0000 7f6f3f400640 1 -- 172.21.15.29:0/3925125116 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f38168770 con 0x7f6f3816cb40 2024-04-17T19:58:29.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:29.454+0000 7f6f34c00640 1 -- 172.21.15.29:0/3925125116 <== mon.2 v2:172.21.15.37:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6084 (secure 0 0 0) 0x7f6f2c0f9850 con 0x7f6f3816cb40 2024-04-17T19:58:29.451 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:58:29.451 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:26.051172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5631,5144,5212,5350]},"id":1}]} 2024-04-17T19:58:29.452 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 48 2024-04-17T19:58:29.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:29.458+0000 7f6f3f400640 1 -- 172.21.15.29:0/3925125116 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6f1c068480 msgr2=0x7f6f1c06a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:29.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:29.458+0000 7f6f3f400640 1 --2- 172.21.15.29:0/3925125116 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6f1c068480 0x7f6f1c06a990 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6f38051840 tx=0x7f6f28008040 comp rx=0 tx=0).stop 2024-04-17T19:58:29.454 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:29.458+0000 7f6f3f400640 1 -- 172.21.15.29:0/3925125116 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6f3816cb40 msgr2=0x7f6f3816bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:29.458+0000 7f6f3f400640 1 --2- 172.21.15.29:0/3925125116 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6f3816cb40 0x7f6f3816bf60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c025730 tx=0x7f6f20005c90 comp rx=0 tx=0).stop 2024-04-17T19:58:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:29.458+0000 7f6f3f400640 1 -- 172.21.15.29:0/3925125116 shutdown_connections 2024-04-17T19:58:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:29.458+0000 7f6f3f400640 1 --2- 172.21.15.29:0/3925125116 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f6f1c068480 0x7f6f1c06a990 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:29.458+0000 7f6f3f400640 1 --2- 172.21.15.29:0/3925125116 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f6f3816c4a0 0x7f6f382027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:29.458+0000 7f6f3f400640 1 --2- 172.21.15.29:0/3925125116 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f6f3816cb40 0x7f6f3816bf60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:29.458+0000 7f6f3f400640 1 --2- 172.21.15.29:0/3925125116 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f20004550 0x7f6f38163930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:29.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:29.458+0000 7f6f3f400640 1 -- 172.21.15.29:0/3925125116 >> 172.21.15.29:0/3925125116 conn(0x7f6f3806de80 msgr2=0x7f6f38160f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:29.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:29.462+0000 7f6f3f400640 1 -- 172.21.15.29:0/3925125116 shutdown_connections 2024-04-17T19:58:29.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:29.462+0000 7f6f3f400640 1 -- 172.21.15.29:0/3925125116 wait complete. 2024-04-17T19:58:29.555 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=1, replacing gid=4302 2024-04-17T19:58:29.555 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.e 2024-04-17T19:58:29.856 INFO:tasks.ceph.mds.c.smithi029.stderr:2024-04-17T19:58:29.854+0000 7ff3b0200640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-17T19:58:35.555 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2024-04-17T19:58:35.556 INFO:tasks.ceph.mds.e:Restarting daemon 2024-04-17T19:58:35.556 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-04-17T19:58:35.559 INFO:tasks.ceph.mds.e:Started 2024-04-17T19:58:35.560 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:58:35.660 INFO:tasks.ceph.mds.e.smithi029.stdout:starting mds.e at 2024-04-17T19:58:35.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.722+0000 7f37c0a00640 1 Processor -- start 2024-04-17T19:58:35.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.722+0000 7f37c0a00640 1 -- start start 2024-04-17T19:58:35.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.722+0000 7f37c0a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f37bc159c90 0x7f37bc0ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:35.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.722+0000 7f37c0a00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f37bc0ca7f0 0x7f37bc0c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:35.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.722+0000 7f37c0a00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37bc15a0a0 con 0x7f37bc0c74b0 2024-04-17T19:58:35.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.722+0000 7f37c0a00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f37bc15abe0 con 0x7f37bc159c90 2024-04-17T19:58:35.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.722+0000 7f37c0a00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f37bc15a340 con 0x7f37bc0ca7f0 2024-04-17T19:58:35.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.722+0000 7f37b9e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f37bc159c90 0x7f37bc0ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:35.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.722+0000 7f37ba800640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f37bc0c74b0 0x7f37bc06c660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43498/0 (socket says 172.21.15.29:43498) 2024-04-17T19:58:35.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.722+0000 7f37ba800640 1 -- 172.21.15.29:0/2043788390 learned_addr learned my addr 172.21.15.29:0/2043788390 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T19:58:35.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.722+0000 7f37b1400640 1 --2- 172.21.15.29:0/2043788390 >> v2:172.21.15.37:3300/0 conn(0x7f37bc0ca7f0 0x7f37bc0c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:35.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.722+0000 7f37b9e00640 1 -- 172.21.15.29:0/2043788390 >> v2:172.21.15.37:3300/0 conn(0x7f37bc0ca7f0 msgr2=0x7f37bc0c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:35.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.722+0000 7f37b9e00640 1 --2- 172.21.15.29:0/2043788390 >> v2:172.21.15.37:3300/0 conn(0x7f37bc0ca7f0 0x7f37bc0c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:35.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.722+0000 7f37b9e00640 1 -- 172.21.15.29:0/2043788390 >> v1:172.21.15.29:6789/0 conn(0x7f37bc0c74b0 legacy=0x7f37bc06c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:35.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.722+0000 7f37b9e00640 1 -- 172.21.15.29:0/2043788390 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37bc15a620 con 0x7f37bc159c90 2024-04-17T19:58:35.724 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.722+0000 7f37b9400640 1 -- 172.21.15.29:0/2043788390 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 884251861 0 0) 0x7f37bc15a0a0 con 0x7f37bc0c74b0 2024-04-17T19:58:35.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.726+0000 7f37b9e00640 1 --2- 172.21.15.29:0/2043788390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37bc159c90 0x7f37bc0ca2b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f37bc0cd310 tx=0x7f37ac01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d923ce0616de0f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:35.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.726+0000 7f37b9400640 1 -- 172.21.15.29:0/2043788390 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ac0b1070 con 0x7f37bc159c90 2024-04-17T19:58:35.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.726+0000 7f37b9400640 1 -- 172.21.15.29:0/2043788390 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37ac0ac5f0 con 0x7f37bc159c90 2024-04-17T19:58:35.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.726+0000 7f37b9400640 1 -- 172.21.15.29:0/2043788390 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ac0ac9c0 con 0x7f37bc159c90 2024-04-17T19:58:35.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.726+0000 7f37c0a00640 1 -- 172.21.15.29:0/2043788390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37bc159c90 msgr2=0x7f37bc0ca2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:35.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.726+0000 7f37c0a00640 1 --2- 172.21.15.29:0/2043788390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37bc159c90 0x7f37bc0ca2b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f37bc0cd310 tx=0x7f37ac01c480 comp rx=0 tx=0).stop 2024-04-17T19:58:35.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.726+0000 7f37c0a00640 1 -- 172.21.15.29:0/2043788390 shutdown_connections 2024-04-17T19:58:35.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.726+0000 7f37c0a00640 1 --2- 172.21.15.29:0/2043788390 >> v2:172.21.15.37:3300/0 conn(0x7f37bc0ca7f0 0x7f37bc0c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:35.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.726+0000 7f37c0a00640 1 --2- 172.21.15.29:0/2043788390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37bc159c90 0x7f37bc0ca2b0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:35.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.726+0000 7f37c0a00640 1 -- 172.21.15.29:0/2043788390 >> 172.21.15.29:0/2043788390 conn(0x7f37bc06fde0 msgr2=0x7f37bc0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:35.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.726+0000 7f37c0a00640 1 -- 172.21.15.29:0/2043788390 shutdown_connections 2024-04-17T19:58:35.725 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.726+0000 7f37c0a00640 1 -- 172.21.15.29:0/2043788390 wait complete. 2024-04-17T19:58:35.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.730+0000 7f37c0a00640 1 Processor -- start 2024-04-17T19:58:35.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.730+0000 7f37c0a00640 1 -- start start 2024-04-17T19:58:35.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.730+0000 7f37c0a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37bc0c74b0 0x7f37bc146d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:35.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.730+0000 7f37c0a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37bc0ca7f0 0x7f37bc1f9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:35.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37c0a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37bc159c90 0x7f37bc1f9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:35.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37c0a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc166fb0 con 0x7f37bc159c90 2024-04-17T19:58:35.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37c0a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f37bc166e30 con 0x7f37bc0ca7f0 2024-04-17T19:58:35.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37c0a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f37bc167130 con 0x7f37bc0c74b0 2024-04-17T19:58:35.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37ba800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37bc159c90 0x7f37bc1f9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:35.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37ba800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37bc159c90 0x7f37bc1f9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33634/0 (socket says 172.21.15.29:33634) 2024-04-17T19:58:35.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37ba800640 1 -- 172.21.15.29:0/1267742054 learned_addr learned my addr 172.21.15.29:0/1267742054 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:35.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37b1400640 1 --2- 172.21.15.29:0/1267742054 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37bc0ca7f0 0x7f37bc1f9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:35.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37b9e00640 1 --2- 172.21.15.29:0/1267742054 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37bc0c74b0 0x7f37bc146d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:35.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37ba800640 1 -- 172.21.15.29:0/1267742054 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37bc0c74b0 msgr2=0x7f37bc146d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:35.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37ba800640 1 --2- 172.21.15.29:0/1267742054 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37bc0c74b0 0x7f37bc146d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:35.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37ba800640 1 -- 172.21.15.29:0/1267742054 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37bc0ca7f0 msgr2=0x7f37bc1f9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:35.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37ba800640 1 --2- 172.21.15.29:0/1267742054 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37bc0ca7f0 0x7f37bc1f9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:35.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37ba800640 1 -- 172.21.15.29:0/1267742054 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37bc14f000 con 0x7f37bc159c90 2024-04-17T19:58:35.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37ba800640 1 --2- 172.21.15.29:0/1267742054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37bc159c90 0x7f37bc1f9540 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f37b4002240 tx=0x7f37b400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:35.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37b2a00640 1 -- 172.21.15.29:0/1267742054 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b4017070 con 0x7f37bc159c90 2024-04-17T19:58:35.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37b2a00640 1 -- 172.21.15.29:0/1267742054 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37b400ac30 con 0x7f37bc159c90 2024-04-17T19:58:35.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37b2a00640 1 -- 172.21.15.29:0/1267742054 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b4012400 con 0x7f37bc159c90 2024-04-17T19:58:35.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37c0a00640 1 -- 172.21.15.29:0/1267742054 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37bc14df20 con 0x7f37bc159c90 2024-04-17T19:58:35.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37c0a00640 1 -- 172.21.15.29:0/1267742054 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37bc072f60 con 0x7f37bc159c90 2024-04-17T19:58:35.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37c0a00640 1 -- 172.21.15.29:0/1267742054 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37bc069b90 con 0x7f37bc159c90 2024-04-17T19:58:35.731 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.734+0000 7f37b2a00640 1 -- 172.21.15.29:0/1267742054 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f37b401a070 con 0x7f37bc159c90 2024-04-17T19:58:35.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.738+0000 7f37b2a00640 1 --2- 172.21.15.29:0/1267742054 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3798068380 0x7f379806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:35.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.738+0000 7f37b2a00640 1 -- 172.21.15.29:0/1267742054 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7f37b4086d40 con 0x7f37bc159c90 2024-04-17T19:58:35.732 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.738+0000 7f37b9e00640 1 --2- 172.21.15.29:0/1267742054 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3798068380 0x7f379806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:35.733 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.738+0000 7f37b9e00640 1 --2- 172.21.15.29:0/1267742054 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3798068380 0x7f379806a890 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f37ac0ae000 tx=0x7f37ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:35.736 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.742+0000 7f37b2a00640 1 -- 172.21.15.29:0/1267742054 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f37bc14f000 con 0x7f37bc159c90 2024-04-17T19:58:35.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.938+0000 7f37c0a00640 1 -- 172.21.15.29:0/1267742054 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37bc14cbe0 con 0x7f37bc159c90 2024-04-17T19:58:35.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.938+0000 7f37b2a00640 1 -- 172.21.15.29:0/1267742054 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+6084 (secure 0 0 0) 0x7f37b40565d0 con 0x7f37bc159c90 2024-04-17T19:58:35.935 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:58:35.935 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:26.051172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[4305,5631,5144,5212,5350]},"id":1}]} 2024-04-17T19:58:35.935 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 48 2024-04-17T19:58:35.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.942+0000 7f37c0a00640 1 -- 172.21.15.29:0/1267742054 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3798068380 msgr2=0x7f379806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:35.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.942+0000 7f37c0a00640 1 --2- 172.21.15.29:0/1267742054 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3798068380 0x7f379806a890 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f37ac0ae000 tx=0x7f37ac008040 comp rx=0 tx=0).stop 2024-04-17T19:58:35.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.942+0000 7f37c0a00640 1 -- 172.21.15.29:0/1267742054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37bc159c90 msgr2=0x7f37bc1f9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:35.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.942+0000 7f37c0a00640 1 --2- 172.21.15.29:0/1267742054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37bc159c90 0x7f37bc1f9540 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f37b4002240 tx=0x7f37b400cf50 comp rx=0 tx=0).stop 2024-04-17T19:58:35.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.942+0000 7f37c0a00640 1 -- 172.21.15.29:0/1267742054 shutdown_connections 2024-04-17T19:58:35.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.942+0000 7f37c0a00640 1 --2- 172.21.15.29:0/1267742054 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3798068380 0x7f379806a890 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:35.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.942+0000 7f37c0a00640 1 --2- 172.21.15.29:0/1267742054 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37bc159c90 0x7f37bc1f9540 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:35.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.942+0000 7f37c0a00640 1 --2- 172.21.15.29:0/1267742054 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f37bc0ca7f0 0x7f37bc1f9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:35.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.942+0000 7f37c0a00640 1 --2- 172.21.15.29:0/1267742054 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f37bc0c74b0 0x7f37bc146d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:35.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.942+0000 7f37c0a00640 1 -- 172.21.15.29:0/1267742054 >> 172.21.15.29:0/1267742054 conn(0x7f37bc06fde0 msgr2=0x7f37bc1581f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:35.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.942+0000 7f37c0a00640 1 -- 172.21.15.29:0/1267742054 shutdown_connections 2024-04-17T19:58:35.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:35.942+0000 7f37c0a00640 1 -- 172.21.15.29:0/1267742054 wait complete. 2024-04-17T19:58:35.957 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2024-04-17T19:58:37.959 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T19:58:38.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.150+0000 7fcc13600640 1 Processor -- start 2024-04-17T19:58:38.147 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.150+0000 7fcc13600640 1 -- start start 2024-04-17T19:58:38.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.154+0000 7fcc13600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c0c6f70 0x7fcc0c0c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:38.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.154+0000 7fcc13600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcc0c06c660 0x7fcc0c06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:38.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.154+0000 7fcc13600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc0c15a0a0 con 0x7fcc0c159c90 2024-04-17T19:58:38.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.154+0000 7fcc13600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fcc0c15abe0 con 0x7fcc0c06c660 2024-04-17T19:58:38.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.154+0000 7fcc13600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcc0c15a340 con 0x7fcc0c0c6f70 2024-04-17T19:58:38.149 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.154+0000 7fcc11600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcc0c06c660 0x7fcc0c06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:38.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.154+0000 7fcc11600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcc0c06c660 0x7fcc0c06ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33642/0 (socket says 172.21.15.29:33642) 2024-04-17T19:58:38.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.154+0000 7fcc0be00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c0c6f70 0x7fcc0c0c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:38.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.154+0000 7fcc11600640 1 -- 172.21.15.29:0/3097163781 learned_addr learned my addr 172.21.15.29:0/3097163781 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:38.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.154+0000 7fcc10c00640 1 --1- 172.21.15.29:0/3097163781 >> v1:172.21.15.37:6789/0 conn(0x7fcc0c159c90 0x7fcc0c0ca2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:57134/0 (socket says 172.21.15.29:57134) 2024-04-17T19:58:38.150 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.154+0000 7fcc11600640 1 -- 172.21.15.29:0/3097163781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c0c6f70 msgr2=0x7fcc0c0c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T19:58:38.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.154+0000 7fcc11600640 1 --2- 172.21.15.29:0/3097163781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c0c6f70 0x7fcc0c0c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:38.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.154+0000 7fcc11600640 1 -- 172.21.15.29:0/3097163781 >> v1:172.21.15.37:6789/0 conn(0x7fcc0c159c90 legacy=0x7fcc0c0ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:38.151 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.154+0000 7fcc11600640 1 -- 172.21.15.29:0/3097163781 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0c15a620 con 0x7fcc0c06c660 2024-04-17T19:58:38.152 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.158+0000 7fcc11600640 1 --2- 172.21.15.29:0/3097163781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c06c660 0x7fcc0c06ca70 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fcc04009e50 tx=0x7fcc0401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f476d345789397a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:38.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.158+0000 7fcc0b400640 1 -- 172.21.15.29:0/3097163781 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc040a5700 con 0x7fcc0c06c660 2024-04-17T19:58:38.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.158+0000 7fcc0b400640 1 -- 172.21.15.29:0/3097163781 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc040a58a0 con 0x7fcc0c06c660 2024-04-17T19:58:38.153 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.158+0000 7fcc0b400640 1 -- 172.21.15.29:0/3097163781 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc040a5c70 con 0x7fcc0c06c660 2024-04-17T19:58:38.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.158+0000 7fcc13600640 1 -- 172.21.15.29:0/3097163781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c06c660 msgr2=0x7fcc0c06ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:38.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.158+0000 7fcc13600640 1 --2- 172.21.15.29:0/3097163781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c06c660 0x7fcc0c06ca70 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fcc04009e50 tx=0x7fcc0401ca30 comp rx=0 tx=0).stop 2024-04-17T19:58:38.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.158+0000 7fcc13600640 1 -- 172.21.15.29:0/3097163781 shutdown_connections 2024-04-17T19:58:38.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.158+0000 7fcc13600640 1 --2- 172.21.15.29:0/3097163781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c06c660 0x7fcc0c06ca70 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:38.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.158+0000 7fcc13600640 1 --2- 172.21.15.29:0/3097163781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c0c6f70 0x7fcc0c0c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:38.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.158+0000 7fcc13600640 1 -- 172.21.15.29:0/3097163781 >> 172.21.15.29:0/3097163781 conn(0x7fcc0c06fde0 msgr2=0x7fcc0c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:38.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.158+0000 7fcc13600640 1 -- 172.21.15.29:0/3097163781 shutdown_connections 2024-04-17T19:58:38.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.158+0000 7fcc13600640 1 -- 172.21.15.29:0/3097163781 wait complete. 2024-04-17T19:58:38.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.162+0000 7fcc13600640 1 Processor -- start 2024-04-17T19:58:38.158 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.162+0000 7fcc13600640 1 -- start start 2024-04-17T19:58:38.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.162+0000 7fcc13600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c06c660 0x7fcc0c138bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:38.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.162+0000 7fcc13600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c0c6f70 0x7fcc0c139130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:38.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.162+0000 7fcc13600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc0c159c90 0x7fcc0c13dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:38.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.162+0000 7fcc13600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c0ca9c0 con 0x7fcc0c0c6f70 2024-04-17T19:58:38.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.162+0000 7fcc13600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcc0c06cfb0 con 0x7fcc0c159c90 2024-04-17T19:58:38.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.162+0000 7fcc13600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcc0c0cab40 con 0x7fcc0c06c660 2024-04-17T19:58:38.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.166+0000 7fcc0be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c0c6f70 0x7fcc0c139130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:38.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.166+0000 7fcc0be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c0c6f70 0x7fcc0c139130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33646/0 (socket says 172.21.15.29:33646) 2024-04-17T19:58:38.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.166+0000 7fcc0be00640 1 -- 172.21.15.29:0/2361502725 learned_addr learned my addr 172.21.15.29:0/2361502725 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T19:58:38.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.166+0000 7fcc10c00640 1 --2- 172.21.15.29:0/2361502725 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c06c660 0x7fcc0c138bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:38.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.166+0000 7fcc11600640 1 --2- 172.21.15.29:0/2361502725 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc0c159c90 0x7fcc0c13dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:38.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.166+0000 7fcc0be00640 1 -- 172.21.15.29:0/2361502725 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c06c660 msgr2=0x7fcc0c138bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:38.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.166+0000 7fcc0be00640 1 --2- 172.21.15.29:0/2361502725 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c06c660 0x7fcc0c138bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:38.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.166+0000 7fcc0be00640 1 -- 172.21.15.29:0/2361502725 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc0c159c90 msgr2=0x7fcc0c13dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:38.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.166+0000 7fcc0be00640 1 --2- 172.21.15.29:0/2361502725 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc0c159c90 0x7fcc0c13dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:38.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.166+0000 7fcc0be00640 1 -- 172.21.15.29:0/2361502725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc0c0c56a0 con 0x7fcc0c0c6f70 2024-04-17T19:58:38.162 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.166+0000 7fcc0be00640 1 --2- 172.21.15.29:0/2361502725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c0c6f70 0x7fcc0c139130 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800e870 tx=0x7fcbf800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:38.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.166+0000 7fcc08c00640 1 -- 172.21.15.29:0/2361502725 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8017070 con 0x7fcc0c0c6f70 2024-04-17T19:58:38.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.166+0000 7fcc08c00640 1 -- 172.21.15.29:0/2361502725 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbf800c940 con 0x7fcc0c0c6f70 2024-04-17T19:58:38.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.166+0000 7fcc13600640 1 -- 172.21.15.29:0/2361502725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc0c14c720 con 0x7fcc0c0c6f70 2024-04-17T19:58:38.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.166+0000 7fcc08c00640 1 -- 172.21.15.29:0/2361502725 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf800cd30 con 0x7fcc0c0c6f70 2024-04-17T19:58:38.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.166+0000 7fcc13600640 1 -- 172.21.15.29:0/2361502725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc0c0c5c20 con 0x7fcc0c0c6f70 2024-04-17T19:58:38.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.166+0000 7fcc13600640 1 -- 172.21.15.29:0/2361502725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc0c14ca40 con 0x7fcc0c0c6f70 2024-04-17T19:58:38.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.170+0000 7fcc08c00640 1 -- 172.21.15.29:0/2361502725 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcbf801a070 con 0x7fcc0c0c6f70 2024-04-17T19:58:38.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.170+0000 7fcc08c00640 1 --2- 172.21.15.29:0/2361502725 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbec0683b0 0x7fcbec06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T19:58:38.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.170+0000 7fcc08c00640 1 -- 172.21.15.29:0/2361502725 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7fcbf80874f0 con 0x7fcc0c0c6f70 2024-04-17T19:58:38.166 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.170+0000 7fcc10c00640 1 --2- 172.21.15.29:0/2361502725 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbec0683b0 0x7fcbec06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T19:58:38.167 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.170+0000 7fcc10c00640 1 --2- 172.21.15.29:0/2361502725 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbec0683b0 0x7fcbec06a8c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcc00000f70 tx=0x7fcc00007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T19:58:38.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.174+0000 7fcc08c00640 1 -- 172.21.15.29:0/2361502725 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fcbf8056f10 con 0x7fcc0c0c6f70 2024-04-17T19:58:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.394+0000 7fcc13600640 1 -- 172.21.15.29:0/2361502725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc0c069b90 con 0x7fcc0c0c6f70 2024-04-17T19:58:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.394+0000 7fcc08c00640 1 -- 172.21.15.29:0/2361502725 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6908 (secure 0 0 0) 0x7fcbf805b210 con 0x7fcc0c0c6f70 2024-04-17T19:58:38.388 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T19:58:38.389 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5775,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:26.051172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5212,5144,5631,4305]},"id":1}]} 2024-04-17T19:58:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 49 2024-04-17T19:58:38.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.394+0000 7fcc13600640 1 -- 172.21.15.29:0/2361502725 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbec0683b0 msgr2=0x7fcbec06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:38.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.394+0000 7fcc13600640 1 --2- 172.21.15.29:0/2361502725 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbec0683b0 0x7fcbec06a8c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcc00000f70 tx=0x7fcc00007e30 comp rx=0 tx=0).stop 2024-04-17T19:58:38.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.394+0000 7fcc13600640 1 -- 172.21.15.29:0/2361502725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c0c6f70 msgr2=0x7fcc0c139130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T19:58:38.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.394+0000 7fcc13600640 1 --2- 172.21.15.29:0/2361502725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c0c6f70 0x7fcc0c139130 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fcbf800e870 tx=0x7fcbf800ec70 comp rx=0 tx=0).stop 2024-04-17T19:58:38.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.394+0000 7fcc13600640 1 -- 172.21.15.29:0/2361502725 shutdown_connections 2024-04-17T19:58:38.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.394+0000 7fcc13600640 1 --2- 172.21.15.29:0/2361502725 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbec0683b0 0x7fcbec06a8c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:38.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.394+0000 7fcc13600640 1 --2- 172.21.15.29:0/2361502725 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcc0c159c90 0x7fcc0c13dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:38.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.394+0000 7fcc13600640 1 --2- 172.21.15.29:0/2361502725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc0c0c6f70 0x7fcc0c139130 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:38.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.394+0000 7fcc13600640 1 --2- 172.21.15.29:0/2361502725 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcc0c06c660 0x7fcc0c138bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T19:58:38.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.394+0000 7fcc13600640 1 -- 172.21.15.29:0/2361502725 >> 172.21.15.29:0/2361502725 conn(0x7fcc0c06fde0 msgr2=0x7fcc0c14d3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T19:58:38.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.394+0000 7fcc13600640 1 -- 172.21.15.29:0/2361502725 shutdown_connections 2024-04-17T19:58:38.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T19:58:38.394+0000 7fcc13600640 1 -- 172.21.15.29:0/2361502725 wait complete. 2024-04-17T19:58:38.411 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2024-04-17T19:58:38.411 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 89 secs before thrashing 2024-04-17T19:58:47.697 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 1% (140/12484) Updating files: 2% (250/12484) Updating files: 3% (375/12484) Updating files: 4% (500/12484) Updating files: 4% (506/12484) Updating files: 5% (625/12484) Updating files: 6% (750/12484) Updating files: 6% (824/12484) Updating files: 7% (874/12484) Updating files: 8% (999/12484) Updating files: 8% (1022/12484) Updating files: 9% (1124/12484) Updating files: 9% (1142/12484) Updating files: 10% (1249/12484) Updating files: 11% (1374/12484) Updating files: 11% (1444/12484) Updating files: 11% (1470/12484) Updating files: 12% (1499/12484) Updating files: 12% (1550/12484) Updating files: 13% (1623/12484) Updating files: 13% (1725/12484) Updating files: 14% (1748/12484) Updating files: 14% (1807/12484) Updating files: 15% (1873/12484) Updating files: 15% (1924/12484) Updating files: 15% (1953/12484) Updating files: 16% (1998/12484) Updating files: 16% (2038/12484) Updating files: 16% (2075/12484) Updating files: 17% (2123/12484) Updating files: 18% (2248/12484) Updating files: 18% (2307/12484) Updating files: 19% (2372/12484) Updating files: 19% (2443/12484) Updating files: 20% (2497/12484) Updating files: 20% (2557/12484) Updating files: 20% (2586/12484) Updating files: 21% (2622/12484) Updating files: 22% (2747/12484) Updating files: 22% (2761/12484) Updating files: 22% (2830/12484) Updating files: 23% (2872/12484) Updating files: 24% (2997/12484) Updating files: 25% (3121/12484) Updating files: 25% (3207/12484) Updating files: 26% (3246/12484) Updating files: 26% (3307/12484) Updating files: 27% (3371/12484) Updating files: 27% (3488/12484) Updating files: 28% (3496/12484) Updating files: 28% (3497/12484) Updating files: 28% (3575/12484) Updating files: 29% (3621/12484) Updating files: 30% (3746/12484) Updating files: 30% (3868/12484) Updating files: 31% (3871/12484) Updating files: 32% (3995/12484) Updating files: 32% (4014/12484) Updating files: 33% (4120/12484) Updating files: 34% (4245/12484) Updating files: 34% (4252/12484) Updating files: 35% (4370/12484) Updating files: 36% (4495/12484) Updating files: 37% (4620/12484) Updating files: 37% (4663/12484) Updating files: 37% (4677/12484) Updating files: 38% (4744/12484) Updating files: 38% (4832/12484) Updating files: 38% (4859/12484) Updating files: 39% (4869/12484) Updating files: 39% (4942/12484) Updating files: 39% (4943/12484) Updating files: 40% (4994/12484) Updating files: 40% (5108/12484) Updating files: 41% (5119/12484) Updating files: 42% (5244/12484) Updating files: 42% (5330/12484) Updating files: 43% (5369/12484) Updating files: 44% (5493/12484) Updating files: 44% (5507/12484) Updating files: 45% (5618/12484) Updating files: 46% (5743/12484) Updating files: 46% (5764/12484) Updating files: 46% (5766/12484) Updating files: 47% (5868/12484) Updating files: 47% (5905/12484) Updating files: 48% (5993/12484) Updating files: 48% (6085/12484) Updating files: 49% (6118/12484) Updating files: 50% (6242/12484) Updating files: 50% (6289/12484) Updating files: 51% (6367/12484) Updating files: 51% (6385/12484) Updating files: 51% (6399/12484) Updating files: 52% (6492/12484) Updating files: 53% (6617/12484) Updating files: 53% (6656/12484) Updating files: 53% (6717/12484) Updating files: 54% (6742/12484) Updating files: 55% (6867/12484) Updating files: 55% (6932/12484) Updating files: 55% (6991/12484) Updating files: 56% (6992/12484) Updating files: 56% (6996/12484) Updating files: 56% (7000/12484) Updating files: 57% (7116/12484) Updating files: 58% (7241/12484) Updating files: 58% (7317/12484) Updating files: 59% (7366/12484) Updating files: 60% (7491/12484) Updating files: 61% (7616/12484) Updating files: 61% (7733/12484) Updating files: 62% (7741/12484) Updating files: 63% (7865/12484) Updating files: 64% (7990/12484) Updating files: 64% (8086/12484) Updating files: 65% (8115/12484) Updating files: 65% (8136/12484) Updating files: 65% (8204/12484) Updating files: 66% (8240/12484) Updating files: 67% (8365/12484) Updating files: 67% (8425/12484) Updating files: 68% (8490/12484) Updating files: 69% (8614/12484) Updating files: 70% (8739/12484) Updating files: 70% (8750/12484) Updating files: 71% (8864/12484) Updating files: 72% (8989/12484) Updating files: 72% (9016/12484) Updating files: 72% (9086/12484) Updating files: 73% (9114/12484) Updating files: 73% (9122/12484) Updating files: 74% (9239/12484) Updating files: 75% (9363/12484) Updating files: 75% (9401/12484) Updating files: 76% (9488/12484) Updating files: 76% (9498/12484) Updating files: 76% (9539/12484) Updating files: 77% (9613/12484) Updating files: 77% (9721/12484) Updating files: 78% (9738/12484) Updating files: 79% (9863/12484) Updating files: 79% (9942/12484) Updating files: 80% (9988/12484) Updating files: 80% (10088/12484) Updating files: 81% (10113/12484) Updating files: 82% (10237/12484) Updating files: 82% (10244/12484) Updating files: 83% (10362/12484) Updating files: 84% (10487/12484) Updating files: 84% (10553/12484) Updating files: 84% (10554/12484) Updating files: 85% (10612/12484) Updating files: 86% (10737/12484) Updating files: 86% (10771/12484) Updating files: 86% (10794/12484) Updating files: 86% (10798/12484) Updating files: 87% (10862/12484) Updating files: 87% (10883/12484) Updating files: 87% (10914/12484) Updating files: 88% (10986/12484) Updating files: 88% (11002/12484) Updating files: 89% (11111/12484) Updating files: 90% (11236/12484) Updating files: 90% (11299/12484) Updating files: 90% (11321/12484) Updating files: 91% (11361/12484) Updating files: 92% (11486/12484) Updating files: 92% (11559/12484) Updating files: 93% (11611/12484) Updating files: 93% (11626/12484) Updating files: 94% (11735/12484) Updating files: 94% (11836/12484) Updating files: 95% (11860/12484) Updating files: 95% (11915/12484) Updating files: 96% (11985/12484) Updating files: 96% (12071/12484) Updating files: 97% (12110/12484) Updating files: 98% (12235/12484) Updating files: 98% (12307/12484) Updating files: 98% (12329/12484) Updating files: 99% (12360/12484) Updating files: 99% (12451/12484) Updating files: 100% (12484/12484) Updating files: 100% (12484/12484), done. 2024-04-17T19:58:48.693 INFO:tasks.workunit.client.0.smithi029.stderr:+ trap - EXIT 2024-04-17T19:58:48.693 INFO:tasks.workunit.client.0.smithi029.stderr:+ cd ceph 2024-04-17T19:58:48.694 INFO:tasks.workunit.client.0.smithi029.stderr:+ seq 1 90 2024-04-17T19:58:48.695 INFO:tasks.workunit.client.0.smithi029.stderr:+ versions=1 2024-04-17T19:58:48.695 INFO:tasks.workunit.client.0.smithi029.stderr:2 2024-04-17T19:58:48.695 INFO:tasks.workunit.client.0.smithi029.stderr:3 2024-04-17T19:58:48.695 INFO:tasks.workunit.client.0.smithi029.stderr:4 2024-04-17T19:58:48.695 INFO:tasks.workunit.client.0.smithi029.stderr:5 2024-04-17T19:58:48.695 INFO:tasks.workunit.client.0.smithi029.stderr:6 2024-04-17T19:58:48.696 INFO:tasks.workunit.client.0.smithi029.stderr:7 2024-04-17T19:58:48.696 INFO:tasks.workunit.client.0.smithi029.stderr:8 2024-04-17T19:58:48.696 INFO:tasks.workunit.client.0.smithi029.stderr:9 2024-04-17T19:58:48.696 INFO:tasks.workunit.client.0.smithi029.stderr:10 2024-04-17T19:58:48.696 INFO:tasks.workunit.client.0.smithi029.stderr:11 2024-04-17T19:58:48.696 INFO:tasks.workunit.client.0.smithi029.stderr:12 2024-04-17T19:58:48.696 INFO:tasks.workunit.client.0.smithi029.stderr:13 2024-04-17T19:58:48.696 INFO:tasks.workunit.client.0.smithi029.stderr:14 2024-04-17T19:58:48.696 INFO:tasks.workunit.client.0.smithi029.stderr:15 2024-04-17T19:58:48.696 INFO:tasks.workunit.client.0.smithi029.stderr:16 2024-04-17T19:58:48.696 INFO:tasks.workunit.client.0.smithi029.stderr:17 2024-04-17T19:58:48.696 INFO:tasks.workunit.client.0.smithi029.stderr:18 2024-04-17T19:58:48.696 INFO:tasks.workunit.client.0.smithi029.stderr:19 2024-04-17T19:58:48.696 INFO:tasks.workunit.client.0.smithi029.stderr:20 2024-04-17T19:58:48.696 INFO:tasks.workunit.client.0.smithi029.stderr:21 2024-04-17T19:58:48.697 INFO:tasks.workunit.client.0.smithi029.stderr:22 2024-04-17T19:58:48.697 INFO:tasks.workunit.client.0.smithi029.stderr:23 2024-04-17T19:58:48.697 INFO:tasks.workunit.client.0.smithi029.stderr:24 2024-04-17T19:58:48.697 INFO:tasks.workunit.client.0.smithi029.stderr:25 2024-04-17T19:58:48.697 INFO:tasks.workunit.client.0.smithi029.stderr:26 2024-04-17T19:58:48.697 INFO:tasks.workunit.client.0.smithi029.stderr:27 2024-04-17T19:58:48.697 INFO:tasks.workunit.client.0.smithi029.stderr:28 2024-04-17T19:58:48.697 INFO:tasks.workunit.client.0.smithi029.stderr:29 2024-04-17T19:58:48.697 INFO:tasks.workunit.client.0.smithi029.stderr:30 2024-04-17T19:58:48.697 INFO:tasks.workunit.client.0.smithi029.stderr:31 2024-04-17T19:58:48.697 INFO:tasks.workunit.client.0.smithi029.stderr:32 2024-04-17T19:58:48.697 INFO:tasks.workunit.client.0.smithi029.stderr:33 2024-04-17T19:58:48.697 INFO:tasks.workunit.client.0.smithi029.stderr:34 2024-04-17T19:58:48.697 INFO:tasks.workunit.client.0.smithi029.stderr:35 2024-04-17T19:58:48.698 INFO:tasks.workunit.client.0.smithi029.stderr:36 2024-04-17T19:58:48.698 INFO:tasks.workunit.client.0.smithi029.stderr:37 2024-04-17T19:58:48.698 INFO:tasks.workunit.client.0.smithi029.stderr:38 2024-04-17T19:58:48.698 INFO:tasks.workunit.client.0.smithi029.stderr:39 2024-04-17T19:58:48.698 INFO:tasks.workunit.client.0.smithi029.stderr:40 2024-04-17T19:58:48.698 INFO:tasks.workunit.client.0.smithi029.stderr:41 2024-04-17T19:58:48.698 INFO:tasks.workunit.client.0.smithi029.stderr:42 2024-04-17T19:58:48.698 INFO:tasks.workunit.client.0.smithi029.stderr:43 2024-04-17T19:58:48.698 INFO:tasks.workunit.client.0.smithi029.stderr:44 2024-04-17T19:58:48.698 INFO:tasks.workunit.client.0.smithi029.stderr:45 2024-04-17T19:58:48.698 INFO:tasks.workunit.client.0.smithi029.stderr:46 2024-04-17T19:58:48.698 INFO:tasks.workunit.client.0.smithi029.stderr:47 2024-04-17T19:58:48.698 INFO:tasks.workunit.client.0.smithi029.stderr:48 2024-04-17T19:58:48.698 INFO:tasks.workunit.client.0.smithi029.stderr:49 2024-04-17T19:58:48.699 INFO:tasks.workunit.client.0.smithi029.stderr:50 2024-04-17T19:58:48.699 INFO:tasks.workunit.client.0.smithi029.stderr:51 2024-04-17T19:58:48.699 INFO:tasks.workunit.client.0.smithi029.stderr:52 2024-04-17T19:58:48.699 INFO:tasks.workunit.client.0.smithi029.stderr:53 2024-04-17T19:58:48.699 INFO:tasks.workunit.client.0.smithi029.stderr:54 2024-04-17T19:58:48.699 INFO:tasks.workunit.client.0.smithi029.stderr:55 2024-04-17T19:58:48.699 INFO:tasks.workunit.client.0.smithi029.stderr:56 2024-04-17T19:58:48.699 INFO:tasks.workunit.client.0.smithi029.stderr:57 2024-04-17T19:58:48.699 INFO:tasks.workunit.client.0.smithi029.stderr:58 2024-04-17T19:58:48.699 INFO:tasks.workunit.client.0.smithi029.stderr:59 2024-04-17T19:58:48.699 INFO:tasks.workunit.client.0.smithi029.stderr:60 2024-04-17T19:58:48.699 INFO:tasks.workunit.client.0.smithi029.stderr:61 2024-04-17T19:58:48.699 INFO:tasks.workunit.client.0.smithi029.stderr:62 2024-04-17T19:58:48.699 INFO:tasks.workunit.client.0.smithi029.stderr:63 2024-04-17T19:58:48.699 INFO:tasks.workunit.client.0.smithi029.stderr:64 2024-04-17T19:58:48.700 INFO:tasks.workunit.client.0.smithi029.stderr:65 2024-04-17T19:58:48.700 INFO:tasks.workunit.client.0.smithi029.stderr:66 2024-04-17T19:58:48.700 INFO:tasks.workunit.client.0.smithi029.stderr:67 2024-04-17T19:58:48.700 INFO:tasks.workunit.client.0.smithi029.stderr:68 2024-04-17T19:58:48.700 INFO:tasks.workunit.client.0.smithi029.stderr:69 2024-04-17T19:58:48.700 INFO:tasks.workunit.client.0.smithi029.stderr:70 2024-04-17T19:58:48.700 INFO:tasks.workunit.client.0.smithi029.stderr:71 2024-04-17T19:58:48.700 INFO:tasks.workunit.client.0.smithi029.stderr:72 2024-04-17T19:58:48.700 INFO:tasks.workunit.client.0.smithi029.stderr:73 2024-04-17T19:58:48.700 INFO:tasks.workunit.client.0.smithi029.stderr:74 2024-04-17T19:58:48.700 INFO:tasks.workunit.client.0.smithi029.stderr:75 2024-04-17T19:58:48.700 INFO:tasks.workunit.client.0.smithi029.stderr:76 2024-04-17T19:58:48.700 INFO:tasks.workunit.client.0.smithi029.stderr:77 2024-04-17T19:58:48.700 INFO:tasks.workunit.client.0.smithi029.stderr:78 2024-04-17T19:58:48.700 INFO:tasks.workunit.client.0.smithi029.stderr:79 2024-04-17T19:58:48.701 INFO:tasks.workunit.client.0.smithi029.stderr:80 2024-04-17T19:58:48.701 INFO:tasks.workunit.client.0.smithi029.stderr:81 2024-04-17T19:58:48.701 INFO:tasks.workunit.client.0.smithi029.stderr:82 2024-04-17T19:58:48.701 INFO:tasks.workunit.client.0.smithi029.stderr:83 2024-04-17T19:58:48.701 INFO:tasks.workunit.client.0.smithi029.stderr:84 2024-04-17T19:58:48.701 INFO:tasks.workunit.client.0.smithi029.stderr:85 2024-04-17T19:58:48.701 INFO:tasks.workunit.client.0.smithi029.stderr:86 2024-04-17T19:58:48.701 INFO:tasks.workunit.client.0.smithi029.stderr:87 2024-04-17T19:58:48.701 INFO:tasks.workunit.client.0.smithi029.stderr:88 2024-04-17T19:58:48.701 INFO:tasks.workunit.client.0.smithi029.stderr:89 2024-04-17T19:58:48.701 INFO:tasks.workunit.client.0.smithi029.stderr:90 2024-04-17T19:58:48.701 INFO:tasks.workunit.client.0.smithi029.stderr:+ [ 1 -eq 48 ] 2024-04-17T19:58:48.701 INFO:tasks.workunit.client.0.smithi029.stderr:+ ver=v0.1 2024-04-17T19:58:48.702 INFO:tasks.workunit.client.0.smithi029.stdout:v0.1 2024-04-17T19:58:48.702 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo v0.1 2024-04-17T19:58:48.702 INFO:tasks.workunit.client.0.smithi029.stderr:+ git reset --hard v0.1 2024-04-17T19:58:52.882 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:58:52.886 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:59:22.956 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:59:22.960 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:59:53.015 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T19:59:53.019 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:00:07.412 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:07.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.569+0000 7f05dbe00640 1 Processor -- start 2024-04-17T20:00:07.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.569+0000 7f05dbe00640 1 -- start start 2024-04-17T20:00:07.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.573+0000 7f05dbe00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f05dc156140 0x7f05dc0c9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:07.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.573+0000 7f05dbe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f05dc15fd90 0x7f05dc0cc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:07.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.573+0000 7f05dbe00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05dc0ca7c0 con 0x7f05dc0cc020 2024-04-17T20:00:07.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.573+0000 7f05dbe00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f05dc06c7e0 con 0x7f05dc15fd90 2024-04-17T20:00:07.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.573+0000 7f05dbe00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f05dc06cac0 con 0x7f05dc156140 2024-04-17T20:00:07.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.573+0000 7f05db400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f05dc15fd90 0x7f05dc0cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:07.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.573+0000 7f05db400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f05dc15fd90 0x7f05dc0cc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41168/0 (socket says 172.21.15.29:41168) 2024-04-17T20:00:07.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.573+0000 7f05db400640 1 -- 172.21.15.29:0/3075118690 learned_addr learned my addr 172.21.15.29:0/3075118690 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:07.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.573+0000 7f05daa00640 1 --2- 172.21.15.29:0/3075118690 >> v2:172.21.15.37:3300/0 conn(0x7f05dc156140 0x7f05dc0c9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:07.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.573+0000 7f05db400640 1 -- 172.21.15.29:0/3075118690 >> v2:172.21.15.37:3300/0 conn(0x7f05dc156140 msgr2=0x7f05dc0c9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:07.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.573+0000 7f05db400640 1 --2- 172.21.15.29:0/3075118690 >> v2:172.21.15.37:3300/0 conn(0x7f05dc156140 0x7f05dc0c9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:07.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.573+0000 7f05db400640 1 -- 172.21.15.29:0/3075118690 >> v1:172.21.15.37:6789/0 conn(0x7f05dc0cc020 legacy=0x7f05dc15f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:07.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.573+0000 7f05db400640 1 -- 172.21.15.29:0/3075118690 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05dc06cda0 con 0x7f05dc15fd90 2024-04-17T20:00:07.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.573+0000 7f05db400640 1 --2- 172.21.15.29:0/3075118690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05dc15fd90 0x7f05dc0cc430 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f05dc0ccb70 tx=0x7f05d401f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b92184b5aab5ee9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:07.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.577+0000 7f05da000640 1 -- 172.21.15.29:0/3075118690 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d40b4070 con 0x7f05dc15fd90 2024-04-17T20:00:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.577+0000 7f05da000640 1 -- 172.21.15.29:0/3075118690 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05d40af400 con 0x7f05dc15fd90 2024-04-17T20:00:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.577+0000 7f05da000640 1 -- 172.21.15.29:0/3075118690 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d40af7d0 con 0x7f05dc15fd90 2024-04-17T20:00:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.577+0000 7f05dbe00640 1 -- 172.21.15.29:0/3075118690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05dc15fd90 msgr2=0x7f05dc0cc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.577+0000 7f05dbe00640 1 --2- 172.21.15.29:0/3075118690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05dc15fd90 0x7f05dc0cc430 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f05dc0ccb70 tx=0x7f05d401f1e0 comp rx=0 tx=0).stop 2024-04-17T20:00:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.577+0000 7f05dbe00640 1 -- 172.21.15.29:0/3075118690 shutdown_connections 2024-04-17T20:00:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.577+0000 7f05dbe00640 1 --2- 172.21.15.29:0/3075118690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05dc15fd90 0x7f05dc0cc430 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.577+0000 7f05dbe00640 1 --2- 172.21.15.29:0/3075118690 >> v2:172.21.15.37:3300/0 conn(0x7f05dc156140 0x7f05dc0c9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.577+0000 7f05dbe00640 1 -- 172.21.15.29:0/3075118690 >> 172.21.15.29:0/3075118690 conn(0x7f05dc06e1a0 msgr2=0x7f05dc06e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.577+0000 7f05dbe00640 1 -- 172.21.15.29:0/3075118690 shutdown_connections 2024-04-17T20:00:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.577+0000 7f05dbe00640 1 -- 172.21.15.29:0/3075118690 wait complete. 2024-04-17T20:00:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.577+0000 7f05dbe00640 1 Processor -- start 2024-04-17T20:00:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.577+0000 7f05dbe00640 1 -- start start 2024-04-17T20:00:07.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05dbe00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05dc0cc020 0x7f05dc1f2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:07.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05dbe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05dc156140 0x7f05dc158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:07.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05dbe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05dc15fd90 0x7f05dc0c5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:07.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05dbe00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc0ca3e0 con 0x7f05dc15fd90 2024-04-17T20:00:07.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05dbe00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f05dc0cb900 con 0x7f05dc156140 2024-04-17T20:00:07.580 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05dbe00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f05dc0ca560 con 0x7f05dc0cc020 2024-04-17T20:00:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05db400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05dc15fd90 0x7f05dc0c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05db400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05dc15fd90 0x7f05dc0c5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41178/0 (socket says 172.21.15.29:41178) 2024-04-17T20:00:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05db400640 1 -- 172.21.15.29:0/2745778052 learned_addr learned my addr 172.21.15.29:0/2745778052 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05d2000640 1 --2- 172.21.15.29:0/2745778052 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05dc156140 0x7f05dc158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05daa00640 1 --2- 172.21.15.29:0/2745778052 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05dc0cc020 0x7f05dc1f2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05db400640 1 -- 172.21.15.29:0/2745778052 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05dc0cc020 msgr2=0x7f05dc1f2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05db400640 1 --2- 172.21.15.29:0/2745778052 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05dc0cc020 0x7f05dc1f2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05db400640 1 -- 172.21.15.29:0/2745778052 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05dc156140 msgr2=0x7f05dc158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05db400640 1 --2- 172.21.15.29:0/2745778052 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05dc156140 0x7f05dc158e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05db400640 1 -- 172.21.15.29:0/2745778052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05dc1f2dd0 con 0x7f05dc15fd90 2024-04-17T20:00:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05db400640 1 --2- 172.21.15.29:0/2745778052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05dc15fd90 0x7f05dc0c5520 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f05d40b2040 tx=0x7f05d401bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05d3400640 1 -- 172.21.15.29:0/2745778052 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d40b4040 con 0x7f05dc15fd90 2024-04-17T20:00:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05d3400640 1 -- 172.21.15.29:0/2745778052 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05d40c0070 con 0x7f05dc15fd90 2024-04-17T20:00:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05d3400640 1 -- 172.21.15.29:0/2745778052 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d4009510 con 0x7f05dc15fd90 2024-04-17T20:00:07.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05dbe00640 1 -- 172.21.15.29:0/2745778052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05dc150820 con 0x7f05dc15fd90 2024-04-17T20:00:07.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05dbe00640 1 -- 172.21.15.29:0/2745778052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05dc150de0 con 0x7f05dc15fd90 2024-04-17T20:00:07.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.581+0000 7f05dbe00640 1 -- 172.21.15.29:0/2745778052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05dc1f2620 con 0x7f05dc15fd90 2024-04-17T20:00:07.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.585+0000 7f05d3400640 1 -- 172.21.15.29:0/2745778052 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f05d40c5460 con 0x7f05dc15fd90 2024-04-17T20:00:07.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.585+0000 7f05d3400640 1 --2- 172.21.15.29:0/2745778052 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f05b8068380 0x7f05b806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:07.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.585+0000 7f05d3400640 1 -- 172.21.15.29:0/2745778052 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7f05d41251f0 con 0x7f05dc15fd90 2024-04-17T20:00:07.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.585+0000 7f05daa00640 1 --2- 172.21.15.29:0/2745778052 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f05b8068380 0x7f05b806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:07.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.585+0000 7f05daa00640 1 --2- 172.21.15.29:0/2745778052 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f05b8068380 0x7f05b806a890 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f05cc00b930 tx=0x7f05cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:07.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.589+0000 7f05d3400640 1 -- 172.21.15.29:0/2745778052 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f05dc1f2620 con 0x7f05dc15fd90 2024-04-17T20:00:07.784 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.789+0000 7f05dbe00640 1 -- 172.21.15.29:0/2745778052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05dc04ff10 con 0x7f05dc15fd90 2024-04-17T20:00:07.786 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.789+0000 7f05d3400640 1 -- 172.21.15.29:0/2745778052 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6908 (secure 0 0 0) 0x7f05d4018d40 con 0x7f05dc15fd90 2024-04-17T20:00:07.787 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:07.787 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5775,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:26.051172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5212,5144,5631,4305]},"id":1}]} 2024-04-17T20:00:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 49 2024-04-17T20:00:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.789+0000 7f05dbe00640 1 -- 172.21.15.29:0/2745778052 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f05b8068380 msgr2=0x7f05b806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.789+0000 7f05dbe00640 1 --2- 172.21.15.29:0/2745778052 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f05b8068380 0x7f05b806a890 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f05cc00b930 tx=0x7f05cc008040 comp rx=0 tx=0).stop 2024-04-17T20:00:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.789+0000 7f05dbe00640 1 -- 172.21.15.29:0/2745778052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05dc15fd90 msgr2=0x7f05dc0c5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.789+0000 7f05dbe00640 1 --2- 172.21.15.29:0/2745778052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05dc15fd90 0x7f05dc0c5520 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f05d40b2040 tx=0x7f05d401bf80 comp rx=0 tx=0).stop 2024-04-17T20:00:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.789+0000 7f05dbe00640 1 -- 172.21.15.29:0/2745778052 shutdown_connections 2024-04-17T20:00:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.789+0000 7f05dbe00640 1 --2- 172.21.15.29:0/2745778052 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f05b8068380 0x7f05b806a890 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.789+0000 7f05dbe00640 1 --2- 172.21.15.29:0/2745778052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05dc15fd90 0x7f05dc0c5520 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.789+0000 7f05dbe00640 1 --2- 172.21.15.29:0/2745778052 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f05dc156140 0x7f05dc158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:07.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.789+0000 7f05dbe00640 1 --2- 172.21.15.29:0/2745778052 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f05dc0cc020 0x7f05dc1f2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:07.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.789+0000 7f05dbe00640 1 -- 172.21.15.29:0/2745778052 >> 172.21.15.29:0/2745778052 conn(0x7f05dc06e1a0 msgr2=0x7f05dc070030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:07.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.789+0000 7f05dbe00640 1 -- 172.21.15.29:0/2745778052 shutdown_connections 2024-04-17T20:00:07.789 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:07.789+0000 7f05dbe00640 1 -- 172.21.15.29:0/2745778052 wait complete. 2024-04-17T20:00:07.807 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2024-04-17T20:00:07.807 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T20:00:07.807 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T20:00:07.835 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T20:00:07.835 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:08.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.029+0000 7f5178400640 1 Processor -- start 2024-04-17T20:00:08.027 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.029+0000 7f5178400640 1 -- start start 2024-04-17T20:00:08.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.033+0000 7f5178400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f51700cc8c0 0x7f51700cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:08.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.033+0000 7f5178400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51701597a0 0x7f51701635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:08.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.033+0000 7f5178400640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51700c88b0 con 0x7f5170159390 2024-04-17T20:00:08.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.033+0000 7f5178400640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f51700c6ce0 con 0x7f51701597a0 2024-04-17T20:00:08.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.033+0000 7f5178400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f51700c6fc0 con 0x7f51700cc8c0 2024-04-17T20:00:08.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.033+0000 7f5176400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51701597a0 0x7f51701635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:08.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.033+0000 7f5176400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51701597a0 0x7f51701635e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41190/0 (socket says 172.21.15.29:41190) 2024-04-17T20:00:08.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.033+0000 7f5176400640 1 -- 172.21.15.29:0/2096290610 learned_addr learned my addr 172.21.15.29:0/2096290610 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:08.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.033+0000 7f5175a00640 1 --2- 172.21.15.29:0/2096290610 >> v2:172.21.15.37:3300/0 conn(0x7f51700cc8c0 0x7f51700cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:08.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.033+0000 7f5176400640 1 -- 172.21.15.29:0/2096290610 >> v2:172.21.15.37:3300/0 conn(0x7f51700cc8c0 msgr2=0x7f51700cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:08.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.033+0000 7f5176400640 1 --2- 172.21.15.29:0/2096290610 >> v2:172.21.15.37:3300/0 conn(0x7f51700cc8c0 0x7f51700cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:08.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.033+0000 7f5176400640 1 -- 172.21.15.29:0/2096290610 >> v1:172.21.15.37:6789/0 conn(0x7f5170159390 legacy=0x7f51700ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:08.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.033+0000 7f5176400640 1 -- 172.21.15.29:0/2096290610 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51700c72a0 con 0x7f51701597a0 2024-04-17T20:00:08.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.037+0000 7f5176400640 1 --2- 172.21.15.29:0/2096290610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51701597a0 0x7f51701635e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f51700721b0 tx=0x7f516801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f3a55e840fe34f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:08.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.037+0000 7f516fe00640 1 -- 172.21.15.29:0/2096290610 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51680a56c0 con 0x7f51701597a0 2024-04-17T20:00:08.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.037+0000 7f516fe00640 1 -- 172.21.15.29:0/2096290610 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51680a5860 con 0x7f51701597a0 2024-04-17T20:00:08.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.037+0000 7f516fe00640 1 -- 172.21.15.29:0/2096290610 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51680a5c30 con 0x7f51701597a0 2024-04-17T20:00:08.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.037+0000 7f5178400640 1 -- 172.21.15.29:0/2096290610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51701597a0 msgr2=0x7f51701635e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:08.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.037+0000 7f5178400640 1 --2- 172.21.15.29:0/2096290610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51701597a0 0x7f51701635e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f51700721b0 tx=0x7f516801c9f0 comp rx=0 tx=0).stop 2024-04-17T20:00:08.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.037+0000 7f5178400640 1 -- 172.21.15.29:0/2096290610 shutdown_connections 2024-04-17T20:00:08.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.037+0000 7f5178400640 1 --2- 172.21.15.29:0/2096290610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51701597a0 0x7f51701635e0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:08.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.037+0000 7f5178400640 1 --2- 172.21.15.29:0/2096290610 >> v2:172.21.15.37:3300/0 conn(0x7f51700cc8c0 0x7f51700cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:08.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.037+0000 7f5178400640 1 -- 172.21.15.29:0/2096290610 >> 172.21.15.29:0/2096290610 conn(0x7f517006fde0 msgr2=0x7f51700701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:08.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.037+0000 7f5178400640 1 -- 172.21.15.29:0/2096290610 shutdown_connections 2024-04-17T20:00:08.034 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.037+0000 7f5178400640 1 -- 172.21.15.29:0/2096290610 wait complete. 2024-04-17T20:00:08.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.037+0000 7f5178400640 1 Processor -- start 2024-04-17T20:00:08.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.037+0000 7f5178400640 1 -- start start 2024-04-17T20:00:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5178400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f51700cc8c0 0x7f51700730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5178400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5170159390 0x7f51700735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5178400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51701597a0 0x7f517015c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5178400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f51700cabc0 con 0x7f5170159390 2024-04-17T20:00:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5178400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f51700caa40 con 0x7f51701597a0 2024-04-17T20:00:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5178400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f51700c6880 con 0x7f51700cc8c0 2024-04-17T20:00:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5175000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5170159390 0x7f51700735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5175000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5170159390 0x7f51700735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41192/0 (socket says 172.21.15.29:41192) 2024-04-17T20:00:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5175000640 1 -- 172.21.15.29:0/79964914 learned_addr learned my addr 172.21.15.29:0/79964914 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5175a00640 1 --2- 172.21.15.29:0/79964914 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f51700cc8c0 0x7f51700730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5176400640 1 --2- 172.21.15.29:0/79964914 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51701597a0 0x7f517015c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5175000640 1 -- 172.21.15.29:0/79964914 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f51700cc8c0 msgr2=0x7f51700730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:08.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5175000640 1 --2- 172.21.15.29:0/79964914 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f51700cc8c0 0x7f51700730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:08.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5175000640 1 -- 172.21.15.29:0/79964914 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51701597a0 msgr2=0x7f517015c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:08.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5175000640 1 --2- 172.21.15.29:0/79964914 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51701597a0 0x7f517015c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:08.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5175000640 1 -- 172.21.15.29:0/79964914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51701f8640 con 0x7f5170159390 2024-04-17T20:00:08.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5175a00640 1 --2- 172.21.15.29:0/79964914 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f51700cc8c0 0x7f51700730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:00:08.039 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5175000640 1 --2- 172.21.15.29:0/79964914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5170159390 0x7f51700735f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f515c00c340 tx=0x7f515c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:08.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f516d600640 1 -- 172.21.15.29:0/79964914 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515c0052d0 con 0x7f5170159390 2024-04-17T20:00:08.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f516d600640 1 -- 172.21.15.29:0/79964914 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f515c005470 con 0x7f5170159390 2024-04-17T20:00:08.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5178400640 1 -- 172.21.15.29:0/79964914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51701f8920 con 0x7f5170159390 2024-04-17T20:00:08.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f516d600640 1 -- 172.21.15.29:0/79964914 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515c007260 con 0x7f5170159390 2024-04-17T20:00:08.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5178400640 1 -- 172.21.15.29:0/79964914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51701f8ee0 con 0x7f5170159390 2024-04-17T20:00:08.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.041+0000 7f5178400640 1 -- 172.21.15.29:0/79964914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51701f5c20 con 0x7f5170159390 2024-04-17T20:00:08.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.045+0000 7f516d600640 1 -- 172.21.15.29:0/79964914 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f515c007400 con 0x7f5170159390 2024-04-17T20:00:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.045+0000 7f516d600640 1 --2- 172.21.15.29:0/79964914 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f51500683d0 0x7f515006a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.045+0000 7f516d600640 1 -- 172.21.15.29:0/79964914 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7f515c0863c0 con 0x7f5170159390 2024-04-17T20:00:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.045+0000 7f5175a00640 1 --2- 172.21.15.29:0/79964914 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f51500683d0 0x7f515006a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:08.042 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.045+0000 7f5175a00640 1 --2- 172.21.15.29:0/79964914 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f51500683d0 0x7f515006a8e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5164009b10 tx=0x7f5164008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:08.046 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.049+0000 7f516d600640 1 -- 172.21.15.29:0/79964914 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f515c0563b0 con 0x7f5170159390 2024-04-17T20:00:08.250 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.253+0000 7f5178400640 1 -- 172.21.15.29:0/79964914 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5170069b90 con 0x7f5170159390 2024-04-17T20:00:08.251 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:08.251 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5775,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:26.051172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5212,5144,5631,4305]},"id":1}]} 2024-04-17T20:00:08.251 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 49 2024-04-17T20:00:08.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.253+0000 7f516d600640 1 -- 172.21.15.29:0/79964914 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6908 (secure 0 0 0) 0x7f515c055cb0 con 0x7f5170159390 2024-04-17T20:00:08.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.257+0000 7f5178400640 1 -- 172.21.15.29:0/79964914 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f51500683d0 msgr2=0x7f515006a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:08.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.257+0000 7f5178400640 1 --2- 172.21.15.29:0/79964914 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f51500683d0 0x7f515006a8e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5164009b10 tx=0x7f5164008040 comp rx=0 tx=0).stop 2024-04-17T20:00:08.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.257+0000 7f5178400640 1 -- 172.21.15.29:0/79964914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5170159390 msgr2=0x7f51700735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:08.921 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.257+0000 7f5178400640 1 --2- 172.21.15.29:0/79964914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5170159390 0x7f51700735f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f515c00c340 tx=0x7f515c00c800 comp rx=0 tx=0).stop 2024-04-17T20:00:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.257+0000 7f5178400640 1 -- 172.21.15.29:0/79964914 shutdown_connections 2024-04-17T20:00:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.257+0000 7f5178400640 1 --2- 172.21.15.29:0/79964914 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f51500683d0 0x7f515006a8e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.257+0000 7f5178400640 1 --2- 172.21.15.29:0/79964914 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f51701597a0 0x7f517015c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.257+0000 7f5178400640 1 --2- 172.21.15.29:0/79964914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5170159390 0x7f51700735f0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.257+0000 7f5178400640 1 --2- 172.21.15.29:0/79964914 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f51700cc8c0 0x7f51700730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.257+0000 7f5178400640 1 -- 172.21.15.29:0/79964914 >> 172.21.15.29:0/79964914 conn(0x7f517006fde0 msgr2=0x7f51700cd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.257+0000 7f5178400640 1 -- 172.21.15.29:0/79964914 shutdown_connections 2024-04-17T20:00:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:08.257+0000 7f5178400640 1 -- 172.21.15.29:0/79964914 wait complete. 2024-04-17T20:00:08.946 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-17T20:00:10.946 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:11.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2560000640 1 Processor -- start 2024-04-17T20:00:11.113 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2560000640 1 -- start start 2024-04-17T20:00:11.114 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2560000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25581625e0 0x7f2558164a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:11.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2560000640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25580cd0e0 con 0x7f25580cccd0 2024-04-17T20:00:11.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2560000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25580c88b0 con 0x7f25580cc8c0 2024-04-17T20:00:11.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2560000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f25580c6ce0 con 0x7f25581625e0 2024-04-17T20:00:11.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f255cc00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f25580cccd0 0x7f25580ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54766/0 (socket says 172.21.15.29:54766) 2024-04-17T20:00:11.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f255cc00640 1 -- 172.21.15.29:0/1249490476 learned_addr learned my addr 172.21.15.29:0/1249490476 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:00:11.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f255e000640 1 --2- 172.21.15.29:0/1249490476 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25581625e0 0x7f2558164a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:11.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2557e00640 1 -- 172.21.15.29:0/1249490476 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4074436807 0 0) 0x7f25580cd0e0 con 0x7f25580cccd0 2024-04-17T20:00:11.115 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2557e00640 1 -- 172.21.15.29:0/1249490476 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2540003530 con 0x7f25580cccd0 2024-04-17T20:00:11.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2557e00640 1 -- 172.21.15.29:0/1249490476 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2197981184 0 0) 0x7f25580c88b0 con 0x7f25580cc8c0 2024-04-17T20:00:11.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2557e00640 1 -- 172.21.15.29:0/1249490476 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25580cd0e0 con 0x7f25580cc8c0 2024-04-17T20:00:11.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2557e00640 1 -- 172.21.15.29:0/1249490476 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3712085118 0 0) 0x7f2540003530 con 0x7f25580cccd0 2024-04-17T20:00:11.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2557e00640 1 -- 172.21.15.29:0/1249490476 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25581625e0 msgr2=0x7f2558164a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:11.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2557e00640 1 --2- 172.21.15.29:0/1249490476 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25581625e0 0x7f2558164a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:11.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2557e00640 1 -- 172.21.15.29:0/1249490476 >> v1:172.21.15.37:6789/0 conn(0x7f25580cc8c0 legacy=0x7f2558159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:11.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2557e00640 1 -- 172.21.15.29:0/1249490476 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25580c6fc0 con 0x7f25580cccd0 2024-04-17T20:00:11.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2557e00640 1 -- 172.21.15.29:0/1249490476 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f2544001680 con 0x7f25580cccd0 2024-04-17T20:00:11.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2557e00640 1 -- 172.21.15.29:0/1249490476 >> v1:172.21.15.29:6789/0 conn(0x7f25580cccd0 legacy=0x7f25580ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:11.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2557e00640 1 --2- 172.21.15.29:0/1249490476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2540003bf0 0x7f2540004040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:11.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2557e00640 1 --2- 172.21.15.29:0/1249490476 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25581625e0 0x7f2540004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:11.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2557e00640 1 --2- 172.21.15.29:0/1249490476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2540004d50 0x7f25400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:11.117 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2557e00640 1 -- 172.21.15.29:0/1249490476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f25580c6ce0 con 0x7f2540004d50 2024-04-17T20:00:11.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2557e00640 1 -- 172.21.15.29:0/1249490476 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f254000bb60 con 0x7f2540003bf0 2024-04-17T20:00:11.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f2557e00640 1 -- 172.21.15.29:0/1249490476 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f254000bd60 con 0x7f25581625e0 2024-04-17T20:00:11.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f255e000640 1 --2- 172.21.15.29:0/1249490476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2540004d50 0x7f25400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:11.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f255cc00640 1 --2- 172.21.15.29:0/1249490476 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25581625e0 0x7f2540004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:11.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.117+0000 7f255d600640 1 --2- 172.21.15.29:0/1249490476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2540003bf0 0x7f2540004040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:11.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.121+0000 7f255e000640 1 -- 172.21.15.29:0/1249490476 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25581625e0 msgr2=0x7f2540004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:11.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.121+0000 7f255e000640 1 --2- 172.21.15.29:0/1249490476 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f25581625e0 0x7f2540004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:11.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.121+0000 7f255e000640 1 -- 172.21.15.29:0/1249490476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2540003bf0 msgr2=0x7f2540004040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:11.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.121+0000 7f255e000640 1 --2- 172.21.15.29:0/1249490476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2540003bf0 0x7f2540004040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:11.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.121+0000 7f255e000640 1 -- 172.21.15.29:0/1249490476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2540003530 con 0x7f2540004d50 2024-04-17T20:00:11.118 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.121+0000 7f255d600640 1 --2- 172.21.15.29:0/1249490476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2540003bf0 0x7f2540004040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:00:11.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.121+0000 7f255e000640 1 --2- 172.21.15.29:0/1249490476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2540004d50 0x7f25400051c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f2550009f90 tx=0x7f255001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e8c05c3db6e4dd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:11.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.121+0000 7f2557e00640 1 -- 172.21.15.29:0/1249490476 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25500b2070 con 0x7f2540004d50 2024-04-17T20:00:11.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.121+0000 7f2557e00640 1 -- 172.21.15.29:0/1249490476 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25500a5b60 con 0x7f2540004d50 2024-04-17T20:00:11.119 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.121+0000 7f2557e00640 1 -- 172.21.15.29:0/1249490476 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2550009c10 con 0x7f2540004d50 2024-04-17T20:00:11.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.121+0000 7f2560000640 1 -- 172.21.15.29:0/1249490476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2540004d50 msgr2=0x7f25400051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:11.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.121+0000 7f2560000640 1 --2- 172.21.15.29:0/1249490476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2540004d50 0x7f25400051c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f2550009f90 tx=0x7f255001cc60 comp rx=0 tx=0).stop 2024-04-17T20:00:11.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.121+0000 7f255e000640 1 -- 172.21.15.29:0/1249490476 reap_dead start 2024-04-17T20:00:11.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.125+0000 7f2560000640 1 -- 172.21.15.29:0/1249490476 shutdown_connections 2024-04-17T20:00:11.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.125+0000 7f2560000640 1 -- 172.21.15.29:0/1249490476 >> 172.21.15.29:0/1249490476 conn(0x7f255806fde0 msgr2=0x7f25580701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:11.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.125+0000 7f2560000640 1 -- 172.21.15.29:0/1249490476 shutdown_connections 2024-04-17T20:00:11.120 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.125+0000 7f2560000640 1 -- 172.21.15.29:0/1249490476 wait complete. 2024-04-17T20:00:11.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.125+0000 7f2560000640 1 Processor -- start 2024-04-17T20:00:11.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.125+0000 7f2560000640 1 -- start start 2024-04-17T20:00:11.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.125+0000 7f2560000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2540004d50 0x7f25581f2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:11.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.125+0000 7f2560000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255815b7f0 0x7f255813cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:11.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.125+0000 7f2560000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25580ca280 0x7f25580ca690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:11.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.125+0000 7f2560000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2558158e70 con 0x7f25580ca280 2024-04-17T20:00:11.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.125+0000 7f2560000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f25580c7580 con 0x7f2540004d50 2024-04-17T20:00:11.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.125+0000 7f2560000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2558159aa0 con 0x7f255815b7f0 2024-04-17T20:00:11.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f255e000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25580ca280 0x7f25580ca690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:11.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f255e000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25580ca280 0x7f25580ca690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46756/0 (socket says 172.21.15.29:46756) 2024-04-17T20:00:11.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f255e000640 1 -- 172.21.15.29:0/828814631 learned_addr learned my addr 172.21.15.29:0/828814631 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:11.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f255d600640 1 --2- 172.21.15.29:0/828814631 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2540004d50 0x7f25581f2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:11.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f255cc00640 1 --2- 172.21.15.29:0/828814631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255815b7f0 0x7f255813cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:11.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f255e000640 1 -- 172.21.15.29:0/828814631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255815b7f0 msgr2=0x7f255813cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:11.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f255e000640 1 --2- 172.21.15.29:0/828814631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255815b7f0 0x7f255813cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:11.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f255e000640 1 -- 172.21.15.29:0/828814631 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2540004d50 msgr2=0x7f25581f2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:11.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f255e000640 1 --2- 172.21.15.29:0/828814631 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2540004d50 0x7f25581f2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:11.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f255e000640 1 -- 172.21.15.29:0/828814631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25581f2fd0 con 0x7f25580ca280 2024-04-17T20:00:11.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f255cc00640 1 --2- 172.21.15.29:0/828814631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255815b7f0 0x7f255813cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:00:11.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f255e000640 1 --2- 172.21.15.29:0/828814631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25580ca280 0x7f25580ca690 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f255001ca90 tx=0x7f2550009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:11.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f2555600640 1 -- 172.21.15.29:0/828814631 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25500b2040 con 0x7f25580ca280 2024-04-17T20:00:11.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f2555600640 1 -- 172.21.15.29:0/828814631 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25500bc070 con 0x7f25580ca280 2024-04-17T20:00:11.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f2555600640 1 -- 172.21.15.29:0/828814631 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2550009c10 con 0x7f25580ca280 2024-04-17T20:00:11.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f2560000640 1 -- 172.21.15.29:0/828814631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f255815c6a0 con 0x7f25580ca280 2024-04-17T20:00:11.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f2560000640 1 -- 172.21.15.29:0/828814631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f255815cc60 con 0x7f25580ca280 2024-04-17T20:00:11.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f2560000640 1 -- 172.21.15.29:0/828814631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2558069b90 con 0x7f25580ca280 2024-04-17T20:00:11.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.129+0000 7f2555600640 1 -- 172.21.15.29:0/828814631 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f25500a5d10 con 0x7f25580ca280 2024-04-17T20:00:11.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.133+0000 7f2555600640 1 --2- 172.21.15.29:0/828814631 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2538068380 0x7f253806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:11.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.133+0000 7f2555600640 1 -- 172.21.15.29:0/828814631 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7f2550121750 con 0x7f25580ca280 2024-04-17T20:00:11.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.133+0000 7f255d600640 1 --2- 172.21.15.29:0/828814631 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2538068380 0x7f253806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:11.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.133+0000 7f255d600640 1 --2- 172.21.15.29:0/828814631 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2538068380 0x7f253806a890 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f254c000c10 tx=0x7f254c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:11.134 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.137+0000 7f2555600640 1 -- 172.21.15.29:0/828814631 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f25581f2fd0 con 0x7f25580ca280 2024-04-17T20:00:11.335 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.337+0000 7f2560000640 1 -- 172.21.15.29:0/828814631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25580c4420 con 0x7f25580ca280 2024-04-17T20:00:11.336 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.341+0000 7f2555600640 1 -- 172.21.15.29:0/828814631 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6908 (secure 0 0 0) 0x7f2550016970 con 0x7f25580ca280 2024-04-17T20:00:11.336 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:11.336 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5775,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:26.051172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5212,5144,5631,4305]},"id":1}]} 2024-04-17T20:00:11.337 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 49 2024-04-17T20:00:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.341+0000 7f2560000640 1 -- 172.21.15.29:0/828814631 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2538068380 msgr2=0x7f253806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.341+0000 7f2560000640 1 --2- 172.21.15.29:0/828814631 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2538068380 0x7f253806a890 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f254c000c10 tx=0x7f254c009040 comp rx=0 tx=0).stop 2024-04-17T20:00:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.341+0000 7f2560000640 1 -- 172.21.15.29:0/828814631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25580ca280 msgr2=0x7f25580ca690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.341+0000 7f2560000640 1 --2- 172.21.15.29:0/828814631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25580ca280 0x7f25580ca690 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f255001ca90 tx=0x7f2550009750 comp rx=0 tx=0).stop 2024-04-17T20:00:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.341+0000 7f2560000640 1 -- 172.21.15.29:0/828814631 shutdown_connections 2024-04-17T20:00:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.341+0000 7f2560000640 1 --2- 172.21.15.29:0/828814631 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2538068380 0x7f253806a890 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.341+0000 7f2560000640 1 --2- 172.21.15.29:0/828814631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25580ca280 0x7f25580ca690 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.341+0000 7f2560000640 1 --2- 172.21.15.29:0/828814631 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f255815b7f0 0x7f255813cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.341+0000 7f2560000640 1 --2- 172.21.15.29:0/828814631 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2540004d50 0x7f25581f2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.341+0000 7f2560000640 1 -- 172.21.15.29:0/828814631 >> 172.21.15.29:0/828814631 conn(0x7f255806fde0 msgr2=0x7f25580c8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.341+0000 7f2560000640 1 -- 172.21.15.29:0/828814631 shutdown_connections 2024-04-17T20:00:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:11.341+0000 7f2560000640 1 -- 172.21.15.29:0/828814631 wait complete. 2024-04-17T20:00:11.359 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-17T20:00:13.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:13.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.529+0000 7f10f9800640 1 Processor -- start 2024-04-17T20:00:13.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.529+0000 7f10f9800640 1 -- start start 2024-04-17T20:00:13.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.529+0000 7f10f9800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f10f40cabe0 0x7f10f40caff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.529+0000 7f10f9800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f10f415c260 0x7f10f415c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.529+0000 7f10f9800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10f406c730 con 0x7f10f4156370 2024-04-17T20:00:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.529+0000 7f10f9800640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f10f406c9f0 con 0x7f10f40cabe0 2024-04-17T20:00:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.529+0000 7f10f9800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f10f406ccd0 con 0x7f10f415c260 2024-04-17T20:00:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.529+0000 7f10ebe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f10f40cabe0 0x7f10f40caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.529+0000 7f10ebe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f10f40cabe0 0x7f10f40caff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46768/0 (socket says 172.21.15.29:46768) 2024-04-17T20:00:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.529+0000 7f10ebe00640 1 -- 172.21.15.29:0/1768571356 learned_addr learned my addr 172.21.15.29:0/1768571356 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:13.527 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.529+0000 7f10f8e00640 1 --2- 172.21.15.29:0/1768571356 >> v2:172.21.15.37:3300/0 conn(0x7f10f415c260 0x7f10f415c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.529+0000 7f10ebe00640 1 -- 172.21.15.29:0/1768571356 >> v2:172.21.15.37:3300/0 conn(0x7f10f415c260 msgr2=0x7f10f415c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.529+0000 7f10ebe00640 1 --2- 172.21.15.29:0/1768571356 >> v2:172.21.15.37:3300/0 conn(0x7f10f415c260 0x7f10f415c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.529+0000 7f10ebe00640 1 -- 172.21.15.29:0/1768571356 >> v1:172.21.15.37:6789/0 conn(0x7f10f4156370 legacy=0x7f10f40ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:13.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.529+0000 7f10ebe00640 1 -- 172.21.15.29:0/1768571356 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f406cfb0 con 0x7f10f40cabe0 2024-04-17T20:00:13.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.533+0000 7f10ebe00640 1 --2- 172.21.15.29:0/1768571356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f40cabe0 0x7f10f40caff0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f10f40714c0 tx=0x7f10dc01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aff38b987ce0186 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:13.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.533+0000 7f10f3400640 1 -- 172.21.15.29:0/1768571356 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc0b5070 con 0x7f10f40cabe0 2024-04-17T20:00:13.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.533+0000 7f10f3400640 1 -- 172.21.15.29:0/1768571356 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10dc0a8a10 con 0x7f10f40cabe0 2024-04-17T20:00:13.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.533+0000 7f10f3400640 1 -- 172.21.15.29:0/1768571356 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc0a8de0 con 0x7f10f40cabe0 2024-04-17T20:00:13.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.533+0000 7f10f9800640 1 -- 172.21.15.29:0/1768571356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f40cabe0 msgr2=0x7f10f40caff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:13.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.533+0000 7f10f9800640 1 --2- 172.21.15.29:0/1768571356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f40cabe0 0x7f10f40caff0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f10f40714c0 tx=0x7f10dc01ed10 comp rx=0 tx=0).stop 2024-04-17T20:00:13.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.533+0000 7f10f9800640 1 -- 172.21.15.29:0/1768571356 shutdown_connections 2024-04-17T20:00:13.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.533+0000 7f10f9800640 1 --2- 172.21.15.29:0/1768571356 >> v2:172.21.15.37:3300/0 conn(0x7f10f415c260 0x7f10f415c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:13.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.533+0000 7f10f9800640 1 --2- 172.21.15.29:0/1768571356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f40cabe0 0x7f10f40caff0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:13.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.533+0000 7f10f9800640 1 -- 172.21.15.29:0/1768571356 >> 172.21.15.29:0/1768571356 conn(0x7f10f40680e0 msgr2=0x7f10f406dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:13.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.533+0000 7f10f9800640 1 -- 172.21.15.29:0/1768571356 shutdown_connections 2024-04-17T20:00:13.531 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.533+0000 7f10f9800640 1 -- 172.21.15.29:0/1768571356 wait complete. 2024-04-17T20:00:13.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10f9800640 1 Processor -- start 2024-04-17T20:00:13.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10f9800640 1 -- start start 2024-04-17T20:00:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10f9800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f40cabe0 0x7f10f41f3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10f9800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10f4156370 0x7f10f4150fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10f9800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10f415c260 0x7f10f414fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10f9800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f10f415cbb0 con 0x7f10f40cabe0 2024-04-17T20:00:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10f9800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f10f406d570 con 0x7f10f4156370 2024-04-17T20:00:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10f9800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f10f406e7c0 con 0x7f10f415c260 2024-04-17T20:00:13.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10f3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f40cabe0 0x7f10f41f3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10f3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f40cabe0 0x7f10f41f3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46784/0 (socket says 172.21.15.29:46784) 2024-04-17T20:00:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10f3e00640 1 -- 172.21.15.29:0/4216859607 learned_addr learned my addr 172.21.15.29:0/4216859607 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10ebe00640 1 --2- 172.21.15.29:0/4216859607 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10f4156370 0x7f10f4150fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10f8e00640 1 --2- 172.21.15.29:0/4216859607 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10f415c260 0x7f10f414fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10f3e00640 1 -- 172.21.15.29:0/4216859607 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10f415c260 msgr2=0x7f10f414fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10f3e00640 1 --2- 172.21.15.29:0/4216859607 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10f415c260 0x7f10f414fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10f3e00640 1 -- 172.21.15.29:0/4216859607 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10f4156370 msgr2=0x7f10f4150fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10f3e00640 1 --2- 172.21.15.29:0/4216859607 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10f4156370 0x7f10f4150fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10f3e00640 1 -- 172.21.15.29:0/4216859607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10f41f35a0 con 0x7f10f40cabe0 2024-04-17T20:00:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10ebe00640 1 --2- 172.21.15.29:0/4216859607 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10f4156370 0x7f10f4150fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:00:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.537+0000 7f10f3e00640 1 --2- 172.21.15.29:0/4216859607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f40cabe0 0x7f10f41f3060 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f10e400c370 tx=0x7f10e400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:13.542 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.541+0000 7f10f0c00640 1 -- 172.21.15.29:0/4216859607 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e40052d0 con 0x7f10f40cabe0 2024-04-17T20:00:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.541+0000 7f10f0c00640 1 -- 172.21.15.29:0/4216859607 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10e4005470 con 0x7f10f40cabe0 2024-04-17T20:00:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.541+0000 7f10f9800640 1 -- 172.21.15.29:0/4216859607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10f40ce980 con 0x7f10f40cabe0 2024-04-17T20:00:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.541+0000 7f10f0c00640 1 -- 172.21.15.29:0/4216859607 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e4007260 con 0x7f10f40cabe0 2024-04-17T20:00:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.541+0000 7f10f9800640 1 -- 172.21.15.29:0/4216859607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10f40cef40 con 0x7f10f40cabe0 2024-04-17T20:00:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.541+0000 7f10f9800640 1 -- 172.21.15.29:0/4216859607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10f404ff10 con 0x7f10f40cabe0 2024-04-17T20:00:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.541+0000 7f10f0c00640 1 -- 172.21.15.29:0/4216859607 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f10e4007400 con 0x7f10f40cabe0 2024-04-17T20:00:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.541+0000 7f10f0c00640 1 --2- 172.21.15.29:0/4216859607 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f10d0068380 0x7f10d006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.541+0000 7f10f0c00640 1 -- 172.21.15.29:0/4216859607 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7f10e4086410 con 0x7f10f40cabe0 2024-04-17T20:00:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.545+0000 7f10ebe00640 1 --2- 172.21.15.29:0/4216859607 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f10d0068380 0x7f10d006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.545+0000 7f10ebe00640 1 --2- 172.21.15.29:0/4216859607 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f10d0068380 0x7f10d006a890 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f10dc0b2000 tx=0x7f10dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:13.548 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.549+0000 7f10f0c00640 1 -- 172.21.15.29:0/4216859607 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f10e4056410 con 0x7f10f40cabe0 2024-04-17T20:00:13.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.749+0000 7f10f9800640 1 -- 172.21.15.29:0/4216859607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10f40c9e50 con 0x7f10f40cabe0 2024-04-17T20:00:13.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.749+0000 7f10f0c00640 1 -- 172.21.15.29:0/4216859607 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6908 (secure 0 0 0) 0x7f10e4055d10 con 0x7f10f40cabe0 2024-04-17T20:00:13.746 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:13.746 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5775,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:26.051172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5212,5144,5631,4305]},"id":1}]} 2024-04-17T20:00:13.747 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 49 2024-04-17T20:00:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.749+0000 7f10f9800640 1 -- 172.21.15.29:0/4216859607 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f10d0068380 msgr2=0x7f10d006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.749+0000 7f10f9800640 1 --2- 172.21.15.29:0/4216859607 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f10d0068380 0x7f10d006a890 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f10dc0b2000 tx=0x7f10dc008040 comp rx=0 tx=0).stop 2024-04-17T20:00:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.749+0000 7f10f9800640 1 -- 172.21.15.29:0/4216859607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f40cabe0 msgr2=0x7f10f41f3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.749+0000 7f10f9800640 1 --2- 172.21.15.29:0/4216859607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f40cabe0 0x7f10f41f3060 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f10e400c370 tx=0x7f10e400c830 comp rx=0 tx=0).stop 2024-04-17T20:00:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.749+0000 7f10f9800640 1 -- 172.21.15.29:0/4216859607 shutdown_connections 2024-04-17T20:00:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.749+0000 7f10f9800640 1 --2- 172.21.15.29:0/4216859607 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f10d0068380 0x7f10d006a890 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.749+0000 7f10f9800640 1 --2- 172.21.15.29:0/4216859607 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f10f415c260 0x7f10f414fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.749+0000 7f10f9800640 1 --2- 172.21.15.29:0/4216859607 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f10f4156370 0x7f10f4150fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.749+0000 7f10f9800640 1 --2- 172.21.15.29:0/4216859607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10f40cabe0 0x7f10f41f3060 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.749+0000 7f10f9800640 1 -- 172.21.15.29:0/4216859607 >> 172.21.15.29:0/4216859607 conn(0x7f10f40680e0 msgr2=0x7f10f40c4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.749+0000 7f10f9800640 1 -- 172.21.15.29:0/4216859607 shutdown_connections 2024-04-17T20:00:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:13.749+0000 7f10f9800640 1 -- 172.21.15.29:0/4216859607 wait complete. 2024-04-17T20:00:13.769 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-17T20:00:15.770 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:15.934 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.937+0000 7fa872200640 1 Processor -- start 2024-04-17T20:00:15.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.937+0000 7fa872200640 1 -- start start 2024-04-17T20:00:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.941+0000 7fa872200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa86c0cc8c0 0x7fa86c0cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.941+0000 7fa872200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa86c1597a0 0x7fa86c1635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.941+0000 7fa872200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa86c0c88b0 con 0x7fa86c159390 2024-04-17T20:00:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.941+0000 7fa872200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa86c0c6ce0 con 0x7fa86c1597a0 2024-04-17T20:00:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.941+0000 7fa872200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa86c0c6fc0 con 0x7fa86c0cc8c0 2024-04-17T20:00:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.941+0000 7fa86be00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa86c1597a0 0x7fa86c1635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.941+0000 7fa862a00640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa86c159390 0x7fa86c0ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:52796/0 (socket says 172.21.15.29:52796) 2024-04-17T20:00:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.941+0000 7fa86b400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa86c0cc8c0 0x7fa86c0cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.941+0000 7fa862a00640 1 -- 172.21.15.29:0/1561153208 learned_addr learned my addr 172.21.15.29:0/1561153208 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:00:15.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.941+0000 7fa86be00640 1 -- 172.21.15.29:0/1561153208 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa86c0cc8c0 msgr2=0x7fa86c0cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:15.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.941+0000 7fa86be00640 1 --2- 172.21.15.29:0/1561153208 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa86c0cc8c0 0x7fa86c0cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:15.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.941+0000 7fa86be00640 1 -- 172.21.15.29:0/1561153208 >> v1:172.21.15.37:6789/0 conn(0x7fa86c159390 legacy=0x7fa86c0ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:15.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.941+0000 7fa86be00640 1 -- 172.21.15.29:0/1561153208 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86c0c72a0 con 0x7fa86c1597a0 2024-04-17T20:00:15.940 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.945+0000 7fa86be00640 1 --2- 172.21.15.29:0/1561153208 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa86c1597a0 0x7fa86c1635e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa864009d90 tx=0x7fa86401c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5732943c967b2f07 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:15.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.945+0000 7fa86aa00640 1 -- 172.21.15.29:0/1561153208 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8640b2070 con 0x7fa86c1597a0 2024-04-17T20:00:15.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.945+0000 7fa86aa00640 1 -- 172.21.15.29:0/1561153208 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8640a5830 con 0x7fa86c1597a0 2024-04-17T20:00:15.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.945+0000 7fa86aa00640 1 -- 172.21.15.29:0/1561153208 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8640a5c00 con 0x7fa86c1597a0 2024-04-17T20:00:15.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.945+0000 7fa872200640 1 -- 172.21.15.29:0/1561153208 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa86c1597a0 msgr2=0x7fa86c1635e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:15.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.945+0000 7fa872200640 1 --2- 172.21.15.29:0/1561153208 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa86c1597a0 0x7fa86c1635e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa864009d90 tx=0x7fa86401c970 comp rx=0 tx=0).stop 2024-04-17T20:00:15.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.945+0000 7fa872200640 1 -- 172.21.15.29:0/1561153208 shutdown_connections 2024-04-17T20:00:15.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.945+0000 7fa872200640 1 --2- 172.21.15.29:0/1561153208 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa86c1597a0 0x7fa86c1635e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:15.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.945+0000 7fa872200640 1 --2- 172.21.15.29:0/1561153208 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa86c0cc8c0 0x7fa86c0cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:15.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.945+0000 7fa872200640 1 -- 172.21.15.29:0/1561153208 >> 172.21.15.29:0/1561153208 conn(0x7fa86c06fde0 msgr2=0x7fa86c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:15.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.945+0000 7fa872200640 1 -- 172.21.15.29:0/1561153208 shutdown_connections 2024-04-17T20:00:15.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.945+0000 7fa872200640 1 -- 172.21.15.29:0/1561153208 wait complete. 2024-04-17T20:00:15.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa872200640 1 Processor -- start 2024-04-17T20:00:15.944 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa872200640 1 -- start start 2024-04-17T20:00:15.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa872200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c0cc8c0 0x7fa86c156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:15.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa872200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa86c159390 0x7fa86c135a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:15.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa872200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa86c1597a0 0x7fa86c135f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:15.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa872200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa86c0cabc0 con 0x7fa86c0cc8c0 2024-04-17T20:00:15.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa872200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa86c0caa40 con 0x7fa86c159390 2024-04-17T20:00:15.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa872200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa86c0c6880 con 0x7fa86c1597a0 2024-04-17T20:00:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa86b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c0cc8c0 0x7fa86c156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa86b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c0cc8c0 0x7fa86c156da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46792/0 (socket says 172.21.15.29:46792) 2024-04-17T20:00:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa86b400640 1 -- 172.21.15.29:0/318203857 learned_addr learned my addr 172.21.15.29:0/318203857 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa86be00640 1 --2- 172.21.15.29:0/318203857 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa86c1597a0 0x7fa86c135f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa862a00640 1 --2- 172.21.15.29:0/318203857 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa86c159390 0x7fa86c135a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa86b400640 1 -- 172.21.15.29:0/318203857 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa86c1597a0 msgr2=0x7fa86c135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa86b400640 1 --2- 172.21.15.29:0/318203857 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa86c1597a0 0x7fa86c135f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa86b400640 1 -- 172.21.15.29:0/318203857 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa86c159390 msgr2=0x7fa86c135a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa86b400640 1 --2- 172.21.15.29:0/318203857 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa86c159390 0x7fa86c135a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa86b400640 1 -- 172.21.15.29:0/318203857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86c1572e0 con 0x7fa86c0cc8c0 2024-04-17T20:00:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa862a00640 1 --2- 172.21.15.29:0/318203857 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa86c159390 0x7fa86c135a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:00:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa86b400640 1 --2- 172.21.15.29:0/318203857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c0cc8c0 0x7fa86c156da0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa85c00e8a0 tx=0x7fa85c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:15.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa863e00640 1 -- 172.21.15.29:0/318203857 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c017070 con 0x7fa86c0cc8c0 2024-04-17T20:00:15.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa863e00640 1 -- 172.21.15.29:0/318203857 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa85c00c8e0 con 0x7fa86c0cc8c0 2024-04-17T20:00:15.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa863e00640 1 -- 172.21.15.29:0/318203857 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c00cc90 con 0x7fa86c0cc8c0 2024-04-17T20:00:15.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa872200640 1 -- 172.21.15.29:0/318203857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa86c137300 con 0x7fa86c0cc8c0 2024-04-17T20:00:15.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.949+0000 7fa872200640 1 -- 172.21.15.29:0/318203857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa86c1375c0 con 0x7fa86c0cc8c0 2024-04-17T20:00:15.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.953+0000 7fa872200640 1 -- 172.21.15.29:0/318203857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa86c069b90 con 0x7fa86c0cc8c0 2024-04-17T20:00:15.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.953+0000 7fa863e00640 1 -- 172.21.15.29:0/318203857 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa85c01a070 con 0x7fa86c0cc8c0 2024-04-17T20:00:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.953+0000 7fa863e00640 1 --2- 172.21.15.29:0/318203857 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa848068380 0x7fa84806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:15.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.953+0000 7fa863e00640 1 -- 172.21.15.29:0/318203857 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7fa85c086e00 con 0x7fa86c0cc8c0 2024-04-17T20:00:15.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.953+0000 7fa862a00640 1 --2- 172.21.15.29:0/318203857 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa848068380 0x7fa84806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:15.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.953+0000 7fa862a00640 1 --2- 172.21.15.29:0/318203857 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa848068380 0x7fa84806a890 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa86c1f9aa0 tx=0x7fa854009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:15.956 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:15.957+0000 7fa863e00640 1 -- 172.21.15.29:0/318203857 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa85c056720 con 0x7fa86c0cc8c0 2024-04-17T20:00:16.155 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:16.157+0000 7fa872200640 1 -- 172.21.15.29:0/318203857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa86c0c5b30 con 0x7fa86c0cc8c0 2024-04-17T20:00:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:16.161+0000 7fa863e00640 1 -- 172.21.15.29:0/318203857 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6908 (secure 0 0 0) 0x7fa85c05aa20 con 0x7fa86c0cc8c0 2024-04-17T20:00:16.156 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:16.156 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5775,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:26.051172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5212,5144,5631,4305]},"id":1}]} 2024-04-17T20:00:16.157 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 49 2024-04-17T20:00:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:16.161+0000 7fa872200640 1 -- 172.21.15.29:0/318203857 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa848068380 msgr2=0x7fa84806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:16.161+0000 7fa872200640 1 --2- 172.21.15.29:0/318203857 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa848068380 0x7fa84806a890 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa86c1f9aa0 tx=0x7fa854009040 comp rx=0 tx=0).stop 2024-04-17T20:00:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:16.161+0000 7fa872200640 1 -- 172.21.15.29:0/318203857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c0cc8c0 msgr2=0x7fa86c156da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:16.161+0000 7fa872200640 1 --2- 172.21.15.29:0/318203857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c0cc8c0 0x7fa86c156da0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fa85c00e8a0 tx=0x7fa85c00eca0 comp rx=0 tx=0).stop 2024-04-17T20:00:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:16.161+0000 7fa872200640 1 -- 172.21.15.29:0/318203857 shutdown_connections 2024-04-17T20:00:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:16.161+0000 7fa872200640 1 --2- 172.21.15.29:0/318203857 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa848068380 0x7fa84806a890 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:16.161+0000 7fa872200640 1 --2- 172.21.15.29:0/318203857 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa86c1597a0 0x7fa86c135f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:16.161+0000 7fa872200640 1 --2- 172.21.15.29:0/318203857 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa86c159390 0x7fa86c135a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:16.161+0000 7fa872200640 1 --2- 172.21.15.29:0/318203857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86c0cc8c0 0x7fa86c156da0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:16.159 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:16.161+0000 7fa872200640 1 -- 172.21.15.29:0/318203857 >> 172.21.15.29:0/318203857 conn(0x7fa86c06fde0 msgr2=0x7fa86c073cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:16.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:16.161+0000 7fa872200640 1 -- 172.21.15.29:0/318203857 shutdown_connections 2024-04-17T20:00:16.160 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:16.161+0000 7fa872200640 1 -- 172.21.15.29:0/318203857 wait complete. 2024-04-17T20:00:16.180 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-17T20:00:18.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:18.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.353+0000 7f5af6c00640 1 Processor -- start 2024-04-17T20:00:18.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.353+0000 7f5af6c00640 1 -- start start 2024-04-17T20:00:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.353+0000 7f5af6c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5af00cc8c0 0x7f5af00cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.353+0000 7f5af6c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5af01597a0 0x7f5af01635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.353+0000 7f5af6c00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5af00c88b0 con 0x7f5af0159390 2024-04-17T20:00:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.353+0000 7f5af6c00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f5af00c6ce0 con 0x7f5af00cc8c0 2024-04-17T20:00:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.353+0000 7f5af6c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5af00c6fc0 con 0x7f5af01597a0 2024-04-17T20:00:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.353+0000 7f5aefe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5af00cc8c0 0x7f5af00cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.353+0000 7f5aefe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5af00cc8c0 0x7f5af00cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46802/0 (socket says 172.21.15.29:46802) 2024-04-17T20:00:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.353+0000 7f5aefe00640 1 -- 172.21.15.29:0/1501328476 learned_addr learned my addr 172.21.15.29:0/1501328476 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.353+0000 7f5af4c00640 1 --2- 172.21.15.29:0/1501328476 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5af01597a0 0x7f5af01635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.353+0000 7f5aefe00640 1 -- 172.21.15.29:0/1501328476 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5af01597a0 msgr2=0x7f5af01635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.353+0000 7f5aefe00640 1 --2- 172.21.15.29:0/1501328476 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5af01597a0 0x7f5af01635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.353+0000 7f5aefe00640 1 -- 172.21.15.29:0/1501328476 >> v1:172.21.15.37:6789/0 conn(0x7f5af0159390 legacy=0x7f5af00ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:18.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.353+0000 7f5aefe00640 1 -- 172.21.15.29:0/1501328476 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af00c72a0 con 0x7f5af00cc8c0 2024-04-17T20:00:18.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.357+0000 7f5aefe00640 1 --2- 172.21.15.29:0/1501328476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5af00cc8c0 0x7f5af00cccd0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0009870 tx=0x7f5ae001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc66a6783cdf3ae6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:18.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.357+0000 7f5aef400640 1 -- 172.21.15.29:0/1501328476 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae00b2070 con 0x7f5af00cc8c0 2024-04-17T20:00:18.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.357+0000 7f5aef400640 1 -- 172.21.15.29:0/1501328476 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae00a9070 con 0x7f5af00cc8c0 2024-04-17T20:00:18.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.357+0000 7f5aef400640 1 -- 172.21.15.29:0/1501328476 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae00ad880 con 0x7f5af00cc8c0 2024-04-17T20:00:18.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.357+0000 7f5af6c00640 1 -- 172.21.15.29:0/1501328476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5af00cc8c0 msgr2=0x7f5af00cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:18.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.357+0000 7f5af6c00640 1 --2- 172.21.15.29:0/1501328476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5af00cc8c0 0x7f5af00cccd0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0009870 tx=0x7f5ae001c560 comp rx=0 tx=0).stop 2024-04-17T20:00:18.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.357+0000 7f5af6c00640 1 -- 172.21.15.29:0/1501328476 shutdown_connections 2024-04-17T20:00:18.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.357+0000 7f5af6c00640 1 --2- 172.21.15.29:0/1501328476 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5af01597a0 0x7f5af01635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:18.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.357+0000 7f5af6c00640 1 --2- 172.21.15.29:0/1501328476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5af00cc8c0 0x7f5af00cccd0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:18.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.357+0000 7f5af6c00640 1 -- 172.21.15.29:0/1501328476 >> 172.21.15.29:0/1501328476 conn(0x7f5af006fde0 msgr2=0x7f5af00701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:18.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.357+0000 7f5af6c00640 1 -- 172.21.15.29:0/1501328476 shutdown_connections 2024-04-17T20:00:18.355 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.357+0000 7f5af6c00640 1 -- 172.21.15.29:0/1501328476 wait complete. 2024-04-17T20:00:18.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5af6c00640 1 Processor -- start 2024-04-17T20:00:18.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5af6c00640 1 -- start start 2024-04-17T20:00:18.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5af6c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af00cc8c0 0x7f5af013e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:18.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5af6c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5af0159390 0x7f5af013c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:18.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5af6c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5af01597a0 0x7f5af01f5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:18.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5af6c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5af00cabc0 con 0x7f5af0159390 2024-04-17T20:00:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5af6c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5af00caa40 con 0x7f5af00cc8c0 2024-04-17T20:00:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5af6c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5af00c6880 con 0x7f5af01597a0 2024-04-17T20:00:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5ae7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5af0159390 0x7f5af013c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5ae7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5af0159390 0x7f5af013c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46814/0 (socket says 172.21.15.29:46814) 2024-04-17T20:00:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5ae7e00640 1 -- 172.21.15.29:0/681498718 learned_addr learned my addr 172.21.15.29:0/681498718 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5aefe00640 1 --2- 172.21.15.29:0/681498718 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af00cc8c0 0x7f5af013e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5af4c00640 1 --2- 172.21.15.29:0/681498718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5af01597a0 0x7f5af01f5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5ae7e00640 1 -- 172.21.15.29:0/681498718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5af01597a0 msgr2=0x7f5af01f5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5ae7e00640 1 --2- 172.21.15.29:0/681498718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5af01597a0 0x7f5af01f5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5ae7e00640 1 -- 172.21.15.29:0/681498718 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af00cc8c0 msgr2=0x7f5af013e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5ae7e00640 1 --2- 172.21.15.29:0/681498718 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af00cc8c0 0x7f5af013e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5ae7e00640 1 -- 172.21.15.29:0/681498718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af01f64f0 con 0x7f5af0159390 2024-04-17T20:00:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5ae7e00640 1 --2- 172.21.15.29:0/681498718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5af0159390 0x7f5af013c830 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5ad800c3c0 tx=0x7f5ad800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5aecc00640 1 -- 172.21.15.29:0/681498718 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad8017070 con 0x7f5af0159390 2024-04-17T20:00:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5aecc00640 1 -- 172.21.15.29:0/681498718 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ad80052d0 con 0x7f5af0159390 2024-04-17T20:00:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5af6c00640 1 -- 172.21.15.29:0/681498718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5af0158720 con 0x7f5af0159390 2024-04-17T20:00:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.361+0000 7f5aecc00640 1 -- 172.21.15.29:0/681498718 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad8003840 con 0x7f5af0159390 2024-04-17T20:00:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.365+0000 7f5af6c00640 1 -- 172.21.15.29:0/681498718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5af01f9780 con 0x7f5af0159390 2024-04-17T20:00:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.365+0000 7f5af6c00640 1 -- 172.21.15.29:0/681498718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5af0069b90 con 0x7f5af0159390 2024-04-17T20:00:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.365+0000 7f5aecc00640 1 -- 172.21.15.29:0/681498718 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5ad801a070 con 0x7f5af0159390 2024-04-17T20:00:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.365+0000 7f5aecc00640 1 --2- 172.21.15.29:0/681498718 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5acc068380 0x7f5acc06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.365+0000 7f5aecc00640 1 -- 172.21.15.29:0/681498718 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7f5ad8086330 con 0x7f5af0159390 2024-04-17T20:00:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.365+0000 7f5aefe00640 1 --2- 172.21.15.29:0/681498718 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5acc068380 0x7f5acc06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:18.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.369+0000 7f5aefe00640 1 --2- 172.21.15.29:0/681498718 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5acc068380 0x7f5acc06a890 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5ae00af000 tx=0x7f5ae0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.369+0000 7f5aecc00640 1 -- 172.21.15.29:0/681498718 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5ad8005480 con 0x7f5af0159390 2024-04-17T20:00:18.567 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.569+0000 7f5af6c00640 1 -- 172.21.15.29:0/681498718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5af00c7e10 con 0x7f5af0159390 2024-04-17T20:00:18.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.569+0000 7f5aecc00640 1 -- 172.21.15.29:0/681498718 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6908 (secure 0 0 0) 0x7f5ad8055bc0 con 0x7f5af0159390 2024-04-17T20:00:18.568 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:18.568 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5775,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:26.051172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5212,5144,5631,4305]},"id":1}]} 2024-04-17T20:00:18.568 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 49 2024-04-17T20:00:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.573+0000 7f5af6c00640 1 -- 172.21.15.29:0/681498718 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5acc068380 msgr2=0x7f5acc06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.573+0000 7f5af6c00640 1 --2- 172.21.15.29:0/681498718 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5acc068380 0x7f5acc06a890 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5ae00af000 tx=0x7f5ae0008040 comp rx=0 tx=0).stop 2024-04-17T20:00:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.573+0000 7f5af6c00640 1 -- 172.21.15.29:0/681498718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5af0159390 msgr2=0x7f5af013c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.573+0000 7f5af6c00640 1 --2- 172.21.15.29:0/681498718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5af0159390 0x7f5af013c830 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f5ad800c3c0 tx=0x7f5ad800c880 comp rx=0 tx=0).stop 2024-04-17T20:00:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.573+0000 7f5af6c00640 1 -- 172.21.15.29:0/681498718 shutdown_connections 2024-04-17T20:00:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.573+0000 7f5af6c00640 1 --2- 172.21.15.29:0/681498718 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5acc068380 0x7f5acc06a890 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.573+0000 7f5af6c00640 1 --2- 172.21.15.29:0/681498718 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5af01597a0 0x7f5af01f5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.573+0000 7f5af6c00640 1 --2- 172.21.15.29:0/681498718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5af0159390 0x7f5af013c830 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.573+0000 7f5af6c00640 1 --2- 172.21.15.29:0/681498718 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5af00cc8c0 0x7f5af013e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.573+0000 7f5af6c00640 1 -- 172.21.15.29:0/681498718 >> 172.21.15.29:0/681498718 conn(0x7f5af006fde0 msgr2=0x7f5af00cb140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:18.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.573+0000 7f5af6c00640 1 -- 172.21.15.29:0/681498718 shutdown_connections 2024-04-17T20:00:18.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:18.573+0000 7f5af6c00640 1 -- 172.21.15.29:0/681498718 wait complete. 2024-04-17T20:00:18.591 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-17T20:00:20.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:20.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.761+0000 7fb88b000640 1 Processor -- start 2024-04-17T20:00:20.757 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.761+0000 7fb88b000640 1 -- start start 2024-04-17T20:00:20.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.761+0000 7fb88b000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb8840cc8c0 0x7fb8840cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:20.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.761+0000 7fb88b000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb8841597a0 0x7fb8841635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:20.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.761+0000 7fb88b000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8840c88b0 con 0x7fb884159390 2024-04-17T20:00:20.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.761+0000 7fb88b000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fb8840c6ce0 con 0x7fb8841597a0 2024-04-17T20:00:20.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.761+0000 7fb88b000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb8840c6fc0 con 0x7fb8840cc8c0 2024-04-17T20:00:20.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.761+0000 7fb889000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb8841597a0 0x7fb8841635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:20.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.761+0000 7fb889000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb8841597a0 0x7fb8841635e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53854/0 (socket says 172.21.15.29:53854) 2024-04-17T20:00:20.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.761+0000 7fb889000640 1 -- 172.21.15.29:0/2422889559 learned_addr learned my addr 172.21.15.29:0/2422889559 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:20.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.761+0000 7fb883e00640 1 --2- 172.21.15.29:0/2422889559 >> v2:172.21.15.37:3300/0 conn(0x7fb8840cc8c0 0x7fb8840cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:20.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.761+0000 7fb889000640 1 -- 172.21.15.29:0/2422889559 >> v2:172.21.15.37:3300/0 conn(0x7fb8840cc8c0 msgr2=0x7fb8840cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:20.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.761+0000 7fb889000640 1 --2- 172.21.15.29:0/2422889559 >> v2:172.21.15.37:3300/0 conn(0x7fb8840cc8c0 0x7fb8840cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:20.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.761+0000 7fb889000640 1 -- 172.21.15.29:0/2422889559 >> v1:172.21.15.37:6789/0 conn(0x7fb884159390 legacy=0x7fb8840ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:20.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.761+0000 7fb889000640 1 -- 172.21.15.29:0/2422889559 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8840c72a0 con 0x7fb8841597a0 2024-04-17T20:00:20.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.765+0000 7fb889000640 1 --2- 172.21.15.29:0/2422889559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8841597a0 0x7fb8841635e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb8840721b0 tx=0x7fb87c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c2a4f404aca4f48 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:20.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.765+0000 7fb883400640 1 -- 172.21.15.29:0/2422889559 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c0a56c0 con 0x7fb8841597a0 2024-04-17T20:00:20.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.765+0000 7fb883400640 1 -- 172.21.15.29:0/2422889559 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb87c0a5860 con 0x7fb8841597a0 2024-04-17T20:00:20.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.765+0000 7fb883400640 1 -- 172.21.15.29:0/2422889559 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c0a5c30 con 0x7fb8841597a0 2024-04-17T20:00:20.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.765+0000 7fb88b000640 1 -- 172.21.15.29:0/2422889559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8841597a0 msgr2=0x7fb8841635e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:20.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.765+0000 7fb88b000640 1 --2- 172.21.15.29:0/2422889559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8841597a0 0x7fb8841635e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fb8840721b0 tx=0x7fb87c01c9f0 comp rx=0 tx=0).stop 2024-04-17T20:00:20.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.765+0000 7fb88b000640 1 -- 172.21.15.29:0/2422889559 shutdown_connections 2024-04-17T20:00:20.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.765+0000 7fb88b000640 1 --2- 172.21.15.29:0/2422889559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8841597a0 0x7fb8841635e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:20.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.765+0000 7fb88b000640 1 --2- 172.21.15.29:0/2422889559 >> v2:172.21.15.37:3300/0 conn(0x7fb8840cc8c0 0x7fb8840cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:20.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.765+0000 7fb88b000640 1 -- 172.21.15.29:0/2422889559 >> 172.21.15.29:0/2422889559 conn(0x7fb88406fde0 msgr2=0x7fb8840701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:20.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.765+0000 7fb88b000640 1 -- 172.21.15.29:0/2422889559 shutdown_connections 2024-04-17T20:00:20.765 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.765+0000 7fb88b000640 1 -- 172.21.15.29:0/2422889559 wait complete. 2024-04-17T20:00:20.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.769+0000 7fb88b000640 1 Processor -- start 2024-04-17T20:00:20.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.769+0000 7fb88b000640 1 -- start start 2024-04-17T20:00:20.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.769+0000 7fb88b000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8840cc8c0 0x7fb8841f3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:20.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.769+0000 7fb88b000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb884159390 0x7fb88413af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:20.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.769+0000 7fb88b000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8841597a0 0x7fb884137190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:20.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.769+0000 7fb88b000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb8840cabc0 con 0x7fb8841597a0 2024-04-17T20:00:20.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.769+0000 7fb88b000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb8840caa40 con 0x7fb884159390 2024-04-17T20:00:20.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.769+0000 7fb88b000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb8840c6880 con 0x7fb8840cc8c0 2024-04-17T20:00:20.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.769+0000 7fb889000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8841597a0 0x7fb884137190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:20.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.769+0000 7fb889000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8841597a0 0x7fb884137190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53868/0 (socket says 172.21.15.29:53868) 2024-04-17T20:00:20.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.769+0000 7fb889000640 1 -- 172.21.15.29:0/1726390250 learned_addr learned my addr 172.21.15.29:0/1726390250 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:20.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.769+0000 7fb87be00640 1 --2- 172.21.15.29:0/1726390250 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb884159390 0x7fb88413af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:20.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.769+0000 7fb883e00640 1 --2- 172.21.15.29:0/1726390250 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8840cc8c0 0x7fb8841f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:20.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.769+0000 7fb889000640 1 -- 172.21.15.29:0/1726390250 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8840cc8c0 msgr2=0x7fb8841f3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:20.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.769+0000 7fb889000640 1 --2- 172.21.15.29:0/1726390250 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8840cc8c0 0x7fb8841f3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:20.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.769+0000 7fb889000640 1 -- 172.21.15.29:0/1726390250 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb884159390 msgr2=0x7fb88413af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:20.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.769+0000 7fb889000640 1 --2- 172.21.15.29:0/1726390250 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb884159390 0x7fb88413af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:20.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.769+0000 7fb889000640 1 -- 172.21.15.29:0/1726390250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8841f3b60 con 0x7fb8841597a0 2024-04-17T20:00:20.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.773+0000 7fb889000640 1 --2- 172.21.15.29:0/1726390250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8841597a0 0x7fb884137190 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fb87c0af040 tx=0x7fb87c019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:20.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.773+0000 7fb880c00640 1 -- 172.21.15.29:0/1726390250 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c009510 con 0x7fb8841597a0 2024-04-17T20:00:20.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.773+0000 7fb880c00640 1 -- 172.21.15.29:0/1726390250 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb87c0be070 con 0x7fb8841597a0 2024-04-17T20:00:20.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.773+0000 7fb88b000640 1 -- 172.21.15.29:0/1726390250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8840c5fb0 con 0x7fb8841597a0 2024-04-17T20:00:20.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.773+0000 7fb880c00640 1 -- 172.21.15.29:0/1726390250 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c0b88f0 con 0x7fb8841597a0 2024-04-17T20:00:20.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.773+0000 7fb88b000640 1 -- 172.21.15.29:0/1726390250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8841f7690 con 0x7fb8841597a0 2024-04-17T20:00:20.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.773+0000 7fb88b000640 1 -- 172.21.15.29:0/1726390250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8841f6b70 con 0x7fb8841597a0 2024-04-17T20:00:20.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.773+0000 7fb880c00640 1 -- 172.21.15.29:0/1726390250 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb87c0b8a90 con 0x7fb8841597a0 2024-04-17T20:00:20.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.773+0000 7fb880c00640 1 --2- 172.21.15.29:0/1726390250 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb860068380 0x7fb86006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:20.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.773+0000 7fb880c00640 1 -- 172.21.15.29:0/1726390250 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7fb87c121e60 con 0x7fb8841597a0 2024-04-17T20:00:20.772 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.773+0000 7fb883e00640 1 --2- 172.21.15.29:0/1726390250 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb860068380 0x7fb86006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:20.773 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.777+0000 7fb883e00640 1 --2- 172.21.15.29:0/1726390250 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb860068380 0x7fb86006a890 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb874009a50 tx=0x7fb874008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:20.776 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.781+0000 7fb880c00640 1 -- 172.21.15.29:0/1726390250 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb8841f6b70 con 0x7fb8841597a0 2024-04-17T20:00:20.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.981+0000 7fb88b000640 1 -- 172.21.15.29:0/1726390250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb884069b90 con 0x7fb8841597a0 2024-04-17T20:00:20.979 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.981+0000 7fb880c00640 1 -- 172.21.15.29:0/1726390250 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6908 (secure 0 0 0) 0x7fb87c0c8020 con 0x7fb8841597a0 2024-04-17T20:00:20.979 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:20.979 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5775,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:26.051172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5212,5144,5631,4305]},"id":1}]} 2024-04-17T20:00:20.979 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 49 2024-04-17T20:00:20.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.985+0000 7fb88b000640 1 -- 172.21.15.29:0/1726390250 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb860068380 msgr2=0x7fb86006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:20.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.985+0000 7fb88b000640 1 --2- 172.21.15.29:0/1726390250 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb860068380 0x7fb86006a890 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb874009a50 tx=0x7fb874008040 comp rx=0 tx=0).stop 2024-04-17T20:00:20.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.985+0000 7fb88b000640 1 -- 172.21.15.29:0/1726390250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8841597a0 msgr2=0x7fb884137190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:20.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.985+0000 7fb88b000640 1 --2- 172.21.15.29:0/1726390250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8841597a0 0x7fb884137190 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fb87c0af040 tx=0x7fb87c019790 comp rx=0 tx=0).stop 2024-04-17T20:00:20.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.985+0000 7fb88b000640 1 -- 172.21.15.29:0/1726390250 shutdown_connections 2024-04-17T20:00:20.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.985+0000 7fb88b000640 1 --2- 172.21.15.29:0/1726390250 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb860068380 0x7fb86006a890 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:20.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.985+0000 7fb88b000640 1 --2- 172.21.15.29:0/1726390250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8841597a0 0x7fb884137190 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:20.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.985+0000 7fb88b000640 1 --2- 172.21.15.29:0/1726390250 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb884159390 0x7fb88413af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:20.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.985+0000 7fb88b000640 1 --2- 172.21.15.29:0/1726390250 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb8840cc8c0 0x7fb8841f3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:20.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.985+0000 7fb88b000640 1 -- 172.21.15.29:0/1726390250 >> 172.21.15.29:0/1726390250 conn(0x7fb88406fde0 msgr2=0x7fb8840cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:20.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.985+0000 7fb88b000640 1 -- 172.21.15.29:0/1726390250 shutdown_connections 2024-04-17T20:00:20.982 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:20.985+0000 7fb88b000640 1 -- 172.21.15.29:0/1726390250 wait complete. 2024-04-17T20:00:21.003 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-17T20:00:23.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:23.094 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:00:23.098 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:00:23.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.172+0000 7f162c800640 1 Processor -- start 2024-04-17T20:00:23.170 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.172+0000 7f162c800640 1 -- start start 2024-04-17T20:00:23.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.176+0000 7f162c800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f16240cc8c0 0x7f16240cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:23.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.176+0000 7f162c800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16240c88b0 con 0x7f16241597a0 2024-04-17T20:00:23.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.176+0000 7f162c800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16240c6ce0 con 0x7f1624159390 2024-04-17T20:00:23.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.176+0000 7f162c800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f16240c6fc0 con 0x7f16240cc8c0 2024-04-17T20:00:23.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.176+0000 7f162a800640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f16241597a0 0x7f16241635e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58324/0 (socket says 172.21.15.29:58324) 2024-04-17T20:00:23.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.176+0000 7f162a800640 1 -- 172.21.15.29:0/3761505882 learned_addr learned my addr 172.21.15.29:0/3761505882 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:00:23.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.176+0000 7f1629e00640 1 --2- 172.21.15.29:0/3761505882 >> v2:172.21.15.37:3300/0 conn(0x7f16240cc8c0 0x7f16240cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:23.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.176+0000 7f1628a00640 1 -- 172.21.15.29:0/3761505882 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1141118110 0 0) 0x7f16240c88b0 con 0x7f16241597a0 2024-04-17T20:00:23.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.176+0000 7f1628a00640 1 -- 172.21.15.29:0/3761505882 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f160c005740 con 0x7f16241597a0 2024-04-17T20:00:23.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.176+0000 7f1629e00640 1 -- 172.21.15.29:0/3761505882 >> v1:172.21.15.37:6789/0 conn(0x7f1624159390 legacy=0x7f16240ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:23.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.176+0000 7f1628a00640 1 -- 172.21.15.29:0/3761505882 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 189647689 0 0) 0x7f160c005740 con 0x7f16241597a0 2024-04-17T20:00:23.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.176+0000 7f1629e00640 1 -- 172.21.15.29:0/3761505882 >> v1:172.21.15.29:6789/0 conn(0x7f16241597a0 legacy=0x7f16241635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:23.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.176+0000 7f1629e00640 1 -- 172.21.15.29:0/3761505882 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16240c72a0 con 0x7f16240cc8c0 2024-04-17T20:00:23.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.180+0000 7f1629e00640 1 --2- 172.21.15.29:0/3761505882 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16240cc8c0 0x7f16240cccd0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f16180097c0 tx=0x7f161801c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0c34d68491588a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:23.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.180+0000 7f1628a00640 1 -- 172.21.15.29:0/3761505882 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16180b0070 con 0x7f16240cc8c0 2024-04-17T20:00:23.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.180+0000 7f1628a00640 1 -- 172.21.15.29:0/3761505882 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16180ab400 con 0x7f16240cc8c0 2024-04-17T20:00:23.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.180+0000 7f1628a00640 1 -- 172.21.15.29:0/3761505882 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1620001a00 con 0x7f16240cc8c0 2024-04-17T20:00:23.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.180+0000 7f162c800640 1 -- 172.21.15.29:0/3761505882 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16240cc8c0 msgr2=0x7f16240cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:23.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.180+0000 7f162c800640 1 --2- 172.21.15.29:0/3761505882 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16240cc8c0 0x7f16240cccd0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f16180097c0 tx=0x7f161801c3c0 comp rx=0 tx=0).stop 2024-04-17T20:00:23.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.180+0000 7f162c800640 1 -- 172.21.15.29:0/3761505882 shutdown_connections 2024-04-17T20:00:23.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.180+0000 7f162c800640 1 --2- 172.21.15.29:0/3761505882 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16240cc8c0 0x7f16240cccd0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:23.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.180+0000 7f162c800640 1 -- 172.21.15.29:0/3761505882 >> 172.21.15.29:0/3761505882 conn(0x7f162406fde0 msgr2=0x7f16240701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:23.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.180+0000 7f162c800640 1 -- 172.21.15.29:0/3761505882 shutdown_connections 2024-04-17T20:00:23.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.180+0000 7f162c800640 1 -- 172.21.15.29:0/3761505882 wait complete. 2024-04-17T20:00:23.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.180+0000 7f162c800640 1 Processor -- start 2024-04-17T20:00:23.180 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f162c800640 1 -- start start 2024-04-17T20:00:23.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f162c800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16240cc8c0 0x7f1624163420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:23.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f162c800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1624159390 0x7f162415c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:23.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f162c800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16241597a0 0x7f1624155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:23.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f162c800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f16240cabc0 con 0x7f16240cc8c0 2024-04-17T20:00:23.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f162c800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f16240caa40 con 0x7f16241597a0 2024-04-17T20:00:23.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f162c800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f16240c6880 con 0x7f1624159390 2024-04-17T20:00:23.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f1629e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16240cc8c0 0x7f1624163420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:23.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f1629e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16240cc8c0 0x7f1624163420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53880/0 (socket says 172.21.15.29:53880) 2024-04-17T20:00:23.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f1629e00640 1 -- 172.21.15.29:0/282815076 learned_addr learned my addr 172.21.15.29:0/282815076 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:23.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f162a800640 1 --2- 172.21.15.29:0/282815076 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16241597a0 0x7f1624155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:23.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f1629400640 1 --2- 172.21.15.29:0/282815076 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1624159390 0x7f162415c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:23.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f1629e00640 1 -- 172.21.15.29:0/282815076 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1624159390 msgr2=0x7f162415c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:23.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f1629e00640 1 --2- 172.21.15.29:0/282815076 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1624159390 0x7f162415c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:23.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f1629e00640 1 -- 172.21.15.29:0/282815076 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16241597a0 msgr2=0x7f1624155880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:23.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f1629e00640 1 --2- 172.21.15.29:0/282815076 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16241597a0 0x7f1624155880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:23.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f1629e00640 1 -- 172.21.15.29:0/282815076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1624068980 con 0x7f16240cc8c0 2024-04-17T20:00:23.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f1629e00640 1 --2- 172.21.15.29:0/282815076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16240cc8c0 0x7f1624163420 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1618000c10 tx=0x7f1618019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:23.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f161e000640 1 -- 172.21.15.29:0/282815076 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16180b0040 con 0x7f16240cc8c0 2024-04-17T20:00:23.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f161e000640 1 -- 172.21.15.29:0/282815076 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16180036a0 con 0x7f16240cc8c0 2024-04-17T20:00:23.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f161e000640 1 -- 172.21.15.29:0/282815076 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1618003a50 con 0x7f16240cc8c0 2024-04-17T20:00:23.196 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f162c800640 1 -- 172.21.15.29:0/282815076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16241f5450 con 0x7f16240cc8c0 2024-04-17T20:00:23.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f162c800640 1 -- 172.21.15.29:0/282815076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16241f59b0 con 0x7f16240cc8c0 2024-04-17T20:00:23.197 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.184+0000 7f162c800640 1 -- 172.21.15.29:0/282815076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1624069b90 con 0x7f16240cc8c0 2024-04-17T20:00:23.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.188+0000 7f161e000640 1 -- 172.21.15.29:0/282815076 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f16180bd020 con 0x7f16240cc8c0 2024-04-17T20:00:23.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.188+0000 7f161e000640 1 --2- 172.21.15.29:0/282815076 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1608068380 0x7f160806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.188+0000 7f161e000640 1 -- 172.21.15.29:0/282815076 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7165+0+0 (secure 0 0 0) 0x7f1618120900 con 0x7f16240cc8c0 2024-04-17T20:00:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.188+0000 7f1629400640 1 --2- 172.21.15.29:0/282815076 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1608068380 0x7f160806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.188+0000 7f1629400640 1 --2- 172.21.15.29:0/282815076 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1608068380 0x7f160806a890 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1624162e90 tx=0x7f1614009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:23.199 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.192+0000 7f161e000640 1 -- 172.21.15.29:0/282815076 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f16241f5450 con 0x7f16240cc8c0 2024-04-17T20:00:23.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.392+0000 7f162c800640 1 -- 172.21.15.29:0/282815076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16240c7e10 con 0x7f16240cc8c0 2024-04-17T20:00:23.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.392+0000 7f161e000640 1 -- 172.21.15.29:0/282815076 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) ==== 94+0+6908 (secure 0 0 0) 0x7f1618015fb0 con 0x7f16240cc8c0 2024-04-17T20:00:23.391 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:23.391 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5775,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":49}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T19:58:26.051172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":4305,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"c","rank":0,"incarnation":5,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2691590158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":2691590158},{"type":"v1","addr":"172.21.15.29:6837","nonce":2691590158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4305,"qdb_cluster":[5350,5212,5144,5631,4305]},"id":1}]} 2024-04-17T20:00:23.391 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 49 2024-04-17T20:00:23.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.396+0000 7f162c800640 1 -- 172.21.15.29:0/282815076 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1608068380 msgr2=0x7f160806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:23.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.396+0000 7f162c800640 1 --2- 172.21.15.29:0/282815076 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1608068380 0x7f160806a890 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1624162e90 tx=0x7f1614009040 comp rx=0 tx=0).stop 2024-04-17T20:00:23.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.396+0000 7f162c800640 1 -- 172.21.15.29:0/282815076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16240cc8c0 msgr2=0x7f1624163420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:23.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.396+0000 7f162c800640 1 --2- 172.21.15.29:0/282815076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16240cc8c0 0x7f1624163420 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1618000c10 tx=0x7f1618019160 comp rx=0 tx=0).stop 2024-04-17T20:00:23.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.396+0000 7f162c800640 1 -- 172.21.15.29:0/282815076 shutdown_connections 2024-04-17T20:00:23.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.396+0000 7f162c800640 1 --2- 172.21.15.29:0/282815076 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f1608068380 0x7f160806a890 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:23.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.396+0000 7f162c800640 1 --2- 172.21.15.29:0/282815076 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f16241597a0 0x7f1624155880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:23.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.396+0000 7f162c800640 1 --2- 172.21.15.29:0/282815076 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f1624159390 0x7f162415c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:23.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.396+0000 7f162c800640 1 --2- 172.21.15.29:0/282815076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16240cc8c0 0x7f1624163420 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:23.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.396+0000 7f162c800640 1 -- 172.21.15.29:0/282815076 >> 172.21.15.29:0/282815076 conn(0x7f162406fde0 msgr2=0x7f1624157810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:23.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.396+0000 7f162c800640 1 -- 172.21.15.29:0/282815076 shutdown_connections 2024-04-17T20:00:23.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:23.396+0000 7f162c800640 1 -- 172.21.15.29:0/282815076 wait complete. 2024-04-17T20:00:23.414 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-17T20:00:25.415 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:25.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.620+0000 7fd09fe00640 1 Processor -- start 2024-04-17T20:00:25.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.620+0000 7fd09fe00640 1 -- start start 2024-04-17T20:00:25.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.620+0000 7fd09fe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd098159c90 0x7fd0980ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:25.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.620+0000 7fd09fe00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0980ca7f0 0x7fd0980c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:25.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.620+0000 7fd09fe00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd09815a0a0 con 0x7fd0980c74b0 2024-04-17T20:00:25.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.620+0000 7fd09fe00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fd09815abe0 con 0x7fd098159c90 2024-04-17T20:00:25.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.620+0000 7fd09fe00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd09815a340 con 0x7fd0980ca7f0 2024-04-17T20:00:25.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.620+0000 7fd09d400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd098159c90 0x7fd0980ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:25.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.620+0000 7fd09d400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd098159c90 0x7fd0980ca2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53890/0 (socket says 172.21.15.29:53890) 2024-04-17T20:00:25.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.620+0000 7fd09d400640 1 -- 172.21.15.29:0/2462668276 learned_addr learned my addr 172.21.15.29:0/2462668276 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:25.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.620+0000 7fd09ca00640 1 --2- 172.21.15.29:0/2462668276 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0980ca7f0 0x7fd0980c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:25.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.620+0000 7fd09d400640 1 -- 172.21.15.29:0/2462668276 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0980ca7f0 msgr2=0x7fd0980c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:25.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.620+0000 7fd09d400640 1 --2- 172.21.15.29:0/2462668276 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0980ca7f0 0x7fd0980c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:25.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.620+0000 7fd09d400640 1 -- 172.21.15.29:0/2462668276 >> v1:172.21.15.37:6789/0 conn(0x7fd0980c74b0 legacy=0x7fd09806c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:25.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.620+0000 7fd09d400640 1 -- 172.21.15.29:0/2462668276 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09815a620 con 0x7fd098159c90 2024-04-17T20:00:25.624 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.624+0000 7fd09d400640 1 --2- 172.21.15.29:0/2462668276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd098159c90 0x7fd0980ca2b0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd08c009870 tx=0x7fd08c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6863734110bb471c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.624+0000 7fd097e00640 1 -- 172.21.15.29:0/2462668276 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c0a7070 con 0x7fd098159c90 2024-04-17T20:00:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.624+0000 7fd097e00640 1 -- 172.21.15.29:0/2462668276 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd08c0b0070 con 0x7fd098159c90 2024-04-17T20:00:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.624+0000 7fd097e00640 1 -- 172.21.15.29:0/2462668276 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c0ab890 con 0x7fd098159c90 2024-04-17T20:00:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.624+0000 7fd09fe00640 1 -- 172.21.15.29:0/2462668276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd098159c90 msgr2=0x7fd0980ca2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.624+0000 7fd09fe00640 1 --2- 172.21.15.29:0/2462668276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd098159c90 0x7fd0980ca2b0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd08c009870 tx=0x7fd08c01c560 comp rx=0 tx=0).stop 2024-04-17T20:00:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.624+0000 7fd09fe00640 1 -- 172.21.15.29:0/2462668276 shutdown_connections 2024-04-17T20:00:25.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.624+0000 7fd09fe00640 1 --2- 172.21.15.29:0/2462668276 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0980ca7f0 0x7fd0980c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:25.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.624+0000 7fd09fe00640 1 --2- 172.21.15.29:0/2462668276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd098159c90 0x7fd0980ca2b0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:25.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.624+0000 7fd09fe00640 1 -- 172.21.15.29:0/2462668276 >> 172.21.15.29:0/2462668276 conn(0x7fd09806fde0 msgr2=0x7fd0980701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:25.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.624+0000 7fd09fe00640 1 -- 172.21.15.29:0/2462668276 shutdown_connections 2024-04-17T20:00:25.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.624+0000 7fd09fe00640 1 -- 172.21.15.29:0/2462668276 wait complete. 2024-04-17T20:00:25.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.628+0000 7fd09fe00640 1 Processor -- start 2024-04-17T20:00:25.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.628+0000 7fd09fe00640 1 -- start start 2024-04-17T20:00:25.637 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.628+0000 7fd09fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0980c74b0 0x7fd098147c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:25.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.628+0000 7fd09fe00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0980ca7f0 0x7fd098072560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:25.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.628+0000 7fd09fe00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd098159c90 0x7fd0981443f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:25.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.628+0000 7fd09fe00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd098166fb0 con 0x7fd0980c74b0 2024-04-17T20:00:25.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.628+0000 7fd09fe00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd098166e30 con 0x7fd098159c90 2024-04-17T20:00:25.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.628+0000 7fd09fe00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd098167130 con 0x7fd0980ca7f0 2024-04-17T20:00:25.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.628+0000 7fd09d400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0980c74b0 0x7fd098147c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:25.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.628+0000 7fd09d400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0980c74b0 0x7fd098147c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53898/0 (socket says 172.21.15.29:53898) 2024-04-17T20:00:25.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.628+0000 7fd09d400640 1 -- 172.21.15.29:0/939661609 learned_addr learned my addr 172.21.15.29:0/939661609 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:25.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.628+0000 7fd09de00640 1 --2- 172.21.15.29:0/939661609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd098159c90 0x7fd0981443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:25.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.628+0000 7fd09ca00640 1 --2- 172.21.15.29:0/939661609 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0980ca7f0 0x7fd098072560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.628+0000 7fd09d400640 1 -- 172.21.15.29:0/939661609 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0980ca7f0 msgr2=0x7fd098072560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.628+0000 7fd09d400640 1 --2- 172.21.15.29:0/939661609 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0980ca7f0 0x7fd098072560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.628+0000 7fd09d400640 1 -- 172.21.15.29:0/939661609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd098159c90 msgr2=0x7fd0981443f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.628+0000 7fd09d400640 1 --2- 172.21.15.29:0/939661609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd098159c90 0x7fd0981443f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.628+0000 7fd09d400640 1 -- 172.21.15.29:0/939661609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd098152040 con 0x7fd0980c74b0 2024-04-17T20:00:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.632+0000 7fd09d400640 1 --2- 172.21.15.29:0/939661609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0980c74b0 0x7fd098147c60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd08c0ad040 tx=0x7fd08c01ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.632+0000 7fd095600640 1 -- 172.21.15.29:0/939661609 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c0be5f0 con 0x7fd0980c74b0 2024-04-17T20:00:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.632+0000 7fd095600640 1 -- 172.21.15.29:0/939661609 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd08c0be790 con 0x7fd0980c74b0 2024-04-17T20:00:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.632+0000 7fd095600640 1 -- 172.21.15.29:0/939661609 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08c0beb80 con 0x7fd0980c74b0 2024-04-17T20:00:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.632+0000 7fd09fe00640 1 -- 172.21.15.29:0/939661609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0981481a0 con 0x7fd0980c74b0 2024-04-17T20:00:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.632+0000 7fd09fe00640 1 -- 172.21.15.29:0/939661609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd098153a60 con 0x7fd0980c74b0 2024-04-17T20:00:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.632+0000 7fd09fe00640 1 -- 172.21.15.29:0/939661609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd098154120 con 0x7fd0980c74b0 2024-04-17T20:00:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.632+0000 7fd095600640 1 -- 172.21.15.29:0/939661609 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd08c0b43f0 con 0x7fd0980c74b0 2024-04-17T20:00:25.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.632+0000 7fd095600640 1 --2- 172.21.15.29:0/939661609 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd0780683b0 0x7fd07806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:25.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.632+0000 7fd095600640 1 -- 172.21.15.29:0/939661609 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7fd08c121c40 con 0x7fd0980c74b0 2024-04-17T20:00:25.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.632+0000 7fd09ca00640 1 --2- 172.21.15.29:0/939661609 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd0780683b0 0x7fd07806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:25.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.636+0000 7fd09ca00640 1 --2- 172.21.15.29:0/939661609 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd0780683b0 0x7fd07806a8c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd09815e2d0 tx=0x7fd084008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:25.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.636+0000 7fd095600640 1 -- 172.21.15.29:0/939661609 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd098154120 con 0x7fd0980c74b0 2024-04-17T20:00:25.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.840+0000 7fd09fe00640 1 -- 172.21.15.29:0/939661609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd098069b90 con 0x7fd0980c74b0 2024-04-17T20:00:25.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.840+0000 7fd095600640 1 -- 172.21.15.29:0/939661609 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+6080 (secure 0 0 0) 0x7fd08c0c4060 con 0x7fd0980c74b0 2024-04-17T20:00:25.853 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:25.853 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:23.943733+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:replay","state_seq":1,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5144,5212,5350,5631]},"id":1}]} 2024-04-17T20:00:25.854 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 50 2024-04-17T20:00:25.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.840+0000 7fd09fe00640 1 -- 172.21.15.29:0/939661609 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd0780683b0 msgr2=0x7fd07806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:25.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.840+0000 7fd09fe00640 1 --2- 172.21.15.29:0/939661609 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd0780683b0 0x7fd07806a8c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd09815e2d0 tx=0x7fd084008040 comp rx=0 tx=0).stop 2024-04-17T20:00:25.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.840+0000 7fd09fe00640 1 -- 172.21.15.29:0/939661609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0980c74b0 msgr2=0x7fd098147c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:25.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.840+0000 7fd09fe00640 1 --2- 172.21.15.29:0/939661609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0980c74b0 0x7fd098147c60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd08c0ad040 tx=0x7fd08c01ada0 comp rx=0 tx=0).stop 2024-04-17T20:00:25.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.840+0000 7fd09fe00640 1 -- 172.21.15.29:0/939661609 shutdown_connections 2024-04-17T20:00:25.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.840+0000 7fd09fe00640 1 --2- 172.21.15.29:0/939661609 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd0780683b0 0x7fd07806a8c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:25.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.840+0000 7fd09fe00640 1 --2- 172.21.15.29:0/939661609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd098159c90 0x7fd0981443f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:25.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.840+0000 7fd09fe00640 1 --2- 172.21.15.29:0/939661609 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd0980ca7f0 0x7fd098072560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:25.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.840+0000 7fd09fe00640 1 --2- 172.21.15.29:0/939661609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0980c74b0 0x7fd098147c60 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:25.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.840+0000 7fd09fe00640 1 -- 172.21.15.29:0/939661609 >> 172.21.15.29:0/939661609 conn(0x7fd09806fde0 msgr2=0x7fd0980701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:25.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.840+0000 7fd09fe00640 1 -- 172.21.15.29:0/939661609 shutdown_connections 2024-04-17T20:00:25.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:25.840+0000 7fd09fe00640 1 -- 172.21.15.29:0/939661609 wait complete. 2024-04-17T20:00:25.859 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2024-04-17T20:00:25.860 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-17T20:00:25.860 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:26.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.016+0000 7f31c1200640 1 Processor -- start 2024-04-17T20:00:26.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.016+0000 7f31c1200640 1 -- start start 2024-04-17T20:00:26.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.020+0000 7f31c1200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f31bc06c7a0 0x7f31bc155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:26.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.020+0000 7f31c1200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31bc15bf00 con 0x7f31bc0cf7a0 2024-04-17T20:00:26.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.020+0000 7f31c1200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31bc0707f0 con 0x7f31bc0cb830 2024-04-17T20:00:26.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.020+0000 7f31c1200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f31bc156430 con 0x7f31bc06c7a0 2024-04-17T20:00:26.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.020+0000 7f31b9e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f31bc06c7a0 0x7f31bc155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:26.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.020+0000 7f31b3e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f31bc0cf7a0 0x7f31bc0cb120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58340/0 (socket says 172.21.15.29:58340) 2024-04-17T20:00:26.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.020+0000 7f31b3e00640 1 -- 172.21.15.29:0/2297275402 learned_addr learned my addr 172.21.15.29:0/2297275402 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:00:26.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.020+0000 7f31b9e00640 1 -- 172.21.15.29:0/2297275402 >> v1:172.21.15.37:6789/0 conn(0x7f31bc0cb830 legacy=0x7f31bc166880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:26.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.020+0000 7f31b9e00640 1 -- 172.21.15.29:0/2297275402 >> v1:172.21.15.29:6789/0 conn(0x7f31bc0cf7a0 legacy=0x7f31bc0cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:26.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.020+0000 7f31b9e00640 1 -- 172.21.15.29:0/2297275402 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31bc15c1c0 con 0x7f31bc06c7a0 2024-04-17T20:00:26.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.020+0000 7f31b9400640 1 -- 172.21.15.29:0/2297275402 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3176973009 0 0) 0x7f31bc15bf00 con 0x7f31bc0cf7a0 2024-04-17T20:00:26.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.024+0000 7f31b9e00640 1 --2- 172.21.15.29:0/2297275402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31bc06c7a0 0x7f31bc155ef0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f31ac0097c0 tx=0x7f31ac01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd5787204576246d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:26.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.024+0000 7f31b9400640 1 -- 172.21.15.29:0/2297275402 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ac0b5070 con 0x7f31bc06c7a0 2024-04-17T20:00:26.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.024+0000 7f31b9400640 1 -- 172.21.15.29:0/2297275402 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31ac0a8990 con 0x7f31bc06c7a0 2024-04-17T20:00:26.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.024+0000 7f31b9400640 1 -- 172.21.15.29:0/2297275402 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ac0a8d60 con 0x7f31bc06c7a0 2024-04-17T20:00:26.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.024+0000 7f31c1200640 1 -- 172.21.15.29:0/2297275402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31bc06c7a0 msgr2=0x7f31bc155ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:26.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.024+0000 7f31c1200640 1 --2- 172.21.15.29:0/2297275402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31bc06c7a0 0x7f31bc155ef0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f31ac0097c0 tx=0x7f31ac01ec90 comp rx=0 tx=0).stop 2024-04-17T20:00:26.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.024+0000 7f31c1200640 1 -- 172.21.15.29:0/2297275402 shutdown_connections 2024-04-17T20:00:26.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.024+0000 7f31c1200640 1 --2- 172.21.15.29:0/2297275402 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31bc06c7a0 0x7f31bc155ef0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:26.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.024+0000 7f31c1200640 1 -- 172.21.15.29:0/2297275402 >> 172.21.15.29:0/2297275402 conn(0x7f31bc06e3d0 msgr2=0x7f31bc06e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:26.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.024+0000 7f31c1200640 1 -- 172.21.15.29:0/2297275402 shutdown_connections 2024-04-17T20:00:26.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.024+0000 7f31c1200640 1 -- 172.21.15.29:0/2297275402 wait complete. 2024-04-17T20:00:26.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31c1200640 1 Processor -- start 2024-04-17T20:00:26.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31c1200640 1 -- start start 2024-04-17T20:00:26.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31c1200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31bc06c7a0 0x7f31bc0c4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:26.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31c1200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31bc0cb830 0x7f31bc0c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:26.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31c1200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31bc0cf7a0 0x7f31bc1f5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:26.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31c1200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f31bc15ca60 con 0x7f31bc06c7a0 2024-04-17T20:00:26.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31c1200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f31bc15c780 con 0x7f31bc0cf7a0 2024-04-17T20:00:26.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31c1200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f31bc0c9ea0 con 0x7f31bc0cb830 2024-04-17T20:00:26.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31b9e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31bc06c7a0 0x7f31bc0c4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:26.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31b9e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31bc06c7a0 0x7f31bc0c4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53916/0 (socket says 172.21.15.29:53916) 2024-04-17T20:00:26.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31b9e00640 1 -- 172.21.15.29:0/4267642580 learned_addr learned my addr 172.21.15.29:0/4267642580 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:26.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31b3e00640 1 --2- 172.21.15.29:0/4267642580 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31bc0cb830 0x7f31bc0c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:26.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31ba800640 1 --2- 172.21.15.29:0/4267642580 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31bc0cf7a0 0x7f31bc1f5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:26.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31b9e00640 1 -- 172.21.15.29:0/4267642580 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31bc0cb830 msgr2=0x7f31bc0c55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:26.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31b9e00640 1 --2- 172.21.15.29:0/4267642580 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31bc0cb830 0x7f31bc0c55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:26.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31b9e00640 1 -- 172.21.15.29:0/4267642580 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31bc0cf7a0 msgr2=0x7f31bc1f5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:26.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31b9e00640 1 --2- 172.21.15.29:0/4267642580 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31bc0cf7a0 0x7f31bc1f5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:26.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31b9e00640 1 -- 172.21.15.29:0/4267642580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31bc0c4840 con 0x7f31bc06c7a0 2024-04-17T20:00:26.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31b9e00640 1 --2- 172.21.15.29:0/4267642580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31bc06c7a0 0x7f31bc0c4300 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f31ac0b3040 tx=0x7f31ac01d4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:26.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31b2000640 1 -- 172.21.15.29:0/4267642580 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ac0b5050 con 0x7f31bc06c7a0 2024-04-17T20:00:26.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31b2000640 1 -- 172.21.15.29:0/4267642580 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31ac0b6040 con 0x7f31bc06c7a0 2024-04-17T20:00:26.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31c1200640 1 -- 172.21.15.29:0/4267642580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31bc1502a0 con 0x7f31bc06c7a0 2024-04-17T20:00:26.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31b2000640 1 -- 172.21.15.29:0/4267642580 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31ac004440 con 0x7f31bc06c7a0 2024-04-17T20:00:26.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.028+0000 7f31c1200640 1 -- 172.21.15.29:0/4267642580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31bc15bc30 con 0x7f31bc06c7a0 2024-04-17T20:00:26.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.032+0000 7f31c1200640 1 -- 172.21.15.29:0/4267642580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31bc069b90 con 0x7f31bc06c7a0 2024-04-17T20:00:26.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.032+0000 7f31b2000640 1 -- 172.21.15.29:0/4267642580 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f31ac0c3060 con 0x7f31bc06c7a0 2024-04-17T20:00:26.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.032+0000 7f31b2000640 1 --2- 172.21.15.29:0/4267642580 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3198068380 0x7f319806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:26.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.032+0000 7f31b2000640 1 -- 172.21.15.29:0/4267642580 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7f31ac1240a0 con 0x7f31bc06c7a0 2024-04-17T20:00:26.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.032+0000 7f31b3e00640 1 --2- 172.21.15.29:0/4267642580 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3198068380 0x7f319806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:26.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.032+0000 7f31b3e00640 1 --2- 172.21.15.29:0/4267642580 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3198068380 0x7f319806a890 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f31bc150c10 tx=0x7f31a4007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:26.035 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.036+0000 7f31b2000640 1 -- 172.21.15.29:0/4267642580 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f31ac0187f0 con 0x7f31bc06c7a0 2024-04-17T20:00:26.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.236+0000 7f31c1200640 1 -- 172.21.15.29:0/4267642580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31bc0cfec0 con 0x7f31bc06c7a0 2024-04-17T20:00:26.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.236+0000 7f31b2000640 1 -- 172.21.15.29:0/4267642580 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+6080 (secure 0 0 0) 0x7f31ac0f3fc0 con 0x7f31bc06c7a0 2024-04-17T20:00:26.236 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:26.236 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:23.943733+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:replay","state_seq":1,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5144,5212,5350,5631]},"id":1}]} 2024-04-17T20:00:26.237 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 50 2024-04-17T20:00:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.240+0000 7f31c1200640 1 -- 172.21.15.29:0/4267642580 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3198068380 msgr2=0x7f319806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.240+0000 7f31c1200640 1 --2- 172.21.15.29:0/4267642580 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3198068380 0x7f319806a890 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f31bc150c10 tx=0x7f31a4007d30 comp rx=0 tx=0).stop 2024-04-17T20:00:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.240+0000 7f31c1200640 1 -- 172.21.15.29:0/4267642580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31bc06c7a0 msgr2=0x7f31bc0c4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.240+0000 7f31c1200640 1 --2- 172.21.15.29:0/4267642580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31bc06c7a0 0x7f31bc0c4300 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f31ac0b3040 tx=0x7f31ac01d4d0 comp rx=0 tx=0).stop 2024-04-17T20:00:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.240+0000 7f31c1200640 1 -- 172.21.15.29:0/4267642580 shutdown_connections 2024-04-17T20:00:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.240+0000 7f31c1200640 1 --2- 172.21.15.29:0/4267642580 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3198068380 0x7f319806a890 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:26.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.240+0000 7f31c1200640 1 --2- 172.21.15.29:0/4267642580 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f31bc0cf7a0 0x7f31bc1f5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:26.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.240+0000 7f31c1200640 1 --2- 172.21.15.29:0/4267642580 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f31bc0cb830 0x7f31bc0c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:26.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.240+0000 7f31c1200640 1 --2- 172.21.15.29:0/4267642580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31bc06c7a0 0x7f31bc0c4300 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:26.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.240+0000 7f31c1200640 1 -- 172.21.15.29:0/4267642580 >> 172.21.15.29:0/4267642580 conn(0x7f31bc06e3d0 msgr2=0x7f31bc155600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:26.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.240+0000 7f31c1200640 1 -- 172.21.15.29:0/4267642580 shutdown_connections 2024-04-17T20:00:26.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:26.240+0000 7f31c1200640 1 -- 172.21.15.29:0/4267642580 wait complete. 2024-04-17T20:00:26.259 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 50, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-17T19:47:49.315515+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5144": { "addr": "172.21.15.37:6839/858161860", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6836", "nonce": 858161860, "type": "v2" }, { "addr": "172.21.15.37:6839", "nonce": 858161860, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5144, "incarnation": 38, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_5212": { "addr": "172.21.15.37:6838/767681664", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6835", "nonce": 767681664, "type": "v2" }, { "addr": "172.21.15.37:6838", "nonce": 767681664, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5212, "incarnation": 26, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 31 }, "gid_5350": { "addr": "172.21.15.37:6837/2495953425", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6834", "nonce": 2495953425, "type": "v2" }, { "addr": "172.21.15.37:6837", "nonce": 2495953425, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5350, "incarnation": 32, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 18 }, "gid_5631": { "addr": "172.21.15.29:6836/2376553580", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6832", "nonce": 2376553580, "type": "v2" }, { "addr": "172.21.15.29:6836", "nonce": 2376553580, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5631, "incarnation": 44, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 42 }, "gid_5775": { "addr": "172.21.15.29:6835/1352350250", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6834", "nonce": 1352350250, "type": "v2" }, { "addr": "172.21.15.29:6835", "nonce": 1352350250, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5775, "incarnation": 50, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 74, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-17T20:00:23.943733+0000", "qdb_cluster": [ 5144, 5212, 5350, 5631 ], "qdb_leader": 5631, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5775, "mds_1": 5631, "mds_2": 5144, "mds_3": 5350, "mds_4": 5212 } } } ], "standbys": [] } 2024-04-17T20:00:28.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:28.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.428+0000 7fcc00e00640 1 Processor -- start 2024-04-17T20:00:28.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.428+0000 7fcc00e00640 1 -- start start 2024-04-17T20:00:28.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.428+0000 7fcc00e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fcbfc0cbf90 0x7fcbfc15ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:28.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.428+0000 7fcc00e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbfc0c7f50 0x7fcbfc0ca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:28.428 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.428+0000 7fcc00e00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbfc15cf70 con 0x7fcbfc155da0 2024-04-17T20:00:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.428+0000 7fcc00e00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fcbfc06bdc0 con 0x7fcbfc0cbf90 2024-04-17T20:00:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.428+0000 7fcc00e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcbfc0cbba0 con 0x7fcbfc0c7f50 2024-04-17T20:00:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.428+0000 7fcbfa800640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcbfc155da0 0x7fcbfc1561b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58344/0 (socket says 172.21.15.29:58344) 2024-04-17T20:00:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.428+0000 7fcbfa800640 1 -- 172.21.15.29:0/2375480071 learned_addr learned my addr 172.21.15.29:0/2375480071 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:00:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.428+0000 7fcbf9e00640 1 --2- 172.21.15.29:0/2375480071 >> v2:172.21.15.37:3300/0 conn(0x7fcbfc0cbf90 0x7fcbfc15ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.428+0000 7fcbf1400640 1 --2- 172.21.15.29:0/2375480071 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbfc0c7f50 0x7fcbfc0ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.428+0000 7fcbf9400640 1 -- 172.21.15.29:0/2375480071 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1576271158 0 0) 0x7fcbfc15cf70 con 0x7fcbfc155da0 2024-04-17T20:00:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.428+0000 7fcbf9400640 1 -- 172.21.15.29:0/2375480071 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbe0003530 con 0x7fcbfc155da0 2024-04-17T20:00:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.428+0000 7fcbf9e00640 1 -- 172.21.15.29:0/2375480071 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbfc0c7f50 msgr2=0x7fcbfc0ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.428+0000 7fcbf9e00640 1 --2- 172.21.15.29:0/2375480071 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbfc0c7f50 0x7fcbfc0ca410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.432+0000 7fcbf9e00640 1 -- 172.21.15.29:0/2375480071 >> v1:172.21.15.29:6789/0 conn(0x7fcbfc155da0 legacy=0x7fcbfc1561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.432+0000 7fcbf9e00640 1 -- 172.21.15.29:0/2375480071 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbfc06c010 con 0x7fcbfc0cbf90 2024-04-17T20:00:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.432+0000 7fcbf1400640 1 --2- 172.21.15.29:0/2375480071 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbfc0c7f50 0x7fcbfc0ca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T20:00:28.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.432+0000 7fcbf9e00640 1 --2- 172.21.15.29:0/2375480071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbfc0cbf90 0x7fcbfc15ca30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009d90 tx=0x7fcbec01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b1401df50bfaf99 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:28.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.432+0000 7fcbf9400640 1 -- 172.21.15.29:0/2375480071 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec0b2070 con 0x7fcbfc0cbf90 2024-04-17T20:00:28.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.432+0000 7fcbf9400640 1 -- 172.21.15.29:0/2375480071 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbec0a5920 con 0x7fcbfc0cbf90 2024-04-17T20:00:28.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.432+0000 7fcbf9400640 1 -- 172.21.15.29:0/2375480071 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec0a5cd0 con 0x7fcbfc0cbf90 2024-04-17T20:00:28.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.436+0000 7fcc00e00640 1 -- 172.21.15.29:0/2375480071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbfc0cbf90 msgr2=0x7fcbfc15ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:28.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.436+0000 7fcc00e00640 1 --2- 172.21.15.29:0/2375480071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbfc0cbf90 0x7fcbfc15ca30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009d90 tx=0x7fcbec01c970 comp rx=0 tx=0).stop 2024-04-17T20:00:28.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.436+0000 7fcc00e00640 1 -- 172.21.15.29:0/2375480071 shutdown_connections 2024-04-17T20:00:28.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.436+0000 7fcc00e00640 1 --2- 172.21.15.29:0/2375480071 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbfc0c7f50 0x7fcbfc0ca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:28.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.436+0000 7fcc00e00640 1 --2- 172.21.15.29:0/2375480071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbfc0cbf90 0x7fcbfc15ca30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:28.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.436+0000 7fcc00e00640 1 -- 172.21.15.29:0/2375480071 >> 172.21.15.29:0/2375480071 conn(0x7fcbfc06fde0 msgr2=0x7fcbfc0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:28.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.436+0000 7fcc00e00640 1 -- 172.21.15.29:0/2375480071 shutdown_connections 2024-04-17T20:00:28.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.436+0000 7fcc00e00640 1 -- 172.21.15.29:0/2375480071 wait complete. 2024-04-17T20:00:28.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.436+0000 7fcc00e00640 1 Processor -- start 2024-04-17T20:00:28.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.436+0000 7fcc00e00640 1 -- start start 2024-04-17T20:00:28.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.436+0000 7fcc00e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbfc0c7f50 0x7fcbfc13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:28.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.436+0000 7fcc00e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbfc0cbf90 0x7fcbfc073300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:28.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.436+0000 7fcc00e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbfc155da0 0x7fcbfc138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:28.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.436+0000 7fcc00e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc0caad0 con 0x7fcbfc155da0 2024-04-17T20:00:28.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.436+0000 7fcc00e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc0ca950 con 0x7fcbfc0cbf90 2024-04-17T20:00:28.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.436+0000 7fcc00e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fcbfc0cb760 con 0x7fcbfc0c7f50 2024-04-17T20:00:28.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.440+0000 7fcbfa800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbfc155da0 0x7fcbfc138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:28.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.440+0000 7fcbfa800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbfc155da0 0x7fcbfc138330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53922/0 (socket says 172.21.15.29:53922) 2024-04-17T20:00:28.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.440+0000 7fcbfa800640 1 -- 172.21.15.29:0/223718122 learned_addr learned my addr 172.21.15.29:0/223718122 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:28.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.440+0000 7fcbf9e00640 1 --2- 172.21.15.29:0/223718122 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbfc0c7f50 0x7fcbfc13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:28.437 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.440+0000 7fcbf1400640 1 --2- 172.21.15.29:0/223718122 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbfc0cbf90 0x7fcbfc073300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:28.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.440+0000 7fcbfa800640 1 -- 172.21.15.29:0/223718122 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbfc0c7f50 msgr2=0x7fcbfc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:28.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.440+0000 7fcbfa800640 1 --2- 172.21.15.29:0/223718122 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbfc0c7f50 0x7fcbfc13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:28.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.440+0000 7fcbfa800640 1 -- 172.21.15.29:0/223718122 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbfc0cbf90 msgr2=0x7fcbfc073300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:28.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.440+0000 7fcbfa800640 1 --2- 172.21.15.29:0/223718122 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbfc0cbf90 0x7fcbfc073300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:28.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.440+0000 7fcbfa800640 1 -- 172.21.15.29:0/223718122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbfc13f2d0 con 0x7fcbfc155da0 2024-04-17T20:00:28.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.440+0000 7fcbfa800640 1 --2- 172.21.15.29:0/223718122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbfc155da0 0x7fcbfc138330 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400c3c0 tx=0x7fcbf400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:28.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.440+0000 7fcbf2a00640 1 -- 172.21.15.29:0/223718122 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4017070 con 0x7fcbfc155da0 2024-04-17T20:00:28.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.440+0000 7fcbf2a00640 1 -- 172.21.15.29:0/223718122 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbf4005cf0 con 0x7fcbfc155da0 2024-04-17T20:00:28.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.440+0000 7fcc00e00640 1 -- 172.21.15.29:0/223718122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbfc15dce0 con 0x7fcbfc155da0 2024-04-17T20:00:28.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.440+0000 7fcbf2a00640 1 -- 172.21.15.29:0/223718122 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf40072d0 con 0x7fcbfc155da0 2024-04-17T20:00:28.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.440+0000 7fcc00e00640 1 -- 172.21.15.29:0/223718122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbfc1410f0 con 0x7fcbfc155da0 2024-04-17T20:00:28.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.440+0000 7fcc00e00640 1 -- 172.21.15.29:0/223718122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbfc069b90 con 0x7fcbfc155da0 2024-04-17T20:00:28.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.440+0000 7fcbf2a00640 1 -- 172.21.15.29:0/223718122 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fcbf401a070 con 0x7fcbfc155da0 2024-04-17T20:00:28.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.444+0000 7fcbf2a00640 1 --2- 172.21.15.29:0/223718122 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbd8068380 0x7fcbd806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:28.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.444+0000 7fcbf2a00640 1 -- 172.21.15.29:0/223718122 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7fcbf4086260 con 0x7fcbfc155da0 2024-04-17T20:00:28.441 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.444+0000 7fcbf9e00640 1 --2- 172.21.15.29:0/223718122 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbd8068380 0x7fcbd806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:28.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.444+0000 7fcbf9e00640 1 --2- 172.21.15.29:0/223718122 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbd8068380 0x7fcbd806a890 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0af000 tx=0x7fcbec0a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:28.445 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.448+0000 7fcbf2a00640 1 -- 172.21.15.29:0/223718122 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fcbf4055be0 con 0x7fcbfc155da0 2024-04-17T20:00:28.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.648+0000 7fcc00e00640 1 -- 172.21.15.29:0/223718122 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcbfc15da70 con 0x7fcbfc155da0 2024-04-17T20:00:28.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.648+0000 7fcbf2a00640 1 -- 172.21.15.29:0/223718122 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+6080 (secure 0 0 0) 0x7fcbf4059ee0 con 0x7fcbfc155da0 2024-04-17T20:00:28.647 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:28.647 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:23.943733+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:replay","state_seq":1,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5144,5212,5350,5631]},"id":1}]} 2024-04-17T20:00:28.648 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 50 2024-04-17T20:00:28.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.648+0000 7fcc00e00640 1 -- 172.21.15.29:0/223718122 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbd8068380 msgr2=0x7fcbd806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:28.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.648+0000 7fcc00e00640 1 --2- 172.21.15.29:0/223718122 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbd8068380 0x7fcbd806a890 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fcbec0af000 tx=0x7fcbec0a7070 comp rx=0 tx=0).stop 2024-04-17T20:00:28.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.648+0000 7fcc00e00640 1 -- 172.21.15.29:0/223718122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbfc155da0 msgr2=0x7fcbfc138330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:28.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.648+0000 7fcc00e00640 1 --2- 172.21.15.29:0/223718122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbfc155da0 0x7fcbfc138330 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fcbf400c3c0 tx=0x7fcbf400c880 comp rx=0 tx=0).stop 2024-04-17T20:00:28.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.648+0000 7fcc00e00640 1 -- 172.21.15.29:0/223718122 shutdown_connections 2024-04-17T20:00:28.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.648+0000 7fcc00e00640 1 --2- 172.21.15.29:0/223718122 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fcbd8068380 0x7fcbd806a890 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:28.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.648+0000 7fcc00e00640 1 --2- 172.21.15.29:0/223718122 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbfc155da0 0x7fcbfc138330 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:28.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.648+0000 7fcc00e00640 1 --2- 172.21.15.29:0/223718122 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fcbfc0cbf90 0x7fcbfc073300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:28.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.648+0000 7fcc00e00640 1 --2- 172.21.15.29:0/223718122 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fcbfc0c7f50 0x7fcbfc13ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:28.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.648+0000 7fcc00e00640 1 -- 172.21.15.29:0/223718122 >> 172.21.15.29:0/223718122 conn(0x7fcbfc06fde0 msgr2=0x7fcbfc0cda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:28.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.648+0000 7fcc00e00640 1 -- 172.21.15.29:0/223718122 shutdown_connections 2024-04-17T20:00:28.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:28.648+0000 7fcc00e00640 1 -- 172.21.15.29:0/223718122 wait complete. 2024-04-17T20:00:28.669 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:00:30.670 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:30.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.832+0000 7f551f600640 1 Processor -- start 2024-04-17T20:00:30.829 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.832+0000 7f551f600640 1 -- start start 2024-04-17T20:00:30.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.832+0000 7f551f600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5518160090 0x7f5518167900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:30.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.832+0000 7f551f600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f551806c900 con 0x7f55180caf80 2024-04-17T20:00:30.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.832+0000 7f551f600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f551806c690 con 0x7f551815fc80 2024-04-17T20:00:30.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.832+0000 7f551f600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f551815c050 con 0x7f5518160090 2024-04-17T20:00:30.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.832+0000 7f551cc00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f55180caf80 0x7f551815c9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47966/0 (socket says 172.21.15.29:47966) 2024-04-17T20:00:30.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.832+0000 7f551cc00640 1 -- 172.21.15.29:0/1917279301 learned_addr learned my addr 172.21.15.29:0/1917279301 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:00:30.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.832+0000 7f551d600640 1 --2- 172.21.15.29:0/1917279301 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5518160090 0x7f5518167900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.832+0000 7f5517400640 1 -- 172.21.15.29:0/1917279301 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3782164467 0 0) 0x7f550c001120 con 0x7f55180caf80 2024-04-17T20:00:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.832+0000 7f5517400640 1 -- 172.21.15.29:0/1917279301 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5500003530 con 0x7f55180caf80 2024-04-17T20:00:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.832+0000 7f5517400640 1 -- 172.21.15.29:0/1917279301 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 916246502 0 0) 0x7f551806c690 con 0x7f551815fc80 2024-04-17T20:00:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.832+0000 7f5517400640 1 -- 172.21.15.29:0/1917279301 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f550c001120 con 0x7f551815fc80 2024-04-17T20:00:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.832+0000 7f551d600640 1 -- 172.21.15.29:0/1917279301 >> v1:172.21.15.37:6789/0 conn(0x7f551815fc80 legacy=0x7f55180cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.832+0000 7f551d600640 1 -- 172.21.15.29:0/1917279301 >> v1:172.21.15.29:6789/0 conn(0x7f55180caf80 legacy=0x7f551815c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.832+0000 7f551d600640 1 -- 172.21.15.29:0/1917279301 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55181604a0 con 0x7f5518160090 2024-04-17T20:00:30.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.836+0000 7f551d600640 1 --2- 172.21.15.29:0/1917279301 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5518160090 0x7f5518167900 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f551806c860 tx=0x7f551001f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e3fa7b171380724 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:30.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.836+0000 7f5517400640 1 -- 172.21.15.29:0/1917279301 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55100b4070 con 0x7f5518160090 2024-04-17T20:00:30.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.836+0000 7f5517400640 1 -- 172.21.15.29:0/1917279301 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55100af600 con 0x7f5518160090 2024-04-17T20:00:30.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.836+0000 7f5517400640 1 -- 172.21.15.29:0/1917279301 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55100af9b0 con 0x7f5518160090 2024-04-17T20:00:30.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.840+0000 7f551f600640 1 -- 172.21.15.29:0/1917279301 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5518160090 msgr2=0x7f5518167900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:30.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.840+0000 7f551f600640 1 --2- 172.21.15.29:0/1917279301 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5518160090 0x7f5518167900 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f551806c860 tx=0x7f551001f170 comp rx=0 tx=0).stop 2024-04-17T20:00:30.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.840+0000 7f551f600640 1 -- 172.21.15.29:0/1917279301 shutdown_connections 2024-04-17T20:00:30.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.840+0000 7f551f600640 1 --2- 172.21.15.29:0/1917279301 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5518160090 0x7f5518167900 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:30.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.840+0000 7f551f600640 1 -- 172.21.15.29:0/1917279301 >> 172.21.15.29:0/1917279301 conn(0x7f551806e3d0 msgr2=0x7f551806e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:30.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.840+0000 7f551f600640 1 -- 172.21.15.29:0/1917279301 shutdown_connections 2024-04-17T20:00:30.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.840+0000 7f551f600640 1 -- 172.21.15.29:0/1917279301 wait complete. 2024-04-17T20:00:30.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.840+0000 7f551f600640 1 Processor -- start 2024-04-17T20:00:30.841 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.844+0000 7f551f600640 1 -- start start 2024-04-17T20:00:30.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.844+0000 7f551f600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55180caf80 0x7f551815a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:30.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.844+0000 7f551f600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f551815fc80 0x7f5518152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:30.844 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.844+0000 7f551f600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5518160090 0x7f5518152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.844+0000 7f551f600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f551806d140 con 0x7f55180caf80 2024-04-17T20:00:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.844+0000 7f551f600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f551806ce60 con 0x7f551815fc80 2024-04-17T20:00:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.844+0000 7f551f600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f551806d420 con 0x7f5518160090 2024-04-17T20:00:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.844+0000 7f551cc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55180caf80 0x7f551815a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.844+0000 7f551cc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55180caf80 0x7f551815a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36080/0 (socket says 172.21.15.29:36080) 2024-04-17T20:00:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.844+0000 7f551cc00640 1 -- 172.21.15.29:0/3084519163 learned_addr learned my addr 172.21.15.29:0/3084519163 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.844+0000 7f5517e00640 1 --2- 172.21.15.29:0/3084519163 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f551815fc80 0x7f5518152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.844+0000 7f551d600640 1 --2- 172.21.15.29:0/3084519163 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5518160090 0x7f5518152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.844+0000 7f551cc00640 1 -- 172.21.15.29:0/3084519163 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5518160090 msgr2=0x7f5518152b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.844+0000 7f551cc00640 1 --2- 172.21.15.29:0/3084519163 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5518160090 0x7f5518152b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.844+0000 7f551cc00640 1 -- 172.21.15.29:0/3084519163 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f551815fc80 msgr2=0x7f5518152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.844+0000 7f551cc00640 1 --2- 172.21.15.29:0/3084519163 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f551815fc80 0x7f5518152650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.844+0000 7f551cc00640 1 -- 172.21.15.29:0/3084519163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f551806c730 con 0x7f55180caf80 2024-04-17T20:00:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.844+0000 7f551cc00640 1 --2- 172.21.15.29:0/3084519163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55180caf80 0x7f551815a900 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f550c00c700 tx=0x7f550c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.848+0000 7f5514c00640 1 -- 172.21.15.29:0/3084519163 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c017070 con 0x7f55180caf80 2024-04-17T20:00:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.848+0000 7f5514c00640 1 -- 172.21.15.29:0/3084519163 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f550c005ae0 con 0x7f55180caf80 2024-04-17T20:00:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.848+0000 7f551f600640 1 -- 172.21.15.29:0/3084519163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55181e8410 con 0x7f55180caf80 2024-04-17T20:00:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.848+0000 7f5514c00640 1 -- 172.21.15.29:0/3084519163 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c007070 con 0x7f55180caf80 2024-04-17T20:00:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.848+0000 7f551f600640 1 -- 172.21.15.29:0/3084519163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55181f57d0 con 0x7f55180caf80 2024-04-17T20:00:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.848+0000 7f551f600640 1 -- 172.21.15.29:0/3084519163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54f40052a0 con 0x7f55180caf80 2024-04-17T20:00:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.848+0000 7f5514c00640 1 -- 172.21.15.29:0/3084519163 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f550c01a070 con 0x7f55180caf80 2024-04-17T20:00:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.852+0000 7f5514c00640 1 --2- 172.21.15.29:0/3084519163 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54f8068380 0x7f54f806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.852+0000 7f5514c00640 1 -- 172.21.15.29:0/3084519163 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7f550c086350 con 0x7f55180caf80 2024-04-17T20:00:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.852+0000 7f5517e00640 1 --2- 172.21.15.29:0/3084519163 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54f8068380 0x7f54f806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.852+0000 7f5517e00640 1 --2- 172.21.15.29:0/3084519163 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54f8068380 0x7f54f806a890 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f55181559c0 tx=0x7f5504009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:30.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:30.860+0000 7f5514c00640 1 -- 172.21.15.29:0/3084519163 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f551806c900 con 0x7f55180caf80 2024-04-17T20:00:31.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:31.064+0000 7f551f600640 1 -- 172.21.15.29:0/3084519163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54f4005860 con 0x7f55180caf80 2024-04-17T20:00:31.065 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:31.068+0000 7f5514c00640 1 -- 172.21.15.29:0/3084519163 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+6080 (secure 0 0 0) 0x7f550c055d40 con 0x7f55180caf80 2024-04-17T20:00:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:31.065 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:23.943733+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:replay","state_seq":1,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5144,5212,5350,5631]},"id":1}]} 2024-04-17T20:00:31.065 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 50 2024-04-17T20:00:31.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:31.068+0000 7f551f600640 1 -- 172.21.15.29:0/3084519163 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54f8068380 msgr2=0x7f54f806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:31.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:31.068+0000 7f551f600640 1 --2- 172.21.15.29:0/3084519163 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54f8068380 0x7f54f806a890 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f55181559c0 tx=0x7f5504009040 comp rx=0 tx=0).stop 2024-04-17T20:00:31.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:31.068+0000 7f551f600640 1 -- 172.21.15.29:0/3084519163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55180caf80 msgr2=0x7f551815a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:31.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:31.068+0000 7f551f600640 1 --2- 172.21.15.29:0/3084519163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55180caf80 0x7f551815a900 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f550c00c700 tx=0x7f550c00cbc0 comp rx=0 tx=0).stop 2024-04-17T20:00:31.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:31.068+0000 7f551f600640 1 -- 172.21.15.29:0/3084519163 shutdown_connections 2024-04-17T20:00:31.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:31.068+0000 7f551f600640 1 --2- 172.21.15.29:0/3084519163 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f54f8068380 0x7f54f806a890 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:31.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:31.068+0000 7f551f600640 1 --2- 172.21.15.29:0/3084519163 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5518160090 0x7f5518152b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:31.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:31.068+0000 7f551f600640 1 --2- 172.21.15.29:0/3084519163 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f551815fc80 0x7f5518152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:31.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:31.068+0000 7f551f600640 1 --2- 172.21.15.29:0/3084519163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55180caf80 0x7f551815a900 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:31.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:31.068+0000 7f551f600640 1 -- 172.21.15.29:0/3084519163 >> 172.21.15.29:0/3084519163 conn(0x7f551806e3d0 msgr2=0x7f551815c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:31.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:31.068+0000 7f551f600640 1 -- 172.21.15.29:0/3084519163 shutdown_connections 2024-04-17T20:00:31.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:31.072+0000 7f551f600640 1 -- 172.21.15.29:0/3084519163 wait complete. 2024-04-17T20:00:31.097 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:00:33.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.264+0000 7fee38800640 1 Processor -- start 2024-04-17T20:00:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.264+0000 7fee38800640 1 -- start start 2024-04-17T20:00:33.262 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.264+0000 7fee38800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee300cc8c0 0x7fee300cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.264+0000 7fee38800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fee30159390 0x7fee301597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.264+0000 7fee38800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee300c88b0 con 0x7fee300ca330 2024-04-17T20:00:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.264+0000 7fee38800640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fee300c6ce0 con 0x7fee30159390 2024-04-17T20:00:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.264+0000 7fee38800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fee300c6fc0 con 0x7fee300cc8c0 2024-04-17T20:00:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.264+0000 7fee35400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fee30159390 0x7fee301597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.264+0000 7fee35400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fee30159390 0x7fee301597a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36082/0 (socket says 172.21.15.29:36082) 2024-04-17T20:00:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.264+0000 7fee35400640 1 -- 172.21.15.29:0/1692490648 learned_addr learned my addr 172.21.15.29:0/1692490648 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.264+0000 7fee35e00640 1 --2- 172.21.15.29:0/1692490648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee300cc8c0 0x7fee300cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.264+0000 7fee35400640 1 -- 172.21.15.29:0/1692490648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee300cc8c0 msgr2=0x7fee300cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:33.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.264+0000 7fee35400640 1 --2- 172.21.15.29:0/1692490648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee300cc8c0 0x7fee300cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:33.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.264+0000 7fee35400640 1 -- 172.21.15.29:0/1692490648 >> v1:172.21.15.37:6789/0 conn(0x7fee300ca330 legacy=0x7fee3015c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:33.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.264+0000 7fee35400640 1 -- 172.21.15.29:0/1692490648 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee300c72a0 con 0x7fee30159390 2024-04-17T20:00:33.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.268+0000 7fee35400640 1 --2- 172.21.15.29:0/1692490648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee30159390 0x7fee301597a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fee20009870 tx=0x7fee2001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f5e956a37d0052 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:33.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.268+0000 7fee34a00640 1 -- 172.21.15.29:0/1692490648 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee200b2070 con 0x7fee30159390 2024-04-17T20:00:33.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.268+0000 7fee34a00640 1 -- 172.21.15.29:0/1692490648 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee200a9070 con 0x7fee30159390 2024-04-17T20:00:33.266 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.268+0000 7fee34a00640 1 -- 172.21.15.29:0/1692490648 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee200ad880 con 0x7fee30159390 2024-04-17T20:00:33.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.268+0000 7fee38800640 1 -- 172.21.15.29:0/1692490648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee30159390 msgr2=0x7fee301597a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:33.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.268+0000 7fee38800640 1 --2- 172.21.15.29:0/1692490648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee30159390 0x7fee301597a0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fee20009870 tx=0x7fee2001c560 comp rx=0 tx=0).stop 2024-04-17T20:00:33.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.268+0000 7fee38800640 1 -- 172.21.15.29:0/1692490648 shutdown_connections 2024-04-17T20:00:33.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.268+0000 7fee38800640 1 --2- 172.21.15.29:0/1692490648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee30159390 0x7fee301597a0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:33.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.268+0000 7fee38800640 1 --2- 172.21.15.29:0/1692490648 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee300cc8c0 0x7fee300cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:33.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.268+0000 7fee38800640 1 -- 172.21.15.29:0/1692490648 >> 172.21.15.29:0/1692490648 conn(0x7fee3006fde0 msgr2=0x7fee300701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:33.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.268+0000 7fee38800640 1 -- 172.21.15.29:0/1692490648 shutdown_connections 2024-04-17T20:00:33.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.268+0000 7fee38800640 1 -- 172.21.15.29:0/1692490648 wait complete. 2024-04-17T20:00:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.272+0000 7fee38800640 1 Processor -- start 2024-04-17T20:00:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.272+0000 7fee38800640 1 -- start start 2024-04-17T20:00:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.272+0000 7fee38800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee300ca330 0x7fee30154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.272+0000 7fee38800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fee300cc8c0 0x7fee3015dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.272+0000 7fee38800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee30159390 0x7fee30139ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.272+0000 7fee38800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fee300c6a00 con 0x7fee30159390 2024-04-17T20:00:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.272+0000 7fee38800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fee300c6880 con 0x7fee300cc8c0 2024-04-17T20:00:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.272+0000 7fee38800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fee3006bdc0 con 0x7fee300ca330 2024-04-17T20:00:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.272+0000 7fee36800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee30159390 0x7fee30139ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.272+0000 7fee35e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee300ca330 0x7fee30154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.272+0000 7fee36800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee30159390 0x7fee30139ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36086/0 (socket says 172.21.15.29:36086) 2024-04-17T20:00:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.272+0000 7fee36800640 1 -- 172.21.15.29:0/177085856 learned_addr learned my addr 172.21.15.29:0/177085856 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.272+0000 7fee35400640 1 --2- 172.21.15.29:0/177085856 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fee300cc8c0 0x7fee3015dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.272+0000 7fee36800640 1 -- 172.21.15.29:0/177085856 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee300ca330 msgr2=0x7fee30154a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.272+0000 7fee36800640 1 --2- 172.21.15.29:0/177085856 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee300ca330 0x7fee30154a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.272+0000 7fee36800640 1 -- 172.21.15.29:0/177085856 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fee300cc8c0 msgr2=0x7fee3015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.272+0000 7fee36800640 1 --2- 172.21.15.29:0/177085856 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fee300cc8c0 0x7fee3015dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.272+0000 7fee36800640 1 -- 172.21.15.29:0/177085856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee30154f70 con 0x7fee30159390 2024-04-17T20:00:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.276+0000 7fee36800640 1 --2- 172.21.15.29:0/177085856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee30159390 0x7fee30139ad0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00c760 tx=0x7fee2c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.276+0000 7fee2a000640 1 -- 172.21.15.29:0/177085856 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c003b80 con 0x7fee30159390 2024-04-17T20:00:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.276+0000 7fee2a000640 1 -- 172.21.15.29:0/177085856 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee2c003d20 con 0x7fee30159390 2024-04-17T20:00:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.276+0000 7fee2a000640 1 -- 172.21.15.29:0/177085856 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2c00aa30 con 0x7fee30159390 2024-04-17T20:00:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.276+0000 7fee38800640 1 -- 172.21.15.29:0/177085856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee30154610 con 0x7fee30159390 2024-04-17T20:00:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.276+0000 7fee38800640 1 -- 172.21.15.29:0/177085856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee301f9370 con 0x7fee30159390 2024-04-17T20:00:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.276+0000 7fee38800640 1 -- 172.21.15.29:0/177085856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee30069b90 con 0x7fee30159390 2024-04-17T20:00:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.276+0000 7fee2a000640 1 -- 172.21.15.29:0/177085856 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fee2c00abd0 con 0x7fee30159390 2024-04-17T20:00:33.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.280+0000 7fee2a000640 1 --2- 172.21.15.29:0/177085856 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fee140683b0 0x7fee1406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:33.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.280+0000 7fee2a000640 1 -- 172.21.15.29:0/177085856 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7fee2c086cd0 con 0x7fee30159390 2024-04-17T20:00:33.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.280+0000 7fee35e00640 1 --2- 172.21.15.29:0/177085856 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fee140683b0 0x7fee1406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.280+0000 7fee35e00640 1 --2- 172.21.15.29:0/177085856 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fee140683b0 0x7fee1406a8c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fee24005c10 tx=0x7fee2400b750 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.284+0000 7fee2a000640 1 -- 172.21.15.29:0/177085856 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fee2c056620 con 0x7fee30159390 2024-04-17T20:00:33.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.484+0000 7fee38800640 1 -- 172.21.15.29:0/177085856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee300c7e10 con 0x7fee30159390 2024-04-17T20:00:33.483 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.484+0000 7fee2a000640 1 -- 172.21.15.29:0/177085856 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+6080 (secure 0 0 0) 0x7fee2c05a920 con 0x7fee30159390 2024-04-17T20:00:33.483 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:33.483 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:23.943733+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:replay","state_seq":1,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5144,5212,5350,5631]},"id":1}]} 2024-04-17T20:00:33.484 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 50 2024-04-17T20:00:33.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.488+0000 7fee38800640 1 -- 172.21.15.29:0/177085856 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fee140683b0 msgr2=0x7fee1406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:33.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.488+0000 7fee38800640 1 --2- 172.21.15.29:0/177085856 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fee140683b0 0x7fee1406a8c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fee24005c10 tx=0x7fee2400b750 comp rx=0 tx=0).stop 2024-04-17T20:00:33.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.488+0000 7fee38800640 1 -- 172.21.15.29:0/177085856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee30159390 msgr2=0x7fee30139ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:33.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.488+0000 7fee38800640 1 --2- 172.21.15.29:0/177085856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee30159390 0x7fee30139ad0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fee2c00c760 tx=0x7fee2c00cc20 comp rx=0 tx=0).stop 2024-04-17T20:00:33.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.488+0000 7fee38800640 1 -- 172.21.15.29:0/177085856 shutdown_connections 2024-04-17T20:00:33.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.488+0000 7fee38800640 1 --2- 172.21.15.29:0/177085856 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fee140683b0 0x7fee1406a8c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:33.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.488+0000 7fee38800640 1 --2- 172.21.15.29:0/177085856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee30159390 0x7fee30139ad0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:33.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.488+0000 7fee38800640 1 --2- 172.21.15.29:0/177085856 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fee300cc8c0 0x7fee3015dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:33.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.488+0000 7fee38800640 1 --2- 172.21.15.29:0/177085856 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fee300ca330 0x7fee30154a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:33.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.488+0000 7fee38800640 1 -- 172.21.15.29:0/177085856 >> 172.21.15.29:0/177085856 conn(0x7fee3006fde0 msgr2=0x7fee3006c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:33.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.488+0000 7fee38800640 1 -- 172.21.15.29:0/177085856 shutdown_connections 2024-04-17T20:00:33.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:33.488+0000 7fee38800640 1 -- 172.21.15.29:0/177085856 wait complete. 2024-04-17T20:00:33.507 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:00:35.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:35.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.664+0000 7f4933000640 1 Processor -- start 2024-04-17T20:00:35.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.664+0000 7f4933000640 1 -- start start 2024-04-17T20:00:35.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.664+0000 7f4933000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f492c159c90 0x7f492c0ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:35.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.664+0000 7f4933000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f492c06c660 0x7f492c06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:35.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.664+0000 7f4933000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f492c15a0a0 con 0x7f492c0ca7f0 2024-04-17T20:00:35.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.664+0000 7f4933000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f492c15abe0 con 0x7f492c06c660 2024-04-17T20:00:35.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.664+0000 7f4933000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f492c15a340 con 0x7f492c159c90 2024-04-17T20:00:35.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.664+0000 7f4931000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f492c06c660 0x7f492c06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:35.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.664+0000 7f4931000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f492c06c660 0x7f492c06ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36098/0 (socket says 172.21.15.29:36098) 2024-04-17T20:00:35.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.664+0000 7f4931000640 1 -- 172.21.15.29:0/2865671910 learned_addr learned my addr 172.21.15.29:0/2865671910 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:35.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.664+0000 7f492be00640 1 --2- 172.21.15.29:0/2865671910 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f492c159c90 0x7f492c0ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:35.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.664+0000 7f4931000640 1 -- 172.21.15.29:0/2865671910 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f492c159c90 msgr2=0x7f492c0ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:35.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.664+0000 7f4931000640 1 --2- 172.21.15.29:0/2865671910 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f492c159c90 0x7f492c0ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:35.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.664+0000 7f4931000640 1 -- 172.21.15.29:0/2865671910 >> v1:172.21.15.37:6789/0 conn(0x7f492c0ca7f0 legacy=0x7f492c0c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:35.664 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.664+0000 7f4931000640 1 -- 172.21.15.29:0/2865671910 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f492c15a620 con 0x7f492c06c660 2024-04-17T20:00:35.665 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.668+0000 7f4931000640 1 --2- 172.21.15.29:0/2865671910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f492c06c660 0x7f492c06ca70 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4924009e50 tx=0x7f492401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933b8971068cf03b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:35.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.668+0000 7f492aa00640 1 -- 172.21.15.29:0/2865671910 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49240a5810 con 0x7f492c06c660 2024-04-17T20:00:35.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.668+0000 7f492aa00640 1 -- 172.21.15.29:0/2865671910 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49240a59b0 con 0x7f492c06c660 2024-04-17T20:00:35.666 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.668+0000 7f492aa00640 1 -- 172.21.15.29:0/2865671910 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49240a5d80 con 0x7f492c06c660 2024-04-17T20:00:35.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.668+0000 7f4933000640 1 -- 172.21.15.29:0/2865671910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f492c06c660 msgr2=0x7f492c06ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:35.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.668+0000 7f4933000640 1 --2- 172.21.15.29:0/2865671910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f492c06c660 0x7f492c06ca70 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f4924009e50 tx=0x7f492401ca30 comp rx=0 tx=0).stop 2024-04-17T20:00:35.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.668+0000 7f4933000640 1 -- 172.21.15.29:0/2865671910 shutdown_connections 2024-04-17T20:00:35.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.668+0000 7f4933000640 1 --2- 172.21.15.29:0/2865671910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f492c06c660 0x7f492c06ca70 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:35.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.668+0000 7f4933000640 1 --2- 172.21.15.29:0/2865671910 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f492c159c90 0x7f492c0ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:35.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.668+0000 7f4933000640 1 -- 172.21.15.29:0/2865671910 >> 172.21.15.29:0/2865671910 conn(0x7f492c06fde0 msgr2=0x7f492c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:35.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.668+0000 7f4933000640 1 -- 172.21.15.29:0/2865671910 shutdown_connections 2024-04-17T20:00:35.668 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.668+0000 7f4933000640 1 -- 172.21.15.29:0/2865671910 wait complete. 2024-04-17T20:00:35.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f4933000640 1 Processor -- start 2024-04-17T20:00:35.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f4933000640 1 -- start start 2024-04-17T20:00:35.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f4933000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f492c06c660 0x7f492c0c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:35.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f4933000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f492c0ca7f0 0x7f492c14ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:35.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f4933000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f492c159c90 0x7f492c0cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:35.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f4933000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f492c0c7680 con 0x7f492c06c660 2024-04-17T20:00:35.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f4933000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f492c06cfb0 con 0x7f492c0ca7f0 2024-04-17T20:00:35.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f4933000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f492c0c7800 con 0x7f492c159c90 2024-04-17T20:00:35.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f492be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f492c06c660 0x7f492c0c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:35.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f492be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f492c06c660 0x7f492c0c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36106/0 (socket says 172.21.15.29:36106) 2024-04-17T20:00:35.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f492be00640 1 -- 172.21.15.29:0/654452181 learned_addr learned my addr 172.21.15.29:0/654452181 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:35.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f492b400640 1 --2- 172.21.15.29:0/654452181 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f492c0ca7f0 0x7f492c14ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:35.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f4931000640 1 --2- 172.21.15.29:0/654452181 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f492c159c90 0x7f492c0cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:35.673 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f492be00640 1 -- 172.21.15.29:0/654452181 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f492c159c90 msgr2=0x7f492c0cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:35.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f492be00640 1 --2- 172.21.15.29:0/654452181 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f492c159c90 0x7f492c0cb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:35.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f492be00640 1 -- 172.21.15.29:0/654452181 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f492c0ca7f0 msgr2=0x7f492c14ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:35.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f492be00640 1 --2- 172.21.15.29:0/654452181 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f492c0ca7f0 0x7f492c14ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:35.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f492be00640 1 -- 172.21.15.29:0/654452181 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f492c0c5fc0 con 0x7f492c06c660 2024-04-17T20:00:35.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f4931000640 1 --2- 172.21.15.29:0/654452181 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f492c159c90 0x7f492c0cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:00:35.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f492be00640 1 --2- 172.21.15.29:0/654452181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f492c06c660 0x7f492c0c5a80 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f491c00d5d0 tx=0x7f491c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:35.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f4923e00640 1 -- 172.21.15.29:0/654452181 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491c004240 con 0x7f492c06c660 2024-04-17T20:00:35.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f4923e00640 1 -- 172.21.15.29:0/654452181 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f491c0043e0 con 0x7f492c06c660 2024-04-17T20:00:35.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f4923e00640 1 -- 172.21.15.29:0/654452181 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f491c003840 con 0x7f492c06c660 2024-04-17T20:00:35.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.672+0000 7f4933000640 1 -- 172.21.15.29:0/654452181 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f492c0c61d0 con 0x7f492c06c660 2024-04-17T20:00:35.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.676+0000 7f4933000640 1 -- 172.21.15.29:0/654452181 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f492c0c4cc0 con 0x7f492c06c660 2024-04-17T20:00:35.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.676+0000 7f4933000640 1 -- 172.21.15.29:0/654452181 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f492c069b90 con 0x7f492c06c660 2024-04-17T20:00:35.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.676+0000 7f4923e00640 1 -- 172.21.15.29:0/654452181 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f491c0039e0 con 0x7f492c06c660 2024-04-17T20:00:35.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.676+0000 7f4923e00640 1 --2- 172.21.15.29:0/654452181 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f490c068380 0x7f490c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:35.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.676+0000 7f4923e00640 1 -- 172.21.15.29:0/654452181 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7f491c087250 con 0x7f492c06c660 2024-04-17T20:00:35.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.676+0000 7f492b400640 1 --2- 172.21.15.29:0/654452181 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f490c068380 0x7f490c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:35.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.680+0000 7f492b400640 1 --2- 172.21.15.29:0/654452181 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f490c068380 0x7f490c06a890 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f492c1f9d80 tx=0x7f4918009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:35.679 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.680+0000 7f4923e00640 1 -- 172.21.15.29:0/654452181 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f491c057160 con 0x7f492c06c660 2024-04-17T20:00:35.878 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.880+0000 7f4933000640 1 -- 172.21.15.29:0/654452181 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f492c0c3e60 con 0x7f492c06c660 2024-04-17T20:00:35.879 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.880+0000 7f4923e00640 1 -- 172.21.15.29:0/654452181 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6082 (secure 0 0 0) 0x7f491c056a60 con 0x7f492c06c660 2024-04-17T20:00:35.879 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:35.879 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:34.297471+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:resolve","state_seq":31,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5631,5350,5212,5144]},"id":1}]} 2024-04-17T20:00:35.879 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 51 2024-04-17T20:00:35.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.884+0000 7f4933000640 1 -- 172.21.15.29:0/654452181 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f490c068380 msgr2=0x7f490c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:35.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.884+0000 7f4933000640 1 --2- 172.21.15.29:0/654452181 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f490c068380 0x7f490c06a890 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f492c1f9d80 tx=0x7f4918009040 comp rx=0 tx=0).stop 2024-04-17T20:00:35.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.884+0000 7f4933000640 1 -- 172.21.15.29:0/654452181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f492c06c660 msgr2=0x7f492c0c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:35.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.884+0000 7f4933000640 1 --2- 172.21.15.29:0/654452181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f492c06c660 0x7f492c0c5a80 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f491c00d5d0 tx=0x7f491c00da90 comp rx=0 tx=0).stop 2024-04-17T20:00:35.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.884+0000 7f4933000640 1 -- 172.21.15.29:0/654452181 shutdown_connections 2024-04-17T20:00:35.881 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.884+0000 7f4933000640 1 --2- 172.21.15.29:0/654452181 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f490c068380 0x7f490c06a890 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:35.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.884+0000 7f4933000640 1 --2- 172.21.15.29:0/654452181 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f492c159c90 0x7f492c0cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:35.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.884+0000 7f4933000640 1 --2- 172.21.15.29:0/654452181 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f492c0ca7f0 0x7f492c14ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:35.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.884+0000 7f4933000640 1 --2- 172.21.15.29:0/654452181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f492c06c660 0x7f492c0c5a80 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:35.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.884+0000 7f4933000640 1 -- 172.21.15.29:0/654452181 >> 172.21.15.29:0/654452181 conn(0x7f492c06fde0 msgr2=0x7f492c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:35.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.884+0000 7f4933000640 1 -- 172.21.15.29:0/654452181 shutdown_connections 2024-04-17T20:00:35.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:35.884+0000 7f4933000640 1 -- 172.21.15.29:0/654452181 wait complete. 2024-04-17T20:00:35.901 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:00:37.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:38.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.068+0000 7fa17b800640 1 Processor -- start 2024-04-17T20:00:38.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.068+0000 7fa17b800640 1 -- start start 2024-04-17T20:00:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.072+0000 7fa17b800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa1740cccd0 0x7fa1740ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.072+0000 7fa17b800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1740ca870 0x7fa1741635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.072+0000 7fa17b800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1740cd0e0 con 0x7fa1740cc8c0 2024-04-17T20:00:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.072+0000 7fa17b800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa1740c88b0 con 0x7fa1740cccd0 2024-04-17T20:00:38.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.072+0000 7fa17b800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa1740c6ce0 con 0x7fa1740ca870 2024-04-17T20:00:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.072+0000 7fa178e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa1740cc8c0 0x7fa174159390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47972/0 (socket says 172.21.15.29:47972) 2024-04-17T20:00:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.072+0000 7fa178e00640 1 -- 172.21.15.29:0/2879995748 learned_addr learned my addr 172.21.15.29:0/2879995748 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:00:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.072+0000 7fa173e00640 1 --2- 172.21.15.29:0/2879995748 >> v2:172.21.15.37:3300/0 conn(0x7fa1740cccd0 0x7fa1740ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.072+0000 7fa179800640 1 --2- 172.21.15.29:0/2879995748 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1740ca870 0x7fa1741635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.072+0000 7fa173400640 1 -- 172.21.15.29:0/2879995748 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1934743362 0 0) 0x7fa1740cd0e0 con 0x7fa1740cc8c0 2024-04-17T20:00:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.072+0000 7fa173400640 1 -- 172.21.15.29:0/2879995748 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa15c003530 con 0x7fa1740cc8c0 2024-04-17T20:00:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.072+0000 7fa173400640 1 -- 172.21.15.29:0/2879995748 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1482337693 0 0) 0x7fa15c003530 con 0x7fa1740cc8c0 2024-04-17T20:00:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.072+0000 7fa173e00640 1 -- 172.21.15.29:0/2879995748 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1740ca870 msgr2=0x7fa1741635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.072+0000 7fa173e00640 1 --2- 172.21.15.29:0/2879995748 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1740ca870 0x7fa1741635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.072+0000 7fa173e00640 1 -- 172.21.15.29:0/2879995748 >> v1:172.21.15.29:6789/0 conn(0x7fa1740cc8c0 legacy=0x7fa174159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:38.071 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.072+0000 7fa173e00640 1 -- 172.21.15.29:0/2879995748 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1740c6fc0 con 0x7fa1740cccd0 2024-04-17T20:00:38.073 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.076+0000 7fa173e00640 1 --2- 172.21.15.29:0/2879995748 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1740cccd0 0x7fa1740ca330 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa1600097c0 tx=0x7fa16001c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eeb40ba2a2034b75 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:38.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.076+0000 7fa173400640 1 -- 172.21.15.29:0/2879995748 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1600b0070 con 0x7fa1740cccd0 2024-04-17T20:00:38.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.076+0000 7fa173400640 1 -- 172.21.15.29:0/2879995748 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1600ab400 con 0x7fa1740cccd0 2024-04-17T20:00:38.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.076+0000 7fa173400640 1 -- 172.21.15.29:0/2879995748 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa168001680 con 0x7fa1740cccd0 2024-04-17T20:00:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.076+0000 7fa17b800640 1 -- 172.21.15.29:0/2879995748 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1740cccd0 msgr2=0x7fa1740ca330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.076+0000 7fa17b800640 1 --2- 172.21.15.29:0/2879995748 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1740cccd0 0x7fa1740ca330 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa1600097c0 tx=0x7fa16001c3c0 comp rx=0 tx=0).stop 2024-04-17T20:00:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.076+0000 7fa17b800640 1 -- 172.21.15.29:0/2879995748 shutdown_connections 2024-04-17T20:00:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.076+0000 7fa17b800640 1 --2- 172.21.15.29:0/2879995748 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1740ca870 0x7fa1741635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.076+0000 7fa17b800640 1 --2- 172.21.15.29:0/2879995748 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1740cccd0 0x7fa1740ca330 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.076+0000 7fa17b800640 1 -- 172.21.15.29:0/2879995748 >> 172.21.15.29:0/2879995748 conn(0x7fa17406fde0 msgr2=0x7fa1740701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.076+0000 7fa17b800640 1 -- 172.21.15.29:0/2879995748 shutdown_connections 2024-04-17T20:00:38.075 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.076+0000 7fa17b800640 1 -- 172.21.15.29:0/2879995748 wait complete. 2024-04-17T20:00:38.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa17b800640 1 Processor -- start 2024-04-17T20:00:38.077 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa17b800640 1 -- start start 2024-04-17T20:00:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa17b800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1740ca870 0x7fa174165380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa17b800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1740cc8c0 0x7fa17413dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa17b800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1740cccd0 0x7fa174153550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa17b800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa174159c20 con 0x7fa1740cccd0 2024-04-17T20:00:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa17b800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa174159aa0 con 0x7fa1740cc8c0 2024-04-17T20:00:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa17b800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa1740c6880 con 0x7fa1740ca870 2024-04-17T20:00:38.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa179800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1740cccd0 0x7fa174153550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa179800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1740cccd0 0x7fa174153550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36112/0 (socket says 172.21.15.29:36112) 2024-04-17T20:00:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa179800640 1 -- 172.21.15.29:0/2239875147 learned_addr learned my addr 172.21.15.29:0/2239875147 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa178e00640 1 --2- 172.21.15.29:0/2239875147 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1740ca870 0x7fa174165380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa173e00640 1 --2- 172.21.15.29:0/2239875147 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1740cc8c0 0x7fa17413dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa179800640 1 -- 172.21.15.29:0/2239875147 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1740ca870 msgr2=0x7fa174165380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa179800640 1 --2- 172.21.15.29:0/2239875147 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1740ca870 0x7fa174165380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa179800640 1 -- 172.21.15.29:0/2239875147 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1740cc8c0 msgr2=0x7fa17413dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa179800640 1 --2- 172.21.15.29:0/2239875147 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1740cc8c0 0x7fa17413dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa179800640 1 -- 172.21.15.29:0/2239875147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1741658c0 con 0x7fa1740cccd0 2024-04-17T20:00:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa173e00640 1 --2- 172.21.15.29:0/2239875147 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1740cc8c0 0x7fa17413dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:00:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa179800640 1 --2- 172.21.15.29:0/2239875147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1740cccd0 0x7fa174153550 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa16c009cb0 tx=0x7fa16c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa170c00640 1 -- 172.21.15.29:0/2239875147 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c018020 con 0x7fa1740cccd0 2024-04-17T20:00:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa170c00640 1 -- 172.21.15.29:0/2239875147 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa16c014480 con 0x7fa1740cccd0 2024-04-17T20:00:38.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa17b800640 1 -- 172.21.15.29:0/2239875147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1741f8800 con 0x7fa1740cccd0 2024-04-17T20:00:38.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa170c00640 1 -- 172.21.15.29:0/2239875147 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16c009e60 con 0x7fa1740cccd0 2024-04-17T20:00:38.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.080+0000 7fa17b800640 1 -- 172.21.15.29:0/2239875147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa174154000 con 0x7fa1740cccd0 2024-04-17T20:00:38.083 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.084+0000 7fa17b800640 1 -- 172.21.15.29:0/2239875147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa174069b90 con 0x7fa1740cccd0 2024-04-17T20:00:38.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.084+0000 7fa170c00640 1 -- 172.21.15.29:0/2239875147 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa16c01b020 con 0x7fa1740cccd0 2024-04-17T20:00:38.084 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.084+0000 7fa170c00640 1 --2- 172.21.15.29:0/2239875147 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa1540683b0 0x7fa15406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:38.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.084+0000 7fa170c00640 1 -- 172.21.15.29:0/2239875147 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7fa16c087de0 con 0x7fa1740cccd0 2024-04-17T20:00:38.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.084+0000 7fa178e00640 1 --2- 172.21.15.29:0/2239875147 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa1540683b0 0x7fa15406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:38.085 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.088+0000 7fa178e00640 1 --2- 172.21.15.29:0/2239875147 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa1540683b0 0x7fa15406a8c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa168007950 tx=0x7fa168009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:38.087 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.088+0000 7fa170c00640 1 -- 172.21.15.29:0/2239875147 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa16c022020 con 0x7fa1740cccd0 2024-04-17T20:00:38.292 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.296+0000 7fa17b800640 1 -- 172.21.15.29:0/2239875147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1740c7e10 con 0x7fa1740cccd0 2024-04-17T20:00:38.293 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.296+0000 7fa170c00640 1 -- 172.21.15.29:0/2239875147 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6082 (secure 0 0 0) 0x7fa16c010040 con 0x7fa1740cccd0 2024-04-17T20:00:38.293 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:38.293 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:34.297471+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:resolve","state_seq":31,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5631,5350,5212,5144]},"id":1}]} 2024-04-17T20:00:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 51 2024-04-17T20:00:38.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.296+0000 7fa17b800640 1 -- 172.21.15.29:0/2239875147 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa1540683b0 msgr2=0x7fa15406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:38.295 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.296+0000 7fa17b800640 1 --2- 172.21.15.29:0/2239875147 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa1540683b0 0x7fa15406a8c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa168007950 tx=0x7fa168009040 comp rx=0 tx=0).stop 2024-04-17T20:00:38.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.296+0000 7fa17b800640 1 -- 172.21.15.29:0/2239875147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1740cccd0 msgr2=0x7fa174153550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:38.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.296+0000 7fa17b800640 1 --2- 172.21.15.29:0/2239875147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1740cccd0 0x7fa174153550 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa16c009cb0 tx=0x7fa16c00c490 comp rx=0 tx=0).stop 2024-04-17T20:00:38.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.296+0000 7fa17b800640 1 -- 172.21.15.29:0/2239875147 shutdown_connections 2024-04-17T20:00:38.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.296+0000 7fa17b800640 1 --2- 172.21.15.29:0/2239875147 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa1540683b0 0x7fa15406a8c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:38.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.296+0000 7fa17b800640 1 --2- 172.21.15.29:0/2239875147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1740cccd0 0x7fa174153550 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:38.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.296+0000 7fa17b800640 1 --2- 172.21.15.29:0/2239875147 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa1740cc8c0 0x7fa17413dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:38.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.296+0000 7fa17b800640 1 --2- 172.21.15.29:0/2239875147 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa1740ca870 0x7fa174165380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:38.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.296+0000 7fa17b800640 1 -- 172.21.15.29:0/2239875147 >> 172.21.15.29:0/2239875147 conn(0x7fa17406fde0 msgr2=0x7fa1740c4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:38.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.296+0000 7fa17b800640 1 -- 172.21.15.29:0/2239875147 shutdown_connections 2024-04-17T20:00:38.296 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:38.296+0000 7fa17b800640 1 -- 172.21.15.29:0/2239875147 wait complete. 2024-04-17T20:00:38.316 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 51, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-17T19:47:49.315515+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 51, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5144": { "addr": "172.21.15.37:6839/858161860", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6836", "nonce": 858161860, "type": "v2" }, { "addr": "172.21.15.37:6839", "nonce": 858161860, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5144, "incarnation": 38, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_5212": { "addr": "172.21.15.37:6838/767681664", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6835", "nonce": 767681664, "type": "v2" }, { "addr": "172.21.15.37:6838", "nonce": 767681664, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5212, "incarnation": 26, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 31 }, "gid_5350": { "addr": "172.21.15.37:6837/2495953425", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6834", "nonce": 2495953425, "type": "v2" }, { "addr": "172.21.15.37:6837", "nonce": 2495953425, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5350, "incarnation": 32, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 18 }, "gid_5631": { "addr": "172.21.15.29:6836/2376553580", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6832", "nonce": 2376553580, "type": "v2" }, { "addr": "172.21.15.29:6836", "nonce": 2376553580, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5631, "incarnation": 44, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 42 }, "gid_5775": { "addr": "172.21.15.29:6835/1352350250", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6834", "nonce": 1352350250, "type": "v2" }, { "addr": "172.21.15.29:6835", "nonce": 1352350250, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5775, "incarnation": 50, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:resolve", "state_seq": 31 } }, "last_failure": 0, "last_failure_osd_epoch": 74, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-17T20:00:34.297471+0000", "qdb_cluster": [ 5631, 5350, 5212, 5144 ], "qdb_leader": 5631, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5775, "mds_1": 5631, "mds_2": 5144, "mds_3": 5350, "mds_4": 5212 } } } ], "standbys": [] } 2024-04-17T20:00:40.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:40.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.484+0000 7f2f66e00640 1 Processor -- start 2024-04-17T20:00:40.482 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.484+0000 7f2f66e00640 1 -- start start 2024-04-17T20:00:40.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.488+0000 7f2f66e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f600cc8c0 0x7f2f600cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:40.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.488+0000 7f2f66e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f60159390 0x7f2f601597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:40.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.488+0000 7f2f66e00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f600c88b0 con 0x7f2f600ca330 2024-04-17T20:00:40.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.488+0000 7f2f66e00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f2f600c6ce0 con 0x7f2f600cc8c0 2024-04-17T20:00:40.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.488+0000 7f2f66e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2f600c6fc0 con 0x7f2f60159390 2024-04-17T20:00:40.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.488+0000 7f2f5fe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f600cc8c0 0x7f2f600cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:40.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.488+0000 7f2f64e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2f600ca330 0x7f2f6015c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49142/0 (socket says 172.21.15.29:49142) 2024-04-17T20:00:40.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.488+0000 7f2f64e00640 1 -- 172.21.15.29:0/4269004382 learned_addr learned my addr 172.21.15.29:0/4269004382 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:00:40.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.488+0000 7f2f5f400640 1 --2- 172.21.15.29:0/4269004382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f60159390 0x7f2f601597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:40.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.488+0000 7f2f5fe00640 1 -- 172.21.15.29:0/4269004382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f60159390 msgr2=0x7f2f601597a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:40.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.488+0000 7f2f5fe00640 1 --2- 172.21.15.29:0/4269004382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f60159390 0x7f2f601597a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:40.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.488+0000 7f2f5fe00640 1 -- 172.21.15.29:0/4269004382 >> v1:172.21.15.29:6789/0 conn(0x7f2f600ca330 legacy=0x7f2f6015c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:40.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.488+0000 7f2f5fe00640 1 -- 172.21.15.29:0/4269004382 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f600c72a0 con 0x7f2f600cc8c0 2024-04-17T20:00:40.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.488+0000 7f2f5f400640 1 --2- 172.21.15.29:0/4269004382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f60159390 0x7f2f601597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:00:40.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.488+0000 7f2f5ea00640 1 -- 172.21.15.29:0/4269004382 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1343756958 0 0) 0x7f2f600c88b0 con 0x7f2f600ca330 2024-04-17T20:00:40.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.492+0000 7f2f5fe00640 1 --2- 172.21.15.29:0/4269004382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f600cc8c0 0x7f2f600cccd0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f2f500097c0 tx=0x7f2f5001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f884f8cde55b76 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:40.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.492+0000 7f2f5ea00640 1 -- 172.21.15.29:0/4269004382 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f500b0070 con 0x7f2f600cc8c0 2024-04-17T20:00:40.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.492+0000 7f2f5ea00640 1 -- 172.21.15.29:0/4269004382 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f500ab600 con 0x7f2f600cc8c0 2024-04-17T20:00:40.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.492+0000 7f2f5ea00640 1 -- 172.21.15.29:0/4269004382 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f500ab9d0 con 0x7f2f600cc8c0 2024-04-17T20:00:40.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.492+0000 7f2f66e00640 1 -- 172.21.15.29:0/4269004382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f600cc8c0 msgr2=0x7f2f600cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:40.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.492+0000 7f2f66e00640 1 --2- 172.21.15.29:0/4269004382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f600cc8c0 0x7f2f600cccd0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f2f500097c0 tx=0x7f2f5001c4a0 comp rx=0 tx=0).stop 2024-04-17T20:00:40.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.492+0000 7f2f66e00640 1 -- 172.21.15.29:0/4269004382 shutdown_connections 2024-04-17T20:00:40.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.492+0000 7f2f66e00640 1 --2- 172.21.15.29:0/4269004382 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f60159390 0x7f2f601597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:40.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.492+0000 7f2f66e00640 1 --2- 172.21.15.29:0/4269004382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f600cc8c0 0x7f2f600cccd0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:40.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.492+0000 7f2f66e00640 1 -- 172.21.15.29:0/4269004382 >> 172.21.15.29:0/4269004382 conn(0x7f2f6006fde0 msgr2=0x7f2f600701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:40.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.492+0000 7f2f66e00640 1 -- 172.21.15.29:0/4269004382 shutdown_connections 2024-04-17T20:00:40.490 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.492+0000 7f2f66e00640 1 -- 172.21.15.29:0/4269004382 wait complete. 2024-04-17T20:00:40.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f66e00640 1 Processor -- start 2024-04-17T20:00:40.492 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f66e00640 1 -- start start 2024-04-17T20:00:40.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f66e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f600ca330 0x7f2f6015a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:40.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f66e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2f600cc8c0 0x7f2f601f9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:40.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f66e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f60159390 0x7f2f600cbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:40.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f66e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2f600c6a00 con 0x7f2f60159390 2024-04-17T20:00:40.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f66e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2f600c6880 con 0x7f2f600cc8c0 2024-04-17T20:00:40.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f66e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2f60166600 con 0x7f2f600ca330 2024-04-17T20:00:40.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f64e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f60159390 0x7f2f600cbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:40.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f64e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f60159390 0x7f2f600cbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56050/0 (socket says 172.21.15.29:56050) 2024-04-17T20:00:40.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f64e00640 1 -- 172.21.15.29:0/1005615328 learned_addr learned my addr 172.21.15.29:0/1005615328 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:40.495 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f5fe00640 1 --2- 172.21.15.29:0/1005615328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f600ca330 0x7f2f6015a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:40.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f5f400640 1 --2- 172.21.15.29:0/1005615328 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2f600cc8c0 0x7f2f601f9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:40.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f64e00640 1 -- 172.21.15.29:0/1005615328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f600ca330 msgr2=0x7f2f6015a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:40.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f64e00640 1 --2- 172.21.15.29:0/1005615328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f600ca330 0x7f2f6015a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:40.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f64e00640 1 -- 172.21.15.29:0/1005615328 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2f600cc8c0 msgr2=0x7f2f601f9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:40.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f64e00640 1 --2- 172.21.15.29:0/1005615328 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2f600cc8c0 0x7f2f601f9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:40.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f64e00640 1 -- 172.21.15.29:0/1005615328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f601f9690 con 0x7f2f60159390 2024-04-17T20:00:40.496 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f64e00640 1 --2- 172.21.15.29:0/1005615328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f60159390 0x7f2f600cbb20 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800c700 tx=0x7f2f5800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:40.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f57e00640 1 -- 172.21.15.29:0/1005615328 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f58018070 con 0x7f2f60159390 2024-04-17T20:00:40.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f57e00640 1 -- 172.21.15.29:0/1005615328 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f5800ce10 con 0x7f2f60159390 2024-04-17T20:00:40.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f57e00640 1 -- 172.21.15.29:0/1005615328 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f5800a820 con 0x7f2f60159390 2024-04-17T20:00:40.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f66e00640 1 -- 172.21.15.29:0/1005615328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f6006be30 con 0x7f2f60159390 2024-04-17T20:00:40.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.496+0000 7f2f66e00640 1 -- 172.21.15.29:0/1005615328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f6015bd80 con 0x7f2f60159390 2024-04-17T20:00:40.497 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.500+0000 7f2f66e00640 1 -- 172.21.15.29:0/1005615328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f60069b90 con 0x7f2f60159390 2024-04-17T20:00:40.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.500+0000 7f2f57e00640 1 -- 172.21.15.29:0/1005615328 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2f5800a9c0 con 0x7f2f60159390 2024-04-17T20:00:40.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.500+0000 7f2f57e00640 1 --2- 172.21.15.29:0/1005615328 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2f40068380 0x7f2f4006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:40.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.500+0000 7f2f57e00640 1 -- 172.21.15.29:0/1005615328 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7f2f580863d0 con 0x7f2f60159390 2024-04-17T20:00:40.500 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.504+0000 7f2f5fe00640 1 --2- 172.21.15.29:0/1005615328 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2f40068380 0x7f2f4006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:40.501 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.504+0000 7f2f5fe00640 1 --2- 172.21.15.29:0/1005615328 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2f40068380 0x7f2f4006a890 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2f500ad000 tx=0x7f2f500a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:40.515 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.512+0000 7f2f57e00640 1 -- 172.21.15.29:0/1005615328 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2f58055d20 con 0x7f2f60159390 2024-04-17T20:00:40.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.712+0000 7f2f66e00640 1 -- 172.21.15.29:0/1005615328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2f600c7e10 con 0x7f2f60159390 2024-04-17T20:00:40.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.716+0000 7f2f57e00640 1 -- 172.21.15.29:0/1005615328 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+6084 (secure 0 0 0) 0x7f2f5805c020 con 0x7f2f60159390 2024-04-17T20:00:40.713 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:40.713 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:40.402611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:reconnect","state_seq":33,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5144,5212,5350,5631]},"id":1}]} 2024-04-17T20:00:40.714 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 52 2024-04-17T20:00:40.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.716+0000 7f2f66e00640 1 -- 172.21.15.29:0/1005615328 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2f40068380 msgr2=0x7f2f4006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:40.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.716+0000 7f2f66e00640 1 --2- 172.21.15.29:0/1005615328 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2f40068380 0x7f2f4006a890 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2f500ad000 tx=0x7f2f500a6000 comp rx=0 tx=0).stop 2024-04-17T20:00:40.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.716+0000 7f2f66e00640 1 -- 172.21.15.29:0/1005615328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f60159390 msgr2=0x7f2f600cbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:40.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.716+0000 7f2f66e00640 1 --2- 172.21.15.29:0/1005615328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f60159390 0x7f2f600cbb20 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f2f5800c700 tx=0x7f2f5800cbc0 comp rx=0 tx=0).stop 2024-04-17T20:00:40.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.716+0000 7f2f66e00640 1 -- 172.21.15.29:0/1005615328 shutdown_connections 2024-04-17T20:00:40.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.716+0000 7f2f66e00640 1 --2- 172.21.15.29:0/1005615328 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2f40068380 0x7f2f4006a890 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:40.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.716+0000 7f2f66e00640 1 --2- 172.21.15.29:0/1005615328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f60159390 0x7f2f600cbb20 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:40.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.716+0000 7f2f66e00640 1 --2- 172.21.15.29:0/1005615328 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2f600cc8c0 0x7f2f601f9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:40.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.716+0000 7f2f66e00640 1 --2- 172.21.15.29:0/1005615328 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2f600ca330 0x7f2f6015a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:40.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.716+0000 7f2f66e00640 1 -- 172.21.15.29:0/1005615328 >> 172.21.15.29:0/1005615328 conn(0x7f2f6006fde0 msgr2=0x7f2f600cd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:40.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.716+0000 7f2f66e00640 1 -- 172.21.15.29:0/1005615328 shutdown_connections 2024-04-17T20:00:40.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:40.716+0000 7f2f66e00640 1 -- 172.21.15.29:0/1005615328 wait complete. 2024-04-17T20:00:40.736 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:00:42.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:42.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.904+0000 7feec3e00640 1 Processor -- start 2024-04-17T20:00:42.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.904+0000 7feec3e00640 1 -- start start 2024-04-17T20:00:42.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.904+0000 7feec3e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feebc0cccd0 0x7feebc0ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:42.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.904+0000 7feec3e00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feebc0cd0e0 con 0x7feebc0cc8c0 2024-04-17T20:00:42.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feec3e00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feebc0c88b0 con 0x7feebc0ca870 2024-04-17T20:00:42.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feec3e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7feebc0c6ce0 con 0x7feebc0cccd0 2024-04-17T20:00:42.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feec1400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7feebc0cc8c0 0x7feebc159390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49146/0 (socket says 172.21.15.29:49146) 2024-04-17T20:00:42.904 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feec1400640 1 -- 172.21.15.29:0/3205503332 learned_addr learned my addr 172.21.15.29:0/3205503332 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:00:42.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feec0a00640 1 --2- 172.21.15.29:0/3205503332 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feebc0cccd0 0x7feebc0ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:42.906 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feebbe00640 1 -- 172.21.15.29:0/3205503332 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4072251427 0 0) 0x7feebc0cd0e0 con 0x7feebc0cc8c0 2024-04-17T20:00:42.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feebbe00640 1 -- 172.21.15.29:0/3205503332 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feea4003530 con 0x7feebc0cc8c0 2024-04-17T20:00:42.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feebbe00640 1 -- 172.21.15.29:0/3205503332 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3289680966 0 0) 0x7feebc0c88b0 con 0x7feebc0ca870 2024-04-17T20:00:42.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feebbe00640 1 -- 172.21.15.29:0/3205503332 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feebc0cd0e0 con 0x7feebc0ca870 2024-04-17T20:00:42.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feebbe00640 1 -- 172.21.15.29:0/3205503332 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3030448990 0 0) 0x7feea4003530 con 0x7feebc0cc8c0 2024-04-17T20:00:42.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feebbe00640 1 -- 172.21.15.29:0/3205503332 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feebc0cccd0 msgr2=0x7feebc0ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:42.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feebbe00640 1 --2- 172.21.15.29:0/3205503332 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feebc0cccd0 0x7feebc0ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:42.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feebbe00640 1 -- 172.21.15.29:0/3205503332 >> v1:172.21.15.37:6789/0 conn(0x7feebc0ca870 legacy=0x7feebc1635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:42.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feebbe00640 1 -- 172.21.15.29:0/3205503332 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feebc0c6fc0 con 0x7feebc0cc8c0 2024-04-17T20:00:42.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feebbe00640 1 -- 172.21.15.29:0/3205503332 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7feeb0001680 con 0x7feebc0cc8c0 2024-04-17T20:00:42.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feebbe00640 1 -- 172.21.15.29:0/3205503332 >> v1:172.21.15.29:6789/0 conn(0x7feebc0cc8c0 legacy=0x7feebc159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:42.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feebbe00640 1 --2- 172.21.15.29:0/3205503332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feea4003bf0 0x7feea4004040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:42.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feebbe00640 1 --2- 172.21.15.29:0/3205503332 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feea4004810 0x7feea4004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:42.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feebbe00640 1 --2- 172.21.15.29:0/3205503332 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feebc0cccd0 0x7feea4005280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:42.907 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feebbe00640 1 -- 172.21.15.29:0/3205503332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feebc0c6ce0 con 0x7feea4003bf0 2024-04-17T20:00:42.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feebbe00640 1 -- 172.21.15.29:0/3205503332 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7feea400bb60 con 0x7feea4004810 2024-04-17T20:00:42.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feebbe00640 1 -- 172.21.15.29:0/3205503332 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7feea400bd60 con 0x7feebc0cccd0 2024-04-17T20:00:42.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feec1400640 1 --2- 172.21.15.29:0/3205503332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feea4003bf0 0x7feea4004040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:42.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feec0a00640 1 --2- 172.21.15.29:0/3205503332 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feea4004810 0x7feea4004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:42.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feec1e00640 1 --2- 172.21.15.29:0/3205503332 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feebc0cccd0 0x7feea4005280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:42.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feec1400640 1 -- 172.21.15.29:0/3205503332 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feebc0cccd0 msgr2=0x7feea4005280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:42.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feec1400640 1 --2- 172.21.15.29:0/3205503332 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feebc0cccd0 0x7feea4005280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:42.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feec1400640 1 -- 172.21.15.29:0/3205503332 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feea4004810 msgr2=0x7feea4004c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:42.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feec1400640 1 --2- 172.21.15.29:0/3205503332 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feea4004810 0x7feea4004c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:42.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feec1400640 1 -- 172.21.15.29:0/3205503332 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feea4003530 con 0x7feea4003bf0 2024-04-17T20:00:42.908 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.908+0000 7feec0a00640 1 --2- 172.21.15.29:0/3205503332 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feea4004810 0x7feea4004c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:00:42.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.912+0000 7feec1400640 1 --2- 172.21.15.29:0/3205503332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feea4003bf0 0x7feea4004040 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7feeb000c190 tx=0x7feeb001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=206a30d635b04468 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:42.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.912+0000 7feebbe00640 1 -- 172.21.15.29:0/3205503332 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb0007cd0 con 0x7feea4003bf0 2024-04-17T20:00:42.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.912+0000 7feebbe00640 1 -- 172.21.15.29:0/3205503332 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feeb00a9020 con 0x7feea4003bf0 2024-04-17T20:00:42.909 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.912+0000 7feebbe00640 1 -- 172.21.15.29:0/3205503332 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feeb0005ae0 con 0x7feea4003bf0 2024-04-17T20:00:42.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.912+0000 7feec3e00640 1 -- 172.21.15.29:0/3205503332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feea4003bf0 msgr2=0x7feea4004040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:42.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.912+0000 7feec3e00640 1 --2- 172.21.15.29:0/3205503332 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feea4003bf0 0x7feea4004040 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7feeb000c190 tx=0x7feeb001cc20 comp rx=0 tx=0).stop 2024-04-17T20:00:42.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.912+0000 7feec1e00640 1 -- 172.21.15.29:0/3205503332 reap_dead start 2024-04-17T20:00:42.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.912+0000 7feec3e00640 1 -- 172.21.15.29:0/3205503332 shutdown_connections 2024-04-17T20:00:42.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.912+0000 7feec3e00640 1 -- 172.21.15.29:0/3205503332 >> 172.21.15.29:0/3205503332 conn(0x7feebc06fde0 msgr2=0x7feebc0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:42.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.912+0000 7feec3e00640 1 -- 172.21.15.29:0/3205503332 shutdown_connections 2024-04-17T20:00:42.910 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.912+0000 7feec3e00640 1 -- 172.21.15.29:0/3205503332 wait complete. 2024-04-17T20:00:42.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec3e00640 1 Processor -- start 2024-04-17T20:00:42.912 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec3e00640 1 -- start start 2024-04-17T20:00:42.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec3e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feea4003bf0 0x7feebc15ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:42.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feebc15c8c0 0x7feebc15db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:42.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec3e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feebc13dfd0 0x7feebc13e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:42.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec3e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feebc159c20 con 0x7feebc15c8c0 2024-04-17T20:00:42.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec3e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7feebc159aa0 con 0x7feea4003bf0 2024-04-17T20:00:42.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec3e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7feebc0c6880 con 0x7feebc13dfd0 2024-04-17T20:00:42.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec0a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feebc15c8c0 0x7feebc15db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:42.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec0a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feebc15c8c0 0x7feebc15db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56074/0 (socket says 172.21.15.29:56074) 2024-04-17T20:00:42.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec0a00640 1 -- 172.21.15.29:0/229440980 learned_addr learned my addr 172.21.15.29:0/229440980 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:42.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec1400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feea4003bf0 0x7feebc15ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:42.915 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec1e00640 1 --2- 172.21.15.29:0/229440980 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feebc13dfd0 0x7feebc13e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:42.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec0a00640 1 -- 172.21.15.29:0/229440980 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feebc13dfd0 msgr2=0x7feebc13e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:42.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec0a00640 1 --2- 172.21.15.29:0/229440980 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feebc13dfd0 0x7feebc13e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:42.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec0a00640 1 -- 172.21.15.29:0/229440980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feea4003bf0 msgr2=0x7feebc15ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:42.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec0a00640 1 --2- 172.21.15.29:0/229440980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feea4003bf0 0x7feebc15ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:42.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec0a00640 1 -- 172.21.15.29:0/229440980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feebc1f59b0 con 0x7feebc15c8c0 2024-04-17T20:00:42.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec1e00640 1 --2- 172.21.15.29:0/229440980 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feebc13dfd0 0x7feebc13e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:00:42.916 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec0a00640 1 --2- 172.21.15.29:0/229440980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feebc15c8c0 0x7feebc15db50 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7feea80116d0 tx=0x7feea8011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:42.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feeb9600640 1 -- 172.21.15.29:0/229440980 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea8004220 con 0x7feebc15c8c0 2024-04-17T20:00:42.917 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feeb9600640 1 -- 172.21.15.29:0/229440980 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feea80043c0 con 0x7feebc15c8c0 2024-04-17T20:00:42.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feeb9600640 1 -- 172.21.15.29:0/229440980 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea800f9c0 con 0x7feebc15c8c0 2024-04-17T20:00:42.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec3e00640 1 -- 172.21.15.29:0/229440980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feebc15b390 con 0x7feebc15c8c0 2024-04-17T20:00:42.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.916+0000 7feec3e00640 1 -- 172.21.15.29:0/229440980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feebc15a650 con 0x7feebc15c8c0 2024-04-17T20:00:42.918 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.920+0000 7feec3e00640 1 -- 172.21.15.29:0/229440980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feebc069b90 con 0x7feebc15c8c0 2024-04-17T20:00:42.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.920+0000 7feeb9600640 1 -- 172.21.15.29:0/229440980 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7feea800fb60 con 0x7feebc15c8c0 2024-04-17T20:00:42.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.920+0000 7feeb9600640 1 --2- 172.21.15.29:0/229440980 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fee9c0683b0 0x7fee9c06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:42.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.920+0000 7feeb9600640 1 -- 172.21.15.29:0/229440980 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7feea80876d0 con 0x7feebc15c8c0 2024-04-17T20:00:42.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.920+0000 7feec1400640 1 --2- 172.21.15.29:0/229440980 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fee9c0683b0 0x7fee9c06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:42.919 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.920+0000 7feec1400640 1 --2- 172.21.15.29:0/229440980 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fee9c0683b0 0x7fee9c06a8c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7feeb000c190 tx=0x7feeb00b1040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:42.923 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:42.924+0000 7feeb9600640 1 -- 172.21.15.29:0/229440980 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7feea8019030 con 0x7feebc15c8c0 2024-04-17T20:00:43.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:43.128+0000 7feec3e00640 1 -- 172.21.15.29:0/229440980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feebc0c50c0 con 0x7feebc15c8c0 2024-04-17T20:00:43.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:43.128+0000 7feeb9600640 1 -- 172.21.15.29:0/229440980 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+6084 (secure 0 0 0) 0x7feea8056f90 con 0x7feebc15c8c0 2024-04-17T20:00:43.125 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:43.126 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:40.402611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:reconnect","state_seq":33,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5144,5212,5350,5631]},"id":1}]} 2024-04-17T20:00:43.126 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 52 2024-04-17T20:00:43.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:43.128+0000 7feec3e00640 1 -- 172.21.15.29:0/229440980 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fee9c0683b0 msgr2=0x7fee9c06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:43.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:43.128+0000 7feec3e00640 1 --2- 172.21.15.29:0/229440980 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fee9c0683b0 0x7fee9c06a8c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7feeb000c190 tx=0x7feeb00b1040 comp rx=0 tx=0).stop 2024-04-17T20:00:43.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:43.128+0000 7feec3e00640 1 -- 172.21.15.29:0/229440980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feebc15c8c0 msgr2=0x7feebc15db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:43.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:43.128+0000 7feec3e00640 1 --2- 172.21.15.29:0/229440980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feebc15c8c0 0x7feebc15db50 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7feea80116d0 tx=0x7feea8011b90 comp rx=0 tx=0).stop 2024-04-17T20:00:43.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:43.128+0000 7feec3e00640 1 -- 172.21.15.29:0/229440980 shutdown_connections 2024-04-17T20:00:43.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:43.128+0000 7feec3e00640 1 --2- 172.21.15.29:0/229440980 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fee9c0683b0 0x7fee9c06a8c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:43.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:43.128+0000 7feec3e00640 1 --2- 172.21.15.29:0/229440980 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feebc13dfd0 0x7feebc13e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:43.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:43.128+0000 7feec3e00640 1 --2- 172.21.15.29:0/229440980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feebc15c8c0 0x7feebc15db50 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:43.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:43.128+0000 7feec3e00640 1 --2- 172.21.15.29:0/229440980 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feea4003bf0 0x7feebc15ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:43.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:43.128+0000 7feec3e00640 1 -- 172.21.15.29:0/229440980 >> 172.21.15.29:0/229440980 conn(0x7feebc06fde0 msgr2=0x7feebc1417d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:43.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:43.128+0000 7feec3e00640 1 -- 172.21.15.29:0/229440980 shutdown_connections 2024-04-17T20:00:43.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:43.128+0000 7feec3e00640 1 -- 172.21.15.29:0/229440980 wait complete. 2024-04-17T20:00:43.148 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:00:45.148 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:45.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.316+0000 7f913bc00640 1 Processor -- start 2024-04-17T20:00:45.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.316+0000 7f913bc00640 1 -- start start 2024-04-17T20:00:45.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.316+0000 7f913bc00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f91341625e0 0x7f9134164a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:45.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.316+0000 7f913bc00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91340cd0e0 con 0x7f91340cc8c0 2024-04-17T20:00:45.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.316+0000 7f913bc00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91340c88b0 con 0x7f91340cccd0 2024-04-17T20:00:45.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.316+0000 7f913bc00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f91340c6ce0 con 0x7f91341625e0 2024-04-17T20:00:45.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.316+0000 7f9139200640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f91340cc8c0 0x7f9134159390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49148/0 (socket says 172.21.15.29:49148) 2024-04-17T20:00:45.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.316+0000 7f9139200640 1 -- 172.21.15.29:0/2083834749 learned_addr learned my addr 172.21.15.29:0/2083834749 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:00:45.315 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.316+0000 7f9139c00640 1 --2- 172.21.15.29:0/2083834749 >> v2:172.21.15.29:3300/0 conn(0x7f91341625e0 0x7f9134164a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:45.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.316+0000 7f9133400640 1 -- 172.21.15.29:0/2083834749 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1136736973 0 0) 0x7f91340cd0e0 con 0x7f91340cc8c0 2024-04-17T20:00:45.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.316+0000 7f9133400640 1 -- 172.21.15.29:0/2083834749 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f911c005740 con 0x7f91340cc8c0 2024-04-17T20:00:45.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.316+0000 7f9139c00640 1 -- 172.21.15.29:0/2083834749 >> v1:172.21.15.37:6789/0 conn(0x7f91340cccd0 legacy=0x7f91340ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:45.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.316+0000 7f9139c00640 1 -- 172.21.15.29:0/2083834749 >> v1:172.21.15.29:6789/0 conn(0x7f91340cc8c0 legacy=0x7f9134159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:45.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.316+0000 7f9139c00640 1 -- 172.21.15.29:0/2083834749 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91340c6fc0 con 0x7f91341625e0 2024-04-17T20:00:45.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.320+0000 7f9139c00640 1 --2- 172.21.15.29:0/2083834749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91341625e0 0x7f9134164a00 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f912c009d90 tx=0x7f912c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d226275cc52f5fef server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:45.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.320+0000 7f9133400640 1 -- 172.21.15.29:0/2083834749 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c0b2070 con 0x7f91341625e0 2024-04-17T20:00:45.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.320+0000 7f9133400640 1 -- 172.21.15.29:0/2083834749 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f912c0a5730 con 0x7f91341625e0 2024-04-17T20:00:45.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.320+0000 7f9133400640 1 -- 172.21.15.29:0/2083834749 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f912c0a5ae0 con 0x7f91341625e0 2024-04-17T20:00:45.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.320+0000 7f913bc00640 1 -- 172.21.15.29:0/2083834749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91341625e0 msgr2=0x7f9134164a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:45.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.320+0000 7f913bc00640 1 --2- 172.21.15.29:0/2083834749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91341625e0 0x7f9134164a00 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f912c009d90 tx=0x7f912c01c970 comp rx=0 tx=0).stop 2024-04-17T20:00:45.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.320+0000 7f913bc00640 1 -- 172.21.15.29:0/2083834749 shutdown_connections 2024-04-17T20:00:45.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.320+0000 7f913bc00640 1 --2- 172.21.15.29:0/2083834749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91341625e0 0x7f9134164a00 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:45.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.320+0000 7f913bc00640 1 -- 172.21.15.29:0/2083834749 >> 172.21.15.29:0/2083834749 conn(0x7f913406fde0 msgr2=0x7f91340701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:45.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.320+0000 7f913bc00640 1 -- 172.21.15.29:0/2083834749 shutdown_connections 2024-04-17T20:00:45.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.320+0000 7f913bc00640 1 -- 172.21.15.29:0/2083834749 wait complete. 2024-04-17T20:00:45.321 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.324+0000 7f913bc00640 1 Processor -- start 2024-04-17T20:00:45.322 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.324+0000 7f913bc00640 1 -- start start 2024-04-17T20:00:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.324+0000 7f913bc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91340cc8c0 0x7f9134140510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.324+0000 7f913bc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91340cccd0 0x7f9134157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.324+0000 7f913bc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91341625e0 0x7f9134151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.324+0000 7f913bc00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9134159c20 con 0x7f91340cc8c0 2024-04-17T20:00:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.324+0000 7f913bc00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9134159aa0 con 0x7f91340cccd0 2024-04-17T20:00:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.324+0000 7f913bc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f91340caa40 con 0x7f91341625e0 2024-04-17T20:00:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.324+0000 7f9139200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91340cc8c0 0x7f9134140510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.324+0000 7f9139200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91340cc8c0 0x7f9134140510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56088/0 (socket says 172.21.15.29:56088) 2024-04-17T20:00:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.324+0000 7f9139200640 1 -- 172.21.15.29:0/2687806926 learned_addr learned my addr 172.21.15.29:0/2687806926 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.324+0000 7f9133e00640 1 --2- 172.21.15.29:0/2687806926 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91340cccd0 0x7f9134157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.324+0000 7f9139c00640 1 --2- 172.21.15.29:0/2687806926 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91341625e0 0x7f9134151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.324+0000 7f9139200640 1 -- 172.21.15.29:0/2687806926 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91341625e0 msgr2=0x7f9134151320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:45.324 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.324+0000 7f9139200640 1 --2- 172.21.15.29:0/2687806926 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91341625e0 0x7f9134151320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:45.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.324+0000 7f9139200640 1 -- 172.21.15.29:0/2687806926 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91340cccd0 msgr2=0x7f9134157440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:45.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.324+0000 7f9139200640 1 --2- 172.21.15.29:0/2687806926 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91340cccd0 0x7f9134157440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:45.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.324+0000 7f9139200640 1 -- 172.21.15.29:0/2687806926 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9134141280 con 0x7f91340cc8c0 2024-04-17T20:00:45.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.328+0000 7f9139200640 1 --2- 172.21.15.29:0/2687806926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91340cc8c0 0x7f9134140510 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f912800c450 tx=0x7f912800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:45.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.328+0000 7f9130c00640 1 -- 172.21.15.29:0/2687806926 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9128017070 con 0x7f91340cc8c0 2024-04-17T20:00:45.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.328+0000 7f9130c00640 1 -- 172.21.15.29:0/2687806926 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9128005cf0 con 0x7f91340cc8c0 2024-04-17T20:00:45.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.328+0000 7f913bc00640 1 -- 172.21.15.29:0/2687806926 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9134141560 con 0x7f91340cc8c0 2024-04-17T20:00:45.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.328+0000 7f9130c00640 1 -- 172.21.15.29:0/2687806926 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9128007270 con 0x7f91340cc8c0 2024-04-17T20:00:45.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.328+0000 7f913bc00640 1 -- 172.21.15.29:0/2687806926 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f913413fd10 con 0x7f91340cc8c0 2024-04-17T20:00:45.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.328+0000 7f913bc00640 1 -- 172.21.15.29:0/2687806926 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9134069b90 con 0x7f91340cc8c0 2024-04-17T20:00:45.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.328+0000 7f9130c00640 1 -- 172.21.15.29:0/2687806926 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f912801a070 con 0x7f91340cc8c0 2024-04-17T20:00:45.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.328+0000 7f9130c00640 1 --2- 172.21.15.29:0/2687806926 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9114068380 0x7f911406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:45.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.328+0000 7f9130c00640 1 -- 172.21.15.29:0/2687806926 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7f9128086320 con 0x7f91340cc8c0 2024-04-17T20:00:45.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.328+0000 7f9133e00640 1 --2- 172.21.15.29:0/2687806926 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9114068380 0x7f911406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:45.329 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.332+0000 7f9133e00640 1 --2- 172.21.15.29:0/2687806926 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9114068380 0x7f911406a890 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f91340cc420 tx=0x7f9120009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:45.332 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.336+0000 7f9130c00640 1 -- 172.21.15.29:0/2687806926 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9128055b50 con 0x7f91340cc8c0 2024-04-17T20:00:45.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.536+0000 7f913bc00640 1 -- 172.21.15.29:0/2687806926 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9134162390 con 0x7f91340cc8c0 2024-04-17T20:00:45.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.536+0000 7f9130c00640 1 -- 172.21.15.29:0/2687806926 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) ==== 94+0+6084 (secure 0 0 0) 0x7f91280860e0 con 0x7f91340cc8c0 2024-04-17T20:00:45.535 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:45.535 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:40.402611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:reconnect","state_seq":33,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5144,5212,5350,5631]},"id":1}]} 2024-04-17T20:00:45.536 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 52 2024-04-17T20:00:45.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.540+0000 7f913bc00640 1 -- 172.21.15.29:0/2687806926 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9114068380 msgr2=0x7f911406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:45.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.540+0000 7f913bc00640 1 --2- 172.21.15.29:0/2687806926 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9114068380 0x7f911406a890 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f91340cc420 tx=0x7f9120009040 comp rx=0 tx=0).stop 2024-04-17T20:00:45.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.540+0000 7f913bc00640 1 -- 172.21.15.29:0/2687806926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91340cc8c0 msgr2=0x7f9134140510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:45.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.540+0000 7f913bc00640 1 --2- 172.21.15.29:0/2687806926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91340cc8c0 0x7f9134140510 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f912800c450 tx=0x7f912800c910 comp rx=0 tx=0).stop 2024-04-17T20:00:45.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.540+0000 7f913bc00640 1 -- 172.21.15.29:0/2687806926 shutdown_connections 2024-04-17T20:00:45.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.540+0000 7f913bc00640 1 --2- 172.21.15.29:0/2687806926 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9114068380 0x7f911406a890 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:45.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.540+0000 7f913bc00640 1 --2- 172.21.15.29:0/2687806926 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f91341625e0 0x7f9134151320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:45.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.540+0000 7f913bc00640 1 --2- 172.21.15.29:0/2687806926 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f91340cccd0 0x7f9134157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:45.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.540+0000 7f913bc00640 1 --2- 172.21.15.29:0/2687806926 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91340cc8c0 0x7f9134140510 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:45.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.540+0000 7f913bc00640 1 -- 172.21.15.29:0/2687806926 >> 172.21.15.29:0/2687806926 conn(0x7f913406fde0 msgr2=0x7f91340c5aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:45.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.540+0000 7f913bc00640 1 -- 172.21.15.29:0/2687806926 shutdown_connections 2024-04-17T20:00:45.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:45.540+0000 7f913bc00640 1 -- 172.21.15.29:0/2687806926 wait complete. 2024-04-17T20:00:45.558 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:00:47.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:47.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.716+0000 7f2c15600640 1 Processor -- start 2024-04-17T20:00:47.714 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.716+0000 7f2c15600640 1 -- start start 2024-04-17T20:00:47.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.716+0000 7f2c15600640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f2c100cb560 0x7f2c100c4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:47.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.716+0000 7f2c15600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c1015a750 0x7f2c100703e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:47.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.716+0000 7f2c15600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c100c9ce0 con 0x7f2c10153c80 2024-04-17T20:00:47.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.716+0000 7f2c15600640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f2c10070920 con 0x7f2c100cb560 2024-04-17T20:00:47.715 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.716+0000 7f2c15600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2c100c9f80 con 0x7f2c1015a750 2024-04-17T20:00:47.716 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.716+0000 7f2c0f400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2c10153c80 0x7f2c100cf810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49156/0 (socket says 172.21.15.29:49156) 2024-04-17T20:00:47.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.716+0000 7f2c0f400640 1 -- 172.21.15.29:0/1840506015 learned_addr learned my addr 172.21.15.29:0/1840506015 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:00:47.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.716+0000 7f2c0ea00640 1 --2- 172.21.15.29:0/1840506015 >> v2:172.21.15.37:3300/0 conn(0x7f2c100cb560 0x7f2c100c4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:47.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.716+0000 7f2c07e00640 1 --2- 172.21.15.29:0/1840506015 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c1015a750 0x7f2c100703e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:47.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.720+0000 7f2c0e000640 1 -- 172.21.15.29:0/1840506015 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1741417251 0 0) 0x7f2c100c9ce0 con 0x7f2c10153c80 2024-04-17T20:00:47.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.720+0000 7f2c0e000640 1 -- 172.21.15.29:0/1840506015 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bf4003530 con 0x7f2c10153c80 2024-04-17T20:00:47.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.720+0000 7f2c0ea00640 1 -- 172.21.15.29:0/1840506015 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c1015a750 msgr2=0x7f2c100703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:47.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.720+0000 7f2c0ea00640 1 --2- 172.21.15.29:0/1840506015 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c1015a750 0x7f2c100703e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:47.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.720+0000 7f2c0ea00640 1 -- 172.21.15.29:0/1840506015 >> v1:172.21.15.29:6789/0 conn(0x7f2c10153c80 legacy=0x7f2c100cf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:47.717 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.720+0000 7f2c0ea00640 1 -- 172.21.15.29:0/1840506015 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c100ca260 con 0x7f2c100cb560 2024-04-17T20:00:47.719 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.720+0000 7f2c0ea00640 1 --2- 172.21.15.29:0/1840506015 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2c100cb560 0x7f2c100c4d10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2c100c9eb0 tx=0x7f2c0001c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a3db57cc9b6e0b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:47.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.724+0000 7f2c0e000640 1 -- 172.21.15.29:0/1840506015 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c000b0070 con 0x7f2c100cb560 2024-04-17T20:00:47.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.724+0000 7f2c0e000640 1 -- 172.21.15.29:0/1840506015 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c000ab600 con 0x7f2c100cb560 2024-04-17T20:00:47.720 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.724+0000 7f2c0e000640 1 -- 172.21.15.29:0/1840506015 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c000ab9d0 con 0x7f2c100cb560 2024-04-17T20:00:47.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.724+0000 7f2c15600640 1 -- 172.21.15.29:0/1840506015 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2c100cb560 msgr2=0x7f2c100c4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:47.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.724+0000 7f2c15600640 1 --2- 172.21.15.29:0/1840506015 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2c100cb560 0x7f2c100c4d10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2c100c9eb0 tx=0x7f2c0001c410 comp rx=0 tx=0).stop 2024-04-17T20:00:47.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.724+0000 7f2c15600640 1 -- 172.21.15.29:0/1840506015 shutdown_connections 2024-04-17T20:00:47.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.724+0000 7f2c15600640 1 --2- 172.21.15.29:0/1840506015 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c1015a750 0x7f2c100703e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:47.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.724+0000 7f2c15600640 1 --2- 172.21.15.29:0/1840506015 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2c100cb560 0x7f2c100c4d10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:47.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.724+0000 7f2c15600640 1 -- 172.21.15.29:0/1840506015 >> 172.21.15.29:0/1840506015 conn(0x7f2c1006e3d0 msgr2=0x7f2c1006e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:47.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.724+0000 7f2c15600640 1 -- 172.21.15.29:0/1840506015 shutdown_connections 2024-04-17T20:00:47.721 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.724+0000 7f2c15600640 1 -- 172.21.15.29:0/1840506015 wait complete. 2024-04-17T20:00:47.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.724+0000 7f2c15600640 1 Processor -- start 2024-04-17T20:00:47.723 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.724+0000 7f2c15600640 1 -- start start 2024-04-17T20:00:47.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c15600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2c100cb560 0x7f2c101587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:47.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c15600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c10153c80 0x7f2c10152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:47.726 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c15600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c1015a750 0x7f2c10157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c15600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2c100cff20 con 0x7f2c10153c80 2024-04-17T20:00:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c15600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2c10159c20 con 0x7f2c100cb560 2024-04-17T20:00:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c15600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2c100d00a0 con 0x7f2c1015a750 2024-04-17T20:00:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c07e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c10153c80 0x7f2c10152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c07e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c10153c80 0x7f2c10152680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56100/0 (socket says 172.21.15.29:56100) 2024-04-17T20:00:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c07e00640 1 -- 172.21.15.29:0/2478868546 learned_addr learned my addr 172.21.15.29:0/2478868546 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c0ea00640 1 --2- 172.21.15.29:0/2478868546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2c100cb560 0x7f2c101587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c0f400640 1 --2- 172.21.15.29:0/2478868546 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c1015a750 0x7f2c10157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c07e00640 1 -- 172.21.15.29:0/2478868546 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c1015a750 msgr2=0x7f2c10157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c07e00640 1 --2- 172.21.15.29:0/2478868546 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c1015a750 0x7f2c10157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:47.727 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c07e00640 1 -- 172.21.15.29:0/2478868546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2c100cb560 msgr2=0x7f2c101587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c07e00640 1 --2- 172.21.15.29:0/2478868546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2c100cb560 0x7f2c101587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c07e00640 1 -- 172.21.15.29:0/2478868546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c100cab30 con 0x7f2c10153c80 2024-04-17T20:00:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c07e00640 1 --2- 172.21.15.29:0/2478868546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c10153c80 0x7f2c10152680 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2bf800e870 tx=0x7f2bf800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c06a00640 1 -- 172.21.15.29:0/2478868546 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf8017070 con 0x7f2c10153c80 2024-04-17T20:00:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c06a00640 1 -- 172.21.15.29:0/2478868546 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bf800c940 con 0x7f2c10153c80 2024-04-17T20:00:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c15600640 1 -- 172.21.15.29:0/2478868546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c10158cf0 con 0x7f2c10153c80 2024-04-17T20:00:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c06a00640 1 -- 172.21.15.29:0/2478868546 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf800ccf0 con 0x7f2c10153c80 2024-04-17T20:00:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c15600640 1 -- 172.21.15.29:0/2478868546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c101f5c30 con 0x7f2c10153c80 2024-04-17T20:00:47.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.728+0000 7f2c15600640 1 -- 172.21.15.29:0/2478868546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c10069b90 con 0x7f2c10153c80 2024-04-17T20:00:47.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.732+0000 7f2c06a00640 1 -- 172.21.15.29:0/2478868546 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2bf801a070 con 0x7f2c10153c80 2024-04-17T20:00:47.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.732+0000 7f2c06a00640 1 --2- 172.21.15.29:0/2478868546 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2bec068380 0x7f2bec06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:47.729 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.732+0000 7f2c06a00640 1 -- 172.21.15.29:0/2478868546 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7f2bf8087250 con 0x7f2c10153c80 2024-04-17T20:00:47.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.732+0000 7f2c0ea00640 1 --2- 172.21.15.29:0/2478868546 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2bec068380 0x7f2bec06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:47.730 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.732+0000 7f2c0ea00640 1 --2- 172.21.15.29:0/2478868546 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2bec068380 0x7f2bec06a890 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2c000ad000 tx=0x7f2c00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:47.734 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.736+0000 7f2c06a00640 1 -- 172.21.15.29:0/2478868546 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2bf80571b0 con 0x7f2c10153c80 2024-04-17T20:00:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.936+0000 7f2c15600640 1 -- 172.21.15.29:0/2478868546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c10164360 con 0x7f2c10153c80 2024-04-17T20:00:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.936+0000 7f2c06a00640 1 -- 172.21.15.29:0/2478868546 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6081 (secure 0 0 0) 0x7f2bf8056ab0 con 0x7f2c10153c80 2024-04-17T20:00:47.936 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:47.936 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:45.462737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:rejoin","state_seq":35,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5631,5350,5212,5144]},"id":1}]} 2024-04-17T20:00:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 53 2024-04-17T20:00:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.940+0000 7f2c15600640 1 -- 172.21.15.29:0/2478868546 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2bec068380 msgr2=0x7f2bec06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.940+0000 7f2c15600640 1 --2- 172.21.15.29:0/2478868546 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2bec068380 0x7f2bec06a890 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2c000ad000 tx=0x7f2c00008040 comp rx=0 tx=0).stop 2024-04-17T20:00:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.940+0000 7f2c15600640 1 -- 172.21.15.29:0/2478868546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c10153c80 msgr2=0x7f2c10152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.940+0000 7f2c15600640 1 --2- 172.21.15.29:0/2478868546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c10153c80 0x7f2c10152680 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2bf800e870 tx=0x7f2bf800ec70 comp rx=0 tx=0).stop 2024-04-17T20:00:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.940+0000 7f2c15600640 1 -- 172.21.15.29:0/2478868546 shutdown_connections 2024-04-17T20:00:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.940+0000 7f2c15600640 1 --2- 172.21.15.29:0/2478868546 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2bec068380 0x7f2bec06a890 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.940+0000 7f2c15600640 1 --2- 172.21.15.29:0/2478868546 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2c1015a750 0x7f2c10157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.940+0000 7f2c15600640 1 --2- 172.21.15.29:0/2478868546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2c10153c80 0x7f2c10152680 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.940+0000 7f2c15600640 1 --2- 172.21.15.29:0/2478868546 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2c100cb560 0x7f2c101587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:47.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.940+0000 7f2c15600640 1 -- 172.21.15.29:0/2478868546 >> 172.21.15.29:0/2478868546 conn(0x7f2c1006e3d0 msgr2=0x7f2c1006f820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:47.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.940+0000 7f2c15600640 1 -- 172.21.15.29:0/2478868546 shutdown_connections 2024-04-17T20:00:47.939 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:47.940+0000 7f2c15600640 1 -- 172.21.15.29:0/2478868546 wait complete. 2024-04-17T20:00:47.958 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:00:49.959 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:50.123 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.124+0000 7f15a5e00640 1 Processor -- start 2024-04-17T20:00:50.124 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.124+0000 7f15a5e00640 1 -- start start 2024-04-17T20:00:50.126 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.128+0000 7f15a5e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f15a00cc8c0 0x7f15a00cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:50.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.128+0000 7f15a5e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f15a01597a0 0x7f15a01635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:50.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.128+0000 7f15a5e00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15a00c88b0 con 0x7f15a0159390 2024-04-17T20:00:50.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.128+0000 7f15a5e00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f15a00c6ce0 con 0x7f15a00cc8c0 2024-04-17T20:00:50.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.128+0000 7f15a5e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f15a00c6fc0 con 0x7f15a01597a0 2024-04-17T20:00:50.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.128+0000 7f159f400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f15a01597a0 0x7f15a01635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:50.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.128+0000 7f159ea00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f15a00cc8c0 0x7f15a00cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:50.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.128+0000 7f1597e00640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f15a0159390 0x7f15a00ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:33492/0 (socket says 172.21.15.29:33492) 2024-04-17T20:00:50.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.128+0000 7f1597e00640 1 -- 172.21.15.29:0/1814916793 learned_addr learned my addr 172.21.15.29:0/1814916793 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:00:50.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.128+0000 7f159ea00640 1 -- 172.21.15.29:0/1814916793 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f15a01597a0 msgr2=0x7f15a01635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:50.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.128+0000 7f159ea00640 1 --2- 172.21.15.29:0/1814916793 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f15a01597a0 0x7f15a01635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:50.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.128+0000 7f159ea00640 1 -- 172.21.15.29:0/1814916793 >> v1:172.21.15.37:6789/0 conn(0x7f15a0159390 legacy=0x7f15a00ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:50.127 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.128+0000 7f159ea00640 1 -- 172.21.15.29:0/1814916793 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a00c72a0 con 0x7f15a00cc8c0 2024-04-17T20:00:50.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.128+0000 7f159f400640 1 --2- 172.21.15.29:0/1814916793 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f15a01597a0 0x7f15a01635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T20:00:50.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.132+0000 7f159ea00640 1 --2- 172.21.15.29:0/1814916793 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15a00cc8c0 0x7f15a00cccd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f159000ba60 tx=0x7f159001eb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=661c61d08a43a8c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:50.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.132+0000 7f159e000640 1 -- 172.21.15.29:0/1814916793 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15900b4070 con 0x7f15a00cc8c0 2024-04-17T20:00:50.130 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.132+0000 7f159e000640 1 -- 172.21.15.29:0/1814916793 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15900a7940 con 0x7f15a00cc8c0 2024-04-17T20:00:50.131 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.132+0000 7f159e000640 1 -- 172.21.15.29:0/1814916793 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15900a7d10 con 0x7f15a00cc8c0 2024-04-17T20:00:50.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.132+0000 7f15a5e00640 1 -- 172.21.15.29:0/1814916793 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15a00cc8c0 msgr2=0x7f15a00cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:50.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.132+0000 7f15a5e00640 1 --2- 172.21.15.29:0/1814916793 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15a00cc8c0 0x7f15a00cccd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f159000ba60 tx=0x7f159001eb70 comp rx=0 tx=0).stop 2024-04-17T20:00:50.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.132+0000 7f15a5e00640 1 -- 172.21.15.29:0/1814916793 shutdown_connections 2024-04-17T20:00:50.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.132+0000 7f15a5e00640 1 --2- 172.21.15.29:0/1814916793 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f15a01597a0 0x7f15a01635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:50.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.132+0000 7f15a5e00640 1 --2- 172.21.15.29:0/1814916793 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15a00cc8c0 0x7f15a00cccd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:50.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.132+0000 7f15a5e00640 1 -- 172.21.15.29:0/1814916793 >> 172.21.15.29:0/1814916793 conn(0x7f15a006fde0 msgr2=0x7f15a00701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:50.132 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.132+0000 7f15a5e00640 1 -- 172.21.15.29:0/1814916793 shutdown_connections 2024-04-17T20:00:50.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.132+0000 7f15a5e00640 1 -- 172.21.15.29:0/1814916793 wait complete. 2024-04-17T20:00:50.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f15a5e00640 1 Processor -- start 2024-04-17T20:00:50.133 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f15a5e00640 1 -- start start 2024-04-17T20:00:50.136 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f15a5e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15a00cc8c0 0x7f15a0163590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f15a5e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15a0159390 0x7f15a0136d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f15a5e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f15a01597a0 0x7f15a0137280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f15a5e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f15a00cabc0 con 0x7f15a0159390 2024-04-17T20:00:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f15a5e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f15a00caa40 con 0x7f15a00cc8c0 2024-04-17T20:00:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f15a5e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f15a00c6880 con 0x7f15a01597a0 2024-04-17T20:00:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f1597e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15a0159390 0x7f15a0136d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f1597e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15a0159390 0x7f15a0136d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33606/0 (socket says 172.21.15.29:33606) 2024-04-17T20:00:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f1597e00640 1 -- 172.21.15.29:0/3432343504 learned_addr learned my addr 172.21.15.29:0/3432343504 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f159ea00640 1 --2- 172.21.15.29:0/3432343504 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15a00cc8c0 0x7f15a0163590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f159f400640 1 --2- 172.21.15.29:0/3432343504 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f15a01597a0 0x7f15a0137280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f1597e00640 1 -- 172.21.15.29:0/3432343504 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f15a01597a0 msgr2=0x7f15a0137280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f1597e00640 1 --2- 172.21.15.29:0/3432343504 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f15a01597a0 0x7f15a0137280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f1597e00640 1 -- 172.21.15.29:0/3432343504 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15a00cc8c0 msgr2=0x7f15a0163590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f1597e00640 1 --2- 172.21.15.29:0/3432343504 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15a00cc8c0 0x7f15a0163590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f1597e00640 1 -- 172.21.15.29:0/3432343504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a01f6a60 con 0x7f15a0159390 2024-04-17T20:00:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f1597e00640 1 --2- 172.21.15.29:0/3432343504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15a0159390 0x7f15a0136d40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f158800c340 tx=0x7f158800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f1597400640 1 -- 172.21.15.29:0/3432343504 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1588017070 con 0x7f15a0159390 2024-04-17T20:00:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f1597400640 1 -- 172.21.15.29:0/3432343504 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1588005cf0 con 0x7f15a0159390 2024-04-17T20:00:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f1597400640 1 -- 172.21.15.29:0/3432343504 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1588007310 con 0x7f15a0159390 2024-04-17T20:00:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f15a5e00640 1 -- 172.21.15.29:0/3432343504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15a01627d0 con 0x7f15a0159390 2024-04-17T20:00:50.140 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.136+0000 7f15a5e00640 1 -- 172.21.15.29:0/3432343504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15a0162d90 con 0x7f15a0159390 2024-04-17T20:00:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.140+0000 7f15a5e00640 1 -- 172.21.15.29:0/3432343504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15a01f9950 con 0x7f15a0159390 2024-04-17T20:00:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.140+0000 7f1597400640 1 -- 172.21.15.29:0/3432343504 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f158801a070 con 0x7f15a0159390 2024-04-17T20:00:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.140+0000 7f1597400640 1 --2- 172.21.15.29:0/3432343504 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f157c0683b0 0x7f157c06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.140+0000 7f1597400640 1 -- 172.21.15.29:0/3432343504 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7f15880862c0 con 0x7f15a0159390 2024-04-17T20:00:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.140+0000 7f159ea00640 1 --2- 172.21.15.29:0/3432343504 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f157c0683b0 0x7f157c06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.144+0000 7f159ea00640 1 --2- 172.21.15.29:0/3432343504 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f157c0683b0 0x7f157c06a8c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f15900b1000 tx=0x7f15900a9070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:50.144 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.144+0000 7f1597400640 1 -- 172.21.15.29:0/3432343504 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f1588055b90 con 0x7f15a0159390 2024-04-17T20:00:50.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.344+0000 7f15a5e00640 1 -- 172.21.15.29:0/3432343504 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15a0069b90 con 0x7f15a0159390 2024-04-17T20:00:50.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.348+0000 7f1597400640 1 -- 172.21.15.29:0/3432343504 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+6081 (secure 0 0 0) 0x7f1588059e90 con 0x7f15a0159390 2024-04-17T20:00:50.351 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:50.351 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:45.462737+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:rejoin","state_seq":35,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5631,5350,5212,5144]},"id":1}]} 2024-04-17T20:00:50.352 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 53 2024-04-17T20:00:50.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.348+0000 7f15a5e00640 1 -- 172.21.15.29:0/3432343504 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f157c0683b0 msgr2=0x7f157c06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:50.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.348+0000 7f15a5e00640 1 --2- 172.21.15.29:0/3432343504 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f157c0683b0 0x7f157c06a8c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f15900b1000 tx=0x7f15900a9070 comp rx=0 tx=0).stop 2024-04-17T20:00:50.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.348+0000 7f15a5e00640 1 -- 172.21.15.29:0/3432343504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15a0159390 msgr2=0x7f15a0136d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:50.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.348+0000 7f15a5e00640 1 --2- 172.21.15.29:0/3432343504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15a0159390 0x7f15a0136d40 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f158800c340 tx=0x7f158800c800 comp rx=0 tx=0).stop 2024-04-17T20:00:50.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.348+0000 7f15a5e00640 1 -- 172.21.15.29:0/3432343504 shutdown_connections 2024-04-17T20:00:50.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.348+0000 7f15a5e00640 1 --2- 172.21.15.29:0/3432343504 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f157c0683b0 0x7f157c06a8c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:50.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.348+0000 7f15a5e00640 1 --2- 172.21.15.29:0/3432343504 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f15a01597a0 0x7f15a0137280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:50.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.348+0000 7f15a5e00640 1 --2- 172.21.15.29:0/3432343504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15a0159390 0x7f15a0136d40 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:50.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.348+0000 7f15a5e00640 1 --2- 172.21.15.29:0/3432343504 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f15a00cc8c0 0x7f15a0163590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:50.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.348+0000 7f15a5e00640 1 -- 172.21.15.29:0/3432343504 >> 172.21.15.29:0/3432343504 conn(0x7f15a006fde0 msgr2=0x7f15a0072320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:50.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.348+0000 7f15a5e00640 1 -- 172.21.15.29:0/3432343504 shutdown_connections 2024-04-17T20:00:50.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:50.348+0000 7f15a5e00640 1 -- 172.21.15.29:0/3432343504 wait complete. 2024-04-17T20:00:50.368 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 53, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-17T19:47:49.315515+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 53, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5144": { "addr": "172.21.15.37:6839/858161860", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6836", "nonce": 858161860, "type": "v2" }, { "addr": "172.21.15.37:6839", "nonce": 858161860, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5144, "incarnation": 38, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 32 }, "gid_5212": { "addr": "172.21.15.37:6838/767681664", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6835", "nonce": 767681664, "type": "v2" }, { "addr": "172.21.15.37:6838", "nonce": 767681664, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5212, "incarnation": 26, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 31 }, "gid_5350": { "addr": "172.21.15.37:6837/2495953425", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6834", "nonce": 2495953425, "type": "v2" }, { "addr": "172.21.15.37:6837", "nonce": 2495953425, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5350, "incarnation": 32, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 18 }, "gid_5631": { "addr": "172.21.15.29:6836/2376553580", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6832", "nonce": 2376553580, "type": "v2" }, { "addr": "172.21.15.29:6836", "nonce": 2376553580, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5631, "incarnation": 44, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 42 }, "gid_5775": { "addr": "172.21.15.29:6835/1352350250", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6834", "nonce": 1352350250, "type": "v2" }, { "addr": "172.21.15.29:6835", "nonce": 1352350250, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5775, "incarnation": 50, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:rejoin", "state_seq": 35 } }, "last_failure": 0, "last_failure_osd_epoch": 74, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-17T20:00:45.462737+0000", "qdb_cluster": [ 5631, 5350, 5212, 5144 ], "qdb_leader": 5631, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5775, "mds_1": 5631, "mds_2": 5144, "mds_3": 5350, "mds_4": 5212 } } } ], "standbys": [] } 2024-04-17T20:00:52.369 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:52.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.524+0000 7f3eaa800640 1 Processor -- start 2024-04-17T20:00:52.522 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.524+0000 7f3eaa800640 1 -- start start 2024-04-17T20:00:52.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.524+0000 7f3eaa800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3ea40cc8c0 0x7f3ea40cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:52.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.524+0000 7f3eaa800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3ea41597a0 0x7f3ea41635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:52.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.524+0000 7f3eaa800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ea40c88b0 con 0x7f3ea4159390 2024-04-17T20:00:52.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.524+0000 7f3eaa800640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f3ea40c6ce0 con 0x7f3ea41597a0 2024-04-17T20:00:52.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.524+0000 7f3eaa800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f3ea40c6fc0 con 0x7f3ea40cc8c0 2024-04-17T20:00:52.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.524+0000 7f3ea3e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3ea41597a0 0x7f3ea41635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:52.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.524+0000 7f3ea3e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3ea41597a0 0x7f3ea41635e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33620/0 (socket says 172.21.15.29:33620) 2024-04-17T20:00:52.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.524+0000 7f3ea3e00640 1 -- 172.21.15.29:0/1798229326 learned_addr learned my addr 172.21.15.29:0/1798229326 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:52.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.524+0000 7f3ea3400640 1 --2- 172.21.15.29:0/1798229326 >> v2:172.21.15.37:3300/0 conn(0x7f3ea40cc8c0 0x7f3ea40cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:52.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.524+0000 7f3ea3e00640 1 -- 172.21.15.29:0/1798229326 >> v2:172.21.15.37:3300/0 conn(0x7f3ea40cc8c0 msgr2=0x7f3ea40cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:52.524 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.524+0000 7f3ea3e00640 1 --2- 172.21.15.29:0/1798229326 >> v2:172.21.15.37:3300/0 conn(0x7f3ea40cc8c0 0x7f3ea40cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:52.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.524+0000 7f3ea3e00640 1 -- 172.21.15.29:0/1798229326 >> v1:172.21.15.37:6789/0 conn(0x7f3ea4159390 legacy=0x7f3ea40ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:52.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.524+0000 7f3ea3e00640 1 -- 172.21.15.29:0/1798229326 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea40c72a0 con 0x7f3ea41597a0 2024-04-17T20:00:52.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.528+0000 7f3ea3e00640 1 --2- 172.21.15.29:0/1798229326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ea41597a0 0x7f3ea41635e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40721b0 tx=0x7f3e9c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=106f56843974e0aa server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:52.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.528+0000 7f3ea2a00640 1 -- 172.21.15.29:0/1798229326 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e9c0a56c0 con 0x7f3ea41597a0 2024-04-17T20:00:52.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.528+0000 7f3ea2a00640 1 -- 172.21.15.29:0/1798229326 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e9c0a5860 con 0x7f3ea41597a0 2024-04-17T20:00:52.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.528+0000 7f3ea2a00640 1 -- 172.21.15.29:0/1798229326 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e9c0a5c30 con 0x7f3ea41597a0 2024-04-17T20:00:52.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.528+0000 7f3eaa800640 1 -- 172.21.15.29:0/1798229326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ea41597a0 msgr2=0x7f3ea41635e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:52.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.528+0000 7f3eaa800640 1 --2- 172.21.15.29:0/1798229326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ea41597a0 0x7f3ea41635e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40721b0 tx=0x7f3e9c01c9f0 comp rx=0 tx=0).stop 2024-04-17T20:00:52.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.528+0000 7f3eaa800640 1 -- 172.21.15.29:0/1798229326 shutdown_connections 2024-04-17T20:00:52.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.528+0000 7f3eaa800640 1 --2- 172.21.15.29:0/1798229326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ea41597a0 0x7f3ea41635e0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:52.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.528+0000 7f3eaa800640 1 --2- 172.21.15.29:0/1798229326 >> v2:172.21.15.37:3300/0 conn(0x7f3ea40cc8c0 0x7f3ea40cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:52.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.528+0000 7f3eaa800640 1 -- 172.21.15.29:0/1798229326 >> 172.21.15.29:0/1798229326 conn(0x7f3ea406fde0 msgr2=0x7f3ea40701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:52.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.528+0000 7f3eaa800640 1 -- 172.21.15.29:0/1798229326 shutdown_connections 2024-04-17T20:00:52.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.528+0000 7f3eaa800640 1 -- 172.21.15.29:0/1798229326 wait complete. 2024-04-17T20:00:52.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.532+0000 7f3eaa800640 1 Processor -- start 2024-04-17T20:00:52.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.532+0000 7f3eaa800640 1 -- start start 2024-04-17T20:00:52.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.532+0000 7f3eaa800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3ea40cc8c0 0x7f3ea41f5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:52.534 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.532+0000 7f3eaa800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ea4159390 0x7f3ea4157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:52.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.532+0000 7f3eaa800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ea41597a0 0x7f3ea41395f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:52.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.532+0000 7f3eaa800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea40cabc0 con 0x7f3ea4159390 2024-04-17T20:00:52.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.532+0000 7f3eaa800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea40caa40 con 0x7f3ea40cc8c0 2024-04-17T20:00:52.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.532+0000 7f3eaa800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3ea40c6880 con 0x7f3ea41597a0 2024-04-17T20:00:52.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.532+0000 7f3e9be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ea4159390 0x7f3ea4157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:52.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.532+0000 7f3e9be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ea4159390 0x7f3ea4157ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33624/0 (socket says 172.21.15.29:33624) 2024-04-17T20:00:52.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.532+0000 7f3e9be00640 1 -- 172.21.15.29:0/3740007705 learned_addr learned my addr 172.21.15.29:0/3740007705 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:52.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.532+0000 7f3ea3400640 1 --2- 172.21.15.29:0/3740007705 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3ea40cc8c0 0x7f3ea41f5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:52.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.532+0000 7f3ea3e00640 1 --2- 172.21.15.29:0/3740007705 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ea41597a0 0x7f3ea41395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:52.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.536+0000 7f3e9be00640 1 -- 172.21.15.29:0/3740007705 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ea41597a0 msgr2=0x7f3ea41395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:52.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.536+0000 7f3e9be00640 1 --2- 172.21.15.29:0/3740007705 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ea41597a0 0x7f3ea41395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:52.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.536+0000 7f3e9be00640 1 -- 172.21.15.29:0/3740007705 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3ea40cc8c0 msgr2=0x7f3ea41f5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:52.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.536+0000 7f3e9be00640 1 --2- 172.21.15.29:0/3740007705 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3ea40cc8c0 0x7f3ea41f5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:52.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.536+0000 7f3e9be00640 1 -- 172.21.15.29:0/3740007705 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea4158220 con 0x7f3ea4159390 2024-04-17T20:00:52.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.536+0000 7f3e9be00640 1 --2- 172.21.15.29:0/3740007705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ea4159390 0x7f3ea4157ce0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00c340 tx=0x7f3e8c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:52.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.536+0000 7f3e9b400640 1 -- 172.21.15.29:0/3740007705 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c0052d0 con 0x7f3ea4159390 2024-04-17T20:00:52.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.536+0000 7f3e9b400640 1 -- 172.21.15.29:0/3740007705 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e8c005470 con 0x7f3ea4159390 2024-04-17T20:00:52.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.536+0000 7f3eaa800640 1 -- 172.21.15.29:0/3740007705 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ea4141700 con 0x7f3ea4159390 2024-04-17T20:00:52.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.536+0000 7f3e9b400640 1 -- 172.21.15.29:0/3740007705 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c007260 con 0x7f3ea4159390 2024-04-17T20:00:52.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.536+0000 7f3eaa800640 1 -- 172.21.15.29:0/3740007705 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ea41574e0 con 0x7f3ea4159390 2024-04-17T20:00:52.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.536+0000 7f3eaa800640 1 -- 172.21.15.29:0/3740007705 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ea4069b90 con 0x7f3ea4159390 2024-04-17T20:00:52.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.536+0000 7f3e9b400640 1 -- 172.21.15.29:0/3740007705 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3e8c007400 con 0x7f3ea4159390 2024-04-17T20:00:52.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.536+0000 7f3e9b400640 1 --2- 172.21.15.29:0/3740007705 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3e800683b0 0x7f3e8006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:52.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.536+0000 7f3e9b400640 1 -- 172.21.15.29:0/3740007705 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7f3e8c086480 con 0x7f3ea4159390 2024-04-17T20:00:52.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.540+0000 7f3ea3400640 1 --2- 172.21.15.29:0/3740007705 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3e800683b0 0x7f3e8006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:52.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.540+0000 7f3ea3400640 1 --2- 172.21.15.29:0/3740007705 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3e800683b0 0x7f3e8006a8c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3e940079c0 tx=0x7f3e94008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:52.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.544+0000 7f3e9b400640 1 -- 172.21.15.29:0/3740007705 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3e8c055cb0 con 0x7f3ea4159390 2024-04-17T20:00:52.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.756+0000 7f3eaa800640 1 -- 172.21.15.29:0/3740007705 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3ea4141490 con 0x7f3ea4159390 2024-04-17T20:00:52.753 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.756+0000 7f3e9b400640 1 -- 172.21.15.29:0/3740007705 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+6087 (secure 0 0 0) 0x7f3e8c059fb0 con 0x7f3ea4159390 2024-04-17T20:00:52.753 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:52.753 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:51.234057+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:clientreplay","state_seq":37,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5144,5212,5350,5631]},"id":1}]} 2024-04-17T20:00:52.754 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 54 2024-04-17T20:00:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.756+0000 7f3eaa800640 1 -- 172.21.15.29:0/3740007705 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3e800683b0 msgr2=0x7f3e8006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.756+0000 7f3eaa800640 1 --2- 172.21.15.29:0/3740007705 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3e800683b0 0x7f3e8006a8c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3e940079c0 tx=0x7f3e94008040 comp rx=0 tx=0).stop 2024-04-17T20:00:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.756+0000 7f3eaa800640 1 -- 172.21.15.29:0/3740007705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ea4159390 msgr2=0x7f3ea4157ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.756+0000 7f3eaa800640 1 --2- 172.21.15.29:0/3740007705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ea4159390 0x7f3ea4157ce0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00c340 tx=0x7f3e8c00c800 comp rx=0 tx=0).stop 2024-04-17T20:00:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.756+0000 7f3eaa800640 1 -- 172.21.15.29:0/3740007705 shutdown_connections 2024-04-17T20:00:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.756+0000 7f3eaa800640 1 --2- 172.21.15.29:0/3740007705 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3e800683b0 0x7f3e8006a8c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.756+0000 7f3eaa800640 1 --2- 172.21.15.29:0/3740007705 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3ea41597a0 0x7f3ea41395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.756+0000 7f3eaa800640 1 --2- 172.21.15.29:0/3740007705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ea4159390 0x7f3ea4157ce0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.756+0000 7f3eaa800640 1 --2- 172.21.15.29:0/3740007705 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3ea40cc8c0 0x7f3ea41f5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.756+0000 7f3eaa800640 1 -- 172.21.15.29:0/3740007705 >> 172.21.15.29:0/3740007705 conn(0x7f3ea406fde0 msgr2=0x7f3ea4075420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.756+0000 7f3eaa800640 1 -- 172.21.15.29:0/3740007705 shutdown_connections 2024-04-17T20:00:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:52.756+0000 7f3eaa800640 1 -- 172.21.15.29:0/3740007705 wait complete. 2024-04-17T20:00:52.777 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:00:53.164 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:00:53.168 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:00:54.779 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:54.946 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.948+0000 7fc61ea00640 1 Processor -- start 2024-04-17T20:00:54.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.948+0000 7fc61ea00640 1 -- start start 2024-04-17T20:00:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.948+0000 7fc61ea00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc6180cccd0 0x7fc6180ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.948+0000 7fc61ea00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc6180ca870 0x7fc6181635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.948+0000 7fc61ea00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6180cd0e0 con 0x7fc6180cc8c0 2024-04-17T20:00:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.948+0000 7fc61ea00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fc6180c88b0 con 0x7fc6180ca870 2024-04-17T20:00:54.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.952+0000 7fc61ea00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc6180c6ce0 con 0x7fc6180cccd0 2024-04-17T20:00:54.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.952+0000 7fc61ca00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc6180ca870 0x7fc6181635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:54.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.952+0000 7fc61ca00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc6180ca870 0x7fc6181635e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33640/0 (socket says 172.21.15.29:33640) 2024-04-17T20:00:54.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.952+0000 7fc61ca00640 1 -- 172.21.15.29:0/2744634858 learned_addr learned my addr 172.21.15.29:0/2744634858 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:54.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.952+0000 7fc617400640 1 --2- 172.21.15.29:0/2744634858 >> v2:172.21.15.37:3300/0 conn(0x7fc6180cccd0 0x7fc6180ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:54.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.952+0000 7fc61ca00640 1 -- 172.21.15.29:0/2744634858 >> v2:172.21.15.37:3300/0 conn(0x7fc6180cccd0 msgr2=0x7fc6180ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:54.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.952+0000 7fc61ca00640 1 --2- 172.21.15.29:0/2744634858 >> v2:172.21.15.37:3300/0 conn(0x7fc6180cccd0 0x7fc6180ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:54.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.952+0000 7fc61ca00640 1 -- 172.21.15.29:0/2744634858 >> v1:172.21.15.37:6789/0 conn(0x7fc6180cc8c0 legacy=0x7fc618159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:54.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.952+0000 7fc61ca00640 1 -- 172.21.15.29:0/2744634858 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6180c6fc0 con 0x7fc6180ca870 2024-04-17T20:00:54.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.952+0000 7fc617400640 1 --2- 172.21.15.29:0/2744634858 >> v2:172.21.15.37:3300/0 conn(0x7fc6180cccd0 0x7fc6180ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:00:54.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.952+0000 7fc61ca00640 1 --2- 172.21.15.29:0/2744634858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6180ca870 0x7fc6181635e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc6180cc7e0 tx=0x7fc61001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937d5448445dd94a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:54.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.952+0000 7fc616a00640 1 -- 172.21.15.29:0/2744634858 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6100a56c0 con 0x7fc6180ca870 2024-04-17T20:00:54.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.956+0000 7fc616a00640 1 -- 172.21.15.29:0/2744634858 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6100a5860 con 0x7fc6180ca870 2024-04-17T20:00:54.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.956+0000 7fc616a00640 1 -- 172.21.15.29:0/2744634858 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6100a5c30 con 0x7fc6180ca870 2024-04-17T20:00:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.956+0000 7fc61ea00640 1 -- 172.21.15.29:0/2744634858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6180ca870 msgr2=0x7fc6181635e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.956+0000 7fc61ea00640 1 --2- 172.21.15.29:0/2744634858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6180ca870 0x7fc6181635e0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fc6180cc7e0 tx=0x7fc61001c9f0 comp rx=0 tx=0).stop 2024-04-17T20:00:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.956+0000 7fc61ea00640 1 -- 172.21.15.29:0/2744634858 shutdown_connections 2024-04-17T20:00:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.956+0000 7fc61ea00640 1 --2- 172.21.15.29:0/2744634858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6180ca870 0x7fc6181635e0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.956+0000 7fc61ea00640 1 --2- 172.21.15.29:0/2744634858 >> v2:172.21.15.37:3300/0 conn(0x7fc6180cccd0 0x7fc6180ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.956+0000 7fc61ea00640 1 -- 172.21.15.29:0/2744634858 >> 172.21.15.29:0/2744634858 conn(0x7fc61806fde0 msgr2=0x7fc6180701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.956+0000 7fc61ea00640 1 -- 172.21.15.29:0/2744634858 shutdown_connections 2024-04-17T20:00:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.956+0000 7fc61ea00640 1 -- 172.21.15.29:0/2744634858 wait complete. 2024-04-17T20:00:54.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.956+0000 7fc61ea00640 1 Processor -- start 2024-04-17T20:00:54.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.956+0000 7fc61ea00640 1 -- start start 2024-04-17T20:00:54.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc61ea00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc6180ca870 0x7fc618072e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:54.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc61ea00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6180cc8c0 0x7fc618139f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:54.957 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc61ea00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6180cccd0 0x7fc618161790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:54.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc61ea00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc618159c20 con 0x7fc6180cccd0 2024-04-17T20:00:54.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc61ea00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc618159aa0 con 0x7fc6180cc8c0 2024-04-17T20:00:54.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc61ea00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc6180c6880 con 0x7fc6180ca870 2024-04-17T20:00:54.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc61ca00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6180cccd0 0x7fc618161790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:54.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc61ca00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6180cccd0 0x7fc618161790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33652/0 (socket says 172.21.15.29:33652) 2024-04-17T20:00:54.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc61ca00640 1 -- 172.21.15.29:0/2910553663 learned_addr learned my addr 172.21.15.29:0/2910553663 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:54.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc617400640 1 --2- 172.21.15.29:0/2910553663 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6180cc8c0 0x7fc618139f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:54.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc617e00640 1 --2- 172.21.15.29:0/2910553663 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc6180ca870 0x7fc618072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:54.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc61ca00640 1 -- 172.21.15.29:0/2910553663 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc6180ca870 msgr2=0x7fc618072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:54.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc61ca00640 1 --2- 172.21.15.29:0/2910553663 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc6180ca870 0x7fc618072e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:54.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc61ca00640 1 -- 172.21.15.29:0/2910553663 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6180cc8c0 msgr2=0x7fc618139f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:54.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc61ca00640 1 --2- 172.21.15.29:0/2910553663 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6180cc8c0 0x7fc618139f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:54.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc61ca00640 1 -- 172.21.15.29:0/2910553663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6180733c0 con 0x7fc6180cccd0 2024-04-17T20:00:54.958 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc61ca00640 1 --2- 172.21.15.29:0/2910553663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6180cccd0 0x7fc618161790 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc61001a620 tx=0x7fc610019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc60fe00640 1 -- 172.21.15.29:0/2910553663 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6100b8670 con 0x7fc6180cccd0 2024-04-17T20:00:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc60fe00640 1 -- 172.21.15.29:0/2910553663 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6100be070 con 0x7fc6180cccd0 2024-04-17T20:00:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc60fe00640 1 -- 172.21.15.29:0/2910553663 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6100b8a70 con 0x7fc6180cccd0 2024-04-17T20:00:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc61ea00640 1 -- 172.21.15.29:0/2910553663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc61813afa0 con 0x7fc6180cccd0 2024-04-17T20:00:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc61ea00640 1 -- 172.21.15.29:0/2910553663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6181f62a0 con 0x7fc6180cccd0 2024-04-17T20:00:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc61ea00640 1 -- 172.21.15.29:0/2910553663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc618069b90 con 0x7fc6180cccd0 2024-04-17T20:00:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.960+0000 7fc60fe00640 1 -- 172.21.15.29:0/2910553663 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc6100b8c10 con 0x7fc6180cccd0 2024-04-17T20:00:54.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.964+0000 7fc60fe00640 1 --2- 172.21.15.29:0/2910553663 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc5f80683b0 0x7fc5f806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:54.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.964+0000 7fc60fe00640 1 -- 172.21.15.29:0/2910553663 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7fc610121ff0 con 0x7fc6180cccd0 2024-04-17T20:00:54.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.964+0000 7fc617e00640 1 --2- 172.21.15.29:0/2910553663 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc5f80683b0 0x7fc5f806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:54.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.964+0000 7fc617e00640 1 --2- 172.21.15.29:0/2910553663 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc5f80683b0 0x7fc5f806a8c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc6080015b0 tx=0x7fc608009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:54.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:54.968+0000 7fc60fe00640 1 -- 172.21.15.29:0/2910553663 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc6100165e0 con 0x7fc6180cccd0 2024-04-17T20:00:55.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:55.176+0000 7fc61ea00640 1 -- 172.21.15.29:0/2910553663 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6180cb560 con 0x7fc6180cccd0 2024-04-17T20:00:55.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:55.180+0000 7fc60fe00640 1 -- 172.21.15.29:0/2910553663 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+6087 (secure 0 0 0) 0x7fc6100f6e20 con 0x7fc6180cccd0 2024-04-17T20:00:55.177 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:55.177 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:51.234057+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:clientreplay","state_seq":37,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5144,5212,5350,5631]},"id":1}]} 2024-04-17T20:00:55.177 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 54 2024-04-17T20:00:55.178 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:55.180+0000 7fc61ea00640 1 -- 172.21.15.29:0/2910553663 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc5f80683b0 msgr2=0x7fc5f806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:55.180+0000 7fc61ea00640 1 --2- 172.21.15.29:0/2910553663 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc5f80683b0 0x7fc5f806a8c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc6080015b0 tx=0x7fc608009040 comp rx=0 tx=0).stop 2024-04-17T20:00:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:55.180+0000 7fc61ea00640 1 -- 172.21.15.29:0/2910553663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6180cccd0 msgr2=0x7fc618161790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:55.180+0000 7fc61ea00640 1 --2- 172.21.15.29:0/2910553663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6180cccd0 0x7fc618161790 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc61001a620 tx=0x7fc610019790 comp rx=0 tx=0).stop 2024-04-17T20:00:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:55.180+0000 7fc61ea00640 1 -- 172.21.15.29:0/2910553663 shutdown_connections 2024-04-17T20:00:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:55.180+0000 7fc61ea00640 1 --2- 172.21.15.29:0/2910553663 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc5f80683b0 0x7fc5f806a8c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:55.180+0000 7fc61ea00640 1 --2- 172.21.15.29:0/2910553663 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6180cccd0 0x7fc618161790 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:55.180+0000 7fc61ea00640 1 --2- 172.21.15.29:0/2910553663 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc6180cc8c0 0x7fc618139f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:55.180+0000 7fc61ea00640 1 --2- 172.21.15.29:0/2910553663 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc6180ca870 0x7fc618072e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:55.180+0000 7fc61ea00640 1 -- 172.21.15.29:0/2910553663 >> 172.21.15.29:0/2910553663 conn(0x7fc61806fde0 msgr2=0x7fc6180cbc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:55.180+0000 7fc61ea00640 1 -- 172.21.15.29:0/2910553663 shutdown_connections 2024-04-17T20:00:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:55.180+0000 7fc61ea00640 1 -- 172.21.15.29:0/2910553663 wait complete. 2024-04-17T20:00:55.199 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:00:57.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:57.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.376+0000 7ff1f4600640 1 Processor -- start 2024-04-17T20:00:57.374 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.376+0000 7ff1f4600640 1 -- start start 2024-04-17T20:00:57.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.376+0000 7ff1f4600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff1ec0c7a20 0x7ff1ec0710d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:57.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.376+0000 7ff1f4600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff1ec0c7e30 0x7ff1ec15a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:57.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.376+0000 7ff1f4600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1ec0c8ae0 con 0x7ff1ec15ab20 2024-04-17T20:00:57.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.376+0000 7ff1f4600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7ff1ec15a150 con 0x7ff1ec0c7a20 2024-04-17T20:00:57.376 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.376+0000 7ff1f4600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff1ec0c9660 con 0x7ff1ec0c7e30 2024-04-17T20:00:57.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.376+0000 7ff1f1c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff1ec0c7a20 0x7ff1ec0710d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:57.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.380+0000 7ff1f1c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff1ec0c7a20 0x7ff1ec0710d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33656/0 (socket says 172.21.15.29:33656) 2024-04-17T20:00:57.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.380+0000 7ff1f1c00640 1 -- 172.21.15.29:0/3011536407 learned_addr learned my addr 172.21.15.29:0/3011536407 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:57.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.380+0000 7ff1f1200640 1 --2- 172.21.15.29:0/3011536407 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff1ec0c7e30 0x7ff1ec15a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:57.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.380+0000 7ff1f1c00640 1 -- 172.21.15.29:0/3011536407 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff1ec0c7e30 msgr2=0x7ff1ec15a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:57.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.380+0000 7ff1f1c00640 1 --2- 172.21.15.29:0/3011536407 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff1ec0c7e30 0x7ff1ec15a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:57.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.380+0000 7ff1f1c00640 1 -- 172.21.15.29:0/3011536407 >> v1:172.21.15.37:6789/0 conn(0x7ff1ec15ab20 legacy=0x7ff1ec15db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:57.377 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.380+0000 7ff1f1c00640 1 -- 172.21.15.29:0/3011536407 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ec0c8dc0 con 0x7ff1ec0c7a20 2024-04-17T20:00:57.379 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.380+0000 7ff1f1c00640 1 --2- 172.21.15.29:0/3011536407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0c7a20 0x7ff1ec0710d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0009870 tx=0x7ff1e001ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6420858695bc1b54 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:57.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.380+0000 7ff1ebe00640 1 -- 172.21.15.29:0/3011536407 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e00b5070 con 0x7ff1ec0c7a20 2024-04-17T20:00:57.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.384+0000 7ff1ebe00640 1 -- 172.21.15.29:0/3011536407 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1e00a8a50 con 0x7ff1ec0c7a20 2024-04-17T20:00:57.380 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.384+0000 7ff1ebe00640 1 -- 172.21.15.29:0/3011536407 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e00a8e20 con 0x7ff1ec0c7a20 2024-04-17T20:00:57.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.384+0000 7ff1f4600640 1 -- 172.21.15.29:0/3011536407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0c7a20 msgr2=0x7ff1ec0710d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:57.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.384+0000 7ff1f4600640 1 --2- 172.21.15.29:0/3011536407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0c7a20 0x7ff1ec0710d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0009870 tx=0x7ff1e001ed50 comp rx=0 tx=0).stop 2024-04-17T20:00:57.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.384+0000 7ff1f4600640 1 -- 172.21.15.29:0/3011536407 shutdown_connections 2024-04-17T20:00:57.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.384+0000 7ff1f4600640 1 --2- 172.21.15.29:0/3011536407 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff1ec0c7e30 0x7ff1ec15a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:57.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.384+0000 7ff1f4600640 1 --2- 172.21.15.29:0/3011536407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0c7a20 0x7ff1ec0710d0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:57.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.384+0000 7ff1f4600640 1 -- 172.21.15.29:0/3011536407 >> 172.21.15.29:0/3011536407 conn(0x7ff1ec072e50 msgr2=0x7ff1ec073260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:57.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.384+0000 7ff1f4600640 1 -- 172.21.15.29:0/3011536407 shutdown_connections 2024-04-17T20:00:57.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.384+0000 7ff1f4600640 1 -- 172.21.15.29:0/3011536407 wait complete. 2024-04-17T20:00:57.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.384+0000 7ff1f4600640 1 Processor -- start 2024-04-17T20:00:57.385 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.384+0000 7ff1f4600640 1 -- start start 2024-04-17T20:00:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f4600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0c7a20 0x7ff1ec150ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f4600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff1ec0c7e30 0x7ff1ec141950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f4600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1ec15ab20 0x7ff1ec137810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f4600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec06bfe0 con 0x7ff1ec0c7a20 2024-04-17T20:00:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f4600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7ff1ec0c9380 con 0x7ff1ec15ab20 2024-04-17T20:00:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f4600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7ff1ec15e210 con 0x7ff1ec0c7e30 2024-04-17T20:00:57.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f1c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0c7a20 0x7ff1ec150ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:57.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f1c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0c7a20 0x7ff1ec150ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33664/0 (socket says 172.21.15.29:33664) 2024-04-17T20:00:57.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f1c00640 1 -- 172.21.15.29:0/3567876962 learned_addr learned my addr 172.21.15.29:0/3567876962 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:57.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f1200640 1 --2- 172.21.15.29:0/3567876962 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff1ec0c7e30 0x7ff1ec141950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:57.388 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f2600640 1 --2- 172.21.15.29:0/3567876962 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1ec15ab20 0x7ff1ec137810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:57.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f1c00640 1 -- 172.21.15.29:0/3567876962 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff1ec0c7e30 msgr2=0x7ff1ec141950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:57.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f1c00640 1 --2- 172.21.15.29:0/3567876962 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff1ec0c7e30 0x7ff1ec141950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:57.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f1c00640 1 -- 172.21.15.29:0/3567876962 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1ec15ab20 msgr2=0x7ff1ec137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:57.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f1c00640 1 --2- 172.21.15.29:0/3567876962 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1ec15ab20 0x7ff1ec137810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:57.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f1c00640 1 -- 172.21.15.29:0/3567876962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1ec151530 con 0x7ff1ec0c7a20 2024-04-17T20:00:57.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f1c00640 1 --2- 172.21.15.29:0/3567876962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0c7a20 0x7ff1ec150ff0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7ff1e001d400 tx=0x7ff1e001baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:57.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1e9600640 1 -- 172.21.15.29:0/3567876962 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e00c0070 con 0x7ff1ec0c7a20 2024-04-17T20:00:57.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1e9600640 1 -- 172.21.15.29:0/3567876962 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1e00ae550 con 0x7ff1ec0c7a20 2024-04-17T20:00:57.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f4600640 1 -- 172.21.15.29:0/3567876962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1ec1401e0 con 0x7ff1ec0c7a20 2024-04-17T20:00:57.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1e9600640 1 -- 172.21.15.29:0/3567876962 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e00b5070 con 0x7ff1ec0c7a20 2024-04-17T20:00:57.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.388+0000 7ff1f4600640 1 -- 172.21.15.29:0/3567876962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1ec0c6c70 con 0x7ff1ec0c7a20 2024-04-17T20:00:57.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.392+0000 7ff1f4600640 1 -- 172.21.15.29:0/3567876962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1ec069b90 con 0x7ff1ec0c7a20 2024-04-17T20:00:57.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.392+0000 7ff1e9600640 1 -- 172.21.15.29:0/3567876962 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7ff1e00c3070 con 0x7ff1ec0c7a20 2024-04-17T20:00:57.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.392+0000 7ff1e9600640 1 --2- 172.21.15.29:0/3567876962 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff1cc0683b0 0x7ff1cc06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:57.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.392+0000 7ff1e9600640 1 -- 172.21.15.29:0/3567876962 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7ff1e0124790 con 0x7ff1ec0c7a20 2024-04-17T20:00:57.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.392+0000 7ff1f1200640 1 --2- 172.21.15.29:0/3567876962 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff1cc0683b0 0x7ff1cc06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.396+0000 7ff1f1200640 1 --2- 172.21.15.29:0/3567876962 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff1cc0683b0 0x7ff1cc06a8c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec070b80 tx=0x7ff1d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.396+0000 7ff1e9600640 1 -- 172.21.15.29:0/3567876962 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7ff1e00188b0 con 0x7ff1ec0c7a20 2024-04-17T20:00:57.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.628+0000 7ff1f4600640 1 -- 172.21.15.29:0/3567876962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1ec154620 con 0x7ff1ec0c7a20 2024-04-17T20:00:57.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.628+0000 7ff1e9600640 1 -- 172.21.15.29:0/3567876962 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+6087 (secure 0 0 0) 0x7ff1e00f4740 con 0x7ff1ec0c7a20 2024-04-17T20:00:57.627 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:00:57.627 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:51.234057+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:clientreplay","state_seq":37,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5631,"qdb_cluster":[5144,5212,5350,5631]},"id":1}]} 2024-04-17T20:00:57.627 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 54 2024-04-17T20:00:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.632+0000 7ff1f4600640 1 -- 172.21.15.29:0/3567876962 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff1cc0683b0 msgr2=0x7ff1cc06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.632+0000 7ff1f4600640 1 --2- 172.21.15.29:0/3567876962 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff1cc0683b0 0x7ff1cc06a8c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec070b80 tx=0x7ff1d8008040 comp rx=0 tx=0).stop 2024-04-17T20:00:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.632+0000 7ff1f4600640 1 -- 172.21.15.29:0/3567876962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0c7a20 msgr2=0x7ff1ec150ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.632+0000 7ff1f4600640 1 --2- 172.21.15.29:0/3567876962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0c7a20 0x7ff1ec150ff0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7ff1e001d400 tx=0x7ff1e001baf0 comp rx=0 tx=0).stop 2024-04-17T20:00:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.632+0000 7ff1f4600640 1 -- 172.21.15.29:0/3567876962 shutdown_connections 2024-04-17T20:00:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.632+0000 7ff1f4600640 1 --2- 172.21.15.29:0/3567876962 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7ff1cc0683b0 0x7ff1cc06a8c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:57.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.632+0000 7ff1f4600640 1 --2- 172.21.15.29:0/3567876962 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7ff1ec15ab20 0x7ff1ec137810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.632+0000 7ff1f4600640 1 --2- 172.21.15.29:0/3567876962 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7ff1ec0c7e30 0x7ff1ec141950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.632+0000 7ff1f4600640 1 --2- 172.21.15.29:0/3567876962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0c7a20 0x7ff1ec150ff0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.632+0000 7ff1f4600640 1 -- 172.21.15.29:0/3567876962 >> 172.21.15.29:0/3567876962 conn(0x7ff1ec072e50 msgr2=0x7ff1ec06c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.632+0000 7ff1f4600640 1 -- 172.21.15.29:0/3567876962 shutdown_connections 2024-04-17T20:00:57.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:57.632+0000 7ff1f4600640 1 -- 172.21.15.29:0/3567876962 wait complete. 2024-04-17T20:00:57.649 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:00:59.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:00:59.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.820+0000 7fd642800640 1 Processor -- start 2024-04-17T20:00:59.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.820+0000 7fd642800640 1 -- start start 2024-04-17T20:00:59.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.820+0000 7fd642800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd63c15fdb0 0x7fd63c1601c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:59.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.820+0000 7fd642800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd63c0cae50 0x7fd63c0cb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:59.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.820+0000 7fd642800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd63c0cbc20 con 0x7fd63c0cc7f0 2024-04-17T20:00:59.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.820+0000 7fd642800640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fd63c15d140 con 0x7fd63c0cae50 2024-04-17T20:00:59.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.820+0000 7fd642800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd63c15c180 con 0x7fd63c15fdb0 2024-04-17T20:00:59.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.820+0000 7fd63be00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd63c0cae50 0x7fd63c0cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:59.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.820+0000 7fd63b400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd63c0cc7f0 0x7fd63c0c9ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57366/0 (socket says 172.21.15.29:57366) 2024-04-17T20:00:59.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.820+0000 7fd63be00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd63c0cae50 0x7fd63c0cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57186/0 (socket says 172.21.15.29:57186) 2024-04-17T20:00:59.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.820+0000 7fd63b400640 1 -- 172.21.15.29:0/1280941504 learned_addr learned my addr 172.21.15.29:0/1280941504 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:00:59.819 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.820+0000 7fd633e00640 1 --2- 172.21.15.29:0/1280941504 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd63c15fdb0 0x7fd63c1601c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:59.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.820+0000 7fd63be00640 1 -- 172.21.15.29:0/1280941504 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd63c15fdb0 msgr2=0x7fd63c1601c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:00:59.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.820+0000 7fd63be00640 1 --2- 172.21.15.29:0/1280941504 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd63c15fdb0 0x7fd63c1601c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:59.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.820+0000 7fd63be00640 1 -- 172.21.15.29:0/1280941504 >> v1:172.21.15.29:6789/0 conn(0x7fd63c0cc7f0 legacy=0x7fd63c0c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:59.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.820+0000 7fd63be00640 1 -- 172.21.15.29:0/1280941504 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd63c06ca70 con 0x7fd63c0cae50 2024-04-17T20:00:59.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.820+0000 7fd63aa00640 1 -- 172.21.15.29:0/1280941504 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3594224880 0 0) 0x7fd62c001120 con 0x7fd63c0cc7f0 2024-04-17T20:00:59.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.820+0000 7fd633e00640 1 --2- 172.21.15.29:0/1280941504 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd63c15fdb0 0x7fd63c1601c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:00:59.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.824+0000 7fd63be00640 1 --2- 172.21.15.29:0/1280941504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63c0cae50 0x7fd63c0cb260 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd634009dc0 tx=0x7fd63401c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b7569db051c081f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:59.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.824+0000 7fd63aa00640 1 -- 172.21.15.29:0/1280941504 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6340a6700 con 0x7fd63c0cae50 2024-04-17T20:00:59.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.824+0000 7fd63aa00640 1 -- 172.21.15.29:0/1280941504 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6340a68a0 con 0x7fd63c0cae50 2024-04-17T20:00:59.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.824+0000 7fd63aa00640 1 -- 172.21.15.29:0/1280941504 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6340a6c70 con 0x7fd63c0cae50 2024-04-17T20:00:59.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.824+0000 7fd642800640 1 -- 172.21.15.29:0/1280941504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63c0cae50 msgr2=0x7fd63c0cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:59.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.824+0000 7fd642800640 1 --2- 172.21.15.29:0/1280941504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63c0cae50 0x7fd63c0cb260 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fd634009dc0 tx=0x7fd63401c830 comp rx=0 tx=0).stop 2024-04-17T20:00:59.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.824+0000 7fd642800640 1 -- 172.21.15.29:0/1280941504 shutdown_connections 2024-04-17T20:00:59.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.824+0000 7fd642800640 1 --2- 172.21.15.29:0/1280941504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63c0cae50 0x7fd63c0cb260 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:59.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.824+0000 7fd642800640 1 --2- 172.21.15.29:0/1280941504 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd63c15fdb0 0x7fd63c1601c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:59.833 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.824+0000 7fd642800640 1 -- 172.21.15.29:0/1280941504 >> 172.21.15.29:0/1280941504 conn(0x7fd63c067eb0 msgr2=0x7fd63c070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:00:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.824+0000 7fd642800640 1 -- 172.21.15.29:0/1280941504 shutdown_connections 2024-04-17T20:00:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.824+0000 7fd642800640 1 -- 172.21.15.29:0/1280941504 wait complete. 2024-04-17T20:00:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.828+0000 7fd642800640 1 Processor -- start 2024-04-17T20:00:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.828+0000 7fd642800640 1 -- start start 2024-04-17T20:00:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.828+0000 7fd642800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd63c0cae50 0x7fd63c0cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.828+0000 7fd642800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63c0cc7f0 0x7fd63c154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.828+0000 7fd642800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd63c15fdb0 0x7fd63c152d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.828+0000 7fd642800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c06c790 con 0x7fd63c0cc7f0 2024-04-17T20:00:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.828+0000 7fd642800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c06c4b0 con 0x7fd63c15fdb0 2024-04-17T20:00:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.828+0000 7fd642800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd63c169240 con 0x7fd63c0cae50 2024-04-17T20:00:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.828+0000 7fd633e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63c0cc7f0 0x7fd63c154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.828+0000 7fd633e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63c0cc7f0 0x7fd63c154f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57192/0 (socket says 172.21.15.29:57192) 2024-04-17T20:00:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.828+0000 7fd633e00640 1 -- 172.21.15.29:0/2190521924 learned_addr learned my addr 172.21.15.29:0/2190521924 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:00:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.828+0000 7fd63be00640 1 --2- 172.21.15.29:0/2190521924 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd63c15fdb0 0x7fd63c152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:59.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.828+0000 7fd63b400640 1 --2- 172.21.15.29:0/2190521924 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd63c0cae50 0x7fd63c0cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:59.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.828+0000 7fd633e00640 1 -- 172.21.15.29:0/2190521924 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd63c0cae50 msgr2=0x7fd63c0cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:59.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.828+0000 7fd633e00640 1 --2- 172.21.15.29:0/2190521924 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd63c0cae50 0x7fd63c0cd3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:59.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.828+0000 7fd633e00640 1 -- 172.21.15.29:0/2190521924 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd63c15fdb0 msgr2=0x7fd63c152d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:00:59.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.828+0000 7fd633e00640 1 --2- 172.21.15.29:0/2190521924 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd63c15fdb0 0x7fd63c152d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:00:59.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.828+0000 7fd633e00640 1 -- 172.21.15.29:0/2190521924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd63c159bb0 con 0x7fd63c0cc7f0 2024-04-17T20:00:59.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.832+0000 7fd633e00640 1 --2- 172.21.15.29:0/2190521924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63c0cc7f0 0x7fd63c154f10 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd62400d830 tx=0x7fd62400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:59.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.832+0000 7fd633400640 1 -- 172.21.15.29:0/2190521924 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd624009640 con 0x7fd63c0cc7f0 2024-04-17T20:00:59.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.832+0000 7fd633400640 1 -- 172.21.15.29:0/2190521924 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd624019070 con 0x7fd63c0cc7f0 2024-04-17T20:00:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.832+0000 7fd633400640 1 -- 172.21.15.29:0/2190521924 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd624009a20 con 0x7fd63c0cc7f0 2024-04-17T20:00:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.832+0000 7fd642800640 1 -- 172.21.15.29:0/2190521924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd63c1f8ec0 con 0x7fd63c0cc7f0 2024-04-17T20:00:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.832+0000 7fd642800640 1 -- 172.21.15.29:0/2190521924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd63c1f9160 con 0x7fd63c0cc7f0 2024-04-17T20:00:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.832+0000 7fd642800640 1 -- 172.21.15.29:0/2190521924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd63c069b90 con 0x7fd63c0cc7f0 2024-04-17T20:00:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.832+0000 7fd633400640 1 -- 172.21.15.29:0/2190521924 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd624009bc0 con 0x7fd63c0cc7f0 2024-04-17T20:00:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.832+0000 7fd633400640 1 --2- 172.21.15.29:0/2190521924 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd618068380 0x7fd61806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:00:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.832+0000 7fd633400640 1 -- 172.21.15.29:0/2190521924 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7fd6240873a0 con 0x7fd63c0cc7f0 2024-04-17T20:00:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.836+0000 7fd63b400640 1 --2- 172.21.15.29:0/2190521924 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd618068380 0x7fd61806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:00:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.836+0000 7fd63b400640 1 --2- 172.21.15.29:0/2190521924 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd618068380 0x7fd61806a890 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd62c000f70 tx=0x7fd62c007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:00:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:00:59.840+0000 7fd633400640 1 -- 172.21.15.29:0/2190521924 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd6240573c0 con 0x7fd63c0cc7f0 2024-04-17T20:01:00.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.040+0000 7fd642800640 1 -- 172.21.15.29:0/2190521924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd63c0cb8b0 con 0x7fd63c0cc7f0 2024-04-17T20:01:00.037 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.040+0000 7fd633400640 1 -- 172.21.15.29:0/2190521924 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+6086 (secure 0 0 0) 0x7fd624056cc0 con 0x7fd63c0cc7f0 2024-04-17T20:01:00.037 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:01:00.038 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:58.083688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5144,5212,5350,5631,5775]},"id":1}]} 2024-04-17T20:01:00.038 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 55 2024-04-17T20:01:00.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.040+0000 7fd642800640 1 -- 172.21.15.29:0/2190521924 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd618068380 msgr2=0x7fd61806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:00.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.040+0000 7fd642800640 1 --2- 172.21.15.29:0/2190521924 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd618068380 0x7fd61806a890 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd62c000f70 tx=0x7fd62c007c10 comp rx=0 tx=0).stop 2024-04-17T20:01:00.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.040+0000 7fd642800640 1 -- 172.21.15.29:0/2190521924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63c0cc7f0 msgr2=0x7fd63c154f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:00.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.040+0000 7fd642800640 1 --2- 172.21.15.29:0/2190521924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63c0cc7f0 0x7fd63c154f10 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd62400d830 tx=0x7fd62400dcf0 comp rx=0 tx=0).stop 2024-04-17T20:01:00.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.040+0000 7fd642800640 1 -- 172.21.15.29:0/2190521924 shutdown_connections 2024-04-17T20:01:00.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.040+0000 7fd642800640 1 --2- 172.21.15.29:0/2190521924 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd618068380 0x7fd61806a890 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:00.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.040+0000 7fd642800640 1 --2- 172.21.15.29:0/2190521924 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd63c15fdb0 0x7fd63c152d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:00.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.040+0000 7fd642800640 1 --2- 172.21.15.29:0/2190521924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63c0cc7f0 0x7fd63c154f10 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:00.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.040+0000 7fd642800640 1 --2- 172.21.15.29:0/2190521924 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd63c0cae50 0x7fd63c0cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:00.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.040+0000 7fd642800640 1 -- 172.21.15.29:0/2190521924 >> 172.21.15.29:0/2190521924 conn(0x7fd63c067eb0 msgr2=0x7fd63c159ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:01:00.040 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.040+0000 7fd642800640 1 -- 172.21.15.29:0/2190521924 shutdown_connections 2024-04-17T20:01:00.041 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.040+0000 7fd642800640 1 -- 172.21.15.29:0/2190521924 wait complete. 2024-04-17T20:01:00.061 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=4305 2024-04-17T20:01:00.062 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.c 2024-04-17T20:01:00.062 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2024-04-17T20:01:00.062 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T20:01:00.062 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T20:01:00.065 INFO:tasks.ceph.mds.c:Started 2024-04-17T20:01:00.066 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:01:00.165 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2024-04-17T20:01:00.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.220+0000 7f0712600640 1 Processor -- start 2024-04-17T20:01:00.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.220+0000 7f0712600640 1 -- start start 2024-04-17T20:01:00.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.224+0000 7f0712600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f070c0cc8c0 0x7f070c0cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:00.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.224+0000 7f0712600640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f070c1597a0 0x7f070c1635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:00.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.224+0000 7f0712600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f070c0c88b0 con 0x7f070c159390 2024-04-17T20:01:00.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.224+0000 7f0712600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f070c0c6ce0 con 0x7f070c0cc8c0 2024-04-17T20:01:00.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.224+0000 7f0712600640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f070c0c6fc0 con 0x7f070c1597a0 2024-04-17T20:01:00.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.224+0000 7f070b400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f070c0cc8c0 0x7f070c0cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:00.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.224+0000 7f070aa00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f070c159390 0x7f070c0ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57382/0 (socket says 172.21.15.29:57382) 2024-04-17T20:01:00.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.224+0000 7f070aa00640 1 -- 172.21.15.29:0/4139055145 learned_addr learned my addr 172.21.15.29:0/4139055145 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:01:00.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.224+0000 7f070be00640 1 --2- 172.21.15.29:0/4139055145 >> v2:172.21.15.37:3300/0 conn(0x7f070c1597a0 0x7f070c1635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:00.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.224+0000 7f070b400640 1 -- 172.21.15.29:0/4139055145 >> v2:172.21.15.37:3300/0 conn(0x7f070c1597a0 msgr2=0x7f070c1635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:01:00.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.224+0000 7f070b400640 1 --2- 172.21.15.29:0/4139055145 >> v2:172.21.15.37:3300/0 conn(0x7f070c1597a0 0x7f070c1635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:00.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.224+0000 7f070b400640 1 -- 172.21.15.29:0/4139055145 >> v1:172.21.15.29:6789/0 conn(0x7f070c159390 legacy=0x7f070c0ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:00.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.224+0000 7f070b400640 1 -- 172.21.15.29:0/4139055145 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f070c0c72a0 con 0x7f070c0cc8c0 2024-04-17T20:01:00.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.224+0000 7f070be00640 1 --2- 172.21.15.29:0/4139055145 >> v2:172.21.15.37:3300/0 conn(0x7f070c1597a0 0x7f070c1635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:01:00.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.228+0000 7f070b400640 1 --2- 172.21.15.29:0/4139055145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f070c0cc8c0 0x7f070c0cccd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f070c0722d0 tx=0x7f06fc01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21bf846a421f2a8e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:00.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.228+0000 7f070a000640 1 -- 172.21.15.29:0/4139055145 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06fc0b0070 con 0x7f070c0cc8c0 2024-04-17T20:01:00.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.228+0000 7f070a000640 1 -- 172.21.15.29:0/4139055145 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06fc0ab600 con 0x7f070c0cc8c0 2024-04-17T20:01:00.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.228+0000 7f070a000640 1 -- 172.21.15.29:0/4139055145 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06fc0ab9d0 con 0x7f070c0cc8c0 2024-04-17T20:01:00.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.228+0000 7f0712600640 1 -- 172.21.15.29:0/4139055145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f070c0cc8c0 msgr2=0x7f070c0cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:00.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.228+0000 7f0712600640 1 --2- 172.21.15.29:0/4139055145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f070c0cc8c0 0x7f070c0cccd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f070c0722d0 tx=0x7f06fc01c480 comp rx=0 tx=0).stop 2024-04-17T20:01:00.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.228+0000 7f0712600640 1 -- 172.21.15.29:0/4139055145 shutdown_connections 2024-04-17T20:01:00.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.228+0000 7f0712600640 1 --2- 172.21.15.29:0/4139055145 >> v2:172.21.15.37:3300/0 conn(0x7f070c1597a0 0x7f070c1635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:00.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.228+0000 7f0712600640 1 --2- 172.21.15.29:0/4139055145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f070c0cc8c0 0x7f070c0cccd0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:00.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.228+0000 7f0712600640 1 -- 172.21.15.29:0/4139055145 >> 172.21.15.29:0/4139055145 conn(0x7f070c06fde0 msgr2=0x7f070c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:01:00.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.228+0000 7f0712600640 1 -- 172.21.15.29:0/4139055145 shutdown_connections 2024-04-17T20:01:00.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.228+0000 7f0712600640 1 -- 172.21.15.29:0/4139055145 wait complete. 2024-04-17T20:01:00.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f0712600640 1 Processor -- start 2024-04-17T20:01:00.229 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f0712600640 1 -- start start 2024-04-17T20:01:00.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f0712600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f070c0cc8c0 0x7f070c1f7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:00.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f0712600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f070c159390 0x7f070c15ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:00.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f0712600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f070c1597a0 0x7f070c15a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:00.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f0712600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f070c0cabc0 con 0x7f070c1597a0 2024-04-17T20:01:00.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f0712600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f070c0caa40 con 0x7f070c159390 2024-04-17T20:01:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f0712600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f070c0c6880 con 0x7f070c0cc8c0 2024-04-17T20:01:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f070be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f070c1597a0 0x7f070c15a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f070be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f070c1597a0 0x7f070c15a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57238/0 (socket says 172.21.15.29:57238) 2024-04-17T20:01:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f070be00640 1 -- 172.21.15.29:0/2954706429 learned_addr learned my addr 172.21.15.29:0/2954706429 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:01:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f070aa00640 1 --2- 172.21.15.29:0/2954706429 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f070c159390 0x7f070c15ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f070b400640 1 --2- 172.21.15.29:0/2954706429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f070c0cc8c0 0x7f070c1f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f070be00640 1 -- 172.21.15.29:0/2954706429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f070c0cc8c0 msgr2=0x7f070c1f7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f070be00640 1 --2- 172.21.15.29:0/2954706429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f070c0cc8c0 0x7f070c1f7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f070be00640 1 -- 172.21.15.29:0/2954706429 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f070c159390 msgr2=0x7f070c15ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f070be00640 1 --2- 172.21.15.29:0/2954706429 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f070c159390 0x7f070c15ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f070be00640 1 -- 172.21.15.29:0/2954706429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f070c15cf70 con 0x7f070c1597a0 2024-04-17T20:01:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f070be00640 1 --2- 172.21.15.29:0/2954706429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f070c1597a0 0x7f070c15a5c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f0704007e90 tx=0x7f070400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f0703400640 1 -- 172.21.15.29:0/2954706429 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f070401a070 con 0x7f070c1597a0 2024-04-17T20:01:00.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f0703400640 1 -- 172.21.15.29:0/2954706429 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0704015400 con 0x7f070c1597a0 2024-04-17T20:01:00.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f0712600640 1 -- 172.21.15.29:0/2954706429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f070c15bc90 con 0x7f070c1597a0 2024-04-17T20:01:00.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f0703400640 1 -- 172.21.15.29:0/2954706429 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07040157f0 con 0x7f070c1597a0 2024-04-17T20:01:00.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.232+0000 7f0712600640 1 -- 172.21.15.29:0/2954706429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f070c15c230 con 0x7f070c1597a0 2024-04-17T20:01:00.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.236+0000 7f0712600640 1 -- 172.21.15.29:0/2954706429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f070c153050 con 0x7f070c1597a0 2024-04-17T20:01:00.233 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.236+0000 7f0703400640 1 -- 172.21.15.29:0/2954706429 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f070401d070 con 0x7f070c1597a0 2024-04-17T20:01:00.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.236+0000 7f0703400640 1 --2- 172.21.15.29:0/2954706429 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f06ec068380 0x7f06ec06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:00.234 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.236+0000 7f0703400640 1 -- 172.21.15.29:0/2954706429 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7f0704087220 con 0x7f070c1597a0 2024-04-17T20:01:00.235 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.236+0000 7f070b400640 1 --2- 172.21.15.29:0/2954706429 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f06ec068380 0x7f06ec06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:00.238 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.236+0000 7f070b400640 1 --2- 172.21.15.29:0/2954706429 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f06ec068380 0x7f06ec06a890 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f06fc0ad000 tx=0x7f06fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:00.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.240+0000 7f0703400640 1 -- 172.21.15.29:0/2954706429 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f0704010070 con 0x7f070c1597a0 2024-04-17T20:01:00.438 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.440+0000 7f0712600640 1 -- 172.21.15.29:0/2954706429 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f070c069b90 con 0x7f070c1597a0 2024-04-17T20:01:00.440 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.440+0000 7f0703400640 1 -- 172.21.15.29:0/2954706429 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+6086 (secure 0 0 0) 0x7f0704056ae0 con 0x7f070c1597a0 2024-04-17T20:01:00.440 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:01:00.440 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:58.083688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5144,5212,5350,5631,5775]},"id":1}]} 2024-04-17T20:01:00.442 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 55 2024-04-17T20:01:00.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.444+0000 7f0712600640 1 -- 172.21.15.29:0/2954706429 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f06ec068380 msgr2=0x7f06ec06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:00.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.444+0000 7f0712600640 1 --2- 172.21.15.29:0/2954706429 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f06ec068380 0x7f06ec06a890 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f06fc0ad000 tx=0x7f06fc008040 comp rx=0 tx=0).stop 2024-04-17T20:01:00.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.444+0000 7f0712600640 1 -- 172.21.15.29:0/2954706429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f070c1597a0 msgr2=0x7f070c15a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:00.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.444+0000 7f0712600640 1 --2- 172.21.15.29:0/2954706429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f070c1597a0 0x7f070c15a5c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f0704007e90 tx=0x7f070400d4a0 comp rx=0 tx=0).stop 2024-04-17T20:01:00.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.444+0000 7f0712600640 1 -- 172.21.15.29:0/2954706429 shutdown_connections 2024-04-17T20:01:00.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.444+0000 7f0712600640 1 --2- 172.21.15.29:0/2954706429 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f06ec068380 0x7f06ec06a890 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:00.442 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.444+0000 7f0712600640 1 --2- 172.21.15.29:0/2954706429 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f070c1597a0 0x7f070c15a5c0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:00.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.444+0000 7f0712600640 1 --2- 172.21.15.29:0/2954706429 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f070c159390 0x7f070c15ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:00.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.444+0000 7f0712600640 1 --2- 172.21.15.29:0/2954706429 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f070c0cc8c0 0x7f070c1f7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:00.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.444+0000 7f0712600640 1 -- 172.21.15.29:0/2954706429 >> 172.21.15.29:0/2954706429 conn(0x7f070c06fde0 msgr2=0x7f070c06bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:01:00.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.444+0000 7f0712600640 1 -- 172.21.15.29:0/2954706429 shutdown_connections 2024-04-17T20:01:00.443 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:00.444+0000 7f0712600640 1 -- 172.21.15.29:0/2954706429 wait complete. 2024-04-17T20:01:00.461 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-04-17T20:01:02.461 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:01:02.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.628+0000 7f821c600640 1 Processor -- start 2024-04-17T20:01:02.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.628+0000 7f821c600640 1 -- start start 2024-04-17T20:01:02.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.628+0000 7f821c600640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f82140cc8c0 0x7f82140cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:02.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.628+0000 7f821c600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82140c88b0 con 0x7f82141597a0 2024-04-17T20:01:02.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.628+0000 7f821c600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82140c6ce0 con 0x7f8214159390 2024-04-17T20:01:02.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.628+0000 7f821c600640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f82140c6fc0 con 0x7f82140cc8c0 2024-04-17T20:01:02.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.628+0000 7f821a600640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f82141597a0 0x7f82141635e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57386/0 (socket says 172.21.15.29:57386) 2024-04-17T20:01:02.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.628+0000 7f821a600640 1 -- 172.21.15.29:0/2737854146 learned_addr learned my addr 172.21.15.29:0/2737854146 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:01:02.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.628+0000 7f8219c00640 1 --2- 172.21.15.29:0/2737854146 >> v2:172.21.15.37:3300/0 conn(0x7f82140cc8c0 0x7f82140cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:02.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8213e00640 1 -- 172.21.15.29:0/2737854146 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 315753373 0 0) 0x7f82140c88b0 con 0x7f82141597a0 2024-04-17T20:01:02.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8213e00640 1 -- 172.21.15.29:0/2737854146 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81fc005740 con 0x7f82141597a0 2024-04-17T20:01:02.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8213e00640 1 -- 172.21.15.29:0/2737854146 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2240655160 0 0) 0x7f82140c6ce0 con 0x7f8214159390 2024-04-17T20:01:02.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8213e00640 1 -- 172.21.15.29:0/2737854146 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82140c88b0 con 0x7f8214159390 2024-04-17T20:01:02.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8213e00640 1 -- 172.21.15.29:0/2737854146 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4159159580 0 0) 0x7f81fc005740 con 0x7f82141597a0 2024-04-17T20:01:02.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8213e00640 1 -- 172.21.15.29:0/2737854146 >> v2:172.21.15.37:3300/0 conn(0x7f82140cc8c0 msgr2=0x7f82140cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:01:02.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8213e00640 1 --2- 172.21.15.29:0/2737854146 >> v2:172.21.15.37:3300/0 conn(0x7f82140cc8c0 0x7f82140cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:02.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8213e00640 1 -- 172.21.15.29:0/2737854146 >> v1:172.21.15.37:6789/0 conn(0x7f8214159390 legacy=0x7f82140ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:02.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8213e00640 1 -- 172.21.15.29:0/2737854146 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82140c72a0 con 0x7f82141597a0 2024-04-17T20:01:02.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8213e00640 1 -- 172.21.15.29:0/2737854146 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f820c001a00 con 0x7f82141597a0 2024-04-17T20:01:02.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8213e00640 1 -- 172.21.15.29:0/2737854146 >> v1:172.21.15.29:6789/0 conn(0x7f82141597a0 legacy=0x7f82141635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:02.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8213e00640 1 --2- 172.21.15.29:0/2737854146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81fc005e20 0x7f81fc006270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:02.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8213e00640 1 --2- 172.21.15.29:0/2737854146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81fc006a20 0x7f81fc006e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:02.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8213e00640 1 --2- 172.21.15.29:0/2737854146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc007480 0x7f81fc00dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:02.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8213e00640 1 -- 172.21.15.29:0/2737854146 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f82140c6fc0 con 0x7f81fc005e20 2024-04-17T20:01:02.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8213e00640 1 -- 172.21.15.29:0/2737854146 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f81fc00e420 con 0x7f81fc006a20 2024-04-17T20:01:02.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8213e00640 1 -- 172.21.15.29:0/2737854146 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f81fc00e620 con 0x7f81fc007480 2024-04-17T20:01:02.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8219c00640 1 --2- 172.21.15.29:0/2737854146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81fc005e20 0x7f81fc006270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:02.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8219200640 1 --2- 172.21.15.29:0/2737854146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81fc006a20 0x7f81fc006e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:02.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f821a600640 1 --2- 172.21.15.29:0/2737854146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc007480 0x7f81fc00dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:02.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8219c00640 1 -- 172.21.15.29:0/2737854146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc007480 msgr2=0x7f81fc00dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:01:02.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8219c00640 1 --2- 172.21.15.29:0/2737854146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc007480 0x7f81fc00dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:02.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8219c00640 1 -- 172.21.15.29:0/2737854146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81fc006a20 msgr2=0x7f81fc006e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:01:02.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8219c00640 1 --2- 172.21.15.29:0/2737854146 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f81fc006a20 0x7f81fc006e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:02.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f8219c00640 1 -- 172.21.15.29:0/2737854146 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81fc005740 con 0x7f81fc005e20 2024-04-17T20:01:02.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f821a600640 1 --2- 172.21.15.29:0/2737854146 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f81fc007480 0x7f81fc00dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T20:01:02.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.632+0000 7f821a600640 1 -- 172.21.15.29:0/2737854146 reap_dead start 2024-04-17T20:01:02.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.636+0000 7f8219c00640 1 --2- 172.21.15.29:0/2737854146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81fc005e20 0x7f81fc006270 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f8208009ae0 tx=0x7f820801ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5891eee4c76f9ea5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:02.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.636+0000 7f8213e00640 1 -- 172.21.15.29:0/2737854146 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82080096b0 con 0x7f81fc005e20 2024-04-17T20:01:02.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.636+0000 7f8213e00640 1 -- 172.21.15.29:0/2737854146 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82080a6450 con 0x7f81fc005e20 2024-04-17T20:01:02.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.636+0000 7f8213e00640 1 -- 172.21.15.29:0/2737854146 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82080a6840 con 0x7f81fc005e20 2024-04-17T20:01:02.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.636+0000 7f821c600640 1 -- 172.21.15.29:0/2737854146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81fc005e20 msgr2=0x7f81fc006270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:02.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.636+0000 7f821c600640 1 --2- 172.21.15.29:0/2737854146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81fc005e20 0x7f81fc006270 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f8208009ae0 tx=0x7f820801ca70 comp rx=0 tx=0).stop 2024-04-17T20:01:02.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.636+0000 7f821c600640 1 -- 172.21.15.29:0/2737854146 shutdown_connections 2024-04-17T20:01:02.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.636+0000 7f821c600640 1 --2- 172.21.15.29:0/2737854146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81fc005e20 0x7f81fc006270 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:02.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.636+0000 7f821c600640 1 -- 172.21.15.29:0/2737854146 >> 172.21.15.29:0/2737854146 conn(0x7f821406fde0 msgr2=0x7f82140701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:01:02.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.636+0000 7f821c600640 1 -- 172.21.15.29:0/2737854146 shutdown_connections 2024-04-17T20:01:02.635 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.636+0000 7f821c600640 1 -- 172.21.15.29:0/2737854146 wait complete. 2024-04-17T20:01:02.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.636+0000 7f821c600640 1 Processor -- start 2024-04-17T20:01:02.636 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.636+0000 7f821c600640 1 -- start start 2024-04-17T20:01:02.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f821c600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81fc005e20 0x7f82141f60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:02.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f821c600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82141e9ad0 0x7f82141e8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:02.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f821c600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f821413a800 0x7f82140c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:02.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f821c600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f82140c6880 con 0x7f81fc005e20 2024-04-17T20:01:02.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f821c600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f8214159bb0 con 0x7f821413a800 2024-04-17T20:01:02.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f821c600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f82140c6a00 con 0x7f82141e9ad0 2024-04-17T20:01:02.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f8219c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81fc005e20 0x7f82141f60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:02.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f8219c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81fc005e20 0x7f82141f60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57242/0 (socket says 172.21.15.29:57242) 2024-04-17T20:01:02.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f8219c00640 1 -- 172.21.15.29:0/4065661476 learned_addr learned my addr 172.21.15.29:0/4065661476 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:01:02.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f821a600640 1 --2- 172.21.15.29:0/4065661476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f821413a800 0x7f82140c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:02.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f8219200640 1 --2- 172.21.15.29:0/4065661476 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82141e9ad0 0x7f82141e8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:02.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f8219c00640 1 -- 172.21.15.29:0/4065661476 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82141e9ad0 msgr2=0x7f82141e8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:02.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f8219c00640 1 --2- 172.21.15.29:0/4065661476 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82141e9ad0 0x7f82141e8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:02.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f8219c00640 1 -- 172.21.15.29:0/4065661476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f821413a800 msgr2=0x7f82140c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:02.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f8219c00640 1 --2- 172.21.15.29:0/4065661476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f821413a800 0x7f82140c8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:02.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f8219c00640 1 -- 172.21.15.29:0/4065661476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82141578b0 con 0x7f81fc005e20 2024-04-17T20:01:02.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f8219200640 1 --2- 172.21.15.29:0/4065661476 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82141e9ad0 0x7f82141e8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T20:01:02.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f8219c00640 1 --2- 172.21.15.29:0/4065661476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81fc005e20 0x7f82141f60e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f8208009ae0 tx=0x7f8208019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:02.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f8211600640 1 -- 172.21.15.29:0/4065661476 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82080a6c80 con 0x7f81fc005e20 2024-04-17T20:01:02.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f8211600640 1 -- 172.21.15.29:0/4065661476 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82080a6e20 con 0x7f81fc005e20 2024-04-17T20:01:02.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f8211600640 1 -- 172.21.15.29:0/4065661476 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82080b9660 con 0x7f81fc005e20 2024-04-17T20:01:02.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f821c600640 1 -- 172.21.15.29:0/4065661476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82141ea5f0 con 0x7f81fc005e20 2024-04-17T20:01:02.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f821c600640 1 -- 172.21.15.29:0/4065661476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82141f5600 con 0x7f81fc005e20 2024-04-17T20:01:02.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.640+0000 7f821c600640 1 -- 172.21.15.29:0/4065661476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8214069b90 con 0x7f81fc005e20 2024-04-17T20:01:02.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.644+0000 7f8211600640 1 -- 172.21.15.29:0/4065661476 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f82080a7430 con 0x7f81fc005e20 2024-04-17T20:01:02.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.644+0000 7f8211600640 1 --2- 172.21.15.29:0/4065661476 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f81f4068380 0x7f81f406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:02.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.644+0000 7f8211600640 1 -- 172.21.15.29:0/4065661476 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7f8208121290 con 0x7f81fc005e20 2024-04-17T20:01:02.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.644+0000 7f8219200640 1 --2- 172.21.15.29:0/4065661476 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f81f4068380 0x7f81f406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:02.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.644+0000 7f8219200640 1 --2- 172.21.15.29:0/4065661476 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f81f4068380 0x7f81f406a890 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8200007fd0 tx=0x7f8200009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:02.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.648+0000 7f8211600640 1 -- 172.21.15.29:0/4065661476 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f82080bf060 con 0x7f81fc005e20 2024-04-17T20:01:02.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.848+0000 7f821c600640 1 -- 172.21.15.29:0/4065661476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82140c7e10 con 0x7f81fc005e20 2024-04-17T20:01:02.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.852+0000 7f8211600640 1 -- 172.21.15.29:0/4065661476 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6910 (secure 0 0 0) 0x7f82080ac020 con 0x7f81fc005e20 2024-04-17T20:01:02.850 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:01:02.850 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5946,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:58.083688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775,5631,5350,5212,5144]},"id":1}]} 2024-04-17T20:01:02.851 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 56 2024-04-17T20:01:02.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.852+0000 7f821c600640 1 -- 172.21.15.29:0/4065661476 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f81f4068380 msgr2=0x7f81f406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:02.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.852+0000 7f821c600640 1 --2- 172.21.15.29:0/4065661476 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f81f4068380 0x7f81f406a890 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8200007fd0 tx=0x7f8200009040 comp rx=0 tx=0).stop 2024-04-17T20:01:02.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.852+0000 7f821c600640 1 -- 172.21.15.29:0/4065661476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81fc005e20 msgr2=0x7f82141f60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:02.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.852+0000 7f821c600640 1 --2- 172.21.15.29:0/4065661476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81fc005e20 0x7f82141f60e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f8208009ae0 tx=0x7f8208019810 comp rx=0 tx=0).stop 2024-04-17T20:01:02.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.852+0000 7f821c600640 1 -- 172.21.15.29:0/4065661476 shutdown_connections 2024-04-17T20:01:02.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.852+0000 7f821c600640 1 --2- 172.21.15.29:0/4065661476 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f81f4068380 0x7f81f406a890 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:02.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.852+0000 7f821c600640 1 --2- 172.21.15.29:0/4065661476 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f821413a800 0x7f82140c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:02.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.852+0000 7f821c600640 1 --2- 172.21.15.29:0/4065661476 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f82141e9ad0 0x7f82141e8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:02.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.852+0000 7f821c600640 1 --2- 172.21.15.29:0/4065661476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81fc005e20 0x7f82141f60e0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:02.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.852+0000 7f821c600640 1 -- 172.21.15.29:0/4065661476 >> 172.21.15.29:0/4065661476 conn(0x7f821406fde0 msgr2=0x7f821413e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:01:02.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.852+0000 7f821c600640 1 -- 172.21.15.29:0/4065661476 shutdown_connections 2024-04-17T20:01:02.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:02.852+0000 7f821c600640 1 -- 172.21.15.29:0/4065661476 wait complete. 2024-04-17T20:01:02.871 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2024-04-17T20:01:02.871 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 48 secs before thrashing 2024-04-17T20:01:07.049 INFO:tasks.ceph.mds.e.smithi029.stderr:2024-04-17T20:01:07.052+0000 7f956e000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T20:01:07.050 INFO:tasks.ceph.mds.e.smithi029.stderr:2024-04-17T20:01:07.052+0000 7f956e000640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-17T20:01:07.050 INFO:tasks.ceph.mds.e.smithi029.stderr:2024-04-17T20:01:07.052+0000 7f956e000640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-17T20:01:07.050 INFO:tasks.ceph.mds.e.smithi029.stderr:2024-04-17T20:01:07.052+0000 7f956e000640 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-17T20:01:07.050 INFO:tasks.ceph.mds.e.smithi029.stderr:2024-04-17T20:01:07.052+0000 7f956e000640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-17T20:01:26.549 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:01:26.554 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:01:50.874 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:01:51.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.044+0000 7fe3afa00640 1 Processor -- start 2024-04-17T20:01:51.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.044+0000 7fe3afa00640 1 -- start start 2024-04-17T20:01:51.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.048+0000 7fe3afa00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe3a80cc8c0 0x7fe3a80cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:51.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.048+0000 7fe3afa00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe3a8159390 0x7fe3a81597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:51.047 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.048+0000 7fe3afa00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3a80c88b0 con 0x7fe3a80ca330 2024-04-17T20:01:51.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.048+0000 7fe3afa00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fe3a80c6ce0 con 0x7fe3a8159390 2024-04-17T20:01:51.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.048+0000 7fe3afa00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe3a80c6fc0 con 0x7fe3a80cc8c0 2024-04-17T20:01:51.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.048+0000 7fe3ad000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe3a80cc8c0 0x7fe3a80cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:51.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.048+0000 7fe3a7e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe3a8159390 0x7fe3a81597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:51.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.048+0000 7fe3ada00640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe3a80ca330 0x7fe3a815c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.29:54396/0 (socket says 172.21.15.29:54396) 2024-04-17T20:01:51.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.048+0000 7fe3ada00640 1 -- 172.21.15.29:0/2906024606 learned_addr learned my addr 172.21.15.29:0/2906024606 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:01:51.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.048+0000 7fe3ad000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe3a80cc8c0 0x7fe3a80cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:33228/0 (socket says 172.21.15.29:33228) 2024-04-17T20:01:51.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.048+0000 7fe3a7400640 1 -- 172.21.15.29:0/2906024606 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2460050307 0 0) 0x7fe3a80c88b0 con 0x7fe3a80ca330 2024-04-17T20:01:51.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.048+0000 7fe3a7400640 1 -- 172.21.15.29:0/2906024606 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe390003530 con 0x7fe3a80ca330 2024-04-17T20:01:51.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.048+0000 7fe3ad000640 1 -- 172.21.15.29:0/2906024606 >> v2:172.21.15.37:3300/0 conn(0x7fe3a8159390 msgr2=0x7fe3a81597a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:01:51.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.048+0000 7fe3ad000640 1 --2- 172.21.15.29:0/2906024606 >> v2:172.21.15.37:3300/0 conn(0x7fe3a8159390 0x7fe3a81597a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.048+0000 7fe3ad000640 1 -- 172.21.15.29:0/2906024606 >> v1:172.21.15.37:6789/0 conn(0x7fe3a80ca330 legacy=0x7fe3a815c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:51.048 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.048+0000 7fe3ad000640 1 -- 172.21.15.29:0/2906024606 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a80c72a0 con 0x7fe3a80cc8c0 2024-04-17T20:01:51.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.048+0000 7fe3a7e00640 1 --2- 172.21.15.29:0/2906024606 >> v2:172.21.15.37:3300/0 conn(0x7fe3a8159390 0x7fe3a81597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T20:01:51.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.052+0000 7fe3ad000640 1 --2- 172.21.15.29:0/2906024606 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe3a80cc8c0 0x7fe3a80cccd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe39c0097c0 tx=0x7fe39c01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31a50ab1ce2ee3ad server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:51.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.052+0000 7fe3a7400640 1 -- 172.21.15.29:0/2906024606 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c0b0070 con 0x7fe3a80cc8c0 2024-04-17T20:01:51.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.052+0000 7fe3a7400640 1 -- 172.21.15.29:0/2906024606 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe39c0ab600 con 0x7fe3a80cc8c0 2024-04-17T20:01:51.051 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.052+0000 7fe3a7400640 1 -- 172.21.15.29:0/2906024606 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39c0ab9b0 con 0x7fe3a80cc8c0 2024-04-17T20:01:51.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.052+0000 7fe3afa00640 1 -- 172.21.15.29:0/2906024606 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe3a80cc8c0 msgr2=0x7fe3a80cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:51.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.052+0000 7fe3afa00640 1 --2- 172.21.15.29:0/2906024606 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe3a80cc8c0 0x7fe3a80cccd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe39c0097c0 tx=0x7fe39c01c4a0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.052+0000 7fe3afa00640 1 -- 172.21.15.29:0/2906024606 shutdown_connections 2024-04-17T20:01:51.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.052+0000 7fe3afa00640 1 --2- 172.21.15.29:0/2906024606 >> v2:172.21.15.37:3300/0 conn(0x7fe3a8159390 0x7fe3a81597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.052+0000 7fe3afa00640 1 --2- 172.21.15.29:0/2906024606 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe3a80cc8c0 0x7fe3a80cccd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.052+0000 7fe3afa00640 1 -- 172.21.15.29:0/2906024606 >> 172.21.15.29:0/2906024606 conn(0x7fe3a806fde0 msgr2=0x7fe3a80701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:01:51.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.052+0000 7fe3afa00640 1 -- 172.21.15.29:0/2906024606 shutdown_connections 2024-04-17T20:01:51.052 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.052+0000 7fe3afa00640 1 -- 172.21.15.29:0/2906024606 wait complete. 2024-04-17T20:01:51.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3afa00640 1 Processor -- start 2024-04-17T20:01:51.053 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3afa00640 1 -- start start 2024-04-17T20:01:51.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3afa00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe3a80ca330 0x7fe3a81f8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:51.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3afa00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe3a80cc8c0 0x7fe3a8159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:51.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3afa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3a8159390 0x7fe3a81410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:51.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3afa00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a80c6a00 con 0x7fe3a8159390 2024-04-17T20:01:51.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3afa00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a80c6880 con 0x7fe3a80cc8c0 2024-04-17T20:01:51.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3afa00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe3a81645f0 con 0x7fe3a80ca330 2024-04-17T20:01:51.055 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3ada00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3a8159390 0x7fe3a81410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:51.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3ada00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3a8159390 0x7fe3a81410f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32962/0 (socket says 172.21.15.29:32962) 2024-04-17T20:01:51.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3ada00640 1 -- 172.21.15.29:0/884045887 learned_addr learned my addr 172.21.15.29:0/884045887 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:01:51.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3a7e00640 1 --2- 172.21.15.29:0/884045887 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe3a80cc8c0 0x7fe3a8159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:51.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3ad000640 1 --2- 172.21.15.29:0/884045887 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe3a80ca330 0x7fe3a81f8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:51.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3ada00640 1 -- 172.21.15.29:0/884045887 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe3a80ca330 msgr2=0x7fe3a81f8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:51.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3ada00640 1 --2- 172.21.15.29:0/884045887 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe3a80ca330 0x7fe3a81f8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3ada00640 1 -- 172.21.15.29:0/884045887 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe3a80cc8c0 msgr2=0x7fe3a8159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:51.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3ada00640 1 --2- 172.21.15.29:0/884045887 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe3a80cc8c0 0x7fe3a8159e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3ada00640 1 -- 172.21.15.29:0/884045887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a80cd7e0 con 0x7fe3a8159390 2024-04-17T20:01:51.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3ada00640 1 --2- 172.21.15.29:0/884045887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3a8159390 0x7fe3a81410f0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe3a000cad0 tx=0x7fe3a000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:51.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3a4c00640 1 -- 172.21.15.29:0/884045887 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0017070 con 0x7fe3a8159390 2024-04-17T20:01:51.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3a4c00640 1 -- 172.21.15.29:0/884045887 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3a000abd0 con 0x7fe3a8159390 2024-04-17T20:01:51.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3a4c00640 1 -- 172.21.15.29:0/884045887 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0012400 con 0x7fe3a8159390 2024-04-17T20:01:51.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3afa00640 1 -- 172.21.15.29:0/884045887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3a81f94f0 con 0x7fe3a8159390 2024-04-17T20:01:51.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.056+0000 7fe3afa00640 1 -- 172.21.15.29:0/884045887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3a815be40 con 0x7fe3a8159390 2024-04-17T20:01:51.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.060+0000 7fe3afa00640 1 -- 172.21.15.29:0/884045887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3a815bba0 con 0x7fe3a8159390 2024-04-17T20:01:51.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.060+0000 7fe3a4c00640 1 -- 172.21.15.29:0/884045887 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe3a001a070 con 0x7fe3a8159390 2024-04-17T20:01:51.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.060+0000 7fe3a4c00640 1 --2- 172.21.15.29:0/884045887 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe388068380 0x7fe38806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:51.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.060+0000 7fe3a4c00640 1 -- 172.21.15.29:0/884045887 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7fe3a0086d80 con 0x7fe3a8159390 2024-04-17T20:01:51.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.060+0000 7fe3ad000640 1 --2- 172.21.15.29:0/884045887 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe388068380 0x7fe38806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:51.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.060+0000 7fe3ad000640 1 --2- 172.21.15.29:0/884045887 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe388068380 0x7fe38806a890 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe39c0ad000 tx=0x7fe39c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:51.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.064+0000 7fe3a4c00640 1 -- 172.21.15.29:0/884045887 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe3a815bba0 con 0x7fe3a8159390 2024-04-17T20:01:51.264 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.268+0000 7fe3afa00640 1 -- 172.21.15.29:0/884045887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3a8069b90 con 0x7fe3a8159390 2024-04-17T20:01:51.265 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.268+0000 7fe3a4c00640 1 -- 172.21.15.29:0/884045887 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6910 (secure 0 0 0) 0x7fe3a00565b0 con 0x7fe3a8159390 2024-04-17T20:01:51.265 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:01:51.266 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5946,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:58.083688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775,5631,5350,5212,5144]},"id":1}]} 2024-04-17T20:01:51.266 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 56 2024-04-17T20:01:51.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.268+0000 7fe3afa00640 1 -- 172.21.15.29:0/884045887 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe388068380 msgr2=0x7fe38806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:51.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.268+0000 7fe3afa00640 1 --2- 172.21.15.29:0/884045887 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe388068380 0x7fe38806a890 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe39c0ad000 tx=0x7fe39c0a6000 comp rx=0 tx=0).stop 2024-04-17T20:01:51.267 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.268+0000 7fe3afa00640 1 -- 172.21.15.29:0/884045887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3a8159390 msgr2=0x7fe3a81410f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:51.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.268+0000 7fe3afa00640 1 --2- 172.21.15.29:0/884045887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3a8159390 0x7fe3a81410f0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe3a000cad0 tx=0x7fe3a000cf90 comp rx=0 tx=0).stop 2024-04-17T20:01:51.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.268+0000 7fe3afa00640 1 -- 172.21.15.29:0/884045887 shutdown_connections 2024-04-17T20:01:51.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.268+0000 7fe3afa00640 1 --2- 172.21.15.29:0/884045887 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe388068380 0x7fe38806a890 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.268+0000 7fe3afa00640 1 --2- 172.21.15.29:0/884045887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3a8159390 0x7fe3a81410f0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.268+0000 7fe3afa00640 1 --2- 172.21.15.29:0/884045887 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe3a80cc8c0 0x7fe3a8159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.268+0000 7fe3afa00640 1 --2- 172.21.15.29:0/884045887 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe3a80ca330 0x7fe3a81f8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.268+0000 7fe3afa00640 1 -- 172.21.15.29:0/884045887 >> 172.21.15.29:0/884045887 conn(0x7fe3a806fde0 msgr2=0x7fe3a80cb140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:01:51.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.268+0000 7fe3afa00640 1 -- 172.21.15.29:0/884045887 shutdown_connections 2024-04-17T20:01:51.268 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.268+0000 7fe3afa00640 1 -- 172.21.15.29:0/884045887 wait complete. 2024-04-17T20:01:51.288 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=2) 2024-04-17T20:01:51.289 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-04-17T20:01:51.289 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T20:01:51.313 INFO:tasks.ceph.mds.f:Stopped 2024-04-17T20:01:51.313 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:01:51.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.472+0000 7fe1aaa00640 1 Processor -- start 2024-04-17T20:01:51.472 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.472+0000 7fe1aaa00640 1 -- start start 2024-04-17T20:01:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.476+0000 7fe1aaa00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe1a40cccd0 0x7fe1a40ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.476+0000 7fe1aaa00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fe1a40ca870 0x7fe1a41635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.476+0000 7fe1aaa00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1a40cd0e0 con 0x7fe1a40cc8c0 2024-04-17T20:01:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.476+0000 7fe1aaa00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fe1a40c88b0 con 0x7fe1a40cccd0 2024-04-17T20:01:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.476+0000 7fe1aaa00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fe1a40c6ce0 con 0x7fe1a40ca870 2024-04-17T20:01:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.476+0000 7fe19be00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe1a40cccd0 0x7fe1a40ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.476+0000 7fe19be00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe1a40cccd0 0x7fe1a40ca330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32964/0 (socket says 172.21.15.29:32964) 2024-04-17T20:01:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.476+0000 7fe19be00640 1 -- 172.21.15.29:0/3236260938 learned_addr learned my addr 172.21.15.29:0/3236260938 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:01:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.476+0000 7fe1a8a00640 1 --2- 172.21.15.29:0/3236260938 >> v2:172.21.15.37:3300/0 conn(0x7fe1a40ca870 0x7fe1a41635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:51.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.476+0000 7fe19be00640 1 -- 172.21.15.29:0/3236260938 >> v2:172.21.15.37:3300/0 conn(0x7fe1a40ca870 msgr2=0x7fe1a41635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:01:51.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.476+0000 7fe19be00640 1 --2- 172.21.15.29:0/3236260938 >> v2:172.21.15.37:3300/0 conn(0x7fe1a40ca870 0x7fe1a41635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.476+0000 7fe19be00640 1 -- 172.21.15.29:0/3236260938 >> v1:172.21.15.29:6789/0 conn(0x7fe1a40cc8c0 legacy=0x7fe1a4159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:51.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.476+0000 7fe19be00640 1 -- 172.21.15.29:0/3236260938 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1a40c6fc0 con 0x7fe1a40cccd0 2024-04-17T20:01:51.476 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.476+0000 7fe1a8a00640 1 --2- 172.21.15.29:0/3236260938 >> v2:172.21.15.37:3300/0 conn(0x7fe1a40ca870 0x7fe1a41635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:01:51.477 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.480+0000 7fe19be00640 1 --2- 172.21.15.29:0/3236260938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a40cccd0 0x7fe1a40ca330 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe1a40722d0 tx=0x7fe18c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcc3b433f5cc25f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:51.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.480+0000 7fe1a3400640 1 -- 172.21.15.29:0/3236260938 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18c0b0070 con 0x7fe1a40cccd0 2024-04-17T20:01:51.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.480+0000 7fe1a3400640 1 -- 172.21.15.29:0/3236260938 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe18c0ab600 con 0x7fe1a40cccd0 2024-04-17T20:01:51.478 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.480+0000 7fe1a3400640 1 -- 172.21.15.29:0/3236260938 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18c0ab9d0 con 0x7fe1a40cccd0 2024-04-17T20:01:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.480+0000 7fe1aaa00640 1 -- 172.21.15.29:0/3236260938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a40cccd0 msgr2=0x7fe1a40ca330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.480+0000 7fe1aaa00640 1 --2- 172.21.15.29:0/3236260938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a40cccd0 0x7fe1a40ca330 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fe1a40722d0 tx=0x7fe18c01c520 comp rx=0 tx=0).stop 2024-04-17T20:01:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.480+0000 7fe1aaa00640 1 -- 172.21.15.29:0/3236260938 shutdown_connections 2024-04-17T20:01:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.480+0000 7fe1aaa00640 1 --2- 172.21.15.29:0/3236260938 >> v2:172.21.15.37:3300/0 conn(0x7fe1a40ca870 0x7fe1a41635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.480+0000 7fe1aaa00640 1 --2- 172.21.15.29:0/3236260938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a40cccd0 0x7fe1a40ca330 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.480+0000 7fe1aaa00640 1 -- 172.21.15.29:0/3236260938 >> 172.21.15.29:0/3236260938 conn(0x7fe1a406fde0 msgr2=0x7fe1a40701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:01:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.480+0000 7fe1aaa00640 1 -- 172.21.15.29:0/3236260938 shutdown_connections 2024-04-17T20:01:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.480+0000 7fe1aaa00640 1 -- 172.21.15.29:0/3236260938 wait complete. 2024-04-17T20:01:51.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1aaa00640 1 Processor -- start 2024-04-17T20:01:51.481 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1aaa00640 1 -- start start 2024-04-17T20:01:51.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1aaa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a40ca870 0x7fe1a4165610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:51.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1aaa00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe1a40cc8c0 0x7fe1a41552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:51.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1aaa00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe1a40cccd0 0x7fe1a41f89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:51.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1aaa00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a4159c20 con 0x7fe1a40ca870 2024-04-17T20:01:51.485 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1aaa00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a4159aa0 con 0x7fe1a40cc8c0 2024-04-17T20:01:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1aaa00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe1a40c6880 con 0x7fe1a40cccd0 2024-04-17T20:01:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1a3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a40ca870 0x7fe1a4165610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1a3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a40ca870 0x7fe1a4165610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32978/0 (socket says 172.21.15.29:32978) 2024-04-17T20:01:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1a3e00640 1 -- 172.21.15.29:0/3216660968 learned_addr learned my addr 172.21.15.29:0/3216660968 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:01:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe19be00640 1 --2- 172.21.15.29:0/3216660968 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe1a40cc8c0 0x7fe1a41552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1a8a00640 1 --2- 172.21.15.29:0/3216660968 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe1a40cccd0 0x7fe1a41f89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1a3e00640 1 -- 172.21.15.29:0/3216660968 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe1a40cccd0 msgr2=0x7fe1a41f89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1a3e00640 1 --2- 172.21.15.29:0/3216660968 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe1a40cccd0 0x7fe1a41f89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1a3e00640 1 -- 172.21.15.29:0/3216660968 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe1a40cc8c0 msgr2=0x7fe1a41552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1a3e00640 1 --2- 172.21.15.29:0/3216660968 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe1a40cc8c0 0x7fe1a41552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1a3e00640 1 -- 172.21.15.29:0/3216660968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1a4166840 con 0x7fe1a40ca870 2024-04-17T20:01:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe19be00640 1 --2- 172.21.15.29:0/3216660968 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe1a40cc8c0 0x7fe1a41552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:01:51.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1a3e00640 1 --2- 172.21.15.29:0/3216660968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a40ca870 0x7fe1a4165610 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe19400c370 tx=0x7fe19400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:51.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1a0c00640 1 -- 172.21.15.29:0/3216660968 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1940052d0 con 0x7fe1a40ca870 2024-04-17T20:01:51.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1a0c00640 1 -- 172.21.15.29:0/3216660968 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe194005470 con 0x7fe1a40ca870 2024-04-17T20:01:51.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1aaa00640 1 -- 172.21.15.29:0/3216660968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1a4165b50 con 0x7fe1a40ca870 2024-04-17T20:01:51.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1a0c00640 1 -- 172.21.15.29:0/3216660968 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe194007260 con 0x7fe1a40ca870 2024-04-17T20:01:51.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.484+0000 7fe1aaa00640 1 -- 172.21.15.29:0/3216660968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1a4156320 con 0x7fe1a40ca870 2024-04-17T20:01:51.487 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.488+0000 7fe1aaa00640 1 -- 172.21.15.29:0/3216660968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1a4069b90 con 0x7fe1a40ca870 2024-04-17T20:01:51.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.488+0000 7fe1a0c00640 1 -- 172.21.15.29:0/3216660968 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe194007400 con 0x7fe1a40ca870 2024-04-17T20:01:51.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.488+0000 7fe1a0c00640 1 --2- 172.21.15.29:0/3216660968 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe180068380 0x7fe18006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:51.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.488+0000 7fe1a0c00640 1 -- 172.21.15.29:0/3216660968 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7fe1940864b0 con 0x7fe1a40ca870 2024-04-17T20:01:51.488 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.488+0000 7fe19be00640 1 --2- 172.21.15.29:0/3216660968 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe180068380 0x7fe18006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:51.489 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.492+0000 7fe19be00640 1 --2- 172.21.15.29:0/3216660968 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe180068380 0x7fe18006a890 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0ad000 tx=0x7fe18c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:51.493 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.496+0000 7fe1a0c00640 1 -- 172.21.15.29:0/3216660968 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe1a4166840 con 0x7fe1a40ca870 2024-04-17T20:01:51.705 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.708+0000 7fe1aaa00640 1 -- 172.21.15.29:0/3216660968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1a40c7e10 con 0x7fe1a40ca870 2024-04-17T20:01:51.706 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.708+0000 7fe1a0c00640 1 -- 172.21.15.29:0/3216660968 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6910 (secure 0 0 0) 0x7fe194056330 con 0x7fe1a40ca870 2024-04-17T20:01:51.706 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:01:51.706 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5946,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:58.083688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775,5631,5350,5212,5144]},"id":1}]} 2024-04-17T20:01:51.706 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 56 2024-04-17T20:01:51.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.708+0000 7fe1aaa00640 1 -- 172.21.15.29:0/3216660968 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe180068380 msgr2=0x7fe18006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:51.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.708+0000 7fe1aaa00640 1 --2- 172.21.15.29:0/3216660968 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe180068380 0x7fe18006a890 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0ad000 tx=0x7fe18c008040 comp rx=0 tx=0).stop 2024-04-17T20:01:51.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.708+0000 7fe1aaa00640 1 -- 172.21.15.29:0/3216660968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a40ca870 msgr2=0x7fe1a4165610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:51.712 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.708+0000 7fe1aaa00640 1 --2- 172.21.15.29:0/3216660968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a40ca870 0x7fe1a4165610 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe19400c370 tx=0x7fe19400c830 comp rx=0 tx=0).stop 2024-04-17T20:01:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.708+0000 7fe1aaa00640 1 -- 172.21.15.29:0/3216660968 shutdown_connections 2024-04-17T20:01:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.708+0000 7fe1aaa00640 1 --2- 172.21.15.29:0/3216660968 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe180068380 0x7fe18006a890 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.708+0000 7fe1aaa00640 1 --2- 172.21.15.29:0/3216660968 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe1a40cccd0 0x7fe1a41f89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.708+0000 7fe1aaa00640 1 --2- 172.21.15.29:0/3216660968 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe1a40cc8c0 0x7fe1a41552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.708+0000 7fe1aaa00640 1 --2- 172.21.15.29:0/3216660968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1a40ca870 0x7fe1a4165610 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.708+0000 7fe1aaa00640 1 -- 172.21.15.29:0/3216660968 >> 172.21.15.29:0/3216660968 conn(0x7fe1a406fde0 msgr2=0x7fe1a40cb3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:01:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.708+0000 7fe1aaa00640 1 -- 172.21.15.29:0/3216660968 shutdown_connections 2024-04-17T20:01:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:51.708+0000 7fe1aaa00640 1 -- 172.21.15.29:0/3216660968 wait complete. 2024-04-17T20:01:51.728 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-17T20:01:53.730 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:01:53.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.968+0000 7f9ca7c00640 1 Processor -- start 2024-04-17T20:01:53.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.968+0000 7f9ca7c00640 1 -- start start 2024-04-17T20:01:53.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.972+0000 7f9ca7c00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9ca00cbf90 0x7f9ca015ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:53.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.972+0000 7f9ca7c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9ca00c7f50 0x7f9ca00ca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:53.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.972+0000 7f9ca7c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ca0155da0 0x7f9ca01561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:53.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.972+0000 7f9ca7c00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f9ca006bdc0 con 0x7f9ca00c7f50 2024-04-17T20:01:53.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.972+0000 7f9ca7c00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f9ca015cf70 con 0x7f9ca00cbf90 2024-04-17T20:01:53.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.972+0000 7f9ca7c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9ca01566f0 con 0x7f9ca0155da0 2024-04-17T20:01:53.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.972+0000 7f9ca5200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9ca00cbf90 0x7f9ca015ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:53.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.972+0000 7f9ca5200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f9ca00cbf90 0x7f9ca015ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:40766/0 (socket says 172.21.15.29:40766) 2024-04-17T20:01:53.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.972+0000 7f9ca5200640 1 -- 172.21.15.29:0/2866147071 learned_addr learned my addr 172.21.15.29:0/2866147071 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:01:53.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.972+0000 7f9ca5c00640 1 --2- 172.21.15.29:0/2866147071 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ca0155da0 0x7f9ca01561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:53.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.972+0000 7f9ca5200640 1 -- 172.21.15.29:0/2866147071 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ca0155da0 msgr2=0x7f9ca01561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:01:53.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.972+0000 7f9ca5200640 1 --2- 172.21.15.29:0/2866147071 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ca0155da0 0x7f9ca01561b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:53.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.972+0000 7f9ca5200640 1 -- 172.21.15.29:0/2866147071 >> v2:172.21.15.29:3300/0 conn(0x7f9ca00c7f50 msgr2=0x7f9ca00ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:01:53.972 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.972+0000 7f9ca5200640 1 --2- 172.21.15.29:0/2866147071 >> v2:172.21.15.29:3300/0 conn(0x7f9ca00c7f50 0x7f9ca00ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:53.973 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.972+0000 7f9ca5200640 1 -- 172.21.15.29:0/2866147071 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca00cbba0 con 0x7f9ca00cbf90 2024-04-17T20:01:53.976 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.976+0000 7f9ca5200640 1 --2- 172.21.15.29:0/2866147071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ca00cbf90 0x7f9ca015ca30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00710e0 tx=0x7f9c9401c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a3eea16c7fbd969 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:53.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.980+0000 7f9c9f400640 1 -- 172.21.15.29:0/2866147071 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c940b1070 con 0x7f9ca00cbf90 2024-04-17T20:01:53.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.980+0000 7f9c9f400640 1 -- 172.21.15.29:0/2866147071 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c940ac630 con 0x7f9ca00cbf90 2024-04-17T20:01:53.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.980+0000 7f9c9f400640 1 -- 172.21.15.29:0/2866147071 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c940a8050 con 0x7f9ca00cbf90 2024-04-17T20:01:53.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.980+0000 7f9ca7c00640 1 -- 172.21.15.29:0/2866147071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ca00cbf90 msgr2=0x7f9ca015ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:53.983 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.980+0000 7f9ca7c00640 1 --2- 172.21.15.29:0/2866147071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ca00cbf90 0x7f9ca015ca30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00710e0 tx=0x7f9c9401c520 comp rx=0 tx=0).stop 2024-04-17T20:01:53.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.980+0000 7f9ca7c00640 1 -- 172.21.15.29:0/2866147071 shutdown_connections 2024-04-17T20:01:53.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.980+0000 7f9ca7c00640 1 --2- 172.21.15.29:0/2866147071 >> v2:172.21.15.29:3300/0 conn(0x7f9ca00c7f50 0x7f9ca00ca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:53.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.980+0000 7f9ca7c00640 1 --2- 172.21.15.29:0/2866147071 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ca0155da0 0x7f9ca01561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:53.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.980+0000 7f9ca7c00640 1 --2- 172.21.15.29:0/2866147071 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ca00cbf90 0x7f9ca015ca30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:53.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.980+0000 7f9ca7c00640 1 -- 172.21.15.29:0/2866147071 >> 172.21.15.29:0/2866147071 conn(0x7f9ca006fde0 msgr2=0x7f9ca00701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:01:53.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.980+0000 7f9ca7c00640 1 -- 172.21.15.29:0/2866147071 shutdown_connections 2024-04-17T20:01:53.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.980+0000 7f9ca7c00640 1 -- 172.21.15.29:0/2866147071 wait complete. 2024-04-17T20:01:53.984 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.984+0000 7f9ca7c00640 1 Processor -- start 2024-04-17T20:01:53.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.988+0000 7f9ca7c00640 1 -- start start 2024-04-17T20:01:53.987 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.988+0000 7f9ca7c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ca00c7f50 0x7f9ca0150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:53.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.988+0000 7f9ca7c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ca00cbf90 0x7f9ca01491c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:53.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.988+0000 7f9ca7c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ca0155da0 0x7f9ca0149700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:53.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.988+0000 7f9ca7c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca00c84e0 con 0x7f9ca00c7f50 2024-04-17T20:01:53.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.988+0000 7f9ca7c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca00c8360 con 0x7f9ca00cbf90 2024-04-17T20:01:53.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.988+0000 7f9ca7c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f9ca00ca950 con 0x7f9ca0155da0 2024-04-17T20:01:53.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.988+0000 7f9ca5200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ca00c7f50 0x7f9ca0150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:53.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.988+0000 7f9ca5200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ca00c7f50 0x7f9ca0150bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32986/0 (socket says 172.21.15.29:32986) 2024-04-17T20:01:53.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.988+0000 7f9ca5200640 1 -- 172.21.15.29:0/993922578 learned_addr learned my addr 172.21.15.29:0/993922578 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:01:53.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.988+0000 7f9ca5c00640 1 --2- 172.21.15.29:0/993922578 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ca0155da0 0x7f9ca0149700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:53.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.988+0000 7f9c9fe00640 1 --2- 172.21.15.29:0/993922578 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ca00cbf90 0x7f9ca01491c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:53.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.992+0000 7f9c9fe00640 1 -- 172.21.15.29:0/993922578 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ca0155da0 msgr2=0x7f9ca0149700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:53.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.992+0000 7f9c9fe00640 1 --2- 172.21.15.29:0/993922578 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ca0155da0 0x7f9ca0149700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:53.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.992+0000 7f9c9fe00640 1 -- 172.21.15.29:0/993922578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ca00c7f50 msgr2=0x7f9ca0150bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:53.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.992+0000 7f9c9fe00640 1 --2- 172.21.15.29:0/993922578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ca00c7f50 0x7f9ca0150bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:53.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.992+0000 7f9c9fe00640 1 -- 172.21.15.29:0/993922578 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca01598f0 con 0x7f9ca00cbf90 2024-04-17T20:01:53.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.992+0000 7f9c9fe00640 1 --2- 172.21.15.29:0/993922578 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ca00cbf90 0x7f9ca01491c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c00b3e0 tx=0x7f9c8c00b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:53.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.992+0000 7f9c9cc00640 1 -- 172.21.15.29:0/993922578 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c015020 con 0x7f9ca00cbf90 2024-04-17T20:01:53.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.992+0000 7f9c9cc00640 1 -- 172.21.15.29:0/993922578 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c8c011850 con 0x7f9ca00cbf90 2024-04-17T20:01:53.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.992+0000 7f9c9cc00640 1 -- 172.21.15.29:0/993922578 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c011c40 con 0x7f9ca00cbf90 2024-04-17T20:01:53.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.996+0000 7f9ca7c00640 1 -- 172.21.15.29:0/993922578 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca0150730 con 0x7f9ca00cbf90 2024-04-17T20:01:53.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.996+0000 7f9ca7c00640 1 -- 172.21.15.29:0/993922578 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca0158e10 con 0x7f9ca00cbf90 2024-04-17T20:01:53.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.996+0000 7f9ca7c00640 1 -- 172.21.15.29:0/993922578 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ca01504a0 con 0x7f9ca00cbf90 2024-04-17T20:01:53.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:53.996+0000 7f9c9cc00640 1 -- 172.21.15.29:0/993922578 <== mon.1 v2:172.21.15.37:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f9c8c018020 con 0x7f9ca00cbf90 2024-04-17T20:01:53.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.000+0000 7f9c9cc00640 1 --2- 172.21.15.29:0/993922578 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9c800683b0 0x7f9c8006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:53.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.000+0000 7f9c9cc00640 1 -- 172.21.15.29:0/993922578 <== mon.1 v2:172.21.15.37:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7f9c8c087240 con 0x7f9ca00cbf90 2024-04-17T20:01:53.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.000+0000 7f9ca5200640 1 --2- 172.21.15.29:0/993922578 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9c800683b0 0x7f9c8006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:54.003 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.004+0000 7f9ca5200640 1 --2- 172.21.15.29:0/993922578 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9c800683b0 0x7f9c8006a8c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9c9401c560 tx=0x7f9c940a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:54.004 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.004+0000 7f9c9cc00640 1 -- 172.21.15.29:0/993922578 <== mon.1 v2:172.21.15.37:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f9c8c01f020 con 0x7f9ca00cbf90 2024-04-17T20:01:54.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.220+0000 7f9ca7c00640 1 -- 172.21.15.29:0/993922578 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ca0069b90 con 0x7f9ca00cbf90 2024-04-17T20:01:54.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.220+0000 7f9c9cc00640 1 -- 172.21.15.29:0/993922578 <== mon.1 v2:172.21.15.37:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6910 (secure 0 0 0) 0x7f9c8c056b80 con 0x7f9ca00cbf90 2024-04-17T20:01:54.220 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:01:54.220 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5946,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:58.083688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775,5631,5350,5212,5144]},"id":1}]} 2024-04-17T20:01:54.220 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 56 2024-04-17T20:01:54.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.224+0000 7f9ca7c00640 1 -- 172.21.15.29:0/993922578 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9c800683b0 msgr2=0x7f9c8006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:54.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.224+0000 7f9ca7c00640 1 --2- 172.21.15.29:0/993922578 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9c800683b0 0x7f9c8006a8c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9c9401c560 tx=0x7f9c940a7000 comp rx=0 tx=0).stop 2024-04-17T20:01:54.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.224+0000 7f9ca7c00640 1 -- 172.21.15.29:0/993922578 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ca00cbf90 msgr2=0x7f9ca01491c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:54.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.224+0000 7f9ca7c00640 1 --2- 172.21.15.29:0/993922578 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ca00cbf90 0x7f9ca01491c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c00b3e0 tx=0x7f9c8c00b7e0 comp rx=0 tx=0).stop 2024-04-17T20:01:54.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.224+0000 7f9ca7c00640 1 -- 172.21.15.29:0/993922578 shutdown_connections 2024-04-17T20:01:54.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.224+0000 7f9ca7c00640 1 --2- 172.21.15.29:0/993922578 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f9c800683b0 0x7f9c8006a8c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:54.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.224+0000 7f9ca7c00640 1 --2- 172.21.15.29:0/993922578 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f9ca0155da0 0x7f9ca0149700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:54.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.224+0000 7f9ca7c00640 1 --2- 172.21.15.29:0/993922578 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f9ca00cbf90 0x7f9ca01491c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:54.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.224+0000 7f9ca7c00640 1 --2- 172.21.15.29:0/993922578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ca00c7f50 0x7f9ca0150bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:54.222 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.224+0000 7f9ca7c00640 1 -- 172.21.15.29:0/993922578 >> 172.21.15.29:0/993922578 conn(0x7f9ca006fde0 msgr2=0x7f9ca00c9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:01:54.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.224+0000 7f9ca7c00640 1 -- 172.21.15.29:0/993922578 shutdown_connections 2024-04-17T20:01:54.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:54.224+0000 7f9ca7c00640 1 -- 172.21.15.29:0/993922578 wait complete. 2024-04-17T20:01:54.242 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-17T20:01:56.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:01:56.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.408+0000 7f44a3e00640 1 Processor -- start 2024-04-17T20:01:56.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.408+0000 7f44a3e00640 1 -- start start 2024-04-17T20:01:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.408+0000 7f44a3e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44a4156370 0x7f44a40ca110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.408+0000 7f44a3e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f44a40cabe0 0x7f44a40caff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.408+0000 7f44a3e00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44a406c730 con 0x7f44a415c260 2024-04-17T20:01:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.408+0000 7f44a3e00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f44a406c9f0 con 0x7f44a40cabe0 2024-04-17T20:01:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.408+0000 7f44a3e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f44a406ccd0 con 0x7f44a4156370 2024-04-17T20:01:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.408+0000 7f449be00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f44a40cabe0 0x7f44a40caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.408+0000 7f449be00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f44a40cabe0 0x7f44a40caff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32998/0 (socket says 172.21.15.29:32998) 2024-04-17T20:01:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.408+0000 7f449be00640 1 -- 172.21.15.29:0/2760410278 learned_addr learned my addr 172.21.15.29:0/2760410278 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:01:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.408+0000 7f44a3400640 1 --1- 172.21.15.29:0/2760410278 >> v1:172.21.15.29:6789/0 conn(0x7f44a415c260 0x7f44a415c670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34408/0 (socket says 172.21.15.29:34408) 2024-04-17T20:01:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.408+0000 7f44a2a00640 1 --2- 172.21.15.29:0/2760410278 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44a4156370 0x7f44a40ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.408+0000 7f449be00640 1 -- 172.21.15.29:0/2760410278 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44a4156370 msgr2=0x7f44a40ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:01:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.408+0000 7f449be00640 1 --2- 172.21.15.29:0/2760410278 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44a4156370 0x7f44a40ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.408+0000 7f449be00640 1 -- 172.21.15.29:0/2760410278 >> v1:172.21.15.29:6789/0 conn(0x7f44a415c260 legacy=0x7f44a415c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.408+0000 7f449be00640 1 -- 172.21.15.29:0/2760410278 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a406cfb0 con 0x7f44a40cabe0 2024-04-17T20:01:56.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.412+0000 7f449be00640 1 --2- 172.21.15.29:0/2760410278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44a40cabe0 0x7f44a40caff0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f448c009870 tx=0x7f448c01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9c8604d3d375141 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:56.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.412+0000 7f44a2000640 1 -- 172.21.15.29:0/2760410278 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448c0b5070 con 0x7f44a40cabe0 2024-04-17T20:01:56.410 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.412+0000 7f44a2000640 1 -- 172.21.15.29:0/2760410278 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f448c0a8a50 con 0x7f44a40cabe0 2024-04-17T20:01:56.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.412+0000 7f44a2000640 1 -- 172.21.15.29:0/2760410278 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448c0a8e20 con 0x7f44a40cabe0 2024-04-17T20:01:56.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.412+0000 7f44a3e00640 1 -- 172.21.15.29:0/2760410278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44a40cabe0 msgr2=0x7f44a40caff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:56.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.412+0000 7f44a3e00640 1 --2- 172.21.15.29:0/2760410278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44a40cabe0 0x7f44a40caff0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f448c009870 tx=0x7f448c01ed50 comp rx=0 tx=0).stop 2024-04-17T20:01:56.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.412+0000 7f44a3e00640 1 -- 172.21.15.29:0/2760410278 shutdown_connections 2024-04-17T20:01:56.411 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.412+0000 7f44a3e00640 1 --2- 172.21.15.29:0/2760410278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44a40cabe0 0x7f44a40caff0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:56.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.412+0000 7f44a3e00640 1 --2- 172.21.15.29:0/2760410278 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44a4156370 0x7f44a40ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:56.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.412+0000 7f44a3e00640 1 -- 172.21.15.29:0/2760410278 >> 172.21.15.29:0/2760410278 conn(0x7f44a40680e0 msgr2=0x7f44a406dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:01:56.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.412+0000 7f44a3e00640 1 -- 172.21.15.29:0/2760410278 shutdown_connections 2024-04-17T20:01:56.412 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.412+0000 7f44a3e00640 1 -- 172.21.15.29:0/2760410278 wait complete. 2024-04-17T20:01:56.413 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f44a3e00640 1 Processor -- start 2024-04-17T20:01:56.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f44a3e00640 1 -- start start 2024-04-17T20:01:56.417 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f44a3e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44a40cabe0 0x7f44a41f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:56.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f44a3e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44a4156370 0x7f44a41f6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:56.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f44a3e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44a415c260 0x7f44a4071800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:56.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f44a3e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f44a40ca650 con 0x7f44a40cabe0 2024-04-17T20:01:56.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f44a3e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f44a406d570 con 0x7f44a4156370 2024-04-17T20:01:56.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f44a3e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f44a406e7c0 con 0x7f44a415c260 2024-04-17T20:01:56.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f44a2a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44a40cabe0 0x7f44a41f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:56.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f44a2a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44a40cabe0 0x7f44a41f31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33010/0 (socket says 172.21.15.29:33010) 2024-04-17T20:01:56.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f44a2a00640 1 -- 172.21.15.29:0/2500941014 learned_addr learned my addr 172.21.15.29:0/2500941014 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:01:56.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f44a3400640 1 --2- 172.21.15.29:0/2500941014 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44a415c260 0x7f44a4071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:56.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f449be00640 1 --2- 172.21.15.29:0/2500941014 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44a4156370 0x7f44a41f6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:56.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f44a2a00640 1 -- 172.21.15.29:0/2500941014 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44a415c260 msgr2=0x7f44a4071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:56.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f44a2a00640 1 --2- 172.21.15.29:0/2500941014 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44a415c260 0x7f44a4071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:56.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f44a2a00640 1 -- 172.21.15.29:0/2500941014 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44a4156370 msgr2=0x7f44a41f6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:56.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f44a2a00640 1 --2- 172.21.15.29:0/2500941014 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44a4156370 0x7f44a41f6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:56.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f44a2a00640 1 -- 172.21.15.29:0/2500941014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44a41f36e0 con 0x7f44a40cabe0 2024-04-17T20:01:56.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f449be00640 1 --2- 172.21.15.29:0/2500941014 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44a4156370 0x7f44a41f6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:01:56.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f44a2a00640 1 --2- 172.21.15.29:0/2500941014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44a40cabe0 0x7f44a41f31a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f449400d5f0 tx=0x7f449400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:56.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.416+0000 7f449aa00640 1 -- 172.21.15.29:0/2500941014 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44940125f0 con 0x7f44a40cabe0 2024-04-17T20:01:56.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.420+0000 7f449aa00640 1 -- 172.21.15.29:0/2500941014 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4494019070 con 0x7f44a40cabe0 2024-04-17T20:01:56.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.420+0000 7f44a3e00640 1 -- 172.21.15.29:0/2500941014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44a4156a00 con 0x7f44a40cabe0 2024-04-17T20:01:56.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.420+0000 7f449aa00640 1 -- 172.21.15.29:0/2500941014 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44940129b0 con 0x7f44a40cabe0 2024-04-17T20:01:56.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.420+0000 7f44a3e00640 1 -- 172.21.15.29:0/2500941014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44a4156cc0 con 0x7f44a40cabe0 2024-04-17T20:01:56.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.420+0000 7f44a3e00640 1 -- 172.21.15.29:0/2500941014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44a404ff10 con 0x7f44a40cabe0 2024-04-17T20:01:56.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.420+0000 7f449aa00640 1 -- 172.21.15.29:0/2500941014 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4494012b50 con 0x7f44a40cabe0 2024-04-17T20:01:56.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.420+0000 7f449aa00640 1 --2- 172.21.15.29:0/2500941014 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4480068380 0x7f448006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:56.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.420+0000 7f449aa00640 1 -- 172.21.15.29:0/2500941014 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7f4494087460 con 0x7f44a40cabe0 2024-04-17T20:01:56.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.420+0000 7f449be00640 1 --2- 172.21.15.29:0/2500941014 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4480068380 0x7f448006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:56.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.424+0000 7f449be00640 1 --2- 172.21.15.29:0/2500941014 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4480068380 0x7f448006a890 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f448c0b2000 tx=0x7f448c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:56.424 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.424+0000 7f449aa00640 1 -- 172.21.15.29:0/2500941014 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4494057370 con 0x7f44a40cabe0 2024-04-17T20:01:56.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.624+0000 7f44a3e00640 1 -- 172.21.15.29:0/2500941014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44a40cf570 con 0x7f44a40cabe0 2024-04-17T20:01:56.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.624+0000 7f449aa00640 1 -- 172.21.15.29:0/2500941014 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6910 (secure 0 0 0) 0x7f449405d030 con 0x7f44a40cabe0 2024-04-17T20:01:56.623 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:01:56.624 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5946,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:58.083688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775,5631,5350,5212,5144]},"id":1}]} 2024-04-17T20:01:56.624 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 56 2024-04-17T20:01:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.628+0000 7f44a3e00640 1 -- 172.21.15.29:0/2500941014 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4480068380 msgr2=0x7f448006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.628+0000 7f44a3e00640 1 --2- 172.21.15.29:0/2500941014 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4480068380 0x7f448006a890 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f448c0b2000 tx=0x7f448c008040 comp rx=0 tx=0).stop 2024-04-17T20:01:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.628+0000 7f44a3e00640 1 -- 172.21.15.29:0/2500941014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44a40cabe0 msgr2=0x7f44a41f31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.628+0000 7f44a3e00640 1 --2- 172.21.15.29:0/2500941014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44a40cabe0 0x7f44a41f31a0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f449400d5f0 tx=0x7f449400dab0 comp rx=0 tx=0).stop 2024-04-17T20:01:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.628+0000 7f44a3e00640 1 -- 172.21.15.29:0/2500941014 shutdown_connections 2024-04-17T20:01:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.628+0000 7f44a3e00640 1 --2- 172.21.15.29:0/2500941014 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4480068380 0x7f448006a890 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.628+0000 7f44a3e00640 1 --2- 172.21.15.29:0/2500941014 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f44a415c260 0x7f44a4071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.628+0000 7f44a3e00640 1 --2- 172.21.15.29:0/2500941014 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f44a4156370 0x7f44a41f6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.628+0000 7f44a3e00640 1 --2- 172.21.15.29:0/2500941014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44a40cabe0 0x7f44a41f31a0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.628+0000 7f44a3e00640 1 -- 172.21.15.29:0/2500941014 >> 172.21.15.29:0/2500941014 conn(0x7f44a40680e0 msgr2=0x7f44a4154c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:01:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.628+0000 7f44a3e00640 1 -- 172.21.15.29:0/2500941014 shutdown_connections 2024-04-17T20:01:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:56.628+0000 7f44a3e00640 1 -- 172.21.15.29:0/2500941014 wait complete. 2024-04-17T20:01:56.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:01:56.641 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:01:56.646 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-17T20:01:58.647 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:01:58.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.799+0000 7fd1a4400640 1 Processor -- start 2024-04-17T20:01:58.800 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.799+0000 7fd1a4400640 1 -- start start 2024-04-17T20:01:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.803+0000 7fd1a4400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd19c159c90 0x7fd19c0ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.803+0000 7fd1a4400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd19c0ca7f0 0x7fd19c0c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.803+0000 7fd1a4400640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd19c15a0a0 con 0x7fd19c0c74b0 2024-04-17T20:01:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.803+0000 7fd1a4400640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fd19c15abe0 con 0x7fd19c0ca7f0 2024-04-17T20:01:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.803+0000 7fd1a4400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd19c15a340 con 0x7fd19c159c90 2024-04-17T20:01:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.803+0000 7fd1a1000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd19c0ca7f0 0x7fd19c0c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.803+0000 7fd1a1000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd19c0ca7f0 0x7fd19c0c6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36764/0 (socket says 172.21.15.29:36764) 2024-04-17T20:01:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.803+0000 7fd1a1000640 1 -- 172.21.15.29:0/317879073 learned_addr learned my addr 172.21.15.29:0/317879073 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:01:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.803+0000 7fd1a1a00640 1 --2- 172.21.15.29:0/317879073 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd19c159c90 0x7fd19c0ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.803+0000 7fd1a1000640 1 -- 172.21.15.29:0/317879073 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd19c159c90 msgr2=0x7fd19c0ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:01:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.803+0000 7fd1a1000640 1 --2- 172.21.15.29:0/317879073 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd19c159c90 0x7fd19c0ca2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.803+0000 7fd1a1000640 1 -- 172.21.15.29:0/317879073 >> v1:172.21.15.29:6789/0 conn(0x7fd19c0c74b0 legacy=0x7fd19c06c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.803+0000 7fd1a1000640 1 -- 172.21.15.29:0/317879073 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd19c15a620 con 0x7fd19c0ca7f0 2024-04-17T20:01:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.807+0000 7fd1a1000640 1 --2- 172.21.15.29:0/317879073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd19c0ca7f0 0x7fd19c0c6f70 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd188009870 tx=0x7fd18801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a963c52584bbe848 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.807+0000 7fd19be00640 1 -- 172.21.15.29:0/317879073 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1880a7070 con 0x7fd19c0ca7f0 2024-04-17T20:01:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.807+0000 7fd19be00640 1 -- 172.21.15.29:0/317879073 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1880b0070 con 0x7fd19c0ca7f0 2024-04-17T20:01:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.807+0000 7fd19be00640 1 -- 172.21.15.29:0/317879073 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1880ab890 con 0x7fd19c0ca7f0 2024-04-17T20:01:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.807+0000 7fd1a4400640 1 -- 172.21.15.29:0/317879073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd19c0ca7f0 msgr2=0x7fd19c0c6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.807+0000 7fd1a4400640 1 --2- 172.21.15.29:0/317879073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd19c0ca7f0 0x7fd19c0c6f70 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd188009870 tx=0x7fd18801c560 comp rx=0 tx=0).stop 2024-04-17T20:01:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.807+0000 7fd1a4400640 1 -- 172.21.15.29:0/317879073 shutdown_connections 2024-04-17T20:01:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.807+0000 7fd1a4400640 1 --2- 172.21.15.29:0/317879073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd19c0ca7f0 0x7fd19c0c6f70 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.807+0000 7fd1a4400640 1 --2- 172.21.15.29:0/317879073 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd19c159c90 0x7fd19c0ca2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.807+0000 7fd1a4400640 1 -- 172.21.15.29:0/317879073 >> 172.21.15.29:0/317879073 conn(0x7fd19c06fde0 msgr2=0x7fd19c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:01:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.807+0000 7fd1a4400640 1 -- 172.21.15.29:0/317879073 shutdown_connections 2024-04-17T20:01:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.807+0000 7fd1a4400640 1 -- 172.21.15.29:0/317879073 wait complete. 2024-04-17T20:01:58.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a4400640 1 Processor -- start 2024-04-17T20:01:58.812 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a4400640 1 -- start start 2024-04-17T20:01:58.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a4400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd19c0c74b0 0x7fd19c0c4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:58.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a4400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd19c0ca7f0 0x7fd19c1f9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:58.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a4400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd19c159c90 0x7fd19c14e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:58.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a4400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c166fb0 con 0x7fd19c159c90 2024-04-17T20:01:58.813 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a4400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd19c166e30 con 0x7fd19c0ca7f0 2024-04-17T20:01:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a4400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd19c167130 con 0x7fd19c0c74b0 2024-04-17T20:01:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a2400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd19c159c90 0x7fd19c14e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a1a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd19c0c74b0 0x7fd19c0c4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a2400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd19c159c90 0x7fd19c14e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36772/0 (socket says 172.21.15.29:36772) 2024-04-17T20:01:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a2400640 1 -- 172.21.15.29:0/1851158491 learned_addr learned my addr 172.21.15.29:0/1851158491 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:01:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a1000640 1 --2- 172.21.15.29:0/1851158491 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd19c0ca7f0 0x7fd19c1f9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a2400640 1 -- 172.21.15.29:0/1851158491 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd19c0c74b0 msgr2=0x7fd19c0c4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a2400640 1 --2- 172.21.15.29:0/1851158491 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd19c0c74b0 0x7fd19c0c4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a2400640 1 -- 172.21.15.29:0/1851158491 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd19c0ca7f0 msgr2=0x7fd19c1f9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a2400640 1 --2- 172.21.15.29:0/1851158491 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd19c0ca7f0 0x7fd19c1f9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:58.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a2400640 1 -- 172.21.15.29:0/1851158491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd19c1f9a40 con 0x7fd19c159c90 2024-04-17T20:01:58.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a2400640 1 --2- 172.21.15.29:0/1851158491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd19c159c90 0x7fd19c14e450 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd19400c760 tx=0x7fd19400cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:58.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd199600640 1 -- 172.21.15.29:0/1851158491 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd194003ba0 con 0x7fd19c159c90 2024-04-17T20:01:58.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd199600640 1 -- 172.21.15.29:0/1851158491 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd194003d40 con 0x7fd19c159c90 2024-04-17T20:01:58.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a4400640 1 -- 172.21.15.29:0/1851158491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd19c149f80 con 0x7fd19c159c90 2024-04-17T20:01:58.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd199600640 1 -- 172.21.15.29:0/1851158491 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd194005310 con 0x7fd19c159c90 2024-04-17T20:01:58.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.811+0000 7fd1a4400640 1 -- 172.21.15.29:0/1851158491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd19c14a540 con 0x7fd19c159c90 2024-04-17T20:01:58.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.815+0000 7fd1a4400640 1 -- 172.21.15.29:0/1851158491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd19c149ce0 con 0x7fd19c159c90 2024-04-17T20:01:58.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.815+0000 7fd199600640 1 -- 172.21.15.29:0/1851158491 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd1940054b0 con 0x7fd19c159c90 2024-04-17T20:01:58.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.815+0000 7fd199600640 1 --2- 172.21.15.29:0/1851158491 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd17c068380 0x7fd17c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:01:58.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.815+0000 7fd199600640 1 -- 172.21.15.29:0/1851158491 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7fd194086ca0 con 0x7fd19c159c90 2024-04-17T20:01:58.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.815+0000 7fd1a1a00640 1 --2- 172.21.15.29:0/1851158491 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd17c068380 0x7fd17c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:01:58.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.819+0000 7fd1a1a00640 1 --2- 172.21.15.29:0/1851158491 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd17c068380 0x7fd17c06a890 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd19c0c8e90 tx=0x7fd190009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:01:58.820 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:58.819+0000 7fd199600640 1 -- 172.21.15.29:0/1851158491 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd19c149ce0 con 0x7fd19c159c90 2024-04-17T20:01:59.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:59.019+0000 7fd1a4400640 1 -- 172.21.15.29:0/1851158491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd19c069b90 con 0x7fd19c159c90 2024-04-17T20:01:59.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:59.019+0000 7fd199600640 1 -- 172.21.15.29:0/1851158491 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6910 (secure 0 0 0) 0x7fd1940565f0 con 0x7fd19c159c90 2024-04-17T20:01:59.023 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:01:59.023 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5946,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:58.083688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775,5631,5350,5212,5144]},"id":1}]} 2024-04-17T20:01:59.024 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 56 2024-04-17T20:01:59.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:59.023+0000 7fd1a4400640 1 -- 172.21.15.29:0/1851158491 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd17c068380 msgr2=0x7fd17c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:59.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:59.023+0000 7fd1a4400640 1 --2- 172.21.15.29:0/1851158491 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd17c068380 0x7fd17c06a890 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd19c0c8e90 tx=0x7fd190009190 comp rx=0 tx=0).stop 2024-04-17T20:01:59.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:59.023+0000 7fd1a4400640 1 -- 172.21.15.29:0/1851158491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd19c159c90 msgr2=0x7fd19c14e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:01:59.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:59.023+0000 7fd1a4400640 1 --2- 172.21.15.29:0/1851158491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd19c159c90 0x7fd19c14e450 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd19400c760 tx=0x7fd19400cb60 comp rx=0 tx=0).stop 2024-04-17T20:01:59.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:59.023+0000 7fd1a4400640 1 -- 172.21.15.29:0/1851158491 shutdown_connections 2024-04-17T20:01:59.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:59.023+0000 7fd1a4400640 1 --2- 172.21.15.29:0/1851158491 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd17c068380 0x7fd17c06a890 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:59.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:59.023+0000 7fd1a4400640 1 --2- 172.21.15.29:0/1851158491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd19c159c90 0x7fd19c14e450 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:59.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:59.023+0000 7fd1a4400640 1 --2- 172.21.15.29:0/1851158491 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd19c0ca7f0 0x7fd19c1f9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:59.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:59.023+0000 7fd1a4400640 1 --2- 172.21.15.29:0/1851158491 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd19c0c74b0 0x7fd19c0c4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:01:59.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:59.023+0000 7fd1a4400640 1 -- 172.21.15.29:0/1851158491 >> 172.21.15.29:0/1851158491 conn(0x7fd19c06fde0 msgr2=0x7fd19c0cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:01:59.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:59.023+0000 7fd1a4400640 1 -- 172.21.15.29:0/1851158491 shutdown_connections 2024-04-17T20:01:59.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:01:59.023+0000 7fd1a4400640 1 -- 172.21.15.29:0/1851158491 wait complete. 2024-04-17T20:01:59.043 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-17T20:02:01.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:02:01.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.211+0000 7f1502200640 1 Processor -- start 2024-04-17T20:02:01.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.211+0000 7f1502200640 1 -- start start 2024-04-17T20:02:01.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.211+0000 7f1502200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f14fc0cc8c0 0x7f14fc0cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:01.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.211+0000 7f1502200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14fc159390 0x7f14fc1597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:01.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.215+0000 7f1502200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14fc0c88b0 con 0x7f14fc0ca330 2024-04-17T20:02:01.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.215+0000 7f1502200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f14fc0c6ce0 con 0x7f14fc0cc8c0 2024-04-17T20:02:01.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.215+0000 7f1502200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f14fc0c6fc0 con 0x7f14fc159390 2024-04-17T20:02:01.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.215+0000 7f14fb400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f14fc0cc8c0 0x7f14fc0cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:01.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.215+0000 7f14fb400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f14fc0cc8c0 0x7f14fc0cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36784/0 (socket says 172.21.15.29:36784) 2024-04-17T20:02:01.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.215+0000 7f14fb400640 1 -- 172.21.15.29:0/1325763488 learned_addr learned my addr 172.21.15.29:0/1325763488 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:01.213 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.215+0000 7f14f3e00640 1 --2- 172.21.15.29:0/1325763488 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14fc159390 0x7f14fc1597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:01.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.215+0000 7f14fb400640 1 -- 172.21.15.29:0/1325763488 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14fc159390 msgr2=0x7f14fc1597a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:01.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.215+0000 7f14fb400640 1 --2- 172.21.15.29:0/1325763488 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14fc159390 0x7f14fc1597a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:01.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.215+0000 7f14fb400640 1 -- 172.21.15.29:0/1325763488 >> v1:172.21.15.37:6789/0 conn(0x7f14fc0ca330 legacy=0x7f14fc15c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:01.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.215+0000 7f14fb400640 1 -- 172.21.15.29:0/1325763488 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14fc0c72a0 con 0x7f14fc0cc8c0 2024-04-17T20:02:01.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.215+0000 7f14fb400640 1 --2- 172.21.15.29:0/1325763488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0cc8c0 0x7f14fc0cccd0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f14ec009870 tx=0x7f14ec01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52197ea7c72fef60 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:01.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.215+0000 7f14faa00640 1 -- 172.21.15.29:0/1325763488 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0b2070 con 0x7f14fc0cc8c0 2024-04-17T20:02:01.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.215+0000 7f14faa00640 1 -- 172.21.15.29:0/1325763488 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14ec0a9070 con 0x7f14fc0cc8c0 2024-04-17T20:02:01.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.215+0000 7f14faa00640 1 -- 172.21.15.29:0/1325763488 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0ad880 con 0x7f14fc0cc8c0 2024-04-17T20:02:01.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.219+0000 7f1502200640 1 -- 172.21.15.29:0/1325763488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0cc8c0 msgr2=0x7f14fc0cccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:01.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.219+0000 7f1502200640 1 --2- 172.21.15.29:0/1325763488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0cc8c0 0x7f14fc0cccd0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f14ec009870 tx=0x7f14ec01c560 comp rx=0 tx=0).stop 2024-04-17T20:02:01.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.219+0000 7f1502200640 1 -- 172.21.15.29:0/1325763488 shutdown_connections 2024-04-17T20:02:01.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.219+0000 7f1502200640 1 --2- 172.21.15.29:0/1325763488 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14fc159390 0x7f14fc1597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:01.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.219+0000 7f1502200640 1 --2- 172.21.15.29:0/1325763488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0cc8c0 0x7f14fc0cccd0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:01.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.219+0000 7f1502200640 1 -- 172.21.15.29:0/1325763488 >> 172.21.15.29:0/1325763488 conn(0x7f14fc06fde0 msgr2=0x7f14fc0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:01.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.219+0000 7f1502200640 1 -- 172.21.15.29:0/1325763488 shutdown_connections 2024-04-17T20:02:01.220 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.219+0000 7f1502200640 1 -- 172.21.15.29:0/1325763488 wait complete. 2024-04-17T20:02:01.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.219+0000 7f1502200640 1 Processor -- start 2024-04-17T20:02:01.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.219+0000 7f1502200640 1 -- start start 2024-04-17T20:02:01.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f1502200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0ca330 0x7f14fc166660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:01.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f1502200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14fc0cc8c0 0x7f14fc1f8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:01.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f1502200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f14fc159390 0x7f14fc13e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:01.223 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f1502200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f14fc0c6a00 con 0x7f14fc0ca330 2024-04-17T20:02:01.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f1502200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f14fc0c6880 con 0x7f14fc159390 2024-04-17T20:02:01.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f1502200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f14fc1645f0 con 0x7f14fc0cc8c0 2024-04-17T20:02:01.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f14fb400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0ca330 0x7f14fc166660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:01.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f14fb400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0ca330 0x7f14fc166660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36786/0 (socket says 172.21.15.29:36786) 2024-04-17T20:02:01.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f14fb400640 1 -- 172.21.15.29:0/4232731370 learned_addr learned my addr 172.21.15.29:0/4232731370 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:01.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f14f3e00640 1 --2- 172.21.15.29:0/4232731370 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14fc0cc8c0 0x7f14fc1f8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:01.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f14fbe00640 1 --2- 172.21.15.29:0/4232731370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f14fc159390 0x7f14fc13e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:01.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f14fb400640 1 -- 172.21.15.29:0/4232731370 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14fc0cc8c0 msgr2=0x7f14fc1f8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:01.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f14fb400640 1 --2- 172.21.15.29:0/4232731370 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14fc0cc8c0 0x7f14fc1f8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:01.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f14fb400640 1 -- 172.21.15.29:0/4232731370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f14fc159390 msgr2=0x7f14fc13e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:01.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f14fb400640 1 --2- 172.21.15.29:0/4232731370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f14fc159390 0x7f14fc13e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:01.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f14fb400640 1 -- 172.21.15.29:0/4232731370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14fc166ba0 con 0x7f14fc0ca330 2024-04-17T20:02:01.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f14fb400640 1 --2- 172.21.15.29:0/4232731370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0ca330 0x7f14fc166660 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0b0040 tx=0x7f14ec01ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:01.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f14f3400640 1 -- 172.21.15.29:0/4232731370 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0bc020 con 0x7f14fc0ca330 2024-04-17T20:02:01.224 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f14f3400640 1 -- 172.21.15.29:0/4232731370 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14ec0ade50 con 0x7f14fc0ca330 2024-04-17T20:02:01.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f14f3400640 1 -- 172.21.15.29:0/4232731370 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ec0b2040 con 0x7f14fc0ca330 2024-04-17T20:02:01.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f1502200640 1 -- 172.21.15.29:0/4232731370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14fc0c8570 con 0x7f14fc0ca330 2024-04-17T20:02:01.225 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f1502200640 1 -- 172.21.15.29:0/4232731370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14fc073e30 con 0x7f14fc0ca330 2024-04-17T20:02:01.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f1502200640 1 -- 172.21.15.29:0/4232731370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14fc069b90 con 0x7f14fc0ca330 2024-04-17T20:02:01.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.223+0000 7f14f3400640 1 -- 172.21.15.29:0/4232731370 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f14ec0bf020 con 0x7f14fc0ca330 2024-04-17T20:02:01.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.227+0000 7f14f3400640 1 --2- 172.21.15.29:0/4232731370 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f14d8068380 0x7f14d806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:01.226 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.227+0000 7f14f3400640 1 -- 172.21.15.29:0/4232731370 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7f14ec121280 con 0x7f14fc0ca330 2024-04-17T20:02:01.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.227+0000 7f14f3e00640 1 --2- 172.21.15.29:0/4232731370 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f14d8068380 0x7f14d806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:01.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.227+0000 7f14f3e00640 1 --2- 172.21.15.29:0/4232731370 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f14d8068380 0x7f14d806a890 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0c4dd0 tx=0x7f14e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:01.231 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.231+0000 7f14f3400640 1 -- 172.21.15.29:0/4232731370 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f14ec0c6030 con 0x7f14fc0ca330 2024-04-17T20:02:01.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.435+0000 7f1502200640 1 -- 172.21.15.29:0/4232731370 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14fc158b80 con 0x7f14fc0ca330 2024-04-17T20:02:01.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.435+0000 7f14f3400640 1 -- 172.21.15.29:0/4232731370 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6910 (secure 0 0 0) 0x7f14ec016150 con 0x7f14fc0ca330 2024-04-17T20:02:01.433 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:02:01.433 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5946,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:58.083688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775,5631,5350,5212,5144]},"id":1}]} 2024-04-17T20:02:01.434 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 56 2024-04-17T20:02:01.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.435+0000 7f1502200640 1 -- 172.21.15.29:0/4232731370 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f14d8068380 msgr2=0x7f14d806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:01.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.435+0000 7f1502200640 1 --2- 172.21.15.29:0/4232731370 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f14d8068380 0x7f14d806a890 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f14fc0c4dd0 tx=0x7f14e4008040 comp rx=0 tx=0).stop 2024-04-17T20:02:01.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.435+0000 7f1502200640 1 -- 172.21.15.29:0/4232731370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0ca330 msgr2=0x7f14fc166660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:01.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.435+0000 7f1502200640 1 --2- 172.21.15.29:0/4232731370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0ca330 0x7f14fc166660 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f14ec0b0040 tx=0x7f14ec01ada0 comp rx=0 tx=0).stop 2024-04-17T20:02:01.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.435+0000 7f1502200640 1 -- 172.21.15.29:0/4232731370 shutdown_connections 2024-04-17T20:02:01.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.435+0000 7f1502200640 1 --2- 172.21.15.29:0/4232731370 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f14d8068380 0x7f14d806a890 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:01.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.435+0000 7f1502200640 1 --2- 172.21.15.29:0/4232731370 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f14fc159390 0x7f14fc13e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:01.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.435+0000 7f1502200640 1 --2- 172.21.15.29:0/4232731370 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f14fc0cc8c0 0x7f14fc1f8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:01.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.435+0000 7f1502200640 1 --2- 172.21.15.29:0/4232731370 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14fc0ca330 0x7f14fc166660 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:01.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.435+0000 7f1502200640 1 -- 172.21.15.29:0/4232731370 >> 172.21.15.29:0/4232731370 conn(0x7f14fc06fde0 msgr2=0x7f14fc0cb370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:01.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.435+0000 7f1502200640 1 -- 172.21.15.29:0/4232731370 shutdown_connections 2024-04-17T20:02:01.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:01.435+0000 7f1502200640 1 -- 172.21.15.29:0/4232731370 wait complete. 2024-04-17T20:02:01.456 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-17T20:02:03.457 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:02:03.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.639+0000 7f59ed200640 1 Processor -- start 2024-04-17T20:02:03.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.639+0000 7f59ed200640 1 -- start start 2024-04-17T20:02:03.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.639+0000 7f59ed200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f59e80cc8c0 0x7f59e80cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:03.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.639+0000 7f59ed200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f59e8159390 0x7f59e81597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:03.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.639+0000 7f59ed200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59e80c88b0 con 0x7f59e80ca330 2024-04-17T20:02:03.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.639+0000 7f59ed200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f59e80c6ce0 con 0x7f59e8159390 2024-04-17T20:02:03.640 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.639+0000 7f59ed200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f59e80c6fc0 con 0x7f59e80cc8c0 2024-04-17T20:02:03.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.639+0000 7f59dfe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f59e8159390 0x7f59e81597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:03.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.639+0000 7f59dfe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f59e8159390 0x7f59e81597a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36798/0 (socket says 172.21.15.29:36798) 2024-04-17T20:02:03.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.643+0000 7f59dfe00640 1 -- 172.21.15.29:0/2085448831 learned_addr learned my addr 172.21.15.29:0/2085448831 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:03.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.639+0000 7f59e6800640 1 --1- 172.21.15.29:0/2085448831 >> v1:172.21.15.29:6789/0 conn(0x7f59e80ca330 0x7f59e815c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40694/0 (socket says 172.21.15.29:40694) 2024-04-17T20:02:03.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.643+0000 7f59e5e00640 1 --2- 172.21.15.29:0/2085448831 >> v2:172.21.15.37:3300/0 conn(0x7f59e80cc8c0 0x7f59e80cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:03.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.643+0000 7f59dfe00640 1 -- 172.21.15.29:0/2085448831 >> v2:172.21.15.37:3300/0 conn(0x7f59e80cc8c0 msgr2=0x7f59e80cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:03.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.643+0000 7f59dfe00640 1 --2- 172.21.15.29:0/2085448831 >> v2:172.21.15.37:3300/0 conn(0x7f59e80cc8c0 0x7f59e80cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:03.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.643+0000 7f59dfe00640 1 -- 172.21.15.29:0/2085448831 >> v1:172.21.15.29:6789/0 conn(0x7f59e80ca330 legacy=0x7f59e815c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:03.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.643+0000 7f59dfe00640 1 -- 172.21.15.29:0/2085448831 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e80c72a0 con 0x7f59e8159390 2024-04-17T20:02:03.643 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.643+0000 7f59dfe00640 1 --2- 172.21.15.29:0/2085448831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59e8159390 0x7f59e81597a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f59e80721b0 tx=0x7f59d001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18369174ec35454f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:03.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.643+0000 7f59e5400640 1 -- 172.21.15.29:0/2085448831 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d00b0070 con 0x7f59e8159390 2024-04-17T20:02:03.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.643+0000 7f59e5400640 1 -- 172.21.15.29:0/2085448831 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d00ab600 con 0x7f59e8159390 2024-04-17T20:02:03.644 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.643+0000 7f59e5400640 1 -- 172.21.15.29:0/2085448831 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d00ab9d0 con 0x7f59e8159390 2024-04-17T20:02:03.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.647+0000 7f59ed200640 1 -- 172.21.15.29:0/2085448831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59e8159390 msgr2=0x7f59e81597a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:03.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.647+0000 7f59ed200640 1 --2- 172.21.15.29:0/2085448831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59e8159390 0x7f59e81597a0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f59e80721b0 tx=0x7f59d001c520 comp rx=0 tx=0).stop 2024-04-17T20:02:03.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.647+0000 7f59ed200640 1 -- 172.21.15.29:0/2085448831 shutdown_connections 2024-04-17T20:02:03.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.647+0000 7f59ed200640 1 --2- 172.21.15.29:0/2085448831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59e8159390 0x7f59e81597a0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:03.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.647+0000 7f59ed200640 1 --2- 172.21.15.29:0/2085448831 >> v2:172.21.15.37:3300/0 conn(0x7f59e80cc8c0 0x7f59e80cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:03.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.647+0000 7f59ed200640 1 -- 172.21.15.29:0/2085448831 >> 172.21.15.29:0/2085448831 conn(0x7f59e806fde0 msgr2=0x7f59e80701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:03.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.647+0000 7f59ed200640 1 -- 172.21.15.29:0/2085448831 shutdown_connections 2024-04-17T20:02:03.645 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.647+0000 7f59ed200640 1 -- 172.21.15.29:0/2085448831 wait complete. 2024-04-17T20:02:03.647 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.647+0000 7f59ed200640 1 Processor -- start 2024-04-17T20:02:03.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.647+0000 7f59ed200640 1 -- start start 2024-04-17T20:02:03.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.647+0000 7f59ed200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59e80ca330 0x7f59e815c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:03.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.647+0000 7f59ed200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59e80cc8c0 0x7f59e81f4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:03.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.647+0000 7f59ed200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f59e8159390 0x7f59e8073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:03.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.647+0000 7f59ed200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f59e80c6a00 con 0x7f59e80ca330 2024-04-17T20:02:03.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.647+0000 7f59ed200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f59e80c6880 con 0x7f59e80cc8c0 2024-04-17T20:02:03.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.647+0000 7f59ed200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f59e806bdc0 con 0x7f59e8159390 2024-04-17T20:02:03.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59e5e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59e80ca330 0x7f59e815c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:03.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59e5e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59e80ca330 0x7f59e815c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36800/0 (socket says 172.21.15.29:36800) 2024-04-17T20:02:03.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59e5e00640 1 -- 172.21.15.29:0/1842553979 learned_addr learned my addr 172.21.15.29:0/1842553979 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:03.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59dfe00640 1 --2- 172.21.15.29:0/1842553979 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59e80cc8c0 0x7f59e81f4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:03.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59e6800640 1 --2- 172.21.15.29:0/1842553979 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f59e8159390 0x7f59e8073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:03.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59e5e00640 1 -- 172.21.15.29:0/1842553979 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f59e8159390 msgr2=0x7f59e8073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:03.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59e5e00640 1 --2- 172.21.15.29:0/1842553979 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f59e8159390 0x7f59e8073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:03.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59e5e00640 1 -- 172.21.15.29:0/1842553979 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59e80cc8c0 msgr2=0x7f59e81f4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:03.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59e5e00640 1 --2- 172.21.15.29:0/1842553979 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59e80cc8c0 0x7f59e81f4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:03.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59e5e00640 1 -- 172.21.15.29:0/1842553979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e81f4e40 con 0x7f59e80ca330 2024-04-17T20:02:03.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59e6800640 1 --2- 172.21.15.29:0/1842553979 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f59e8159390 0x7f59e8073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T20:02:03.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59e5e00640 1 --2- 172.21.15.29:0/1842553979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59e80ca330 0x7f59e815c3c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f59d800d5f0 tx=0x7f59d800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:03.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59de000640 1 -- 172.21.15.29:0/1842553979 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d80125f0 con 0x7f59e80ca330 2024-04-17T20:02:03.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59de000640 1 -- 172.21.15.29:0/1842553979 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d8019070 con 0x7f59e80ca330 2024-04-17T20:02:03.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59de000640 1 -- 172.21.15.29:0/1842553979 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d80129b0 con 0x7f59e80ca330 2024-04-17T20:02:03.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59ed200640 1 -- 172.21.15.29:0/1842553979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e815a1a0 con 0x7f59e80ca330 2024-04-17T20:02:03.651 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59ed200640 1 -- 172.21.15.29:0/1842553979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e8138360 con 0x7f59e80ca330 2024-04-17T20:02:03.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59ed200640 1 -- 172.21.15.29:0/1842553979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59e8069b90 con 0x7f59e80ca330 2024-04-17T20:02:03.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.651+0000 7f59de000640 1 -- 172.21.15.29:0/1842553979 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f59d8012b50 con 0x7f59e80ca330 2024-04-17T20:02:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.655+0000 7f59de000640 1 --2- 172.21.15.29:0/1842553979 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f59c4068380 0x7f59c406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.655+0000 7f59de000640 1 -- 172.21.15.29:0/1842553979 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7251+0+0 (secure 0 0 0) 0x7f59d8087530 con 0x7f59e80ca330 2024-04-17T20:02:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.655+0000 7f59dfe00640 1 --2- 172.21.15.29:0/1842553979 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f59c4068380 0x7f59c406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.655+0000 7f59dfe00640 1 --2- 172.21.15.29:0/1842553979 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f59c4068380 0x7f59c406a890 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f59d00ad000 tx=0x7f59d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.659+0000 7f59de000640 1 -- 172.21.15.29:0/1842553979 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f59d80573b0 con 0x7f59e80ca330 2024-04-17T20:02:03.857 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.859+0000 7f59ed200640 1 -- 172.21.15.29:0/1842553979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59e80c7e10 con 0x7f59e80ca330 2024-04-17T20:02:03.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.859+0000 7f59de000640 1 -- 172.21.15.29:0/1842553979 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6910 (secure 0 0 0) 0x7f59d8056cb0 con 0x7f59e80ca330 2024-04-17T20:02:03.858 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:02:03.858 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5946,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:00:58.083688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5144,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5144":{"gid":5144,"name":"f","rank":2,"incarnation":38,"state":"up:active","state_seq":32,"addr":"172.21.15.37:6839/858161860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":858161860},{"type":"v1","addr":"172.21.15.37:6839","nonce":858161860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775,5631,5350,5212,5144]},"id":1}]} 2024-04-17T20:02:03.859 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 56 2024-04-17T20:02:03.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.859+0000 7f59ed200640 1 -- 172.21.15.29:0/1842553979 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f59c4068380 msgr2=0x7f59c406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:03.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.859+0000 7f59ed200640 1 --2- 172.21.15.29:0/1842553979 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f59c4068380 0x7f59c406a890 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f59d00ad000 tx=0x7f59d0008040 comp rx=0 tx=0).stop 2024-04-17T20:02:03.860 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.859+0000 7f59ed200640 1 -- 172.21.15.29:0/1842553979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59e80ca330 msgr2=0x7f59e815c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:03.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.859+0000 7f59ed200640 1 --2- 172.21.15.29:0/1842553979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59e80ca330 0x7f59e815c3c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f59d800d5f0 tx=0x7f59d800dab0 comp rx=0 tx=0).stop 2024-04-17T20:02:03.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.859+0000 7f59ed200640 1 -- 172.21.15.29:0/1842553979 shutdown_connections 2024-04-17T20:02:03.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.859+0000 7f59ed200640 1 --2- 172.21.15.29:0/1842553979 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f59c4068380 0x7f59c406a890 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:03.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.859+0000 7f59ed200640 1 --2- 172.21.15.29:0/1842553979 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f59e8159390 0x7f59e8073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:03.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.859+0000 7f59ed200640 1 --2- 172.21.15.29:0/1842553979 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f59e80cc8c0 0x7f59e81f4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:03.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.859+0000 7f59ed200640 1 --2- 172.21.15.29:0/1842553979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59e80ca330 0x7f59e815c3c0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:03.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.859+0000 7f59ed200640 1 -- 172.21.15.29:0/1842553979 >> 172.21.15.29:0/1842553979 conn(0x7f59e806fde0 msgr2=0x7f59e80c5b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:03.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.859+0000 7f59ed200640 1 -- 172.21.15.29:0/1842553979 shutdown_connections 2024-04-17T20:02:03.861 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:03.859+0000 7f59ed200640 1 -- 172.21.15.29:0/1842553979 wait complete. 2024-04-17T20:02:03.884 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2024-04-17T20:02:05.886 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:02:06.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.055+0000 7fd04e200640 1 Processor -- start 2024-04-17T20:02:06.054 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.055+0000 7fd04e200640 1 -- start start 2024-04-17T20:02:06.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.055+0000 7fd04e200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd04815dea0 0x7fd048165720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:06.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.055+0000 7fd04e200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0480ca700 con 0x7fd04815d380 2024-04-17T20:02:06.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.055+0000 7fd04e200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd04806f780 con 0x7fd0480cc3b0 2024-04-17T20:02:06.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.055+0000 7fd04e200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd048070540 con 0x7fd04815dea0 2024-04-17T20:02:06.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.055+0000 7fd03fe00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd04815d380 0x7fd04815d790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40702/0 (socket says 172.21.15.29:40702) 2024-04-17T20:02:06.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.055+0000 7fd03fe00640 1 -- 172.21.15.29:0/1856367243 learned_addr learned my addr 172.21.15.29:0/1856367243 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:02:06.056 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd047e00640 1 --2- 172.21.15.29:0/1856367243 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd04815dea0 0x7fd048165720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:06.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd046a00640 1 -- 172.21.15.29:0/1856367243 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 753257793 0 0) 0x7fd0480ca700 con 0x7fd04815d380 2024-04-17T20:02:06.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd046a00640 1 -- 172.21.15.29:0/1856367243 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd02c003530 con 0x7fd04815d380 2024-04-17T20:02:06.058 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd046a00640 1 -- 172.21.15.29:0/1856367243 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1986942 0 0) 0x7fd04806f780 con 0x7fd0480cc3b0 2024-04-17T20:02:06.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd046a00640 1 -- 172.21.15.29:0/1856367243 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0480ca700 con 0x7fd0480cc3b0 2024-04-17T20:02:06.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd046a00640 1 -- 172.21.15.29:0/1856367243 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1394124554 0 0) 0x7fd02c003530 con 0x7fd04815d380 2024-04-17T20:02:06.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd046a00640 1 -- 172.21.15.29:0/1856367243 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd04815dea0 msgr2=0x7fd048165720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:06.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd046a00640 1 --2- 172.21.15.29:0/1856367243 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd04815dea0 0x7fd048165720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd046a00640 1 -- 172.21.15.29:0/1856367243 >> v1:172.21.15.37:6789/0 conn(0x7fd0480cc3b0 legacy=0x7fd0480c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:06.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd046a00640 1 -- 172.21.15.29:0/1856367243 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd04806fa60 con 0x7fd04815d380 2024-04-17T20:02:06.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd046a00640 1 -- 172.21.15.29:0/1856367243 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fd030001680 con 0x7fd04815d380 2024-04-17T20:02:06.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd046a00640 1 -- 172.21.15.29:0/1856367243 >> v1:172.21.15.29:6789/0 conn(0x7fd04815d380 legacy=0x7fd04815d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:06.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd046a00640 1 --2- 172.21.15.29:0/1856367243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd02c003bf0 0x7fd02c004040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:06.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd046a00640 1 --2- 172.21.15.29:0/1856367243 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd04815dea0 0x7fd02c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:06.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd046a00640 1 --2- 172.21.15.29:0/1856367243 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd02c004d50 0x7fd02c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:06.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd046a00640 1 -- 172.21.15.29:0/1856367243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd02c00bb60 con 0x7fd02c003bf0 2024-04-17T20:02:06.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd046a00640 1 -- 172.21.15.29:0/1856367243 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd02c00bd10 con 0x7fd02c004d50 2024-04-17T20:02:06.059 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd046a00640 1 -- 172.21.15.29:0/1856367243 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd02c00bf10 con 0x7fd04815dea0 2024-04-17T20:02:06.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd047400640 1 --2- 172.21.15.29:0/1856367243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd02c003bf0 0x7fd02c004040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:06.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd047e00640 1 --2- 172.21.15.29:0/1856367243 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd02c004d50 0x7fd02c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:06.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd03fe00640 1 --2- 172.21.15.29:0/1856367243 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd04815dea0 0x7fd02c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:06.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd047400640 1 -- 172.21.15.29:0/1856367243 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd04815dea0 msgr2=0x7fd02c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:06.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd047400640 1 --2- 172.21.15.29:0/1856367243 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd04815dea0 0x7fd02c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd047400640 1 -- 172.21.15.29:0/1856367243 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd02c004d50 msgr2=0x7fd02c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:06.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd047400640 1 --2- 172.21.15.29:0/1856367243 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd02c004d50 0x7fd02c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.059+0000 7fd047400640 1 -- 172.21.15.29:0/1856367243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd02c003530 con 0x7fd02c003bf0 2024-04-17T20:02:06.060 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.063+0000 7fd047400640 1 --2- 172.21.15.29:0/1856367243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd02c003bf0 0x7fd02c004040 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd0380018d0 tx=0x7fd03801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9b04bbac3ba1540 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:06.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.063+0000 7fd046a00640 1 -- 172.21.15.29:0/1856367243 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd038001520 con 0x7fd02c003bf0 2024-04-17T20:02:06.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.063+0000 7fd046a00640 1 -- 172.21.15.29:0/1856367243 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0380052f0 con 0x7fd02c003bf0 2024-04-17T20:02:06.061 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.063+0000 7fd046a00640 1 -- 172.21.15.29:0/1856367243 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0380a8950 con 0x7fd02c003bf0 2024-04-17T20:02:06.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.063+0000 7fd04e200640 1 -- 172.21.15.29:0/1856367243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd02c003bf0 msgr2=0x7fd02c004040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:06.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.063+0000 7fd04e200640 1 --2- 172.21.15.29:0/1856367243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd02c003bf0 0x7fd02c004040 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd0380018d0 tx=0x7fd03801ecf0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.062 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.063+0000 7fd047e00640 1 -- 172.21.15.29:0/1856367243 reap_dead start 2024-04-17T20:02:06.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.063+0000 7fd04e200640 1 -- 172.21.15.29:0/1856367243 shutdown_connections 2024-04-17T20:02:06.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.063+0000 7fd04e200640 1 -- 172.21.15.29:0/1856367243 >> 172.21.15.29:0/1856367243 conn(0x7fd048071ac0 msgr2=0x7fd048071ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:06.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.063+0000 7fd04e200640 1 -- 172.21.15.29:0/1856367243 shutdown_connections 2024-04-17T20:02:06.063 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.063+0000 7fd04e200640 1 -- 172.21.15.29:0/1856367243 wait complete. 2024-04-17T20:02:06.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.063+0000 7fd04e200640 1 Processor -- start 2024-04-17T20:02:06.064 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.063+0000 7fd04e200640 1 -- start start 2024-04-17T20:02:06.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd04e200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd02c003bf0 0x7fd0480c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:06.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd04e200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd048153110 0x7fd048150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:06.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd04e200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd048153520 0x7fd048150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:06.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd04e200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd0480cc7c0 con 0x7fd048153110 2024-04-17T20:02:06.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd04e200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd0480700b0 con 0x7fd048153520 2024-04-17T20:02:06.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd04e200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd0480c4b10 con 0x7fd02c003bf0 2024-04-17T20:02:06.067 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd03fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd048153110 0x7fd048150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:06.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd03fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd048153110 0x7fd048150010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36810/0 (socket says 172.21.15.29:36810) 2024-04-17T20:02:06.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd03fe00640 1 -- 172.21.15.29:0/3108830469 learned_addr learned my addr 172.21.15.29:0/3108830469 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:06.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd047e00640 1 --2- 172.21.15.29:0/3108830469 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd048153520 0x7fd048150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:06.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd047400640 1 --2- 172.21.15.29:0/3108830469 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd02c003bf0 0x7fd0480c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:06.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd03fe00640 1 -- 172.21.15.29:0/3108830469 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd02c003bf0 msgr2=0x7fd0480c85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:06.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd03fe00640 1 --2- 172.21.15.29:0/3108830469 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd02c003bf0 0x7fd0480c85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd03fe00640 1 -- 172.21.15.29:0/3108830469 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd048153520 msgr2=0x7fd048150550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:06.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd03fe00640 1 --2- 172.21.15.29:0/3108830469 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd048153520 0x7fd048150550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd03fe00640 1 -- 172.21.15.29:0/3108830469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0480c8b20 con 0x7fd048153110 2024-04-17T20:02:06.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd047e00640 1 --2- 172.21.15.29:0/3108830469 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd048153520 0x7fd048150550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:02:06.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd047400640 1 --2- 172.21.15.29:0/3108830469 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd02c003bf0 0x7fd0480c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:02:06.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd03fe00640 1 --2- 172.21.15.29:0/3108830469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd048153110 0x7fd048150010 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd03000e700 tx=0x7fd03000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:06.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd03f400640 1 -- 172.21.15.29:0/3108830469 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03000c6c0 con 0x7fd048153110 2024-04-17T20:02:06.068 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd03f400640 1 -- 172.21.15.29:0/3108830469 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd030018070 con 0x7fd048153110 2024-04-17T20:02:06.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd04e200640 1 -- 172.21.15.29:0/3108830469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0480c7440 con 0x7fd048153110 2024-04-17T20:02:06.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd03f400640 1 -- 172.21.15.29:0/3108830469 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03000ca60 con 0x7fd048153110 2024-04-17T20:02:06.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd04e200640 1 -- 172.21.15.29:0/3108830469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd048150f40 con 0x7fd048153110 2024-04-17T20:02:06.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.067+0000 7fd04e200640 1 -- 172.21.15.29:0/3108830469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd048069b90 con 0x7fd048153110 2024-04-17T20:02:06.069 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.071+0000 7fd03f400640 1 -- 172.21.15.29:0/3108830469 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd03000cc00 con 0x7fd048153110 2024-04-17T20:02:06.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.071+0000 7fd03f400640 1 --2- 172.21.15.29:0/3108830469 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd024068380 0x7fd02406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:06.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.071+0000 7fd03f400640 1 -- 172.21.15.29:0/3108830469 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7fd030086f90 con 0x7fd048153110 2024-04-17T20:02:06.070 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.071+0000 7fd047400640 1 --2- 172.21.15.29:0/3108830469 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd024068380 0x7fd02406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:06.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.071+0000 7fd047400640 1 --2- 172.21.15.29:0/3108830469 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd024068380 0x7fd02406a890 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd0380b0000 tx=0x7fd0380aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:06.081 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.075+0000 7fd03f400640 1 -- 172.21.15.29:0/3108830469 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd030056dc0 con 0x7fd048153110 2024-04-17T20:02:06.275 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.275+0000 7fd04e200640 1 -- 172.21.15.29:0/3108830469 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0480cb570 con 0x7fd048153110 2024-04-17T20:02:06.276 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.275+0000 7fd03f400640 1 -- 172.21.15.29:0/3108830469 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+6083 (secure 0 0 0) 0x7fd0300566c0 con 0x7fd048153110 2024-04-17T20:02:06.276 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:02:06.276 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:04.111387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:replay","state_seq":1,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5212,5350,5631,5775]},"id":1}]} 2024-04-17T20:02:06.276 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 57 2024-04-17T20:02:06.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.279+0000 7fd04e200640 1 -- 172.21.15.29:0/3108830469 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd024068380 msgr2=0x7fd02406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:06.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.279+0000 7fd04e200640 1 --2- 172.21.15.29:0/3108830469 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd024068380 0x7fd02406a890 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd0380b0000 tx=0x7fd0380aa070 comp rx=0 tx=0).stop 2024-04-17T20:02:06.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.279+0000 7fd04e200640 1 -- 172.21.15.29:0/3108830469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd048153110 msgr2=0x7fd048150010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:06.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.279+0000 7fd04e200640 1 --2- 172.21.15.29:0/3108830469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd048153110 0x7fd048150010 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd03000e700 tx=0x7fd03000ebc0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.279+0000 7fd04e200640 1 -- 172.21.15.29:0/3108830469 shutdown_connections 2024-04-17T20:02:06.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.279+0000 7fd04e200640 1 --2- 172.21.15.29:0/3108830469 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd024068380 0x7fd02406a890 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.279+0000 7fd04e200640 1 --2- 172.21.15.29:0/3108830469 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd048153520 0x7fd048150550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.279+0000 7fd04e200640 1 --2- 172.21.15.29:0/3108830469 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd048153110 0x7fd048150010 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.279+0000 7fd04e200640 1 --2- 172.21.15.29:0/3108830469 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd02c003bf0 0x7fd0480c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.279+0000 7fd04e200640 1 -- 172.21.15.29:0/3108830469 >> 172.21.15.29:0/3108830469 conn(0x7fd048071ac0 msgr2=0x7fd0480c57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:06.279 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.279+0000 7fd04e200640 1 -- 172.21.15.29:0/3108830469 shutdown_connections 2024-04-17T20:02:06.280 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.279+0000 7fd04e200640 1 -- 172.21.15.29:0/3108830469 wait complete. 2024-04-17T20:02:06.297 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2024-04-17T20:02:06.298 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-17T20:02:06.298 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:02:06.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.451+0000 7feb8f600640 1 Processor -- start 2024-04-17T20:02:06.451 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.451+0000 7feb8f600640 1 -- start start 2024-04-17T20:02:06.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb8f600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb880cc8c0 0x7feb880cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:06.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb8f600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb880c88b0 con 0x7feb88159390 2024-04-17T20:02:06.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb8f600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb880c6ce0 con 0x7feb881597a0 2024-04-17T20:02:06.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb8f600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7feb880c6fc0 con 0x7feb880cc8c0 2024-04-17T20:02:06.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7feb88159390 0x7feb880ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40716/0 (socket says 172.21.15.29:40716) 2024-04-17T20:02:06.453 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87e00640 1 -- 172.21.15.29:0/2827132606 learned_addr learned my addr 172.21.15.29:0/2827132606 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:02:06.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb8cc00640 1 --2- 172.21.15.29:0/2827132606 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb880cc8c0 0x7feb880cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:06.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87400640 1 -- 172.21.15.29:0/2827132606 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1428778221 0 0) 0x7feb880c88b0 con 0x7feb88159390 2024-04-17T20:02:06.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87400640 1 -- 172.21.15.29:0/2827132606 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb70005740 con 0x7feb88159390 2024-04-17T20:02:06.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87400640 1 -- 172.21.15.29:0/2827132606 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 307872934 0 0) 0x7feb880c6ce0 con 0x7feb881597a0 2024-04-17T20:02:06.455 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87400640 1 -- 172.21.15.29:0/2827132606 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb880c88b0 con 0x7feb881597a0 2024-04-17T20:02:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87400640 1 -- 172.21.15.29:0/2827132606 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4268839426 0 0) 0x7feb70005740 con 0x7feb88159390 2024-04-17T20:02:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87400640 1 -- 172.21.15.29:0/2827132606 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb880cc8c0 msgr2=0x7feb880cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87400640 1 --2- 172.21.15.29:0/2827132606 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb880cc8c0 0x7feb880cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87400640 1 -- 172.21.15.29:0/2827132606 >> v1:172.21.15.37:6789/0 conn(0x7feb881597a0 legacy=0x7feb881635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87400640 1 -- 172.21.15.29:0/2827132606 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb880c72a0 con 0x7feb88159390 2024-04-17T20:02:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87400640 1 -- 172.21.15.29:0/2827132606 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7feb74001680 con 0x7feb88159390 2024-04-17T20:02:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87400640 1 -- 172.21.15.29:0/2827132606 >> v1:172.21.15.29:6789/0 conn(0x7feb88159390 legacy=0x7feb880ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87400640 1 --2- 172.21.15.29:0/2827132606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb70005e00 0x7feb70006250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87400640 1 --2- 172.21.15.29:0/2827132606 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb70006a20 0x7feb70006e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87400640 1 --2- 172.21.15.29:0/2827132606 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb880cc8c0 0x7feb70007490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87400640 1 -- 172.21.15.29:0/2827132606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feb880c6fc0 con 0x7feb70005e00 2024-04-17T20:02:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87400640 1 -- 172.21.15.29:0/2827132606 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7feb7000dd70 con 0x7feb70006a20 2024-04-17T20:02:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87400640 1 -- 172.21.15.29:0/2827132606 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7feb7000df70 con 0x7feb880cc8c0 2024-04-17T20:02:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb8cc00640 1 --2- 172.21.15.29:0/2827132606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb70005e00 0x7feb70006250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb87e00640 1 --2- 172.21.15.29:0/2827132606 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb70006a20 0x7feb70006e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb8d600640 1 --2- 172.21.15.29:0/2827132606 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb880cc8c0 0x7feb70007490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb8cc00640 1 -- 172.21.15.29:0/2827132606 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb880cc8c0 msgr2=0x7feb70007490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb8cc00640 1 --2- 172.21.15.29:0/2827132606 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb880cc8c0 0x7feb70007490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb8cc00640 1 -- 172.21.15.29:0/2827132606 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb70006a20 msgr2=0x7feb70006e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb8cc00640 1 --2- 172.21.15.29:0/2827132606 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb70006a20 0x7feb70006e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.455+0000 7feb8cc00640 1 -- 172.21.15.29:0/2827132606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb70005740 con 0x7feb70005e00 2024-04-17T20:02:06.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.459+0000 7feb8cc00640 1 --2- 172.21.15.29:0/2827132606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb70005e00 0x7feb70006250 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7feb7c009ae0 tx=0x7feb7c01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c75e7027f4b4408e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:06.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.459+0000 7feb87400640 1 -- 172.21.15.29:0/2827132606 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb7c0096b0 con 0x7feb70005e00 2024-04-17T20:02:06.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.459+0000 7feb87400640 1 -- 172.21.15.29:0/2827132606 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb7c0a9470 con 0x7feb70005e00 2024-04-17T20:02:06.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.459+0000 7feb87400640 1 -- 172.21.15.29:0/2827132606 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb7c0a9860 con 0x7feb70005e00 2024-04-17T20:02:06.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.459+0000 7feb8f600640 1 -- 172.21.15.29:0/2827132606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb70005e00 msgr2=0x7feb70006250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:06.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.459+0000 7feb8f600640 1 --2- 172.21.15.29:0/2827132606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb70005e00 0x7feb70006250 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7feb7c009ae0 tx=0x7feb7c01ca90 comp rx=0 tx=0).stop 2024-04-17T20:02:06.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.459+0000 7feb8d600640 1 -- 172.21.15.29:0/2827132606 reap_dead start 2024-04-17T20:02:06.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.459+0000 7feb8f600640 1 -- 172.21.15.29:0/2827132606 shutdown_connections 2024-04-17T20:02:06.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.459+0000 7feb8f600640 1 -- 172.21.15.29:0/2827132606 >> 172.21.15.29:0/2827132606 conn(0x7feb8806fde0 msgr2=0x7feb880701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:06.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.459+0000 7feb8f600640 1 -- 172.21.15.29:0/2827132606 shutdown_connections 2024-04-17T20:02:06.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.459+0000 7feb8f600640 1 -- 172.21.15.29:0/2827132606 wait complete. 2024-04-17T20:02:06.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8f600640 1 Processor -- start 2024-04-17T20:02:06.461 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8f600640 1 -- start start 2024-04-17T20:02:06.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8f600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb70005e00 0x7feb8815c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:06.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8f600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb880ba530 0x7feb8815cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:06.463 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8f600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb881537f0 0x7feb881f1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8f600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feb880caa40 con 0x7feb881537f0 2024-04-17T20:02:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8f600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7feb88159bb0 con 0x7feb70005e00 2024-04-17T20:02:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8f600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7feb880cabc0 con 0x7feb880ba530 2024-04-17T20:02:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8cc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb70005e00 0x7feb8815c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8d600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb881537f0 0x7feb881f1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8d600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb881537f0 0x7feb881f1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36830/0 (socket says 172.21.15.29:36830) 2024-04-17T20:02:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8cc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb70005e00 0x7feb8815c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:44580/0 (socket says 172.21.15.29:44580) 2024-04-17T20:02:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8d600640 1 -- 172.21.15.29:0/2814565297 learned_addr learned my addr 172.21.15.29:0/2814565297 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb87e00640 1 --2- 172.21.15.29:0/2814565297 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb880ba530 0x7feb8815cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8d600640 1 -- 172.21.15.29:0/2814565297 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb880ba530 msgr2=0x7feb8815cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8d600640 1 --2- 172.21.15.29:0/2814565297 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb880ba530 0x7feb8815cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8d600640 1 -- 172.21.15.29:0/2814565297 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb70005e00 msgr2=0x7feb8815c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8d600640 1 --2- 172.21.15.29:0/2814565297 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb70005e00 0x7feb8815c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.464 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8d600640 1 -- 172.21.15.29:0/2814565297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb881f1e70 con 0x7feb881537f0 2024-04-17T20:02:06.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8d600640 1 --2- 172.21.15.29:0/2814565297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb881537f0 0x7feb881f1120 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7feb88070990 tx=0x7feb8000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:06.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb84c00640 1 -- 172.21.15.29:0/2814565297 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb80003980 con 0x7feb881537f0 2024-04-17T20:02:06.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb84c00640 1 -- 172.21.15.29:0/2814565297 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb80018070 con 0x7feb881537f0 2024-04-17T20:02:06.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8f600640 1 -- 172.21.15.29:0/2814565297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb880c3f80 con 0x7feb881537f0 2024-04-17T20:02:06.465 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb84c00640 1 -- 172.21.15.29:0/2814565297 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb80003d80 con 0x7feb881537f0 2024-04-17T20:02:06.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.463+0000 7feb8f600640 1 -- 172.21.15.29:0/2814565297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb880c4260 con 0x7feb881537f0 2024-04-17T20:02:06.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.467+0000 7feb8f600640 1 -- 172.21.15.29:0/2814565297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb88069b90 con 0x7feb881537f0 2024-04-17T20:02:06.466 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.467+0000 7feb84c00640 1 -- 172.21.15.29:0/2814565297 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7feb800050d0 con 0x7feb881537f0 2024-04-17T20:02:06.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.467+0000 7feb84c00640 1 --2- 172.21.15.29:0/2814565297 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7feb68068380 0x7feb6806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:06.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.467+0000 7feb84c00640 1 -- 172.21.15.29:0/2814565297 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7feb80086ca0 con 0x7feb881537f0 2024-04-17T20:02:06.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.467+0000 7feb8cc00640 1 --2- 172.21.15.29:0/2814565297 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7feb68068380 0x7feb6806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:06.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.467+0000 7feb8cc00640 1 --2- 172.21.15.29:0/2814565297 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7feb68068380 0x7feb6806a890 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7feb7c009ae0 tx=0x7feb7c0046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:06.471 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.471+0000 7feb84c00640 1 -- 172.21.15.29:0/2814565297 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7feb800565a0 con 0x7feb881537f0 2024-04-17T20:02:06.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.671+0000 7feb8f600640 1 -- 172.21.15.29:0/2814565297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb881f2040 con 0x7feb881537f0 2024-04-17T20:02:06.671 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.671+0000 7feb84c00640 1 -- 172.21.15.29:0/2814565297 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+6083 (secure 0 0 0) 0x7feb8005a8a0 con 0x7feb881537f0 2024-04-17T20:02:06.672 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:02:06.672 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:04.111387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:replay","state_seq":1,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5212,5350,5631,5775]},"id":1}]} 2024-04-17T20:02:06.672 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 57 2024-04-17T20:02:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.675+0000 7feb8f600640 1 -- 172.21.15.29:0/2814565297 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7feb68068380 msgr2=0x7feb6806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.675+0000 7feb8f600640 1 --2- 172.21.15.29:0/2814565297 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7feb68068380 0x7feb6806a890 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7feb7c009ae0 tx=0x7feb7c0046d0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.675+0000 7feb8f600640 1 -- 172.21.15.29:0/2814565297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb881537f0 msgr2=0x7feb881f1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.675+0000 7feb8f600640 1 --2- 172.21.15.29:0/2814565297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb881537f0 0x7feb881f1120 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7feb88070990 tx=0x7feb8000ec00 comp rx=0 tx=0).stop 2024-04-17T20:02:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.675+0000 7feb8f600640 1 -- 172.21.15.29:0/2814565297 shutdown_connections 2024-04-17T20:02:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.675+0000 7feb8f600640 1 --2- 172.21.15.29:0/2814565297 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7feb68068380 0x7feb6806a890 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.675+0000 7feb8f600640 1 --2- 172.21.15.29:0/2814565297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb881537f0 0x7feb881f1120 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.675+0000 7feb8f600640 1 --2- 172.21.15.29:0/2814565297 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb880ba530 0x7feb8815cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.675+0000 7feb8f600640 1 --2- 172.21.15.29:0/2814565297 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb70005e00 0x7feb8815c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:06.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.675+0000 7feb8f600640 1 -- 172.21.15.29:0/2814565297 >> 172.21.15.29:0/2814565297 conn(0x7feb8806fde0 msgr2=0x7feb88072520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:06.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.675+0000 7feb8f600640 1 -- 172.21.15.29:0/2814565297 shutdown_connections 2024-04-17T20:02:06.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:06.675+0000 7feb8f600640 1 -- 172.21.15.29:0/2814565297 wait complete. 2024-04-17T20:02:06.694 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 57, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-17T19:47:49.315515+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 57, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5212": { "addr": "172.21.15.37:6838/767681664", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6835", "nonce": 767681664, "type": "v2" }, { "addr": "172.21.15.37:6838", "nonce": 767681664, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5212, "incarnation": 26, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 31 }, "gid_5350": { "addr": "172.21.15.37:6837/2495953425", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6834", "nonce": 2495953425, "type": "v2" }, { "addr": "172.21.15.37:6837", "nonce": 2495953425, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5350, "incarnation": 32, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 18 }, "gid_5631": { "addr": "172.21.15.29:6836/2376553580", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6832", "nonce": 2376553580, "type": "v2" }, { "addr": "172.21.15.29:6836", "nonce": 2376553580, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5631, "incarnation": 44, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 42 }, "gid_5775": { "addr": "172.21.15.29:6835/1352350250", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6834", "nonce": 1352350250, "type": "v2" }, { "addr": "172.21.15.29:6835", "nonce": 1352350250, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5775, "incarnation": 50, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 39 }, "gid_5946": { "addr": "172.21.15.29:6837/3087508289", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6833", "nonce": 3087508289, "type": "v2" }, { "addr": "172.21.15.29:6837", "nonce": 3087508289, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5946, "incarnation": 57, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 75, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-17T20:02:04.111387+0000", "qdb_cluster": [ 5212, 5350, 5631, 5775 ], "qdb_leader": 5775, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5775, "mds_1": 5631, "mds_2": 5946, "mds_3": 5350, "mds_4": 5212 } } } ], "standbys": [] } 2024-04-17T20:02:08.696 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:02:08.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd061e00640 1 Processor -- start 2024-04-17T20:02:08.849 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd061e00640 1 -- start start 2024-04-17T20:02:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd061e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd05c159e60 0x7fd05c0cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd061e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd05c153d20 0x7fd05c071940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd061e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd05c071e80 0x7fd05c0c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd061e00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fd05c0d0320 con 0x7fd05c159e60 2024-04-17T20:02:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd061e00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd05c0c4930 con 0x7fd05c153d20 2024-04-17T20:02:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd061e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd05c0cf620 con 0x7fd05c071e80 2024-04-17T20:02:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd05aa00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd05c159e60 0x7fd05c0cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd05aa00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd05c159e60 0x7fd05c0cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41392/0 (socket says 172.21.15.29:41392) 2024-04-17T20:02:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd05aa00640 1 -- 172.21.15.29:0/715164760 learned_addr learned my addr 172.21.15.29:0/715164760 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd05b400640 1 --2- 172.21.15.29:0/715164760 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd05c071e80 0x7fd05c0c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:08.851 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd05a000640 1 --2- 172.21.15.29:0/715164760 >> v2:172.21.15.37:3300/0 conn(0x7fd05c153d20 0x7fd05c071940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:08.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd05aa00640 1 -- 172.21.15.29:0/715164760 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd05c071e80 msgr2=0x7fd05c0c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:08.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd05aa00640 1 --2- 172.21.15.29:0/715164760 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd05c071e80 0x7fd05c0c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:08.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd05aa00640 1 -- 172.21.15.29:0/715164760 >> v2:172.21.15.37:3300/0 conn(0x7fd05c153d20 msgr2=0x7fd05c071940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:08.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd05aa00640 1 --2- 172.21.15.29:0/715164760 >> v2:172.21.15.37:3300/0 conn(0x7fd05c153d20 0x7fd05c071940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:08.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd05aa00640 1 -- 172.21.15.29:0/715164760 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd05c0c5190 con 0x7fd05c159e60 2024-04-17T20:02:08.852 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd05b400640 1 --2- 172.21.15.29:0/715164760 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd05c071e80 0x7fd05c0c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:02:08.853 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.851+0000 7fd05a000640 1 --2- 172.21.15.29:0/715164760 >> v2:172.21.15.37:3300/0 conn(0x7fd05c153d20 0x7fd05c071940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:02:08.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.855+0000 7fd05aa00640 1 --2- 172.21.15.29:0/715164760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd05c159e60 0x7fd05c0cfde0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd05c068560 tx=0x7fd04c01c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e5b05145a177ef server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:08.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.855+0000 7fd059600640 1 -- 172.21.15.29:0/715164760 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04c0b0070 con 0x7fd05c159e60 2024-04-17T20:02:08.854 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.855+0000 7fd059600640 1 -- 172.21.15.29:0/715164760 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd04c0ab600 con 0x7fd05c159e60 2024-04-17T20:02:08.855 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.855+0000 7fd059600640 1 -- 172.21.15.29:0/715164760 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04c0ab9d0 con 0x7fd05c159e60 2024-04-17T20:02:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.855+0000 7fd061e00640 1 -- 172.21.15.29:0/715164760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd05c159e60 msgr2=0x7fd05c0cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.855+0000 7fd061e00640 1 --2- 172.21.15.29:0/715164760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd05c159e60 0x7fd05c0cfde0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fd05c068560 tx=0x7fd04c01c260 comp rx=0 tx=0).stop 2024-04-17T20:02:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.855+0000 7fd061e00640 1 -- 172.21.15.29:0/715164760 shutdown_connections 2024-04-17T20:02:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.855+0000 7fd061e00640 1 --2- 172.21.15.29:0/715164760 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd05c071e80 0x7fd05c0c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.855+0000 7fd061e00640 1 --2- 172.21.15.29:0/715164760 >> v2:172.21.15.37:3300/0 conn(0x7fd05c153d20 0x7fd05c071940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.855+0000 7fd061e00640 1 --2- 172.21.15.29:0/715164760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd05c159e60 0x7fd05c0cfde0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.855+0000 7fd061e00640 1 -- 172.21.15.29:0/715164760 >> 172.21.15.29:0/715164760 conn(0x7fd05c067eb0 msgr2=0x7fd05c070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.855+0000 7fd061e00640 1 -- 172.21.15.29:0/715164760 shutdown_connections 2024-04-17T20:02:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.855+0000 7fd061e00640 1 -- 172.21.15.29:0/715164760 wait complete. 2024-04-17T20:02:08.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.859+0000 7fd061e00640 1 Processor -- start 2024-04-17T20:02:08.858 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.859+0000 7fd061e00640 1 -- start start 2024-04-17T20:02:08.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.859+0000 7fd061e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd05c071e80 0x7fd05c0cee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:08.872 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.859+0000 7fd061e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd05c153d20 0x7fd05c06c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:08.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.859+0000 7fd061e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd05c159e60 0x7fd05c1f91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:08.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.859+0000 7fd061e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd05c0cbd10 con 0x7fd05c159e60 2024-04-17T20:02:08.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.859+0000 7fd061e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd05c0cbb90 con 0x7fd05c153d20 2024-04-17T20:02:08.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.859+0000 7fd061e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd05c0c4510 con 0x7fd05c071e80 2024-04-17T20:02:08.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.859+0000 7fd05b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd05c159e60 0x7fd05c1f91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:08.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.859+0000 7fd05b400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd05c159e60 0x7fd05c1f91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41406/0 (socket says 172.21.15.29:41406) 2024-04-17T20:02:08.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.859+0000 7fd05b400640 1 -- 172.21.15.29:0/4181085789 learned_addr learned my addr 172.21.15.29:0/4181085789 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:08.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.859+0000 7fd05aa00640 1 --2- 172.21.15.29:0/4181085789 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd05c071e80 0x7fd05c0cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:08.873 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.859+0000 7fd05a000640 1 --2- 172.21.15.29:0/4181085789 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd05c153d20 0x7fd05c06c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.863+0000 7fd05b400640 1 -- 172.21.15.29:0/4181085789 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd05c071e80 msgr2=0x7fd05c0cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.863+0000 7fd05b400640 1 --2- 172.21.15.29:0/4181085789 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd05c071e80 0x7fd05c0cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.863+0000 7fd05b400640 1 -- 172.21.15.29:0/4181085789 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd05c153d20 msgr2=0x7fd05c06c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.863+0000 7fd05b400640 1 --2- 172.21.15.29:0/4181085789 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd05c153d20 0x7fd05c06c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.863+0000 7fd05b400640 1 -- 172.21.15.29:0/4181085789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd05c06cdd0 con 0x7fd05c159e60 2024-04-17T20:02:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.863+0000 7fd05b400640 1 --2- 172.21.15.29:0/4181085789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd05c159e60 0x7fd05c1f91b0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fd054007e10 tx=0x7fd05400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.863+0000 7fd053400640 1 -- 172.21.15.29:0/4181085789 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05401a070 con 0x7fd05c159e60 2024-04-17T20:02:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.863+0000 7fd053400640 1 -- 172.21.15.29:0/4181085789 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd054003bc0 con 0x7fd05c159e60 2024-04-17T20:02:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.863+0000 7fd061e00640 1 -- 172.21.15.29:0/4181085789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd05c06c030 con 0x7fd05c159e60 2024-04-17T20:02:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.863+0000 7fd053400640 1 -- 172.21.15.29:0/4181085789 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05400de80 con 0x7fd05c159e60 2024-04-17T20:02:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.863+0000 7fd061e00640 1 -- 172.21.15.29:0/4181085789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd05c0ce1d0 con 0x7fd05c159e60 2024-04-17T20:02:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.863+0000 7fd061e00640 1 -- 172.21.15.29:0/4181085789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd05c069b90 con 0x7fd05c159e60 2024-04-17T20:02:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.863+0000 7fd053400640 1 -- 172.21.15.29:0/4181085789 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd05401d070 con 0x7fd05c159e60 2024-04-17T20:02:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.867+0000 7fd053400640 1 --2- 172.21.15.29:0/4181085789 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd03c0683b0 0x7fd03c06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.867+0000 7fd053400640 1 -- 172.21.15.29:0/4181085789 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7fd054087150 con 0x7fd05c159e60 2024-04-17T20:02:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.867+0000 7fd05aa00640 1 --2- 172.21.15.29:0/4181085789 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd03c0683b0 0x7fd03c06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.867+0000 7fd05aa00640 1 --2- 172.21.15.29:0/4181085789 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd03c0683b0 0x7fd03c06a8c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd04c0ad000 tx=0x7fd04c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:08.871+0000 7fd053400640 1 -- 172.21.15.29:0/4181085789 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd054010070 con 0x7fd05c159e60 2024-04-17T20:02:09.074 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:09.075+0000 7fd061e00640 1 -- 172.21.15.29:0/4181085789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd05c0cb6a0 con 0x7fd05c159e60 2024-04-17T20:02:09.161 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:09.075+0000 7fd053400640 1 -- 172.21.15.29:0/4181085789 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+6083 (secure 0 0 0) 0x7fd0540569c0 con 0x7fd05c159e60 2024-04-17T20:02:09.162 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:02:09.162 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:04.111387+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:replay","state_seq":1,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5212,5350,5631,5775]},"id":1}]} 2024-04-17T20:02:09.164 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 57 2024-04-17T20:02:09.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:09.075+0000 7fd061e00640 1 -- 172.21.15.29:0/4181085789 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd03c0683b0 msgr2=0x7fd03c06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:09.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:09.075+0000 7fd061e00640 1 --2- 172.21.15.29:0/4181085789 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd03c0683b0 0x7fd03c06a8c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd04c0ad000 tx=0x7fd04c008040 comp rx=0 tx=0).stop 2024-04-17T20:02:09.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:09.075+0000 7fd061e00640 1 -- 172.21.15.29:0/4181085789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd05c159e60 msgr2=0x7fd05c1f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:09.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:09.075+0000 7fd061e00640 1 --2- 172.21.15.29:0/4181085789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd05c159e60 0x7fd05c1f91b0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fd054007e10 tx=0x7fd05400d4a0 comp rx=0 tx=0).stop 2024-04-17T20:02:09.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:09.075+0000 7fd061e00640 1 -- 172.21.15.29:0/4181085789 shutdown_connections 2024-04-17T20:02:09.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:09.075+0000 7fd061e00640 1 --2- 172.21.15.29:0/4181085789 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd03c0683b0 0x7fd03c06a8c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:09.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:09.075+0000 7fd061e00640 1 --2- 172.21.15.29:0/4181085789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd05c159e60 0x7fd05c1f91b0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:09.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:09.075+0000 7fd061e00640 1 --2- 172.21.15.29:0/4181085789 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd05c153d20 0x7fd05c06c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:09.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:09.075+0000 7fd061e00640 1 --2- 172.21.15.29:0/4181085789 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd05c071e80 0x7fd05c0cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:09.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:09.075+0000 7fd061e00640 1 -- 172.21.15.29:0/4181085789 >> 172.21.15.29:0/4181085789 conn(0x7fd05c067eb0 msgr2=0x7fd05c151270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:09.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:09.075+0000 7fd061e00640 1 -- 172.21.15.29:0/4181085789 shutdown_connections 2024-04-17T20:02:09.165 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:09.075+0000 7fd061e00640 1 -- 172.21.15.29:0/4181085789 wait complete. 2024-04-17T20:02:09.167 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:02:11.168 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:02:11.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.331+0000 7fe531600640 1 Processor -- start 2024-04-17T20:02:11.337 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.331+0000 7fe531600640 1 -- start start 2024-04-17T20:02:11.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe531600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe52c0cbf90 0x7fe52c15ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:11.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe531600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe52c15cf70 con 0x7fe52c155da0 2024-04-17T20:02:11.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe531600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe52c0cbba0 con 0x7fe52c0c7f50 2024-04-17T20:02:11.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe531600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe52c06bdc0 con 0x7fe52c0cbf90 2024-04-17T20:02:11.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52b400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe52c155da0 0x7fe52c1561b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60574/0 (socket says 172.21.15.29:60574) 2024-04-17T20:02:11.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52b400640 1 -- 172.21.15.29:0/4281529168 learned_addr learned my addr 172.21.15.29:0/4281529168 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:02:11.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52aa00640 1 --2- 172.21.15.29:0/4281529168 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe52c0cbf90 0x7fe52c15ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:11.338 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52a000640 1 -- 172.21.15.29:0/4281529168 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3671031378 0 0) 0x7fe52c15cf70 con 0x7fe52c155da0 2024-04-17T20:02:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52a000640 1 -- 172.21.15.29:0/4281529168 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe510003530 con 0x7fe52c155da0 2024-04-17T20:02:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52a000640 1 -- 172.21.15.29:0/4281529168 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1881609230 0 0) 0x7fe52c0cbba0 con 0x7fe52c0c7f50 2024-04-17T20:02:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52a000640 1 -- 172.21.15.29:0/4281529168 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe52c15cf70 con 0x7fe52c0c7f50 2024-04-17T20:02:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52a000640 1 -- 172.21.15.29:0/4281529168 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 915278756 0 0) 0x7fe510003530 con 0x7fe52c155da0 2024-04-17T20:02:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52a000640 1 -- 172.21.15.29:0/4281529168 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe52c0cbf90 msgr2=0x7fe52c15ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52a000640 1 --2- 172.21.15.29:0/4281529168 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe52c0cbf90 0x7fe52c15ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52a000640 1 -- 172.21.15.29:0/4281529168 >> v1:172.21.15.37:6789/0 conn(0x7fe52c0c7f50 legacy=0x7fe52c0ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:11.339 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52a000640 1 -- 172.21.15.29:0/4281529168 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe52c06c010 con 0x7fe52c155da0 2024-04-17T20:02:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52a000640 1 -- 172.21.15.29:0/4281529168 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fe524001a00 con 0x7fe52c155da0 2024-04-17T20:02:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52a000640 1 -- 172.21.15.29:0/4281529168 >> v1:172.21.15.29:6789/0 conn(0x7fe52c155da0 legacy=0x7fe52c1561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52a000640 1 --2- 172.21.15.29:0/4281529168 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe510003bf0 0x7fe510004040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52a000640 1 --2- 172.21.15.29:0/4281529168 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe52c0cbf90 0x7fe510004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52a000640 1 --2- 172.21.15.29:0/4281529168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe510004d50 0x7fe510005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52a000640 1 -- 172.21.15.29:0/4281529168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe52c06bdc0 con 0x7fe510004d50 2024-04-17T20:02:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52a000640 1 -- 172.21.15.29:0/4281529168 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe51000bb90 con 0x7fe510003bf0 2024-04-17T20:02:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52a000640 1 -- 172.21.15.29:0/4281529168 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe51000bd90 con 0x7fe52c0cbf90 2024-04-17T20:02:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52aa00640 1 --2- 172.21.15.29:0/4281529168 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe510003bf0 0x7fe510004040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52b400640 1 --2- 172.21.15.29:0/4281529168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe510004d50 0x7fe510005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe523e00640 1 --2- 172.21.15.29:0/4281529168 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe52c0cbf90 0x7fe510004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52b400640 1 -- 172.21.15.29:0/4281529168 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe52c0cbf90 msgr2=0x7fe510004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52b400640 1 --2- 172.21.15.29:0/4281529168 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe52c0cbf90 0x7fe510004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52b400640 1 -- 172.21.15.29:0/4281529168 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe510003bf0 msgr2=0x7fe510004040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52b400640 1 --2- 172.21.15.29:0/4281529168 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe510003bf0 0x7fe510004040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.335+0000 7fe52b400640 1 -- 172.21.15.29:0/4281529168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe510003530 con 0x7fe510004d50 2024-04-17T20:02:11.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.339+0000 7fe52b400640 1 --2- 172.21.15.29:0/4281529168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe510004d50 0x7fe510005200 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fe52400a750 tx=0x7fe52401f9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47aec283260e92fe server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:11.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.339+0000 7fe52a000640 1 -- 172.21.15.29:0/4281529168 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe52400a370 con 0x7fe510004d50 2024-04-17T20:02:11.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.339+0000 7fe52a000640 1 -- 172.21.15.29:0/4281529168 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5240b1070 con 0x7fe510004d50 2024-04-17T20:02:11.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.339+0000 7fe52a000640 1 -- 172.21.15.29:0/4281529168 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5240ac8b0 con 0x7fe510004d50 2024-04-17T20:02:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.339+0000 7fe531600640 1 -- 172.21.15.29:0/4281529168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe510004d50 msgr2=0x7fe510005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.339+0000 7fe531600640 1 --2- 172.21.15.29:0/4281529168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe510004d50 0x7fe510005200 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fe52400a750 tx=0x7fe52401f9c0 comp rx=0 tx=0).stop 2024-04-17T20:02:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.339+0000 7fe52b400640 1 -- 172.21.15.29:0/4281529168 reap_dead start 2024-04-17T20:02:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.339+0000 7fe531600640 1 -- 172.21.15.29:0/4281529168 shutdown_connections 2024-04-17T20:02:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.339+0000 7fe531600640 1 -- 172.21.15.29:0/4281529168 >> 172.21.15.29:0/4281529168 conn(0x7fe52c06fde0 msgr2=0x7fe52c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.339+0000 7fe531600640 1 -- 172.21.15.29:0/4281529168 shutdown_connections 2024-04-17T20:02:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.339+0000 7fe531600640 1 -- 172.21.15.29:0/4281529168 wait complete. 2024-04-17T20:02:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe531600640 1 Processor -- start 2024-04-17T20:02:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe531600640 1 -- start start 2024-04-17T20:02:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe531600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe52c15d1a0 0x7fe52c1601c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe531600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe52c15b070 0x7fe52c1f2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:11.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe531600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe52c15b480 0x7fe52c14bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:11.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe531600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe52c06c5d0 con 0x7fe52c15d1a0 2024-04-17T20:02:11.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe531600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe52c0cab20 con 0x7fe52c15b070 2024-04-17T20:02:11.350 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe531600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe52c0c8360 con 0x7fe52c15b480 2024-04-17T20:02:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe52aa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe52c15d1a0 0x7fe52c1601c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe52aa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe52c15d1a0 0x7fe52c1601c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41428/0 (socket says 172.21.15.29:41428) 2024-04-17T20:02:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe52aa00640 1 -- 172.21.15.29:0/2592563774 learned_addr learned my addr 172.21.15.29:0/2592563774 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe523e00640 1 --2- 172.21.15.29:0/2592563774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe52c15b070 0x7fe52c1f2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe52b400640 1 --2- 172.21.15.29:0/2592563774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe52c15b480 0x7fe52c14bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe52aa00640 1 -- 172.21.15.29:0/2592563774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe52c15b480 msgr2=0x7fe52c14bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe52aa00640 1 --2- 172.21.15.29:0/2592563774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe52c15b480 0x7fe52c14bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe52aa00640 1 -- 172.21.15.29:0/2592563774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe52c15b070 msgr2=0x7fe52c1f2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe52aa00640 1 --2- 172.21.15.29:0/2592563774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe52c15b070 0x7fe52c1f2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe52aa00640 1 -- 172.21.15.29:0/2592563774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe52c1586c0 con 0x7fe52c15d1a0 2024-04-17T20:02:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.343+0000 7fe523e00640 1 --2- 172.21.15.29:0/2592563774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe52c15b070 0x7fe52c1f2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:02:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.347+0000 7fe52aa00640 1 --2- 172.21.15.29:0/2592563774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe52c15d1a0 0x7fe52c1601c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0119d0 tx=0x7fe51c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.347+0000 7fe522a00640 1 -- 172.21.15.29:0/2592563774 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c00fb50 con 0x7fe52c15d1a0 2024-04-17T20:02:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.347+0000 7fe522a00640 1 -- 172.21.15.29:0/2592563774 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe51c00fcf0 con 0x7fe52c15d1a0 2024-04-17T20:02:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.347+0000 7fe531600640 1 -- 172.21.15.29:0/2592563774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe52c1589a0 con 0x7fe52c15d1a0 2024-04-17T20:02:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.347+0000 7fe522a00640 1 -- 172.21.15.29:0/2592563774 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe51c00fea0 con 0x7fe52c15d1a0 2024-04-17T20:02:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.347+0000 7fe531600640 1 -- 172.21.15.29:0/2592563774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe52c158f60 con 0x7fe52c15d1a0 2024-04-17T20:02:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.347+0000 7fe531600640 1 -- 172.21.15.29:0/2592563774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe52c069b90 con 0x7fe52c15d1a0 2024-04-17T20:02:11.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.347+0000 7fe522a00640 1 -- 172.21.15.29:0/2592563774 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe51c012040 con 0x7fe52c15d1a0 2024-04-17T20:02:11.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.347+0000 7fe522a00640 1 --2- 172.21.15.29:0/2592563774 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe508068380 0x7fe50806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:11.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.347+0000 7fe522a00640 1 -- 172.21.15.29:0/2592563774 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7fe51c0877d0 con 0x7fe52c15d1a0 2024-04-17T20:02:11.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.347+0000 7fe523e00640 1 --2- 172.21.15.29:0/2592563774 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe508068380 0x7fe50806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:11.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.351+0000 7fe523e00640 1 --2- 172.21.15.29:0/2592563774 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe508068380 0x7fe50806a890 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe52c155cf0 tx=0x7fe514009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:11.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.351+0000 7fe522a00640 1 -- 172.21.15.29:0/2592563774 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe51c019030 con 0x7fe52c15d1a0 2024-04-17T20:02:11.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.555+0000 7fe531600640 1 -- 172.21.15.29:0/2592563774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe52c153710 con 0x7fe52c15d1a0 2024-04-17T20:02:11.554 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.555+0000 7fe522a00640 1 -- 172.21.15.29:0/2592563774 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+6085 (secure 0 0 0) 0x7fe51c057100 con 0x7fe52c15d1a0 2024-04-17T20:02:11.555 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:02:11.555 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:09.124906+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:resolve","state_seq":19,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775,5631,5350,5212]},"id":1}]} 2024-04-17T20:02:11.555 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 58 2024-04-17T20:02:11.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.555+0000 7fe531600640 1 -- 172.21.15.29:0/2592563774 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe508068380 msgr2=0x7fe50806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:11.556 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.555+0000 7fe531600640 1 --2- 172.21.15.29:0/2592563774 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe508068380 0x7fe50806a890 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe52c155cf0 tx=0x7fe514009040 comp rx=0 tx=0).stop 2024-04-17T20:02:11.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.555+0000 7fe531600640 1 -- 172.21.15.29:0/2592563774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe52c15d1a0 msgr2=0x7fe52c1601c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:11.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.555+0000 7fe531600640 1 --2- 172.21.15.29:0/2592563774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe52c15d1a0 0x7fe52c1601c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0119d0 tx=0x7fe51c011e90 comp rx=0 tx=0).stop 2024-04-17T20:02:11.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.555+0000 7fe531600640 1 -- 172.21.15.29:0/2592563774 shutdown_connections 2024-04-17T20:02:11.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.555+0000 7fe531600640 1 --2- 172.21.15.29:0/2592563774 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe508068380 0x7fe50806a890 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:11.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.555+0000 7fe531600640 1 --2- 172.21.15.29:0/2592563774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe52c15b480 0x7fe52c14bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:11.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.555+0000 7fe531600640 1 --2- 172.21.15.29:0/2592563774 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe52c15b070 0x7fe52c1f2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:11.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.555+0000 7fe531600640 1 --2- 172.21.15.29:0/2592563774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe52c15d1a0 0x7fe52c1601c0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:11.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.555+0000 7fe531600640 1 -- 172.21.15.29:0/2592563774 >> 172.21.15.29:0/2592563774 conn(0x7fe52c06fde0 msgr2=0x7fe52c0cdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:11.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.559+0000 7fe531600640 1 -- 172.21.15.29:0/2592563774 shutdown_connections 2024-04-17T20:02:11.557 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:11.559+0000 7fe531600640 1 -- 172.21.15.29:0/2592563774 wait complete. 2024-04-17T20:02:11.576 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:02:13.578 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:02:13.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.747+0000 7f45b2000640 1 Processor -- start 2024-04-17T20:02:13.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.747+0000 7f45b2000640 1 -- start start 2024-04-17T20:02:13.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.747+0000 7f45b2000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f45ac153c00 0x7f45ac072200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:13.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.747+0000 7f45b2000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45ac06d620 0x7f45ac1644a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:13.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.747+0000 7f45b2000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45ac154010 con 0x7f45ac06d210 2024-04-17T20:02:13.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.747+0000 7f45b2000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f45ac159c30 con 0x7f45ac153c00 2024-04-17T20:02:13.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.747+0000 7f45b2000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f45ac071000 con 0x7f45ac06d620 2024-04-17T20:02:13.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.747+0000 7f45b0c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f45ac153c00 0x7f45ac072200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:13.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.747+0000 7f45b0c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f45ac153c00 0x7f45ac072200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41434/0 (socket says 172.21.15.29:41434) 2024-04-17T20:02:13.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.747+0000 7f45b0c00640 1 -- 172.21.15.29:0/2941760871 learned_addr learned my addr 172.21.15.29:0/2941760871 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:13.747 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.747+0000 7f45b1600640 1 --2- 172.21.15.29:0/2941760871 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45ac06d620 0x7f45ac1644a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:13.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.747+0000 7f45b0c00640 1 -- 172.21.15.29:0/2941760871 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45ac06d620 msgr2=0x7f45ac1644a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:13.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.747+0000 7f45b0c00640 1 --2- 172.21.15.29:0/2941760871 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45ac06d620 0x7f45ac1644a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:13.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.747+0000 7f45b0c00640 1 -- 172.21.15.29:0/2941760871 >> v1:172.21.15.37:6789/0 conn(0x7f45ac06d210 legacy=0x7f45ac0c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:13.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.747+0000 7f45b0c00640 1 -- 172.21.15.29:0/2941760871 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ac072b20 con 0x7f45ac153c00 2024-04-17T20:02:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.751+0000 7f45b0c00640 1 --2- 172.21.15.29:0/2941760871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45ac153c00 0x7f45ac072200 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f45a0009870 tx=0x7f45a001c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149f5d2982bd814d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.751+0000 7f45ab400640 1 -- 172.21.15.29:0/2941760871 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a00a95f0 con 0x7f45ac153c00 2024-04-17T20:02:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.751+0000 7f45ab400640 1 -- 172.21.15.29:0/2941760871 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45a00a9790 con 0x7f45ac153c00 2024-04-17T20:02:13.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.751+0000 7f45ab400640 1 -- 172.21.15.29:0/2941760871 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a00a9b60 con 0x7f45ac153c00 2024-04-17T20:02:13.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.751+0000 7f45b2000640 1 -- 172.21.15.29:0/2941760871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45ac153c00 msgr2=0x7f45ac072200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:13.751 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.751+0000 7f45b2000640 1 --2- 172.21.15.29:0/2941760871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45ac153c00 0x7f45ac072200 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f45a0009870 tx=0x7f45a001c530 comp rx=0 tx=0).stop 2024-04-17T20:02:13.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.751+0000 7f45b2000640 1 -- 172.21.15.29:0/2941760871 shutdown_connections 2024-04-17T20:02:13.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.751+0000 7f45b2000640 1 --2- 172.21.15.29:0/2941760871 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45ac06d620 0x7f45ac1644a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:13.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.751+0000 7f45b2000640 1 --2- 172.21.15.29:0/2941760871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45ac153c00 0x7f45ac072200 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:13.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.751+0000 7f45b2000640 1 -- 172.21.15.29:0/2941760871 >> 172.21.15.29:0/2941760871 conn(0x7f45ac0ccf80 msgr2=0x7f45ac0cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:13.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.751+0000 7f45b2000640 1 -- 172.21.15.29:0/2941760871 shutdown_connections 2024-04-17T20:02:13.752 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.751+0000 7f45b2000640 1 -- 172.21.15.29:0/2941760871 wait complete. 2024-04-17T20:02:13.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b2000640 1 Processor -- start 2024-04-17T20:02:13.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b2000640 1 -- start start 2024-04-17T20:02:13.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b2000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45ac06d210 0x7f45ac1f96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b2000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45ac06d620 0x7f45ac14e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b2000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45ac153c00 0x7f45ac0cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b2000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac072f80 con 0x7f45ac06d210 2024-04-17T20:02:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b2000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f45ac072e00 con 0x7f45ac153c00 2024-04-17T20:02:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b2000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f45ac15a9b0 con 0x7f45ac06d620 2024-04-17T20:02:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b0c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45ac06d210 0x7f45ac1f96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b0c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45ac06d210 0x7f45ac1f96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41446/0 (socket says 172.21.15.29:41446) 2024-04-17T20:02:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b0c00640 1 -- 172.21.15.29:0/3094605620 learned_addr learned my addr 172.21.15.29:0/3094605620 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b1600640 1 --2- 172.21.15.29:0/3094605620 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45ac153c00 0x7f45ac0cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45abe00640 1 --2- 172.21.15.29:0/3094605620 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45ac06d620 0x7f45ac14e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b0c00640 1 -- 172.21.15.29:0/3094605620 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45ac06d620 msgr2=0x7f45ac14e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b0c00640 1 --2- 172.21.15.29:0/3094605620 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45ac06d620 0x7f45ac14e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b0c00640 1 -- 172.21.15.29:0/3094605620 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45ac153c00 msgr2=0x7f45ac0cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b0c00640 1 --2- 172.21.15.29:0/3094605620 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45ac153c00 0x7f45ac0cc230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b0c00640 1 -- 172.21.15.29:0/3094605620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45ac153580 con 0x7f45ac06d210 2024-04-17T20:02:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b0c00640 1 --2- 172.21.15.29:0/3094605620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45ac06d210 0x7f45ac1f96b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f45a001abe0 tx=0x7f45a00192d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45a8c00640 1 -- 172.21.15.29:0/3094605620 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a00b1800 con 0x7f45ac06d210 2024-04-17T20:02:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45a8c00640 1 -- 172.21.15.29:0/3094605620 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45a00b19a0 con 0x7f45ac06d210 2024-04-17T20:02:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45a8c00640 1 -- 172.21.15.29:0/3094605620 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a00b1d70 con 0x7f45ac06d210 2024-04-17T20:02:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b2000640 1 -- 172.21.15.29:0/3094605620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45ac1f9bf0 con 0x7f45ac06d210 2024-04-17T20:02:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.755+0000 7f45b2000640 1 -- 172.21.15.29:0/3094605620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45ac0c3280 con 0x7f45ac06d210 2024-04-17T20:02:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.759+0000 7f45b2000640 1 -- 172.21.15.29:0/3094605620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45ac04ff10 con 0x7f45ac06d210 2024-04-17T20:02:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.759+0000 7f45a8c00640 1 -- 172.21.15.29:0/3094605620 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f45a00b83d0 con 0x7f45ac06d210 2024-04-17T20:02:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.759+0000 7f45a8c00640 1 --2- 172.21.15.29:0/3094605620 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f458c0683b0 0x7f458c06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.759+0000 7f45a8c00640 1 -- 172.21.15.29:0/3094605620 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7f45a0120ad0 con 0x7f45ac06d210 2024-04-17T20:02:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.759+0000 7f45abe00640 1 --2- 172.21.15.29:0/3094605620 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f458c0683b0 0x7f458c06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.763+0000 7f45abe00640 1 --2- 172.21.15.29:0/3094605620 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f458c0683b0 0x7f458c06a8c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f45ac14dbc0 tx=0x7f4598009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:13.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.763+0000 7f45a8c00640 1 -- 172.21.15.29:0/3094605620 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f45a00ad020 con 0x7f45ac06d210 2024-04-17T20:02:13.967 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.967+0000 7f45b2000640 1 -- 172.21.15.29:0/3094605620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45ac1f5b30 con 0x7f45ac06d210 2024-04-17T20:02:13.968 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.967+0000 7f45a8c00640 1 -- 172.21.15.29:0/3094605620 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+6085 (secure 0 0 0) 0x7f45a0016150 con 0x7f45ac06d210 2024-04-17T20:02:13.968 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:02:13.968 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:09.124906+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:resolve","state_seq":19,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775,5631,5350,5212]},"id":1}]} 2024-04-17T20:02:13.968 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 58 2024-04-17T20:02:13.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.971+0000 7f45b2000640 1 -- 172.21.15.29:0/3094605620 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f458c0683b0 msgr2=0x7f458c06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:13.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.971+0000 7f45b2000640 1 --2- 172.21.15.29:0/3094605620 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f458c0683b0 0x7f458c06a8c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f45ac14dbc0 tx=0x7f4598009040 comp rx=0 tx=0).stop 2024-04-17T20:02:13.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.971+0000 7f45b2000640 1 -- 172.21.15.29:0/3094605620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45ac06d210 msgr2=0x7f45ac1f96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:13.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.971+0000 7f45b2000640 1 --2- 172.21.15.29:0/3094605620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45ac06d210 0x7f45ac1f96b0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f45a001abe0 tx=0x7f45a00192d0 comp rx=0 tx=0).stop 2024-04-17T20:02:13.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.971+0000 7f45b2000640 1 -- 172.21.15.29:0/3094605620 shutdown_connections 2024-04-17T20:02:13.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.971+0000 7f45b2000640 1 --2- 172.21.15.29:0/3094605620 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f458c0683b0 0x7f458c06a8c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:13.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.971+0000 7f45b2000640 1 --2- 172.21.15.29:0/3094605620 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45ac153c00 0x7f45ac0cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:13.970 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.971+0000 7f45b2000640 1 --2- 172.21.15.29:0/3094605620 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45ac06d620 0x7f45ac14e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:13.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.971+0000 7f45b2000640 1 --2- 172.21.15.29:0/3094605620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45ac06d210 0x7f45ac1f96b0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:13.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.971+0000 7f45b2000640 1 -- 172.21.15.29:0/3094605620 >> 172.21.15.29:0/3094605620 conn(0x7f45ac0ccf80 msgr2=0x7f45ac072200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:13.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.971+0000 7f45b2000640 1 -- 172.21.15.29:0/3094605620 shutdown_connections 2024-04-17T20:02:13.971 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:13.971+0000 7f45b2000640 1 -- 172.21.15.29:0/3094605620 wait complete. 2024-04-17T20:02:13.990 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:02:15.991 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:02:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.155+0000 7faf56000640 1 Processor -- start 2024-04-17T20:02:16.156 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.155+0000 7faf56000640 1 -- start start 2024-04-17T20:02:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.159+0000 7faf56000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7faf500ca2e0 0x7faf500ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.159+0000 7faf56000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faf500c6f70 0x7faf500c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.159+0000 7faf56000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf5015a0f0 con 0x7faf5006c410 2024-04-17T20:02:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.159+0000 7faf56000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7faf5015ac70 con 0x7faf500c6f70 2024-04-17T20:02:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.159+0000 7faf56000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7faf500c4520 con 0x7faf500ca2e0 2024-04-17T20:02:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.159+0000 7faf47e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faf500c6f70 0x7faf500c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.159+0000 7faf47e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faf500c6f70 0x7faf500c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41460/0 (socket says 172.21.15.29:41460) 2024-04-17T20:02:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.159+0000 7faf47e00640 1 -- 172.21.15.29:0/1011311234 learned_addr learned my addr 172.21.15.29:0/1011311234 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.159+0000 7faf4f400640 1 --2- 172.21.15.29:0/1011311234 >> v2:172.21.15.37:3300/0 conn(0x7faf500ca2e0 0x7faf500ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:16.163 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.159+0000 7faf47e00640 1 -- 172.21.15.29:0/1011311234 >> v2:172.21.15.37:3300/0 conn(0x7faf500ca2e0 msgr2=0x7faf500ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.159+0000 7faf47e00640 1 --2- 172.21.15.29:0/1011311234 >> v2:172.21.15.37:3300/0 conn(0x7faf500ca2e0 0x7faf500ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.159+0000 7faf47e00640 1 -- 172.21.15.29:0/1011311234 >> v1:172.21.15.37:6789/0 conn(0x7faf5006c410 legacy=0x7faf5006c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.159+0000 7faf47e00640 1 -- 172.21.15.29:0/1011311234 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf5015a3d0 con 0x7faf500c6f70 2024-04-17T20:02:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.163+0000 7faf47e00640 1 --2- 172.21.15.29:0/1011311234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf500c6f70 0x7faf500c7380 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7faf500713c0 tx=0x7faf3801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418732a41fffd938 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.163+0000 7faf4ea00640 1 -- 172.21.15.29:0/1011311234 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf380b0070 con 0x7faf500c6f70 2024-04-17T20:02:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.163+0000 7faf4ea00640 1 -- 172.21.15.29:0/1011311234 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf380ab600 con 0x7faf500c6f70 2024-04-17T20:02:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.163+0000 7faf4ea00640 1 -- 172.21.15.29:0/1011311234 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf380ab9d0 con 0x7faf500c6f70 2024-04-17T20:02:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.163+0000 7faf56000640 1 -- 172.21.15.29:0/1011311234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf500c6f70 msgr2=0x7faf500c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.163+0000 7faf56000640 1 --2- 172.21.15.29:0/1011311234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf500c6f70 0x7faf500c7380 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7faf500713c0 tx=0x7faf3801c490 comp rx=0 tx=0).stop 2024-04-17T20:02:16.164 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.163+0000 7faf56000640 1 -- 172.21.15.29:0/1011311234 shutdown_connections 2024-04-17T20:02:16.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.163+0000 7faf56000640 1 --2- 172.21.15.29:0/1011311234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf500c6f70 0x7faf500c7380 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:16.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.163+0000 7faf56000640 1 --2- 172.21.15.29:0/1011311234 >> v2:172.21.15.37:3300/0 conn(0x7faf500ca2e0 0x7faf500ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:16.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.163+0000 7faf56000640 1 -- 172.21.15.29:0/1011311234 >> 172.21.15.29:0/1011311234 conn(0x7faf5006fde0 msgr2=0x7faf500701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:16.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.163+0000 7faf56000640 1 -- 172.21.15.29:0/1011311234 shutdown_connections 2024-04-17T20:02:16.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.163+0000 7faf56000640 1 -- 172.21.15.29:0/1011311234 wait complete. 2024-04-17T20:02:16.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf56000640 1 Processor -- start 2024-04-17T20:02:16.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf56000640 1 -- start start 2024-04-17T20:02:16.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf56000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faf5006c410 0x7faf501f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:16.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf56000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf500c6f70 0x7faf501f6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:16.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf56000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faf500ca2e0 0x7faf50158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:16.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf56000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faf50159de0 con 0x7faf500c6f70 2024-04-17T20:02:16.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf56000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7faf50159c60 con 0x7faf500ca2e0 2024-04-17T20:02:16.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf56000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7faf50166e00 con 0x7faf5006c410 2024-04-17T20:02:16.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf47e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf500c6f70 0x7faf501f6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:16.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf47e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf500c6f70 0x7faf501f6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41466/0 (socket says 172.21.15.29:41466) 2024-04-17T20:02:16.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf47e00640 1 -- 172.21.15.29:0/60129609 learned_addr learned my addr 172.21.15.29:0/60129609 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:16.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf4f400640 1 --2- 172.21.15.29:0/60129609 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faf5006c410 0x7faf501f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:16.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf4fe00640 1 --2- 172.21.15.29:0/60129609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faf500ca2e0 0x7faf50158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:16.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf47e00640 1 -- 172.21.15.29:0/60129609 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faf5006c410 msgr2=0x7faf501f3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:16.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf47e00640 1 --2- 172.21.15.29:0/60129609 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faf5006c410 0x7faf501f3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:16.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf47e00640 1 -- 172.21.15.29:0/60129609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faf500ca2e0 msgr2=0x7faf50158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:16.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf47e00640 1 --2- 172.21.15.29:0/60129609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faf500ca2e0 0x7faf50158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:16.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf47e00640 1 -- 172.21.15.29:0/60129609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf501f3120 con 0x7faf500c6f70 2024-04-17T20:02:16.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf4f400640 1 --2- 172.21.15.29:0/60129609 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faf5006c410 0x7faf501f3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:02:16.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf47e00640 1 --2- 172.21.15.29:0/60129609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf500c6f70 0x7faf501f6f90 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7faf380ae040 tx=0x7faf38019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:16.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf47400640 1 -- 172.21.15.29:0/60129609 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf380b0040 con 0x7faf500c6f70 2024-04-17T20:02:16.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf47400640 1 -- 172.21.15.29:0/60129609 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf380b4640 con 0x7faf500c6f70 2024-04-17T20:02:16.173 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf47400640 1 -- 172.21.15.29:0/60129609 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf380b4a30 con 0x7faf500c6f70 2024-04-17T20:02:16.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf56000640 1 -- 172.21.15.29:0/60129609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf5014eff0 con 0x7faf500c6f70 2024-04-17T20:02:16.174 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.167+0000 7faf56000640 1 -- 172.21.15.29:0/60129609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf50149c90 con 0x7faf500c6f70 2024-04-17T20:02:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.171+0000 7faf56000640 1 -- 172.21.15.29:0/60129609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf5014ed60 con 0x7faf500c6f70 2024-04-17T20:02:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.171+0000 7faf47400640 1 -- 172.21.15.29:0/60129609 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7faf380ab4c0 con 0x7faf500c6f70 2024-04-17T20:02:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.171+0000 7faf47400640 1 --2- 172.21.15.29:0/60129609 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faf2c0683b0 0x7faf2c06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.171+0000 7faf47400640 1 -- 172.21.15.29:0/60129609 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7faf38120bc0 con 0x7faf500c6f70 2024-04-17T20:02:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.171+0000 7faf4f400640 1 --2- 172.21.15.29:0/60129609 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faf2c0683b0 0x7faf2c06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.171+0000 7faf4f400640 1 --2- 172.21.15.29:0/60129609 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faf2c0683b0 0x7faf2c06a8c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7faf40009b10 tx=0x7faf40008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:16.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.175+0000 7faf47400640 1 -- 172.21.15.29:0/60129609 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7faf38016110 con 0x7faf500c6f70 2024-04-17T20:02:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.383+0000 7faf56000640 1 -- 172.21.15.29:0/60129609 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf50069b90 con 0x7faf500c6f70 2024-04-17T20:02:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.383+0000 7faf47400640 1 -- 172.21.15.29:0/60129609 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+6087 (secure 0 0 0) 0x7faf380f0bb0 con 0x7faf500c6f70 2024-04-17T20:02:16.382 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:02:16.382 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:14.179475+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5212,5350,5631,5775]},"id":1}]} 2024-04-17T20:02:16.382 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 59 2024-04-17T20:02:16.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.383+0000 7faf56000640 1 -- 172.21.15.29:0/60129609 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faf2c0683b0 msgr2=0x7faf2c06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:16.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.383+0000 7faf56000640 1 --2- 172.21.15.29:0/60129609 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faf2c0683b0 0x7faf2c06a8c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7faf40009b10 tx=0x7faf40008040 comp rx=0 tx=0).stop 2024-04-17T20:02:16.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.383+0000 7faf56000640 1 -- 172.21.15.29:0/60129609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf500c6f70 msgr2=0x7faf501f6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:16.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.383+0000 7faf56000640 1 --2- 172.21.15.29:0/60129609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf500c6f70 0x7faf501f6f90 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7faf380ae040 tx=0x7faf38019230 comp rx=0 tx=0).stop 2024-04-17T20:02:16.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.383+0000 7faf56000640 1 -- 172.21.15.29:0/60129609 shutdown_connections 2024-04-17T20:02:16.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.383+0000 7faf56000640 1 --2- 172.21.15.29:0/60129609 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faf2c0683b0 0x7faf2c06a8c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:16.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.383+0000 7faf56000640 1 --2- 172.21.15.29:0/60129609 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faf500ca2e0 0x7faf50158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:16.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.383+0000 7faf56000640 1 --2- 172.21.15.29:0/60129609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf500c6f70 0x7faf501f6f90 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:16.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.383+0000 7faf56000640 1 --2- 172.21.15.29:0/60129609 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faf5006c410 0x7faf501f3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:16.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.383+0000 7faf56000640 1 -- 172.21.15.29:0/60129609 >> 172.21.15.29:0/60129609 conn(0x7faf5006fde0 msgr2=0x7faf50073880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:16.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.383+0000 7faf56000640 1 -- 172.21.15.29:0/60129609 shutdown_connections 2024-04-17T20:02:16.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:16.383+0000 7faf56000640 1 -- 172.21.15.29:0/60129609 wait complete. 2024-04-17T20:02:16.404 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:02:18.405 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:02:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.571+0000 7f60d6e00640 1 Processor -- start 2024-04-17T20:02:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.571+0000 7f60d6e00640 1 -- start start 2024-04-17T20:02:18.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.571+0000 7f60d6e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f60d00c71b0 0x7f60d015c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:18.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.571+0000 7f60d6e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f60d01523d0 0x7f60d01527e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:18.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.571+0000 7f60d6e00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60d00c60e0 con 0x7f60d00c75c0 2024-04-17T20:02:18.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.571+0000 7f60d6e00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f60d015be90 con 0x7f60d01523d0 2024-04-17T20:02:18.572 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.571+0000 7f60d6e00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f60d006c6a0 con 0x7f60d00c71b0 2024-04-17T20:02:18.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.571+0000 7f60d4e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f60d01523d0 0x7f60d01527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:18.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.571+0000 7f60d4e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f60d01523d0 0x7f60d01527e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41468/0 (socket says 172.21.15.29:41468) 2024-04-17T20:02:18.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.571+0000 7f60d4e00640 1 -- 172.21.15.29:0/2861131621 learned_addr learned my addr 172.21.15.29:0/2861131621 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:18.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.571+0000 7f60cfe00640 1 --2- 172.21.15.29:0/2861131621 >> v2:172.21.15.37:3300/0 conn(0x7f60d00c71b0 0x7f60d015c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:18.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.575+0000 7f60d4e00640 1 -- 172.21.15.29:0/2861131621 >> v2:172.21.15.37:3300/0 conn(0x7f60d00c71b0 msgr2=0x7f60d015c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:18.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.575+0000 7f60d4e00640 1 --2- 172.21.15.29:0/2861131621 >> v2:172.21.15.37:3300/0 conn(0x7f60d00c71b0 0x7f60d015c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:18.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.575+0000 7f60d4e00640 1 -- 172.21.15.29:0/2861131621 >> v1:172.21.15.37:6789/0 conn(0x7f60d00c75c0 legacy=0x7f60d006bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:18.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.575+0000 7f60d4e00640 1 -- 172.21.15.29:0/2861131621 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d00c6350 con 0x7f60d01523d0 2024-04-17T20:02:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.575+0000 7f60d4e00640 1 --2- 172.21.15.29:0/2861131621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60d01523d0 0x7f60d01527e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f60d0071f40 tx=0x7f60c80213e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff54a3431e06940e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.575+0000 7f60cea00640 1 -- 172.21.15.29:0/2861131621 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c80b8070 con 0x7f60d01523d0 2024-04-17T20:02:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.575+0000 7f60cea00640 1 -- 172.21.15.29:0/2861131621 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60c80b3410 con 0x7f60d01523d0 2024-04-17T20:02:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.575+0000 7f60cea00640 1 -- 172.21.15.29:0/2861131621 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c80b37e0 con 0x7f60d01523d0 2024-04-17T20:02:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.579+0000 7f60d6e00640 1 -- 172.21.15.29:0/2861131621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60d01523d0 msgr2=0x7f60d01527e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.579+0000 7f60d6e00640 1 --2- 172.21.15.29:0/2861131621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60d01523d0 0x7f60d01527e0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f60d0071f40 tx=0x7f60c80213e0 comp rx=0 tx=0).stop 2024-04-17T20:02:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.579+0000 7f60d6e00640 1 -- 172.21.15.29:0/2861131621 shutdown_connections 2024-04-17T20:02:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.579+0000 7f60d6e00640 1 --2- 172.21.15.29:0/2861131621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60d01523d0 0x7f60d01527e0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:18.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.579+0000 7f60d6e00640 1 --2- 172.21.15.29:0/2861131621 >> v2:172.21.15.37:3300/0 conn(0x7f60d00c71b0 0x7f60d015c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:18.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.579+0000 7f60d6e00640 1 -- 172.21.15.29:0/2861131621 >> 172.21.15.29:0/2861131621 conn(0x7f60d006fde0 msgr2=0x7f60d00701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:18.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.579+0000 7f60d6e00640 1 -- 172.21.15.29:0/2861131621 shutdown_connections 2024-04-17T20:02:18.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.579+0000 7f60d6e00640 1 -- 172.21.15.29:0/2861131621 wait complete. 2024-04-17T20:02:18.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.579+0000 7f60d6e00640 1 Processor -- start 2024-04-17T20:02:18.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.579+0000 7f60d6e00640 1 -- start start 2024-04-17T20:02:18.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60d6e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60d00c71b0 0x7f60d01f1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60d6e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d00c75c0 0x7f60d015b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60d6e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60d01523d0 0x7f60d0154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60d6e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f60d00cb140 con 0x7f60d00c71b0 2024-04-17T20:02:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60d6e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f60d015bb20 con 0x7f60d01523d0 2024-04-17T20:02:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60d6e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f60d00c6910 con 0x7f60d00c75c0 2024-04-17T20:02:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60cfe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60d00c71b0 0x7f60d01f1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60cfe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60d00c71b0 0x7f60d01f1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41470/0 (socket says 172.21.15.29:41470) 2024-04-17T20:02:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60cfe00640 1 -- 172.21.15.29:0/1754262327 learned_addr learned my addr 172.21.15.29:0/1754262327 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60d4e00640 1 --2- 172.21.15.29:0/1754262327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60d01523d0 0x7f60d0154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60cf400640 1 --2- 172.21.15.29:0/1754262327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d00c75c0 0x7f60d015b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60cfe00640 1 -- 172.21.15.29:0/1754262327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d00c75c0 msgr2=0x7f60d015b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:18.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60cfe00640 1 --2- 172.21.15.29:0/1754262327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d00c75c0 0x7f60d015b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:18.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60cfe00640 1 -- 172.21.15.29:0/1754262327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60d01523d0 msgr2=0x7f60d0154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:18.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60cfe00640 1 --2- 172.21.15.29:0/1754262327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60d01523d0 0x7f60d0154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:18.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60cfe00640 1 -- 172.21.15.29:0/1754262327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d01f24a0 con 0x7f60d00c71b0 2024-04-17T20:02:18.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60cfe00640 1 --2- 172.21.15.29:0/1754262327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60d00c71b0 0x7f60d01f1f60 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f60b800e8a0 tx=0x7f60b800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:18.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60c7e00640 1 -- 172.21.15.29:0/1754262327 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b8017070 con 0x7f60d00c71b0 2024-04-17T20:02:18.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60c7e00640 1 -- 172.21.15.29:0/1754262327 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60b800c8e0 con 0x7f60d00c71b0 2024-04-17T20:02:18.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60c7e00640 1 -- 172.21.15.29:0/1754262327 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b800ccb0 con 0x7f60d00c71b0 2024-04-17T20:02:18.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60d6e00640 1 -- 172.21.15.29:0/1754262327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60d014a850 con 0x7f60d00c71b0 2024-04-17T20:02:18.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60d6e00640 1 -- 172.21.15.29:0/1754262327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60d0148e00 con 0x7f60d00c71b0 2024-04-17T20:02:18.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60d6e00640 1 -- 172.21.15.29:0/1754262327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60d0069b90 con 0x7f60d00c71b0 2024-04-17T20:02:18.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.583+0000 7f60c7e00640 1 -- 172.21.15.29:0/1754262327 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f60b801a070 con 0x7f60d00c71b0 2024-04-17T20:02:18.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.587+0000 7f60c7e00640 1 --2- 172.21.15.29:0/1754262327 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f60ac068380 0x7f60ac06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:18.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.587+0000 7f60c7e00640 1 -- 172.21.15.29:0/1754262327 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7f60b80873b0 con 0x7f60d00c71b0 2024-04-17T20:02:18.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.587+0000 7f60cf400640 1 --2- 172.21.15.29:0/1754262327 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f60ac068380 0x7f60ac06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.587+0000 7f60cf400640 1 --2- 172.21.15.29:0/1754262327 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f60ac068380 0x7f60ac06a890 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f60d0075d20 tx=0x7f60c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:18.589 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.591+0000 7f60c7e00640 1 -- 172.21.15.29:0/1754262327 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f60b80571e0 con 0x7f60d00c71b0 2024-04-17T20:02:18.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.795+0000 7f60d6e00640 1 -- 172.21.15.29:0/1754262327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60d00764e0 con 0x7f60d00c71b0 2024-04-17T20:02:18.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.795+0000 7f60c7e00640 1 -- 172.21.15.29:0/1754262327 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) ==== 94+0+6087 (secure 0 0 0) 0x7f60b8056ae0 con 0x7f60d00c71b0 2024-04-17T20:02:18.795 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:02:18.795 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:14.179475+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5212,5350,5631,5775]},"id":1}]} 2024-04-17T20:02:18.796 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 59 2024-04-17T20:02:18.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.799+0000 7f60d6e00640 1 -- 172.21.15.29:0/1754262327 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f60ac068380 msgr2=0x7f60ac06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:18.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.799+0000 7f60d6e00640 1 --2- 172.21.15.29:0/1754262327 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f60ac068380 0x7f60ac06a890 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f60d0075d20 tx=0x7f60c0009040 comp rx=0 tx=0).stop 2024-04-17T20:02:18.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.799+0000 7f60d6e00640 1 -- 172.21.15.29:0/1754262327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60d00c71b0 msgr2=0x7f60d01f1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:18.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.799+0000 7f60d6e00640 1 --2- 172.21.15.29:0/1754262327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60d00c71b0 0x7f60d01f1f60 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f60b800e8a0 tx=0x7f60b800ed60 comp rx=0 tx=0).stop 2024-04-17T20:02:18.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.799+0000 7f60d6e00640 1 -- 172.21.15.29:0/1754262327 shutdown_connections 2024-04-17T20:02:18.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.799+0000 7f60d6e00640 1 --2- 172.21.15.29:0/1754262327 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f60ac068380 0x7f60ac06a890 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:18.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.799+0000 7f60d6e00640 1 --2- 172.21.15.29:0/1754262327 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f60d01523d0 0x7f60d0154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:18.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.799+0000 7f60d6e00640 1 --2- 172.21.15.29:0/1754262327 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f60d00c75c0 0x7f60d015b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:18.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.799+0000 7f60d6e00640 1 --2- 172.21.15.29:0/1754262327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60d00c71b0 0x7f60d01f1f60 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:18.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.799+0000 7f60d6e00640 1 -- 172.21.15.29:0/1754262327 >> 172.21.15.29:0/1754262327 conn(0x7f60d006fde0 msgr2=0x7f60d00c8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:18.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.799+0000 7f60d6e00640 1 -- 172.21.15.29:0/1754262327 shutdown_connections 2024-04-17T20:02:18.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:18.799+0000 7f60d6e00640 1 -- 172.21.15.29:0/1754262327 wait complete. 2024-04-17T20:02:18.828 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 59, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-17T19:47:49.315515+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 59, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5212": { "addr": "172.21.15.37:6838/767681664", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6835", "nonce": 767681664, "type": "v2" }, { "addr": "172.21.15.37:6838", "nonce": 767681664, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5212, "incarnation": 26, "join_fscid": -1, "name": "b", "rank": 4, "state": "up:active", "state_seq": 31 }, "gid_5350": { "addr": "172.21.15.37:6837/2495953425", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6834", "nonce": 2495953425, "type": "v2" }, { "addr": "172.21.15.37:6837", "nonce": 2495953425, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5350, "incarnation": 32, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 18 }, "gid_5631": { "addr": "172.21.15.29:6836/2376553580", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6832", "nonce": 2376553580, "type": "v2" }, { "addr": "172.21.15.29:6836", "nonce": 2376553580, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5631, "incarnation": 44, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 42 }, "gid_5775": { "addr": "172.21.15.29:6835/1352350250", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6834", "nonce": 1352350250, "type": "v2" }, { "addr": "172.21.15.29:6835", "nonce": 1352350250, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5775, "incarnation": 50, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 39 }, "gid_5946": { "addr": "172.21.15.29:6837/3087508289", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6833", "nonce": 3087508289, "type": "v2" }, { "addr": "172.21.15.29:6837", "nonce": 3087508289, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5946, "incarnation": 57, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:reconnect", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 75, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-17T20:02:14.179475+0000", "qdb_cluster": [ 5212, 5350, 5631, 5775 ], "qdb_leader": 5775, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5775, "mds_1": 5631, "mds_2": 5946, "mds_3": 5350, "mds_4": 5212 } } } ], "standbys": [] } 2024-04-17T20:02:20.830 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:02:20.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.991+0000 7f4f4f800640 1 Processor -- start 2024-04-17T20:02:20.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.991+0000 7f4f4f800640 1 -- start start 2024-04-17T20:02:20.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.991+0000 7f4f4f800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f480cc8c0 0x7f4f480cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:20.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.991+0000 7f4f4f800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f480c88b0 con 0x7f4f48159390 2024-04-17T20:02:20.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.991+0000 7f4f4f800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f480c6ce0 con 0x7f4f481597a0 2024-04-17T20:02:20.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.991+0000 7f4f4f800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4f480c6fc0 con 0x7f4f480cc8c0 2024-04-17T20:02:20.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.991+0000 7f4f47e00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4f48159390 0x7f4f480ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59822/0 (socket says 172.21.15.29:59822) 2024-04-17T20:02:20.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.991+0000 7f4f47e00640 1 -- 172.21.15.29:0/1647575958 learned_addr learned my addr 172.21.15.29:0/1647575958 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:02:20.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f4ce00640 1 --2- 172.21.15.29:0/1647575958 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f480cc8c0 0x7f4f480cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:20.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47400640 1 -- 172.21.15.29:0/1647575958 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4181162892 0 0) 0x7f4f480c88b0 con 0x7f4f48159390 2024-04-17T20:02:20.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47400640 1 -- 172.21.15.29:0/1647575958 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f30003530 con 0x7f4f48159390 2024-04-17T20:02:20.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47400640 1 -- 172.21.15.29:0/1647575958 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1591283742 0 0) 0x7f4f480c6ce0 con 0x7f4f481597a0 2024-04-17T20:02:20.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47400640 1 -- 172.21.15.29:0/1647575958 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f480c88b0 con 0x7f4f481597a0 2024-04-17T20:02:20.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47400640 1 -- 172.21.15.29:0/1647575958 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 530658935 0 0) 0x7f4f30003530 con 0x7f4f48159390 2024-04-17T20:02:20.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47400640 1 -- 172.21.15.29:0/1647575958 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f480cc8c0 msgr2=0x7f4f480cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:20.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47400640 1 --2- 172.21.15.29:0/1647575958 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f480cc8c0 0x7f4f480cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:20.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47400640 1 -- 172.21.15.29:0/1647575958 >> v1:172.21.15.37:6789/0 conn(0x7f4f481597a0 legacy=0x7f4f481635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:20.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47400640 1 -- 172.21.15.29:0/1647575958 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f480c72a0 con 0x7f4f48159390 2024-04-17T20:02:20.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47400640 1 -- 172.21.15.29:0/1647575958 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f4f34001680 con 0x7f4f48159390 2024-04-17T20:02:20.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47400640 1 -- 172.21.15.29:0/1647575958 >> v1:172.21.15.29:6789/0 conn(0x7f4f48159390 legacy=0x7f4f480ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:20.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47400640 1 --2- 172.21.15.29:0/1647575958 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f30003bf0 0x7f4f30004040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:20.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47400640 1 --2- 172.21.15.29:0/1647575958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f30004810 0x7f4f30004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:20.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47400640 1 --2- 172.21.15.29:0/1647575958 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f480cc8c0 0x7f4f300052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:20.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47400640 1 -- 172.21.15.29:0/1647575958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4f480c6fc0 con 0x7f4f30004810 2024-04-17T20:02:20.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47400640 1 -- 172.21.15.29:0/1647575958 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3000bbb0 con 0x7f4f30003bf0 2024-04-17T20:02:20.996 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47400640 1 -- 172.21.15.29:0/1647575958 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4f3000bdb0 con 0x7f4f480cc8c0 2024-04-17T20:02:20.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47e00640 1 --2- 172.21.15.29:0/1647575958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f30004810 0x7f4f30004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:20.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f4ce00640 1 --2- 172.21.15.29:0/1647575958 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f30003bf0 0x7f4f30004040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:20.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f4d800640 1 --2- 172.21.15.29:0/1647575958 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f480cc8c0 0x7f4f300052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:20.997 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47e00640 1 -- 172.21.15.29:0/1647575958 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f480cc8c0 msgr2=0x7f4f300052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:21.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47e00640 1 --2- 172.21.15.29:0/1647575958 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f480cc8c0 0x7f4f300052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:21.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47e00640 1 -- 172.21.15.29:0/1647575958 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f30003bf0 msgr2=0x7f4f30004040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:21.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47e00640 1 --2- 172.21.15.29:0/1647575958 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f30003bf0 0x7f4f30004040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:21.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.995+0000 7f4f47e00640 1 -- 172.21.15.29:0/1647575958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f30003530 con 0x7f4f30004810 2024-04-17T20:02:21.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.999+0000 7f4f47e00640 1 --2- 172.21.15.29:0/1647575958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f30004810 0x7f4f30004c20 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4f34007fa0 tx=0x7f4f3401cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c819629ed4d349f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:21.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.999+0000 7f4f47400640 1 -- 172.21.15.29:0/1647575958 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f34007bf0 con 0x7f4f30004810 2024-04-17T20:02:21.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.999+0000 7f4f47400640 1 -- 172.21.15.29:0/1647575958 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f340a9020 con 0x7f4f30004810 2024-04-17T20:02:21.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.999+0000 7f4f47400640 1 -- 172.21.15.29:0/1647575958 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f34005b00 con 0x7f4f30004810 2024-04-17T20:02:21.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.999+0000 7f4f4f800640 1 -- 172.21.15.29:0/1647575958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f30004810 msgr2=0x7f4f30004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:21.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.999+0000 7f4f4f800640 1 --2- 172.21.15.29:0/1647575958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f30004810 0x7f4f30004c20 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4f34007fa0 tx=0x7f4f3401cb00 comp rx=0 tx=0).stop 2024-04-17T20:02:21.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.999+0000 7f4f4d800640 1 -- 172.21.15.29:0/1647575958 reap_dead start 2024-04-17T20:02:21.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.999+0000 7f4f4f800640 1 -- 172.21.15.29:0/1647575958 shutdown_connections 2024-04-17T20:02:21.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.999+0000 7f4f4f800640 1 -- 172.21.15.29:0/1647575958 >> 172.21.15.29:0/1647575958 conn(0x7f4f4806fde0 msgr2=0x7f4f480701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:21.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.999+0000 7f4f4f800640 1 -- 172.21.15.29:0/1647575958 shutdown_connections 2024-04-17T20:02:21.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.999+0000 7f4f4f800640 1 -- 172.21.15.29:0/1647575958 wait complete. 2024-04-17T20:02:21.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.999+0000 7f4f4f800640 1 Processor -- start 2024-04-17T20:02:21.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:20.999+0000 7f4f4f800640 1 -- start start 2024-04-17T20:02:21.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4f800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f30004810 0x7f4f48140ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:21.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4f800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f481f2970 0x7f4f4815caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:21.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4f800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f4815ac10 0x7f4f4813cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:21.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4f800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f4f48159bb0 con 0x7f4f30004810 2024-04-17T20:02:21.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4f800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f4f48158e70 con 0x7f4f481f2970 2024-04-17T20:02:21.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4f800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f4f480caa40 con 0x7f4f4815ac10 2024-04-17T20:02:21.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4ce00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f30004810 0x7f4f48140ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:21.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4ce00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f30004810 0x7f4f48140ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55202/0 (socket says 172.21.15.29:55202) 2024-04-17T20:02:21.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4ce00640 1 -- 172.21.15.29:0/2783446497 learned_addr learned my addr 172.21.15.29:0/2783446497 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:21.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4d800640 1 --2- 172.21.15.29:0/2783446497 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f4815ac10 0x7f4f4813cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:21.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f47e00640 1 --2- 172.21.15.29:0/2783446497 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f481f2970 0x7f4f4815caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:21.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4ce00640 1 -- 172.21.15.29:0/2783446497 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f4815ac10 msgr2=0x7f4f4813cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:21.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4ce00640 1 --2- 172.21.15.29:0/2783446497 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f4815ac10 0x7f4f4813cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:21.020 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4ce00640 1 -- 172.21.15.29:0/2783446497 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f481f2970 msgr2=0x7f4f4815caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:21.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4ce00640 1 --2- 172.21.15.29:0/2783446497 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f481f2970 0x7f4f4815caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:21.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4ce00640 1 -- 172.21.15.29:0/2783446497 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f4815a350 con 0x7f4f30004810 2024-04-17T20:02:21.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f47e00640 1 --2- 172.21.15.29:0/2783446497 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f481f2970 0x7f4f4815caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:02:21.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4ce00640 1 --2- 172.21.15.29:0/2783446497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f30004810 0x7f4f48140ff0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c011a00 tx=0x7f4f3c011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:21.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f44c00640 1 -- 172.21.15.29:0/2783446497 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c00fb50 con 0x7f4f30004810 2024-04-17T20:02:21.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f44c00640 1 -- 172.21.15.29:0/2783446497 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f3c00fcf0 con 0x7f4f30004810 2024-04-17T20:02:21.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f44c00640 1 -- 172.21.15.29:0/2783446497 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c00fea0 con 0x7f4f30004810 2024-04-17T20:02:21.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4f800640 1 -- 172.21.15.29:0/2783446497 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f4815a630 con 0x7f4f30004810 2024-04-17T20:02:21.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4f800640 1 -- 172.21.15.29:0/2783446497 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f481f54f0 con 0x7f4f30004810 2024-04-17T20:02:21.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.003+0000 7f4f4f800640 1 -- 172.21.15.29:0/2783446497 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f481f5720 con 0x7f4f30004810 2024-04-17T20:02:21.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.007+0000 7f4f44c00640 1 -- 172.21.15.29:0/2783446497 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f4f3c012040 con 0x7f4f30004810 2024-04-17T20:02:21.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.007+0000 7f4f44c00640 1 --2- 172.21.15.29:0/2783446497 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4f28068380 0x7f4f2806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:21.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.007+0000 7f4f44c00640 1 -- 172.21.15.29:0/2783446497 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7f4f3c087810 con 0x7f4f30004810 2024-04-17T20:02:21.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.007+0000 7f4f47e00640 1 --2- 172.21.15.29:0/2783446497 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4f28068380 0x7f4f2806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:21.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.007+0000 7f4f47e00640 1 --2- 172.21.15.29:0/2783446497 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4f28068380 0x7f4f2806a890 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4f340aa000 tx=0x7f4f340b0040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:21.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.011+0000 7f4f44c00640 1 -- 172.21.15.29:0/2783446497 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f4f481f5720 con 0x7f4f30004810 2024-04-17T20:02:21.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.211+0000 7f4f4f800640 1 -- 172.21.15.29:0/2783446497 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f48069b90 con 0x7f4f30004810 2024-04-17T20:02:21.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.215+0000 7f4f44c00640 1 -- 172.21.15.29:0/2783446497 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+6084 (secure 0 0 0) 0x7f4f3c019030 con 0x7f4f30004810 2024-04-17T20:02:21.213 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:02:21.213 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:19.188764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775,5631,5350,5212]},"id":1}]} 2024-04-17T20:02:21.213 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 60 2024-04-17T20:02:21.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.215+0000 7f4f4f800640 1 -- 172.21.15.29:0/2783446497 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4f28068380 msgr2=0x7f4f2806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:21.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.215+0000 7f4f4f800640 1 --2- 172.21.15.29:0/2783446497 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4f28068380 0x7f4f2806a890 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4f340aa000 tx=0x7f4f340b0040 comp rx=0 tx=0).stop 2024-04-17T20:02:21.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.215+0000 7f4f4f800640 1 -- 172.21.15.29:0/2783446497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f30004810 msgr2=0x7f4f48140ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:21.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.215+0000 7f4f4f800640 1 --2- 172.21.15.29:0/2783446497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f30004810 0x7f4f48140ff0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c011a00 tx=0x7f4f3c011ec0 comp rx=0 tx=0).stop 2024-04-17T20:02:21.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.215+0000 7f4f4f800640 1 -- 172.21.15.29:0/2783446497 shutdown_connections 2024-04-17T20:02:21.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.215+0000 7f4f4f800640 1 --2- 172.21.15.29:0/2783446497 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f4f28068380 0x7f4f2806a890 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:21.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.215+0000 7f4f4f800640 1 --2- 172.21.15.29:0/2783446497 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f4f4815ac10 0x7f4f4813cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:21.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.215+0000 7f4f4f800640 1 --2- 172.21.15.29:0/2783446497 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f4f481f2970 0x7f4f4815caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:21.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.215+0000 7f4f4f800640 1 --2- 172.21.15.29:0/2783446497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f30004810 0x7f4f48140ff0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:21.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.215+0000 7f4f4f800640 1 -- 172.21.15.29:0/2783446497 >> 172.21.15.29:0/2783446497 conn(0x7f4f4806fde0 msgr2=0x7f4f480ca210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:21.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.215+0000 7f4f4f800640 1 -- 172.21.15.29:0/2783446497 shutdown_connections 2024-04-17T20:02:21.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:21.215+0000 7f4f4f800640 1 -- 172.21.15.29:0/2783446497 wait complete. 2024-04-17T20:02:21.235 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:02:23.237 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:02:23.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.399+0000 7faa47600640 1 Processor -- start 2024-04-17T20:02:23.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.403+0000 7faa47600640 1 -- start start 2024-04-17T20:02:23.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.403+0000 7faa47600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa400cccd0 0x7faa400ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.403+0000 7faa47600640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7faa400ca870 0x7faa401635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.403+0000 7faa47600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa400cd0e0 con 0x7faa400cc8c0 2024-04-17T20:02:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.403+0000 7faa47600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7faa400c88b0 con 0x7faa400cccd0 2024-04-17T20:02:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.403+0000 7faa47600640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7faa400c6ce0 con 0x7faa400ca870 2024-04-17T20:02:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.403+0000 7faa3fe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa400cccd0 0x7faa400ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.403+0000 7faa3fe00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa400cccd0 0x7faa400ca330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55208/0 (socket says 172.21.15.29:55208) 2024-04-17T20:02:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.403+0000 7faa3fe00640 1 -- 172.21.15.29:0/1379489278 learned_addr learned my addr 172.21.15.29:0/1379489278 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.403+0000 7faa45600640 1 --2- 172.21.15.29:0/1379489278 >> v2:172.21.15.37:3300/0 conn(0x7faa400ca870 0x7faa401635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.403+0000 7faa3fe00640 1 -- 172.21.15.29:0/1379489278 >> v2:172.21.15.37:3300/0 conn(0x7faa400ca870 msgr2=0x7faa401635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.403+0000 7faa3fe00640 1 --2- 172.21.15.29:0/1379489278 >> v2:172.21.15.37:3300/0 conn(0x7faa400ca870 0x7faa401635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.403+0000 7faa3fe00640 1 -- 172.21.15.29:0/1379489278 >> v1:172.21.15.37:6789/0 conn(0x7faa400cc8c0 legacy=0x7faa40159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.403+0000 7faa3fe00640 1 -- 172.21.15.29:0/1379489278 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa400c6fc0 con 0x7faa400cccd0 2024-04-17T20:02:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.407+0000 7faa3fe00640 1 --2- 172.21.15.29:0/1379489278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa400cccd0 0x7faa400ca330 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7faa400722d0 tx=0x7faa2c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8708370cbc8da52a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:23.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.407+0000 7faa3f400640 1 -- 172.21.15.29:0/1379489278 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c0b0070 con 0x7faa400cccd0 2024-04-17T20:02:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.407+0000 7faa3f400640 1 -- 172.21.15.29:0/1379489278 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa2c0ab600 con 0x7faa400cccd0 2024-04-17T20:02:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.407+0000 7faa3f400640 1 -- 172.21.15.29:0/1379489278 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c0ab9d0 con 0x7faa400cccd0 2024-04-17T20:02:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.407+0000 7faa47600640 1 -- 172.21.15.29:0/1379489278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa400cccd0 msgr2=0x7faa400ca330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.407+0000 7faa47600640 1 --2- 172.21.15.29:0/1379489278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa400cccd0 0x7faa400ca330 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7faa400722d0 tx=0x7faa2c01c520 comp rx=0 tx=0).stop 2024-04-17T20:02:23.408 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.407+0000 7faa47600640 1 -- 172.21.15.29:0/1379489278 shutdown_connections 2024-04-17T20:02:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.407+0000 7faa47600640 1 --2- 172.21.15.29:0/1379489278 >> v2:172.21.15.37:3300/0 conn(0x7faa400ca870 0x7faa401635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.407+0000 7faa47600640 1 --2- 172.21.15.29:0/1379489278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa400cccd0 0x7faa400ca330 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.407+0000 7faa47600640 1 -- 172.21.15.29:0/1379489278 >> 172.21.15.29:0/1379489278 conn(0x7faa4006fde0 msgr2=0x7faa400701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.407+0000 7faa47600640 1 -- 172.21.15.29:0/1379489278 shutdown_connections 2024-04-17T20:02:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.407+0000 7faa47600640 1 -- 172.21.15.29:0/1379489278 wait complete. 2024-04-17T20:02:23.409 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa47600640 1 Processor -- start 2024-04-17T20:02:23.414 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa47600640 1 -- start start 2024-04-17T20:02:23.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa47600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa400ca870 0x7faa40157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:23.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa47600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa400cc8c0 0x7faa40154f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:23.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa47600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa400cccd0 0x7faa401584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:23.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa47600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faa40159c20 con 0x7faa400cc8c0 2024-04-17T20:02:23.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa47600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7faa40159aa0 con 0x7faa400cccd0 2024-04-17T20:02:23.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa47600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7faa400c6880 con 0x7faa400ca870 2024-04-17T20:02:23.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa3fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa400cc8c0 0x7faa40154f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:23.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa3fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa400cc8c0 0x7faa40154f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55214/0 (socket says 172.21.15.29:55214) 2024-04-17T20:02:23.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa3fe00640 1 -- 172.21.15.29:0/1319056252 learned_addr learned my addr 172.21.15.29:0/1319056252 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:23.418 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa45600640 1 --2- 172.21.15.29:0/1319056252 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa400cccd0 0x7faa401584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa44c00640 1 --2- 172.21.15.29:0/1319056252 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa400ca870 0x7faa40157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa3fe00640 1 -- 172.21.15.29:0/1319056252 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa400ca870 msgr2=0x7faa40157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa3fe00640 1 --2- 172.21.15.29:0/1319056252 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa400ca870 0x7faa40157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa3fe00640 1 -- 172.21.15.29:0/1319056252 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa400cccd0 msgr2=0x7faa401584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa3fe00640 1 --2- 172.21.15.29:0/1319056252 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa400cccd0 0x7faa401584e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa3fe00640 1 -- 172.21.15.29:0/1319056252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa40155490 con 0x7faa400cc8c0 2024-04-17T20:02:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa3fe00640 1 --2- 172.21.15.29:0/1319056252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa400cc8c0 0x7faa40154f50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7faa2c0ae040 tx=0x7faa2c0192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa3cc00640 1 -- 172.21.15.29:0/1319056252 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c0b8020 con 0x7faa400cc8c0 2024-04-17T20:02:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa3cc00640 1 -- 172.21.15.29:0/1319056252 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa2c0b4410 con 0x7faa400cc8c0 2024-04-17T20:02:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa3cc00640 1 -- 172.21.15.29:0/1319056252 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2c0b0040 con 0x7faa400cc8c0 2024-04-17T20:02:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa47600640 1 -- 172.21.15.29:0/1319056252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4015c160 con 0x7faa400cc8c0 2024-04-17T20:02:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.411+0000 7faa47600640 1 -- 172.21.15.29:0/1319056252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4015c6e0 con 0x7faa400cc8c0 2024-04-17T20:02:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.415+0000 7faa47600640 1 -- 172.21.15.29:0/1319056252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa40157ff0 con 0x7faa400cc8c0 2024-04-17T20:02:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.415+0000 7faa3cc00640 1 -- 172.21.15.29:0/1319056252 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7faa2c0bb020 con 0x7faa400cc8c0 2024-04-17T20:02:23.419 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.415+0000 7faa3cc00640 1 --2- 172.21.15.29:0/1319056252 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faa20068380 0x7faa2006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:23.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.415+0000 7faa3cc00640 1 -- 172.21.15.29:0/1319056252 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7faa2c120840 con 0x7faa400cc8c0 2024-04-17T20:02:23.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.415+0000 7faa44c00640 1 --2- 172.21.15.29:0/1319056252 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faa20068380 0x7faa2006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:23.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.419+0000 7faa44c00640 1 --2- 172.21.15.29:0/1319056252 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faa20068380 0x7faa2006a890 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7faa340015b0 tx=0x7faa34009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:23.421 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.419+0000 7faa3cc00640 1 -- 172.21.15.29:0/1319056252 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7faa40157ff0 con 0x7faa400cc8c0 2024-04-17T20:02:23.669 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.671+0000 7faa47600640 1 -- 172.21.15.29:0/1319056252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa40069b90 con 0x7faa400cc8c0 2024-04-17T20:02:23.670 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.671+0000 7faa3cc00640 1 -- 172.21.15.29:0/1319056252 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+6084 (secure 0 0 0) 0x7faa2c0c2070 con 0x7faa400cc8c0 2024-04-17T20:02:23.671 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:02:23.671 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:19.188764+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:rejoin","state_seq":23,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775,5631,5350,5212]},"id":1}]} 2024-04-17T20:02:23.671 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 60 2024-04-17T20:02:23.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.675+0000 7faa47600640 1 -- 172.21.15.29:0/1319056252 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faa20068380 msgr2=0x7faa2006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:23.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.675+0000 7faa47600640 1 --2- 172.21.15.29:0/1319056252 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faa20068380 0x7faa2006a890 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7faa340015b0 tx=0x7faa34009040 comp rx=0 tx=0).stop 2024-04-17T20:02:23.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.675+0000 7faa47600640 1 -- 172.21.15.29:0/1319056252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa400cc8c0 msgr2=0x7faa40154f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:23.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.675+0000 7faa47600640 1 --2- 172.21.15.29:0/1319056252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa400cc8c0 0x7faa40154f50 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7faa2c0ae040 tx=0x7faa2c0192c0 comp rx=0 tx=0).stop 2024-04-17T20:02:23.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.675+0000 7faa47600640 1 -- 172.21.15.29:0/1319056252 shutdown_connections 2024-04-17T20:02:23.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.675+0000 7faa47600640 1 --2- 172.21.15.29:0/1319056252 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faa20068380 0x7faa2006a890 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:23.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.675+0000 7faa47600640 1 --2- 172.21.15.29:0/1319056252 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa400cccd0 0x7faa401584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:23.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.675+0000 7faa47600640 1 --2- 172.21.15.29:0/1319056252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa400cc8c0 0x7faa40154f50 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:23.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.675+0000 7faa47600640 1 --2- 172.21.15.29:0/1319056252 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa400ca870 0x7faa40157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:23.675 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.675+0000 7faa47600640 1 -- 172.21.15.29:0/1319056252 >> 172.21.15.29:0/1319056252 conn(0x7faa4006fde0 msgr2=0x7faa4013ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:23.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.675+0000 7faa47600640 1 -- 172.21.15.29:0/1319056252 shutdown_connections 2024-04-17T20:02:23.676 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:23.675+0000 7faa47600640 1 -- 172.21.15.29:0/1319056252 wait complete. 2024-04-17T20:02:23.707 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:02:25.709 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:02:25.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.879+0000 7fd936400640 1 Processor -- start 2024-04-17T20:02:25.880 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.879+0000 7fd936400640 1 -- start start 2024-04-17T20:02:25.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.883+0000 7fd936400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fd93006fb50 0x7fd9300cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:25.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.883+0000 7fd936400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd93015a840 0x7fd93015dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:25.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.883+0000 7fd936400640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd93015e220 con 0x7fd9300ca1e0 2024-04-17T20:02:25.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.883+0000 7fd936400640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fd9300c4470 con 0x7fd93015a840 2024-04-17T20:02:25.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.883+0000 7fd936400640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fd93015a3b0 con 0x7fd93006fb50 2024-04-17T20:02:25.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.883+0000 7fd927e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd93015a840 0x7fd93015dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:25.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.883+0000 7fd927e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd93015a840 0x7fd93015dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55220/0 (socket says 172.21.15.29:55220) 2024-04-17T20:02:25.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.883+0000 7fd927e00640 1 -- 172.21.15.29:0/4245246811 learned_addr learned my addr 172.21.15.29:0/4245246811 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:25.882 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.883+0000 7fd92f400640 1 --2- 172.21.15.29:0/4245246811 >> v2:172.21.15.37:3300/0 conn(0x7fd93006fb50 0x7fd9300cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:25.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.883+0000 7fd927e00640 1 -- 172.21.15.29:0/4245246811 >> v2:172.21.15.37:3300/0 conn(0x7fd93006fb50 msgr2=0x7fd9300cb6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:25.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.883+0000 7fd927e00640 1 --2- 172.21.15.29:0/4245246811 >> v2:172.21.15.37:3300/0 conn(0x7fd93006fb50 0x7fd9300cb6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:25.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.883+0000 7fd927e00640 1 -- 172.21.15.29:0/4245246811 >> v1:172.21.15.37:6789/0 conn(0x7fd9300ca1e0 legacy=0x7fd9300c4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:25.883 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.883+0000 7fd927e00640 1 -- 172.21.15.29:0/4245246811 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9300c46c0 con 0x7fd93015a840 2024-04-17T20:02:25.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.887+0000 7fd927e00640 1 --2- 172.21.15.29:0/4245246811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd93015a840 0x7fd93015dce0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd9300d0460 tx=0x7fd91801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade804ec7f993170 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:25.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.887+0000 7fd92ea00640 1 -- 172.21.15.29:0/4245246811 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9180b1070 con 0x7fd93015a840 2024-04-17T20:02:25.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.887+0000 7fd92ea00640 1 -- 172.21.15.29:0/4245246811 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9180ac630 con 0x7fd93015a840 2024-04-17T20:02:25.885 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.887+0000 7fd92ea00640 1 -- 172.21.15.29:0/4245246811 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9180a8050 con 0x7fd93015a840 2024-04-17T20:02:25.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.887+0000 7fd936400640 1 -- 172.21.15.29:0/4245246811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd93015a840 msgr2=0x7fd93015dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:25.887 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.887+0000 7fd936400640 1 --2- 172.21.15.29:0/4245246811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd93015a840 0x7fd93015dce0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd9300d0460 tx=0x7fd91801c3d0 comp rx=0 tx=0).stop 2024-04-17T20:02:25.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.887+0000 7fd936400640 1 -- 172.21.15.29:0/4245246811 shutdown_connections 2024-04-17T20:02:25.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.887+0000 7fd936400640 1 --2- 172.21.15.29:0/4245246811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd93015a840 0x7fd93015dce0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:25.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.887+0000 7fd936400640 1 --2- 172.21.15.29:0/4245246811 >> v2:172.21.15.37:3300/0 conn(0x7fd93006fb50 0x7fd9300cb6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:25.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.887+0000 7fd936400640 1 -- 172.21.15.29:0/4245246811 >> 172.21.15.29:0/4245246811 conn(0x7fd93006e3d0 msgr2=0x7fd93006e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:25.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.887+0000 7fd936400640 1 -- 172.21.15.29:0/4245246811 shutdown_connections 2024-04-17T20:02:25.889 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.887+0000 7fd936400640 1 -- 172.21.15.29:0/4245246811 wait complete. 2024-04-17T20:02:25.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.887+0000 7fd936400640 1 Processor -- start 2024-04-17T20:02:25.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd936400640 1 -- start start 2024-04-17T20:02:25.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd936400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd93006fb50 0x7fd9301f6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:25.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd936400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9300ca1e0 0x7fd930155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:25.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd936400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd93015a840 0x7fd930142770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:25.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd936400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd930159d10 con 0x7fd93015a840 2024-04-17T20:02:25.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd936400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd93015ac50 con 0x7fd9300ca1e0 2024-04-17T20:02:25.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd936400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd9300b51d0 con 0x7fd93006fb50 2024-04-17T20:02:25.892 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd92fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd93015a840 0x7fd930142770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:25.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd92fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd93015a840 0x7fd930142770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55232/0 (socket says 172.21.15.29:55232) 2024-04-17T20:02:25.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd92fe00640 1 -- 172.21.15.29:0/1069924308 learned_addr learned my addr 172.21.15.29:0/1069924308 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:25.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd927e00640 1 --2- 172.21.15.29:0/1069924308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9300ca1e0 0x7fd930155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:25.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd92f400640 1 --2- 172.21.15.29:0/1069924308 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd93006fb50 0x7fd9301f6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:25.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd92fe00640 1 -- 172.21.15.29:0/1069924308 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd93006fb50 msgr2=0x7fd9301f6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:25.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd92fe00640 1 --2- 172.21.15.29:0/1069924308 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd93006fb50 0x7fd9301f6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:25.893 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd92fe00640 1 -- 172.21.15.29:0/1069924308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9300ca1e0 msgr2=0x7fd930155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:25.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd92fe00640 1 --2- 172.21.15.29:0/1069924308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9300ca1e0 0x7fd930155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:25.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd92fe00640 1 -- 172.21.15.29:0/1069924308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9301f7150 con 0x7fd93015a840 2024-04-17T20:02:25.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd92fe00640 1 --2- 172.21.15.29:0/1069924308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd93015a840 0x7fd930142770 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fd92800c730 tx=0x7fd92800cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:25.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd927400640 1 -- 172.21.15.29:0/1069924308 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd928003ba0 con 0x7fd93015a840 2024-04-17T20:02:25.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd927400640 1 -- 172.21.15.29:0/1069924308 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd928003d40 con 0x7fd93015a840 2024-04-17T20:02:25.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd927400640 1 -- 172.21.15.29:0/1069924308 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd928005310 con 0x7fd93015a840 2024-04-17T20:02:25.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd936400640 1 -- 172.21.15.29:0/1069924308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd930154f10 con 0x7fd93015a840 2024-04-17T20:02:25.894 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.891+0000 7fd936400640 1 -- 172.21.15.29:0/1069924308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd930148570 con 0x7fd93015a840 2024-04-17T20:02:25.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.895+0000 7fd936400640 1 -- 172.21.15.29:0/1069924308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd930069b90 con 0x7fd93015a840 2024-04-17T20:02:25.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.895+0000 7fd927400640 1 -- 172.21.15.29:0/1069924308 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd9280054b0 con 0x7fd93015a840 2024-04-17T20:02:25.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.895+0000 7fd927400640 1 --2- 172.21.15.29:0/1069924308 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd90c068380 0x7fd90c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:25.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.895+0000 7fd927400640 1 -- 172.21.15.29:0/1069924308 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7fd928086cf0 con 0x7fd93015a840 2024-04-17T20:02:25.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.895+0000 7fd92f400640 1 --2- 172.21.15.29:0/1069924308 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd90c068380 0x7fd90c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:25.897 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.899+0000 7fd92f400640 1 --2- 172.21.15.29:0/1069924308 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd90c068380 0x7fd90c06a890 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd9200079c0 tx=0x7fd920008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:25.902 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:25.899+0000 7fd927400640 1 -- 172.21.15.29:0/1069924308 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd9280565f0 con 0x7fd93015a840 2024-04-17T20:02:26.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:26.127+0000 7fd936400640 1 -- 172.21.15.29:0/1069924308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9300b4fb0 con 0x7fd93015a840 2024-04-17T20:02:26.125 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:26.127+0000 7fd927400640 1 -- 172.21.15.29:0/1069924308 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6089 (secure 0 0 0) 0x7fd92805a8f0 con 0x7fd93015a840 2024-04-17T20:02:26.126 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:02:26.126 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:25.246297+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5212,5350,5631,5775,5946]},"id":1}]} 2024-04-17T20:02:26.126 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 61 2024-04-17T20:02:26.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:26.127+0000 7fd936400640 1 -- 172.21.15.29:0/1069924308 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd90c068380 msgr2=0x7fd90c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:26.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:26.127+0000 7fd936400640 1 --2- 172.21.15.29:0/1069924308 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd90c068380 0x7fd90c06a890 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd9200079c0 tx=0x7fd920008040 comp rx=0 tx=0).stop 2024-04-17T20:02:26.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:26.127+0000 7fd936400640 1 -- 172.21.15.29:0/1069924308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd93015a840 msgr2=0x7fd930142770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:26.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:26.127+0000 7fd936400640 1 --2- 172.21.15.29:0/1069924308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd93015a840 0x7fd930142770 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fd92800c730 tx=0x7fd92800cb30 comp rx=0 tx=0).stop 2024-04-17T20:02:26.128 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:26.127+0000 7fd936400640 1 -- 172.21.15.29:0/1069924308 shutdown_connections 2024-04-17T20:02:26.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:26.127+0000 7fd936400640 1 --2- 172.21.15.29:0/1069924308 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd90c068380 0x7fd90c06a890 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:26.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:26.127+0000 7fd936400640 1 --2- 172.21.15.29:0/1069924308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd93015a840 0x7fd930142770 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:26.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:26.127+0000 7fd936400640 1 --2- 172.21.15.29:0/1069924308 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd9300ca1e0 0x7fd930155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:26.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:26.127+0000 7fd936400640 1 --2- 172.21.15.29:0/1069924308 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd93006fb50 0x7fd9301f6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:26.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:26.127+0000 7fd936400640 1 -- 172.21.15.29:0/1069924308 >> 172.21.15.29:0/1069924308 conn(0x7fd93006e3d0 msgr2=0x7fd9300ce410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:26.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:26.127+0000 7fd936400640 1 -- 172.21.15.29:0/1069924308 shutdown_connections 2024-04-17T20:02:26.129 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:26.127+0000 7fd936400640 1 -- 172.21.15.29:0/1069924308 wait complete. 2024-04-17T20:02:26.150 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=2, replacing gid=5144 2024-04-17T20:02:26.150 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.f 2024-04-17T20:02:26.696 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:02:26.700 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:02:27.481 INFO:tasks.ceph.mds.e.smithi029.stderr:2024-04-17T20:02:27.479+0000 7f956e000640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-04-17T20:02:33.151 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2024-04-17T20:02:33.151 INFO:tasks.ceph.mds.f:Restarting daemon 2024-04-17T20:02:33.152 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-04-17T20:02:33.156 INFO:tasks.ceph.mds.f:Started 2024-04-17T20:02:33.156 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:02:33.267 INFO:tasks.ceph.mds.f.smithi037.stdout:starting mds.f at 2024-04-17T20:02:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.311+0000 7fc195c00640 1 Processor -- start 2024-04-17T20:02:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.311+0000 7fc195c00640 1 -- start start 2024-04-17T20:02:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.311+0000 7fc195c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1900cc8c0 0x7fc1900cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.311+0000 7fc195c00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fc190159390 0x7fc1901597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.311+0000 7fc195c00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1900c88b0 con 0x7fc1900ca330 2024-04-17T20:02:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.311+0000 7fc195c00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fc1900c6ce0 con 0x7fc190159390 2024-04-17T20:02:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.311+0000 7fc195c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc1900c6fc0 con 0x7fc1900cc8c0 2024-04-17T20:02:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.315+0000 7fc18f400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc1900ca330 0x7fc19015c250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54324/0 (socket says 172.21.15.29:54324) 2024-04-17T20:02:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.315+0000 7fc18f400640 1 -- 172.21.15.29:0/1877084701 learned_addr learned my addr 172.21.15.29:0/1877084701 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:02:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.315+0000 7fc187e00640 1 --2- 172.21.15.29:0/1877084701 >> v2:172.21.15.37:3300/0 conn(0x7fc190159390 0x7fc1901597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.315+0000 7fc18ea00640 1 --2- 172.21.15.29:0/1877084701 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1900cc8c0 0x7fc1900cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.315+0000 7fc18e000640 1 -- 172.21.15.29:0/1877084701 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4040965790 0 0) 0x7fc1900c88b0 con 0x7fc1900ca330 2024-04-17T20:02:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.315+0000 7fc18e000640 1 -- 172.21.15.29:0/1877084701 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc174003530 con 0x7fc1900ca330 2024-04-17T20:02:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.315+0000 7fc187e00640 1 -- 172.21.15.29:0/1877084701 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1900cc8c0 msgr2=0x7fc1900cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:02:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.315+0000 7fc187e00640 1 --2- 172.21.15.29:0/1877084701 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1900cc8c0 0x7fc1900cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.315+0000 7fc187e00640 1 -- 172.21.15.29:0/1877084701 >> v1:172.21.15.29:6789/0 conn(0x7fc1900ca330 legacy=0x7fc19015c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.315+0000 7fc187e00640 1 -- 172.21.15.29:0/1877084701 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1900c72a0 con 0x7fc190159390 2024-04-17T20:02:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.315+0000 7fc187e00640 1 --2- 172.21.15.29:0/1877084701 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc190159390 0x7fc1901597a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc1780097c0 tx=0x7fc17801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=741f399d0b83bdbd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.319+0000 7fc18e000640 1 -- 172.21.15.29:0/1877084701 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1780b0070 con 0x7fc190159390 2024-04-17T20:02:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.319+0000 7fc18e000640 1 -- 172.21.15.29:0/1877084701 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1780ab600 con 0x7fc190159390 2024-04-17T20:02:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.319+0000 7fc18e000640 1 -- 172.21.15.29:0/1877084701 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1780ab9b0 con 0x7fc190159390 2024-04-17T20:02:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.319+0000 7fc195c00640 1 -- 172.21.15.29:0/1877084701 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc190159390 msgr2=0x7fc1901597a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.319+0000 7fc195c00640 1 --2- 172.21.15.29:0/1877084701 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc190159390 0x7fc1901597a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc1780097c0 tx=0x7fc17801c4a0 comp rx=0 tx=0).stop 2024-04-17T20:02:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.319+0000 7fc195c00640 1 -- 172.21.15.29:0/1877084701 shutdown_connections 2024-04-17T20:02:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.319+0000 7fc195c00640 1 --2- 172.21.15.29:0/1877084701 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc190159390 0x7fc1901597a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.319+0000 7fc195c00640 1 --2- 172.21.15.29:0/1877084701 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1900cc8c0 0x7fc1900cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.319+0000 7fc195c00640 1 -- 172.21.15.29:0/1877084701 >> 172.21.15.29:0/1877084701 conn(0x7fc19006fde0 msgr2=0x7fc1900701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.319+0000 7fc195c00640 1 -- 172.21.15.29:0/1877084701 shutdown_connections 2024-04-17T20:02:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.319+0000 7fc195c00640 1 -- 172.21.15.29:0/1877084701 wait complete. 2024-04-17T20:02:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.319+0000 7fc195c00640 1 Processor -- start 2024-04-17T20:02:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.319+0000 7fc195c00640 1 -- start start 2024-04-17T20:02:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc195c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1900ca330 0x7fc190155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc195c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1900cc8c0 0x7fc190154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc195c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc190159390 0x7fc190157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc195c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fc1900c6a00 con 0x7fc190159390 2024-04-17T20:02:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc195c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fc1900c6880 con 0x7fc1900cc8c0 2024-04-17T20:02:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc195c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fc190166600 con 0x7fc1900ca330 2024-04-17T20:02:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc18f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc190159390 0x7fc190157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc18f400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc190159390 0x7fc190157c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38924/0 (socket says 172.21.15.29:38924) 2024-04-17T20:02:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc18f400640 1 -- 172.21.15.29:0/4192731574 learned_addr learned my addr 172.21.15.29:0/4192731574 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc187e00640 1 --2- 172.21.15.29:0/4192731574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1900cc8c0 0x7fc190154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc18ea00640 1 --2- 172.21.15.29:0/4192731574 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1900ca330 0x7fc190155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc18f400640 1 -- 172.21.15.29:0/4192731574 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1900ca330 msgr2=0x7fc190155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc18f400640 1 --2- 172.21.15.29:0/4192731574 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1900ca330 0x7fc190155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc18f400640 1 -- 172.21.15.29:0/4192731574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1900cc8c0 msgr2=0x7fc190154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc18f400640 1 --2- 172.21.15.29:0/4192731574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1900cc8c0 0x7fc190154700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc18f400640 1 -- 172.21.15.29:0/4192731574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc190154c40 con 0x7fc190159390 2024-04-17T20:02:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc187e00640 1 --2- 172.21.15.29:0/4192731574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1900cc8c0 0x7fc190154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T20:02:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc18f400640 1 --2- 172.21.15.29:0/4192731574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc190159390 0x7fc190157c50 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fc18800cad0 tx=0x7fc18800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc186a00640 1 -- 172.21.15.29:0/4192731574 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc188017070 con 0x7fc190159390 2024-04-17T20:02:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc186a00640 1 -- 172.21.15.29:0/4192731574 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc18800ac30 con 0x7fc190159390 2024-04-17T20:02:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc195c00640 1 -- 172.21.15.29:0/4192731574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1900c4400 con 0x7fc190159390 2024-04-17T20:02:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc186a00640 1 -- 172.21.15.29:0/4192731574 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc188012400 con 0x7fc190159390 2024-04-17T20:02:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc195c00640 1 -- 172.21.15.29:0/4192731574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc190153940 con 0x7fc190159390 2024-04-17T20:02:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.323+0000 7fc195c00640 1 -- 172.21.15.29:0/4192731574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc190069b90 con 0x7fc190159390 2024-04-17T20:02:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.327+0000 7fc186a00640 1 -- 172.21.15.29:0/4192731574 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fc18801a070 con 0x7fc190159390 2024-04-17T20:02:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.327+0000 7fc186a00640 1 --2- 172.21.15.29:0/4192731574 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc16c068380 0x7fc16c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.327+0000 7fc186a00640 1 -- 172.21.15.29:0/4192731574 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7fc188086e20 con 0x7fc190159390 2024-04-17T20:02:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.327+0000 7fc18ea00640 1 --2- 172.21.15.29:0/4192731574 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc16c068380 0x7fc16c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.327+0000 7fc18ea00640 1 --2- 172.21.15.29:0/4192731574 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc16c068380 0x7fc16c06a890 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc180009ac0 tx=0x7fc180009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:33.330 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.331+0000 7fc186a00640 1 -- 172.21.15.29:0/4192731574 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fc188056610 con 0x7fc190159390 2024-04-17T20:02:33.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.531+0000 7fc195c00640 1 -- 172.21.15.29:0/4192731574 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1900c46f0 con 0x7fc190159390 2024-04-17T20:02:33.533 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.535+0000 7fc186a00640 1 -- 172.21.15.29:0/4192731574 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6089 (secure 0 0 0) 0x7fc18805a910 con 0x7fc190159390 2024-04-17T20:02:33.533 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:02:33.533 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:25.246297+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5212,5350,5631,5775,5946]},"id":1}]} 2024-04-17T20:02:33.533 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 61 2024-04-17T20:02:33.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.535+0000 7fc195c00640 1 -- 172.21.15.29:0/4192731574 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc16c068380 msgr2=0x7fc16c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:33.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.535+0000 7fc195c00640 1 --2- 172.21.15.29:0/4192731574 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc16c068380 0x7fc16c06a890 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc180009ac0 tx=0x7fc180009190 comp rx=0 tx=0).stop 2024-04-17T20:02:33.535 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.535+0000 7fc195c00640 1 -- 172.21.15.29:0/4192731574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc190159390 msgr2=0x7fc190157c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:33.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.535+0000 7fc195c00640 1 --2- 172.21.15.29:0/4192731574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc190159390 0x7fc190157c50 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fc18800cad0 tx=0x7fc18800cf90 comp rx=0 tx=0).stop 2024-04-17T20:02:33.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.535+0000 7fc195c00640 1 -- 172.21.15.29:0/4192731574 shutdown_connections 2024-04-17T20:02:33.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.535+0000 7fc195c00640 1 --2- 172.21.15.29:0/4192731574 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fc16c068380 0x7fc16c06a890 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:33.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.535+0000 7fc195c00640 1 --2- 172.21.15.29:0/4192731574 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc190159390 0x7fc190157c50 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:33.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.535+0000 7fc195c00640 1 --2- 172.21.15.29:0/4192731574 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fc1900cc8c0 0x7fc190154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:33.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.535+0000 7fc195c00640 1 --2- 172.21.15.29:0/4192731574 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fc1900ca330 0x7fc190155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:33.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.535+0000 7fc195c00640 1 -- 172.21.15.29:0/4192731574 >> 172.21.15.29:0/4192731574 conn(0x7fc19006fde0 msgr2=0x7fc19006bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:33.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.535+0000 7fc195c00640 1 -- 172.21.15.29:0/4192731574 shutdown_connections 2024-04-17T20:02:33.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:33.535+0000 7fc195c00640 1 -- 172.21.15.29:0/4192731574 wait complete. 2024-04-17T20:02:33.556 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2024-04-17T20:02:35.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:02:35.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.759+0000 7f46ee200640 1 Processor -- start 2024-04-17T20:02:35.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.759+0000 7f46ee200640 1 -- start start 2024-04-17T20:02:35.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.763+0000 7f46ee200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f46e806e960 0x7f46e8070d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:35.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.763+0000 7f46ee200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46e815bf10 con 0x7f46e8155da0 2024-04-17T20:02:35.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.763+0000 7f46ee200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46e80c98e0 con 0x7f46e815cae0 2024-04-17T20:02:35.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.763+0000 7f46ee200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f46e80c9bc0 con 0x7f46e806e960 2024-04-17T20:02:35.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.763+0000 7f46e7400640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f46e806e960 0x7f46e8070d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:35.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.763+0000 7f46e6a00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f46e8155da0 0x7f46e815c3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54326/0 (socket says 172.21.15.29:54326) 2024-04-17T20:02:35.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.763+0000 7f46e6a00640 1 -- 172.21.15.29:0/482760949 learned_addr learned my addr 172.21.15.29:0/482760949 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:02:35.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.763+0000 7f46e7400640 1 -- 172.21.15.29:0/482760949 >> v1:172.21.15.37:6789/0 conn(0x7f46e815cae0 legacy=0x7f46e8166810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:35.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.763+0000 7f46e7400640 1 -- 172.21.15.29:0/482760949 >> v1:172.21.15.29:6789/0 conn(0x7f46e8155da0 legacy=0x7f46e815c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:35.763 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.763+0000 7f46e7400640 1 -- 172.21.15.29:0/482760949 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e80c9ea0 con 0x7f46e806e960 2024-04-17T20:02:35.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.767+0000 7f46e7400640 1 --2- 172.21.15.29:0/482760949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46e806e960 0x7f46e8070d70 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f46d80097c0 tx=0x7f46d801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc02bcef7f7c9de5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:35.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.767+0000 7f46e6000640 1 -- 172.21.15.29:0/482760949 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d80b2070 con 0x7f46e806e960 2024-04-17T20:02:35.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.767+0000 7f46e6000640 1 -- 172.21.15.29:0/482760949 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46d80a9070 con 0x7f46e806e960 2024-04-17T20:02:35.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.767+0000 7f46e6000640 1 -- 172.21.15.29:0/482760949 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46d80ad880 con 0x7f46e806e960 2024-04-17T20:02:35.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.767+0000 7f46ee200640 1 -- 172.21.15.29:0/482760949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46e806e960 msgr2=0x7f46e8070d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:35.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.767+0000 7f46ee200640 1 --2- 172.21.15.29:0/482760949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46e806e960 0x7f46e8070d70 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f46d80097c0 tx=0x7f46d801c3c0 comp rx=0 tx=0).stop 2024-04-17T20:02:35.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.767+0000 7f46ee200640 1 -- 172.21.15.29:0/482760949 shutdown_connections 2024-04-17T20:02:35.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.767+0000 7f46ee200640 1 --2- 172.21.15.29:0/482760949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46e806e960 0x7f46e8070d70 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:35.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.767+0000 7f46ee200640 1 -- 172.21.15.29:0/482760949 >> 172.21.15.29:0/482760949 conn(0x7f46e80ce5e0 msgr2=0x7f46e80ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:35.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.767+0000 7f46ee200640 1 -- 172.21.15.29:0/482760949 shutdown_connections 2024-04-17T20:02:35.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.767+0000 7f46ee200640 1 -- 172.21.15.29:0/482760949 wait complete. 2024-04-17T20:02:35.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46ee200640 1 Processor -- start 2024-04-17T20:02:35.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46ee200640 1 -- start start 2024-04-17T20:02:35.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46ee200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f46e806e960 0x7f46e814a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:35.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46ee200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f46e8155da0 0x7f46e81f6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:35.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46ee200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46e815cae0 0x7f46e8144c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:35.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46ee200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f46e806f7b0 con 0x7f46e815cae0 2024-04-17T20:02:35.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46ee200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f46e80ca460 con 0x7f46e8155da0 2024-04-17T20:02:35.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46ee200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f46e80712b0 con 0x7f46e806e960 2024-04-17T20:02:35.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46e7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46e815cae0 0x7f46e8144c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:35.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46e7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46e815cae0 0x7f46e8144c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38936/0 (socket says 172.21.15.29:38936) 2024-04-17T20:02:35.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46e7e00640 1 -- 172.21.15.29:0/211741781 learned_addr learned my addr 172.21.15.29:0/211741781 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:02:35.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46e7400640 1 --2- 172.21.15.29:0/211741781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f46e806e960 0x7f46e814a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:35.778 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46e6a00640 1 --2- 172.21.15.29:0/211741781 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f46e8155da0 0x7f46e81f6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:35.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46e7e00640 1 -- 172.21.15.29:0/211741781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f46e806e960 msgr2=0x7f46e814a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:35.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46e7e00640 1 --2- 172.21.15.29:0/211741781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f46e806e960 0x7f46e814a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:35.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46e7e00640 1 -- 172.21.15.29:0/211741781 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f46e8155da0 msgr2=0x7f46e81f6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:35.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46e7e00640 1 --2- 172.21.15.29:0/211741781 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f46e8155da0 0x7f46e81f6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:35.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46e7e00640 1 -- 172.21.15.29:0/211741781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46e814ad80 con 0x7f46e815cae0 2024-04-17T20:02:35.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46e7e00640 1 --2- 172.21.15.29:0/211741781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46e815cae0 0x7f46e8144c50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f46e000c760 tx=0x7f46e000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:35.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46df400640 1 -- 172.21.15.29:0/211741781 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e0003b80 con 0x7f46e815cae0 2024-04-17T20:02:35.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.771+0000 7f46df400640 1 -- 172.21.15.29:0/211741781 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46e0003d20 con 0x7f46e815cae0 2024-04-17T20:02:35.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.775+0000 7f46ee200640 1 -- 172.21.15.29:0/211741781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46e814cdf0 con 0x7f46e815cae0 2024-04-17T20:02:35.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.775+0000 7f46df400640 1 -- 172.21.15.29:0/211741781 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46e000aa90 con 0x7f46e815cae0 2024-04-17T20:02:35.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.775+0000 7f46ee200640 1 -- 172.21.15.29:0/211741781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46e8074180 con 0x7f46e815cae0 2024-04-17T20:02:35.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.775+0000 7f46ee200640 1 -- 172.21.15.29:0/211741781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46e8069b90 con 0x7f46e815cae0 2024-04-17T20:02:35.779 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.775+0000 7f46df400640 1 -- 172.21.15.29:0/211741781 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f46e000ac30 con 0x7f46e815cae0 2024-04-17T20:02:35.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.775+0000 7f46df400640 1 --2- 172.21.15.29:0/211741781 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f46c8068380 0x7f46c806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:02:35.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.775+0000 7f46df400640 1 -- 172.21.15.29:0/211741781 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7f46e0086d30 con 0x7f46e815cae0 2024-04-17T20:02:35.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.779+0000 7f46e7400640 1 --2- 172.21.15.29:0/211741781 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f46c8068380 0x7f46c806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:02:35.780 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.779+0000 7f46e7400640 1 --2- 172.21.15.29:0/211741781 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f46c8068380 0x7f46c806a890 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f46d80af000 tx=0x7f46d80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:02:35.781 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.783+0000 7f46df400640 1 -- 172.21.15.29:0/211741781 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f46e0056630 con 0x7f46e815cae0 2024-04-17T20:02:35.985 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.987+0000 7f46ee200640 1 -- 172.21.15.29:0/211741781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46e8069250 con 0x7f46e815cae0 2024-04-17T20:02:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.987+0000 7f46df400640 1 -- 172.21.15.29:0/211741781 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6913 (secure 0 0 0) 0x7f46e005a930 con 0x7f46e815cae0 2024-04-17T20:02:35.986 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:02:35.986 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6076,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:25.246297+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5946,5775,5631,5350,5212]},"id":1}]} 2024-04-17T20:02:35.986 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 62 2024-04-17T20:02:35.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.987+0000 7f46ee200640 1 -- 172.21.15.29:0/211741781 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f46c8068380 msgr2=0x7f46c806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:35.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.987+0000 7f46ee200640 1 --2- 172.21.15.29:0/211741781 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f46c8068380 0x7f46c806a890 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f46d80af000 tx=0x7f46d80a7000 comp rx=0 tx=0).stop 2024-04-17T20:02:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.987+0000 7f46ee200640 1 -- 172.21.15.29:0/211741781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46e815cae0 msgr2=0x7f46e8144c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:02:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.987+0000 7f46ee200640 1 --2- 172.21.15.29:0/211741781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46e815cae0 0x7f46e8144c50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f46e000c760 tx=0x7f46e000cc20 comp rx=0 tx=0).stop 2024-04-17T20:02:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.987+0000 7f46ee200640 1 -- 172.21.15.29:0/211741781 shutdown_connections 2024-04-17T20:02:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.987+0000 7f46ee200640 1 --2- 172.21.15.29:0/211741781 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f46c8068380 0x7f46c806a890 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.987+0000 7f46ee200640 1 --2- 172.21.15.29:0/211741781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46e815cae0 0x7f46e8144c50 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.987+0000 7f46ee200640 1 --2- 172.21.15.29:0/211741781 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f46e8155da0 0x7f46e81f6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.987+0000 7f46ee200640 1 --2- 172.21.15.29:0/211741781 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f46e806e960 0x7f46e814a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:02:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.987+0000 7f46ee200640 1 -- 172.21.15.29:0/211741781 >> 172.21.15.29:0/211741781 conn(0x7f46e80ce5e0 msgr2=0x7f46e815b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:02:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.987+0000 7f46ee200640 1 -- 172.21.15.29:0/211741781 shutdown_connections 2024-04-17T20:02:35.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:02:35.987+0000 7f46ee200640 1 -- 172.21.15.29:0/211741781 wait complete. 2024-04-17T20:02:36.012 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2024-04-17T20:02:36.012 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 103 secs before thrashing 2024-04-17T20:02:58.018 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:02:58.022 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:03:29.533 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:03:29.541 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:03:55.918 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 0% (97/13274) Updating files: 0% (113/13274) Updating files: 1% (133/13274) Updating files: 1% (230/13274) Updating files: 1% (247/13274) Updating files: 2% (266/13274) Updating files: 2% (278/13274) Updating files: 2% (361/13274) Updating files: 2% (390/13274) Updating files: 3% (399/13274) Updating files: 3% (489/13274) Updating files: 4% (531/13274) Updating files: 4% (586/13274) Updating files: 5% (664/13274) Updating files: 5% (717/13274) Updating files: 5% (788/13274) Updating files: 6% (797/13274) Updating files: 6% (891/13274) Updating files: 7% (930/13274) Updating files: 7% (991/13274) Updating files: 7% (1008/13274) Updating files: 8% (1062/13274) Updating files: 8% (1081/13274) Updating files: 8% (1099/13274) Updating files: 8% (1133/13274) Updating files: 8% (1183/13274) Updating files: 9% (1195/13274) Updating files: 9% (1212/13274) Updating files: 9% (1278/13274) Updating files: 10% (1328/13274) Updating files: 10% (1370/13274) Updating files: 10% (1431/13274) Updating files: 11% (1461/13274) Updating files: 11% (1576/13274) Updating files: 12% (1593/13274) Updating files: 12% (1634/13274) Updating files: 12% (1639/13274) Updating files: 12% (1670/13274) Updating files: 13% (1726/13274) Updating files: 13% (1764/13274) Updating files: 13% (1794/13274) Updating files: 14% (1859/13274) Updating files: 14% (1877/13274) Updating files: 14% (1924/13274) Updating files: 14% (1930/13274) Updating files: 15% (1992/13274) Updating files: 15% (2070/13274) Updating files: 15% (2123/13274) Updating files: 16% (2124/13274) Updating files: 16% (2233/13274) Updating files: 17% (2257/13274) Updating files: 17% (2322/13274) Updating files: 18% (2390/13274) Updating files: 18% (2400/13274) Updating files: 18% (2403/13274) Updating files: 18% (2405/13274) Updating files: 18% (2434/13274) Updating files: 18% (2452/13274) Updating files: 18% (2506/13274) Updating files: 19% (2523/13274) Updating files: 19% (2555/13274) Updating files: 19% (2635/13274) Updating files: 20% (2655/13274) Updating files: 20% (2688/13274) Updating files: 20% (2735/13274) Updating files: 20% (2780/13274) Updating files: 21% (2788/13274) Updating files: 21% (2855/13274) Updating files: 22% (2921/13274) Updating files: 22% (2926/13274) Updating files: 22% (2955/13274) Updating files: 22% (3004/13274) Updating files: 22% (3019/13274) Updating files: 22% (3045/13274) Updating files: 23% (3054/13274) Updating files: 23% (3127/13274) Updating files: 24% (3186/13274) Updating files: 24% (3229/13274) Updating files: 24% (3308/13274) Updating files: 24% (3317/13274) Updating files: 25% (3319/13274) Updating files: 25% (3389/13274) Updating files: 25% (3391/13274) Updating files: 25% (3426/13274) Updating files: 26% (3452/13274) Updating files: 26% (3552/13274) Updating files: 27% (3584/13274) Updating files: 27% (3586/13274) Updating files: 27% (3649/13274) Updating files: 28% (3717/13274) Updating files: 28% (3756/13274) Updating files: 28% (3767/13274) Updating files: 28% (3792/13274) Updating files: 28% (3848/13274) Updating files: 29% (3850/13274) Updating files: 29% (3885/13274) Updating files: 30% (3983/13274) Updating files: 30% (4020/13274) Updating files: 30% (4039/13274) Updating files: 31% (4115/13274) Updating files: 31% (4124/13274) Updating files: 31% (4195/13274) Updating files: 32% (4248/13274) Updating files: 32% (4305/13274) Updating files: 32% (4316/13274) Updating files: 33% (4381/13274) Updating files: 33% (4419/13274) Updating files: 33% (4443/13274) Updating files: 34% (4514/13274) Updating files: 34% (4534/13274) Updating files: 34% (4626/13274) Updating files: 35% (4646/13274) Updating files: 35% (4778/13274) Updating files: 36% (4779/13274) Updating files: 36% (4786/13274) Updating files: 36% (4791/13274) Updating files: 36% (4862/13274) Updating files: 36% (4882/13274) Updating files: 36% (4899/13274) Updating files: 37% (4912/13274) Updating files: 37% (4981/13274) Updating files: 37% (5021/13274) Updating files: 38% (5045/13274) Updating files: 38% (5046/13274) Updating files: 38% (5176/13274) Updating files: 39% (5177/13274) Updating files: 39% (5222/13274) Updating files: 39% (5300/13274) Updating files: 40% (5310/13274) Updating files: 40% (5367/13274) Updating files: 41% (5443/13274) Updating files: 41% (5524/13274) Updating files: 41% (5526/13274) Updating files: 42% (5576/13274) Updating files: 42% (5646/13274) Updating files: 42% (5680/13274) Updating files: 42% (5703/13274) Updating files: 43% (5708/13274) Updating files: 44% (5841/13274) Updating files: 44% (5918/13274) Updating files: 44% (5941/13274) Updating files: 44% (5961/13274) Updating files: 45% (5974/13274) Updating files: 45% (6098/13274) Updating files: 46% (6107/13274) Updating files: 46% (6142/13274) Updating files: 47% (6239/13274) Updating files: 47% (6240/13274) Updating files: 47% (6270/13274) Updating files: 47% (6326/13274) Updating files: 48% (6372/13274) Updating files: 48% (6430/13274) Updating files: 48% (6475/13274) Updating files: 49% (6505/13274) Updating files: 50% (6637/13274) Updating files: 50% (6650/13274) Updating files: 50% (6703/13274) Updating files: 50% (6728/13274) Updating files: 51% (6770/13274) Updating files: 51% (6801/13274) Updating files: 51% (6838/13274) Updating files: 51% (6889/13274) Updating files: 52% (6903/13274) Updating files: 52% (6905/13274) Updating files: 52% (6910/13274) Updating files: 53% (7036/13274) Updating files: 53% (7042/13274) Updating files: 53% (7066/13274) Updating files: 53% (7076/13274) Updating files: 53% (7115/13274) Updating files: 53% (7165/13274) Updating files: 54% (7168/13274) Updating files: 54% (7194/13274) Updating files: 54% (7199/13274) Updating files: 54% (7279/13274) Updating files: 55% (7301/13274) Updating files: 55% (7322/13274) Updating files: 56% (7434/13274) Updating files: 56% (7487/13274) Updating files: 57% (7567/13274) Updating files: 57% (7658/13274) Updating files: 58% (7699/13274) Updating files: 58% (7767/13274) Updating files: 59% (7832/13274) Updating files: 59% (7876/13274) Updating files: 59% (7900/13274) Updating files: 60% (7965/13274) Updating files: 60% (8005/13274) Updating files: 60% (8029/13274) Updating files: 60% (8049/13274) Updating files: 61% (8098/13274) Updating files: 61% (8113/13274) Updating files: 61% (8137/13274) Updating files: 61% (8181/13274) Updating files: 61% (8197/13274) Updating files: 62% (8230/13274) Updating files: 62% (8258/13274) Updating files: 62% (8347/13274) Updating files: 63% (8363/13274) Updating files: 63% (8406/13274) Updating files: 64% (8496/13274) Updating files: 64% (8556/13274) Updating files: 64% (8587/13274) Updating files: 64% (8600/13274) Updating files: 65% (8629/13274) Updating files: 65% (8637/13274) Updating files: 65% (8738/13274) Updating files: 66% (8761/13274) Updating files: 66% (8785/13274) Updating files: 66% (8850/13274) Updating files: 66% (8881/13274) Updating files: 67% (8894/13274) Updating files: 67% (8971/13274) Updating files: 68% (9027/13274) Updating files: 68% (9030/13274) Updating files: 68% (9105/13274) Updating files: 69% (9160/13274) Updating files: 69% (9172/13274) Updating files: 69% (9198/13274) Updating files: 70% (9292/13274) Updating files: 70% (9351/13274) Updating files: 71% (9425/13274) Updating files: 71% (9430/13274) Updating files: 71% (9450/13274) Updating files: 71% (9523/13274) Updating files: 71% (9544/13274) Updating files: 72% (9558/13274) Updating files: 72% (9576/13274) Updating files: 72% (9651/13274) Updating files: 73% (9691/13274) Updating files: 73% (9727/13274) Updating files: 74% (9823/13274) Updating files: 74% (9939/13274) Updating files: 75% (9956/13274) Updating files: 76% (10089/13274) Updating files: 76% (10119/13274) Updating files: 77% (10221/13274) Updating files: 77% (10237/13274) Updating files: 78% (10354/13274) Updating files: 78% (10381/13274) Updating files: 78% (10401/13274) Updating files: 79% (10487/13274) Updating files: 79% (10587/13274) Updating files: 80% (10620/13274) Updating files: 80% (10648/13274) Updating files: 80% (10661/13274) Updating files: 81% (10752/13274) Updating files: 81% (10820/13274) Updating files: 81% (10827/13274) Updating files: 81% (10846/13274) Updating files: 81% (10878/13274) Updating files: 82% (10885/13274) Updating files: 82% (10931/13274) Updating files: 82% (11001/13274) Updating files: 83% (11018/13274) Updating files: 84% (11151/13274) Updating files: 84% (11166/13274) Updating files: 84% (11229/13274) Updating files: 84% (11250/13274) Updating files: 85% (11283/13274) Updating files: 85% (11324/13274) Updating files: 85% (11403/13274) Updating files: 86% (11416/13274) Updating files: 86% (11473/13274) Updating files: 86% (11519/13274) Updating files: 87% (11549/13274) Updating files: 87% (11625/13274) Updating files: 88% (11682/13274) Updating files: 88% (11719/13274) Updating files: 88% (11764/13274) Updating files: 88% (11785/13274) Updating files: 88% (11806/13274) Updating files: 89% (11814/13274) Updating files: 89% (11847/13274) Updating files: 89% (11861/13274) Updating files: 90% (11947/13274) Updating files: 90% (11968/13274) Updating files: 91% (12080/13274) Updating files: 91% (12165/13274) Updating files: 91% (12191/13274) Updating files: 92% (12213/13274) Updating files: 93% (12345/13274) Updating files: 93% (12367/13274) Updating files: 93% (12393/13274) Updating files: 94% (12478/13274) Updating files: 94% (12523/13274) Updating files: 94% (12600/13274) Updating files: 95% (12611/13274) Updating files: 96% (12744/13274) Updating files: 96% (12760/13274) Updating files: 96% (12784/13274) Updating files: 96% (12840/13274) Updating files: 97% (12876/13274) Updating files: 97% (12886/13274) Updating files: 97% (12942/13274) Updating files: 97% (12990/13274) Updating files: 98% (13009/13274) Updating files: 98% (13108/13274) Updating files: 99% (13142/13274) Updating files: 99% (13166/13274) Updating files: 100% (13274/13274) Updating files: 100% (13274/13274), done. 2024-04-17T20:03:57.050 INFO:tasks.workunit.client.0.smithi029.stdout:HEAD is now at 204c175eea3 msg: blobhash on entity_name_t causes strange badness... should look into that, but avoid for now 2024-04-17T20:03:57.068 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/v0.1 2024-04-17T20:03:57.515 INFO:tasks.workunit.client.0.smithi029.stderr:+ [ 2 -eq 48 ] 2024-04-17T20:03:57.515 INFO:tasks.workunit.client.0.smithi029.stderr:+ ver=v0.2 2024-04-17T20:03:57.516 INFO:tasks.workunit.client.0.smithi029.stdout:v0.2 2024-04-17T20:03:57.516 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo v0.2 2024-04-17T20:03:57.516 INFO:tasks.workunit.client.0.smithi029.stderr:+ git reset --hard v0.2 2024-04-17T20:03:59.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:03:59.586 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:04:03.073 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 21% (48/222) Updating files: 22% (49/222) Updating files: 23% (52/222) Updating files: 24% (54/222) Updating files: 25% (56/222) Updating files: 26% (58/222) Updating files: 27% (60/222) Updating files: 28% (63/222) Updating files: 29% (65/222) Updating files: 30% (67/222) Updating files: 31% (69/222) Updating files: 32% (72/222) Updating files: 33% (74/222) Updating files: 34% (76/222) Updating files: 35% (78/222) Updating files: 36% (80/222) Updating files: 37% (83/222) Updating files: 38% (85/222) Updating files: 39% (87/222) Updating files: 40% (89/222) Updating files: 41% (92/222) Updating files: 41% (93/222) Updating files: 42% (94/222) Updating files: 43% (96/222) Updating files: 44% (98/222) Updating files: 45% (100/222) Updating files: 46% (103/222) Updating files: 47% (105/222) Updating files: 48% (107/222) Updating files: 49% (109/222) Updating files: 50% (111/222) Updating files: 51% (114/222) Updating files: 52% (116/222) Updating files: 53% (118/222) Updating files: 54% (120/222) Updating files: 55% (123/222) Updating files: 56% (125/222) Updating files: 57% (127/222) Updating files: 58% (129/222) Updating files: 59% (131/222) Updating files: 60% (134/222) Updating files: 61% (136/222) Updating files: 62% (138/222) Updating files: 63% (140/222) Updating files: 64% (143/222) Updating files: 65% (145/222) Updating files: 66% (147/222) Updating files: 67% (149/222) Updating files: 68% (151/222) Updating files: 69% (154/222) Updating files: 70% (156/222) Updating files: 71% (158/222) Updating files: 72% (160/222) Updating files: 73% (163/222) Updating files: 74% (165/222) Updating files: 75% (167/222) Updating files: 76% (169/222) Updating files: 77% (171/222) Updating files: 78% (174/222) Updating files: 79% (176/222) Updating files: 80% (178/222) Updating files: 81% (180/222) Updating files: 82% (183/222) Updating files: 83% (185/222) Updating files: 84% (187/222) Updating files: 85% (189/222) Updating files: 86% (191/222) Updating files: 87% (194/222) Updating files: 88% (196/222) Updating files: 89% (198/222) Updating files: 90% (200/222) Updating files: 90% (201/222) Updating files: 91% (203/222) Updating files: 92% (205/222) Updating files: 93% (207/222) Updating files: 94% (209/222) Updating files: 95% (211/222) Updating files: 96% (214/222) Updating files: 97% (216/222) Updating files: 98% (218/222) Updating files: 99% (220/222) Updating files: 100% (222/222) Updating files: 100% (222/222), done. 2024-04-17T20:04:03.096 INFO:tasks.workunit.client.0.smithi029.stdout:HEAD is now at 2b5f76f48e9 readme 2024-04-17T20:04:03.101 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/v0.2 2024-04-17T20:04:03.307 INFO:tasks.workunit.client.0.smithi029.stderr:+ [ 3 -eq 48 ] 2024-04-17T20:04:03.307 INFO:tasks.workunit.client.0.smithi029.stderr:+ ver=v0.3 2024-04-17T20:04:03.307 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo v0.3 2024-04-17T20:04:03.307 INFO:tasks.workunit.client.0.smithi029.stderr:+ git reset --hard v0.3 2024-04-17T20:04:03.307 INFO:tasks.workunit.client.0.smithi029.stdout:v0.3 2024-04-17T20:04:11.800 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 44% (132/295) Updating files: 45% (133/295) Updating files: 46% (136/295) Updating files: 47% (139/295) Updating files: 48% (142/295) Updating files: 49% (145/295) Updating files: 49% (147/295) Updating files: 50% (148/295) Updating files: 51% (151/295) Updating files: 52% (154/295) Updating files: 53% (157/295) Updating files: 54% (160/295) Updating files: 55% (163/295) Updating files: 56% (166/295) Updating files: 57% (169/295) Updating files: 58% (172/295) Updating files: 59% (175/295) Updating files: 60% (177/295) Updating files: 61% (180/295) Updating files: 62% (183/295) Updating files: 63% (186/295) Updating files: 64% (189/295) Updating files: 65% (192/295) Updating files: 66% (195/295) Updating files: 67% (198/295) Updating files: 68% (201/295) Updating files: 69% (204/295) Updating files: 70% (207/295) Updating files: 71% (210/295) Updating files: 72% (213/295) Updating files: 73% (216/295) Updating files: 74% (219/295) Updating files: 75% (222/295) Updating files: 76% (225/295) Updating files: 77% (228/295) Updating files: 78% (231/295) Updating files: 79% (234/295) Updating files: 80% (236/295) Updating files: 81% (239/295) Updating files: 82% (242/295) Updating files: 83% (245/295) Updating files: 84% (248/295) Updating files: 85% (251/295) Updating files: 86% (254/295) Updating files: 87% (257/295) Updating files: 88% (260/295) Updating files: 89% (263/295) Updating files: 90% (266/295) Updating files: 91% (269/295) Updating files: 92% (272/295) Updating files: 93% (275/295) Updating files: 94% (278/295) Updating files: 95% (281/295) Updating files: 96% (284/295) Updating files: 97% (287/295) Updating files: 98% (290/295) Updating files: 99% (293/295) Updating files: 100% (295/295) Updating files: 100% (295/295), done. 2024-04-17T20:04:11.832 INFO:tasks.workunit.client.0.smithi029.stdout:HEAD is now at 2e986fe9ddf include ceph.spec in configure.ac 2024-04-17T20:04:11.841 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/v0.3 2024-04-17T20:04:13.445 INFO:tasks.workunit.client.0.smithi029.stderr:+ [ 4 -eq 48 ] 2024-04-17T20:04:13.445 INFO:tasks.workunit.client.0.smithi029.stderr:+ ver=v0.4 2024-04-17T20:04:13.445 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo v0.4 2024-04-17T20:04:13.445 INFO:tasks.workunit.client.0.smithi029.stdout:v0.4 2024-04-17T20:04:13.446 INFO:tasks.workunit.client.0.smithi029.stderr:+ git reset --hard v0.4 2024-04-17T20:04:16.976 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 44% (111/247) Updating files: 45% (112/247) Updating files: 46% (114/247) Updating files: 47% (117/247) Updating files: 48% (119/247) Updating files: 49% (122/247) Updating files: 50% (124/247) Updating files: 51% (126/247) Updating files: 52% (129/247) Updating files: 53% (131/247) Updating files: 54% (134/247) Updating files: 55% (136/247) Updating files: 56% (139/247) Updating files: 57% (141/247) Updating files: 58% (144/247) Updating files: 59% (146/247) Updating files: 60% (149/247) Updating files: 61% (151/247) Updating files: 62% (154/247) Updating files: 63% (156/247) Updating files: 64% (159/247) Updating files: 65% (161/247) Updating files: 66% (164/247) Updating files: 67% (166/247) Updating files: 68% (168/247) Updating files: 69% (171/247) Updating files: 70% (173/247) Updating files: 71% (176/247) Updating files: 72% (178/247) Updating files: 73% (181/247) Updating files: 74% (183/247) Updating files: 75% (186/247) Updating files: 76% (188/247) Updating files: 77% (191/247) Updating files: 78% (193/247) Updating files: 79% (196/247) Updating files: 80% (198/247) Updating files: 81% (201/247) Updating files: 82% (203/247) Updating files: 83% (206/247) Updating files: 84% (208/247) Updating files: 85% (210/247) Updating files: 86% (213/247) Updating files: 87% (215/247) Updating files: 88% (218/247) Updating files: 89% (220/247) Updating files: 90% (223/247) Updating files: 91% (225/247) Updating files: 92% (228/247) Updating files: 93% (230/247) Updating files: 94% (233/247) Updating files: 95% (235/247) Updating files: 96% (238/247) Updating files: 97% (240/247) Updating files: 98% (243/247) Updating files: 99% (245/247) Updating files: 100% (247/247) Updating files: 100% (247/247), done. 2024-04-17T20:04:16.994 INFO:tasks.workunit.client.0.smithi029.stdout:HEAD is now at 13b1bf7c492 kclient: initializing kaddr (merge fix) 2024-04-17T20:04:17.000 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/v0.4 2024-04-17T20:04:17.051 INFO:tasks.workunit.client.0.smithi029.stderr:+ [ 5 -eq 48 ] 2024-04-17T20:04:17.051 INFO:tasks.workunit.client.0.smithi029.stderr:+ ver=v0.5 2024-04-17T20:04:17.051 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo v0.5 2024-04-17T20:04:17.051 INFO:tasks.workunit.client.0.smithi029.stderr:+ git reset --hard v0.5 2024-04-17T20:04:17.051 INFO:tasks.workunit.client.0.smithi029.stdout:v0.5 2024-04-17T20:04:19.013 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:19.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.181+0000 7fd1e7600640 1 Processor -- start 2024-04-17T20:04:19.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.181+0000 7fd1e7600640 1 -- start start 2024-04-17T20:04:19.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.181+0000 7fd1e7600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd1e01625e0 0x7fd1e0164a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:19.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.181+0000 7fd1e7600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e00cd0e0 con 0x7fd1e00cccd0 2024-04-17T20:04:19.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.181+0000 7fd1e7600640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e00c88b0 con 0x7fd1e00cc8c0 2024-04-17T20:04:19.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.181+0000 7fd1e7600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fd1e00c6ce0 con 0x7fd1e01625e0 2024-04-17T20:04:19.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.181+0000 7fd1dfe00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd1e00cccd0 0x7fd1e00ca330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51496/0 (socket says 172.21.15.29:51496) 2024-04-17T20:04:19.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.181+0000 7fd1dfe00640 1 -- 172.21.15.29:0/2573477173 learned_addr learned my addr 172.21.15.29:0/2573477173 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:04:19.181 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.181+0000 7fd1e5600640 1 --2- 172.21.15.29:0/2573477173 >> v2:172.21.15.29:3300/0 conn(0x7fd1e01625e0 0x7fd1e0164a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:19.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.181+0000 7fd1e5600640 1 -- 172.21.15.29:0/2573477173 >> v1:172.21.15.37:6789/0 conn(0x7fd1e00cc8c0 legacy=0x7fd1e0159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:19.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.181+0000 7fd1e5600640 1 -- 172.21.15.29:0/2573477173 >> v1:172.21.15.29:6789/0 conn(0x7fd1e00cccd0 legacy=0x7fd1e00ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:19.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.181+0000 7fd1e5600640 1 -- 172.21.15.29:0/2573477173 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e00c6fc0 con 0x7fd1e01625e0 2024-04-17T20:04:19.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.181+0000 7fd1df400640 1 -- 172.21.15.29:0/2573477173 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 715966159 0 0) 0x7fd1e00cd0e0 con 0x7fd1e00cccd0 2024-04-17T20:04:19.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.185+0000 7fd1e5600640 1 --2- 172.21.15.29:0/2573477173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1e01625e0 0x7fd1e0164a00 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8009d90 tx=0x7fd1d801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85dedaf3bc7839bf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:19.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.185+0000 7fd1df400640 1 -- 172.21.15.29:0/2573477173 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d80b2070 con 0x7fd1e01625e0 2024-04-17T20:04:19.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.185+0000 7fd1df400640 1 -- 172.21.15.29:0/2573477173 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1d80a5830 con 0x7fd1e01625e0 2024-04-17T20:04:19.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.185+0000 7fd1df400640 1 -- 172.21.15.29:0/2573477173 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d80a5c00 con 0x7fd1e01625e0 2024-04-17T20:04:19.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.185+0000 7fd1e7600640 1 -- 172.21.15.29:0/2573477173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1e01625e0 msgr2=0x7fd1e0164a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:19.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.185+0000 7fd1e7600640 1 --2- 172.21.15.29:0/2573477173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1e01625e0 0x7fd1e0164a00 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8009d90 tx=0x7fd1d801c970 comp rx=0 tx=0).stop 2024-04-17T20:04:19.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.185+0000 7fd1e7600640 1 -- 172.21.15.29:0/2573477173 shutdown_connections 2024-04-17T20:04:19.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.185+0000 7fd1e7600640 1 --2- 172.21.15.29:0/2573477173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1e01625e0 0x7fd1e0164a00 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:19.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.185+0000 7fd1e7600640 1 -- 172.21.15.29:0/2573477173 >> 172.21.15.29:0/2573477173 conn(0x7fd1e006fde0 msgr2=0x7fd1e00701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:19.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.185+0000 7fd1e7600640 1 -- 172.21.15.29:0/2573477173 shutdown_connections 2024-04-17T20:04:19.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.185+0000 7fd1e7600640 1 -- 172.21.15.29:0/2573477173 wait complete. 2024-04-17T20:04:19.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.189+0000 7fd1e7600640 1 Processor -- start 2024-04-17T20:04:19.188 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.189+0000 7fd1e7600640 1 -- start start 2024-04-17T20:04:19.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.189+0000 7fd1e7600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1e00cc8c0 0x7fd1e01f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:19.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.189+0000 7fd1e7600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1e00cccd0 0x7fd1e0137970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:19.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.189+0000 7fd1e7600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd1e01625e0 0x7fd1e0151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:19.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.189+0000 7fd1e7600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e0159c20 con 0x7fd1e00cc8c0 2024-04-17T20:04:19.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.189+0000 7fd1e7600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e0159aa0 con 0x7fd1e00cccd0 2024-04-17T20:04:19.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.189+0000 7fd1e7600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fd1e00caa40 con 0x7fd1e01625e0 2024-04-17T20:04:19.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.189+0000 7fd1e4c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1e00cc8c0 0x7fd1e01f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:19.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.189+0000 7fd1e4c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1e00cc8c0 0x7fd1e01f6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33950/0 (socket says 172.21.15.29:33950) 2024-04-17T20:04:19.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.189+0000 7fd1e4c00640 1 -- 172.21.15.29:0/305048920 learned_addr learned my addr 172.21.15.29:0/305048920 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:19.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.189+0000 7fd1dfe00640 1 --2- 172.21.15.29:0/305048920 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1e00cccd0 0x7fd1e0137970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:19.190 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.189+0000 7fd1e5600640 1 --2- 172.21.15.29:0/305048920 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd1e01625e0 0x7fd1e0151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:19.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.193+0000 7fd1e4c00640 1 -- 172.21.15.29:0/305048920 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd1e01625e0 msgr2=0x7fd1e0151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:19.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.193+0000 7fd1e4c00640 1 --2- 172.21.15.29:0/305048920 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd1e01625e0 0x7fd1e0151f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:19.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.193+0000 7fd1e4c00640 1 -- 172.21.15.29:0/305048920 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1e00cccd0 msgr2=0x7fd1e0137970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:19.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.193+0000 7fd1e4c00640 1 --2- 172.21.15.29:0/305048920 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1e00cccd0 0x7fd1e0137970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:19.191 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.193+0000 7fd1e4c00640 1 -- 172.21.15.29:0/305048920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e01f7090 con 0x7fd1e00cc8c0 2024-04-17T20:04:19.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.193+0000 7fd1e4c00640 1 --2- 172.21.15.29:0/305048920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1e00cc8c0 0x7fd1e01f6b50 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd1d400c3c0 tx=0x7fd1d400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:19.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.193+0000 7fd1dcc00640 1 -- 172.21.15.29:0/305048920 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4017070 con 0x7fd1e00cc8c0 2024-04-17T20:04:19.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.193+0000 7fd1dcc00640 1 -- 172.21.15.29:0/305048920 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1d40052d0 con 0x7fd1e00cc8c0 2024-04-17T20:04:19.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.193+0000 7fd1e7600640 1 -- 172.21.15.29:0/305048920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e0137eb0 con 0x7fd1e00cc8c0 2024-04-17T20:04:19.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.193+0000 7fd1dcc00640 1 -- 172.21.15.29:0/305048920 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d4003840 con 0x7fd1e00cc8c0 2024-04-17T20:04:19.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.193+0000 7fd1e7600640 1 -- 172.21.15.29:0/305048920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e013b7a0 con 0x7fd1e00cc8c0 2024-04-17T20:04:19.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.193+0000 7fd1e7600640 1 -- 172.21.15.29:0/305048920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1e0069b90 con 0x7fd1e00cc8c0 2024-04-17T20:04:19.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.193+0000 7fd1dcc00640 1 -- 172.21.15.29:0/305048920 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fd1d401a070 con 0x7fd1e00cc8c0 2024-04-17T20:04:19.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.193+0000 7fd1dcc00640 1 --2- 172.21.15.29:0/305048920 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd1c0068380 0x7fd1c006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:19.194 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.193+0000 7fd1dcc00640 1 -- 172.21.15.29:0/305048920 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7fd1d4086420 con 0x7fd1e00cc8c0 2024-04-17T20:04:19.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.197+0000 7fd1dfe00640 1 --2- 172.21.15.29:0/305048920 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd1c0068380 0x7fd1c006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:19.195 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.197+0000 7fd1dfe00640 1 --2- 172.21.15.29:0/305048920 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd1c0068380 0x7fd1c006a890 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00cbc00 tx=0x7fd1cc007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:19.198 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.201+0000 7fd1dcc00640 1 -- 172.21.15.29:0/305048920 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fd1d4005480 con 0x7fd1e00cc8c0 2024-04-17T20:04:19.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.397+0000 7fd1e7600640 1 -- 172.21.15.29:0/305048920 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1e00c7e10 con 0x7fd1e00cc8c0 2024-04-17T20:04:19.397 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.397+0000 7fd1dcc00640 1 -- 172.21.15.29:0/305048920 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6913 (secure 0 0 0) 0x7fd1d40562e0 con 0x7fd1e00cc8c0 2024-04-17T20:04:19.397 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:19.397 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6076,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:25.246297+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5946,5775,5631,5350,5212]},"id":1}]} 2024-04-17T20:04:19.398 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 62 2024-04-17T20:04:19.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.401+0000 7fd1e7600640 1 -- 172.21.15.29:0/305048920 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd1c0068380 msgr2=0x7fd1c006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:19.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.401+0000 7fd1e7600640 1 --2- 172.21.15.29:0/305048920 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd1c0068380 0x7fd1c006a890 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd1e00cbc00 tx=0x7fd1cc007d30 comp rx=0 tx=0).stop 2024-04-17T20:04:19.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.401+0000 7fd1e7600640 1 -- 172.21.15.29:0/305048920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1e00cc8c0 msgr2=0x7fd1e01f6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:19.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.401+0000 7fd1e7600640 1 --2- 172.21.15.29:0/305048920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1e00cc8c0 0x7fd1e01f6b50 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd1d400c3c0 tx=0x7fd1d400c880 comp rx=0 tx=0).stop 2024-04-17T20:04:19.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.401+0000 7fd1e7600640 1 -- 172.21.15.29:0/305048920 shutdown_connections 2024-04-17T20:04:19.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.401+0000 7fd1e7600640 1 --2- 172.21.15.29:0/305048920 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fd1c0068380 0x7fd1c006a890 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:19.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.401+0000 7fd1e7600640 1 --2- 172.21.15.29:0/305048920 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fd1e01625e0 0x7fd1e0151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:19.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.401+0000 7fd1e7600640 1 --2- 172.21.15.29:0/305048920 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fd1e00cccd0 0x7fd1e0137970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:19.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.401+0000 7fd1e7600640 1 --2- 172.21.15.29:0/305048920 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1e00cc8c0 0x7fd1e01f6b50 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:19.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.401+0000 7fd1e7600640 1 -- 172.21.15.29:0/305048920 >> 172.21.15.29:0/305048920 conn(0x7fd1e006fde0 msgr2=0x7fd1e0159390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:19.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.401+0000 7fd1e7600640 1 -- 172.21.15.29:0/305048920 shutdown_connections 2024-04-17T20:04:19.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.401+0000 7fd1e7600640 1 -- 172.21.15.29:0/305048920 wait complete. 2024-04-17T20:04:19.419 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=4) 2024-04-17T20:04:19.420 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T20:04:19.420 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T20:04:19.447 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T20:04:19.447 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:19.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.601+0000 7f5cd6800640 1 Processor -- start 2024-04-17T20:04:19.601 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.601+0000 7f5cd6800640 1 -- start start 2024-04-17T20:04:19.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.605+0000 7f5cd6800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5cd0159c90 0x7f5cd00ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:19.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.605+0000 7f5cd6800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5cd006c660 0x7f5cd006ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:19.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.605+0000 7f5cd6800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cd015a0a0 con 0x7f5cd00ca7f0 2024-04-17T20:04:19.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.605+0000 7f5cd6800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f5cd015abe0 con 0x7f5cd006c660 2024-04-17T20:04:19.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.605+0000 7f5cd6800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5cd015a340 con 0x7f5cd0159c90 2024-04-17T20:04:19.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.605+0000 7f5ccfe00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5cd006c660 0x7f5cd006ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:19.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.605+0000 7f5ccf400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5cd0159c90 0x7f5cd00ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:19.607 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.605+0000 7f5ccfe00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f5cd006c660 0x7f5cd006ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:53178/0 (socket says 172.21.15.29:53178) 2024-04-17T20:04:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.605+0000 7f5ccf400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5cd0159c90 0x7f5cd00ca2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:42964/0 (socket says 172.21.15.29:42964) 2024-04-17T20:04:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.605+0000 7f5ccfe00640 1 -- 172.21.15.29:0/1009711688 learned_addr learned my addr 172.21.15.29:0/1009711688 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.605+0000 7f5ccf400640 1 -- 172.21.15.29:0/1009711688 >> v2:172.21.15.37:3300/0 conn(0x7f5cd006c660 msgr2=0x7f5cd006ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.605+0000 7f5ccf400640 1 --2- 172.21.15.29:0/1009711688 >> v2:172.21.15.37:3300/0 conn(0x7f5cd006c660 0x7f5cd006ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.605+0000 7f5ccf400640 1 -- 172.21.15.29:0/1009711688 >> v1:172.21.15.37:6789/0 conn(0x7f5cd00ca7f0 legacy=0x7f5cd00c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.605+0000 7f5ccf400640 1 -- 172.21.15.29:0/1009711688 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cd015a620 con 0x7f5cd0159c90 2024-04-17T20:04:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.605+0000 7f5ccfe00640 1 --2- 172.21.15.29:0/1009711688 >> v2:172.21.15.37:3300/0 conn(0x7f5cd006c660 0x7f5cd006ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T20:04:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.609+0000 7f5ccf400640 1 --2- 172.21.15.29:0/1009711688 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5cd0159c90 0x7f5cd00ca2b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5cc00097c0 tx=0x7f5cc001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1673820498ec3b7b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.609+0000 7f5cce000640 1 -- 172.21.15.29:0/1009711688 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc00b1070 con 0x7f5cd0159c90 2024-04-17T20:04:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.609+0000 7f5cce000640 1 -- 172.21.15.29:0/1009711688 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cc00ac410 con 0x7f5cd0159c90 2024-04-17T20:04:19.608 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.609+0000 7f5cce000640 1 -- 172.21.15.29:0/1009711688 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc00ac7e0 con 0x7f5cd0159c90 2024-04-17T20:04:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.609+0000 7f5cd6800640 1 -- 172.21.15.29:0/1009711688 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5cd0159c90 msgr2=0x7f5cd00ca2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.609+0000 7f5cd6800640 1 --2- 172.21.15.29:0/1009711688 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5cd0159c90 0x7f5cd00ca2b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5cc00097c0 tx=0x7f5cc001c4a0 comp rx=0 tx=0).stop 2024-04-17T20:04:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.609+0000 7f5cd6800640 1 -- 172.21.15.29:0/1009711688 shutdown_connections 2024-04-17T20:04:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.609+0000 7f5cd6800640 1 --2- 172.21.15.29:0/1009711688 >> v2:172.21.15.37:3300/0 conn(0x7f5cd006c660 0x7f5cd006ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.609+0000 7f5cd6800640 1 --2- 172.21.15.29:0/1009711688 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5cd0159c90 0x7f5cd00ca2b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.609+0000 7f5cd6800640 1 -- 172.21.15.29:0/1009711688 >> 172.21.15.29:0/1009711688 conn(0x7f5cd006fde0 msgr2=0x7f5cd00701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:19.609 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.609+0000 7f5cd6800640 1 -- 172.21.15.29:0/1009711688 shutdown_connections 2024-04-17T20:04:19.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.609+0000 7f5cd6800640 1 -- 172.21.15.29:0/1009711688 wait complete. 2024-04-17T20:04:19.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5cd6800640 1 Processor -- start 2024-04-17T20:04:19.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5cd6800640 1 -- start start 2024-04-17T20:04:19.612 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5cd6800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cd006c660 0x7f5cd014bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5cd6800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5cd00ca7f0 0x7f5cd0148ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5cd6800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5cd0159c90 0x7f5cd01f8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5cd6800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd00c7680 con 0x7f5cd006c660 2024-04-17T20:04:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5cd6800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f5cd006cfb0 con 0x7f5cd00ca7f0 2024-04-17T20:04:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5cd6800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f5cd00c7800 con 0x7f5cd0159c90 2024-04-17T20:04:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5ccf400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cd006c660 0x7f5cd014bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5ccf400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cd006c660 0x7f5cd014bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33956/0 (socket says 172.21.15.29:33956) 2024-04-17T20:04:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5ccf400640 1 -- 172.21.15.29:0/2198319575 learned_addr learned my addr 172.21.15.29:0/2198319575 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5ccea00640 1 --2- 172.21.15.29:0/2198319575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5cd00ca7f0 0x7f5cd0148ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5ccfe00640 1 --2- 172.21.15.29:0/2198319575 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5cd0159c90 0x7f5cd01f8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5ccf400640 1 -- 172.21.15.29:0/2198319575 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5cd0159c90 msgr2=0x7f5cd01f8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5ccf400640 1 --2- 172.21.15.29:0/2198319575 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5cd0159c90 0x7f5cd01f8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5ccf400640 1 -- 172.21.15.29:0/2198319575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5cd00ca7f0 msgr2=0x7f5cd0148ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5ccf400640 1 --2- 172.21.15.29:0/2198319575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5cd00ca7f0 0x7f5cd0148ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:19.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5ccf400640 1 -- 172.21.15.29:0/2198319575 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cd014c4e0 con 0x7f5cd006c660 2024-04-17T20:04:19.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5ccea00640 1 --2- 172.21.15.29:0/2198319575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5cd00ca7f0 0x7f5cd0148ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:04:19.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5ccf400640 1 --2- 172.21.15.29:0/2198319575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cd006c660 0x7f5cd014bfa0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5cc001a0d0 tx=0x7f5cc0019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:19.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5cc7400640 1 -- 172.21.15.29:0/2198319575 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc00b1040 con 0x7f5cd006c660 2024-04-17T20:04:19.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5cd6800640 1 -- 172.21.15.29:0/2198319575 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cd00cc790 con 0x7f5cd006c660 2024-04-17T20:04:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5cc7400640 1 -- 172.21.15.29:0/2198319575 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cc00bb070 con 0x7f5cd006c660 2024-04-17T20:04:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5cc7400640 1 -- 172.21.15.29:0/2198319575 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc00b5410 con 0x7f5cd006c660 2024-04-17T20:04:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.613+0000 7f5cd6800640 1 -- 172.21.15.29:0/2198319575 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cd014a6f0 con 0x7f5cd006c660 2024-04-17T20:04:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.617+0000 7f5cd6800640 1 -- 172.21.15.29:0/2198319575 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cd0069b90 con 0x7f5cd006c660 2024-04-17T20:04:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.617+0000 7f5cc7400640 1 -- 172.21.15.29:0/2198319575 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f5cc00b55b0 con 0x7f5cd006c660 2024-04-17T20:04:19.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.617+0000 7f5cc7400640 1 --2- 172.21.15.29:0/2198319575 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5cb00683b0 0x7f5cb006a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:19.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.617+0000 7f5cc7400640 1 -- 172.21.15.29:0/2198319575 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7f5cc0120b60 con 0x7f5cd006c660 2024-04-17T20:04:19.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.617+0000 7f5ccea00640 1 --2- 172.21.15.29:0/2198319575 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5cb00683b0 0x7f5cb006a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:19.617 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.617+0000 7f5ccea00640 1 --2- 172.21.15.29:0/2198319575 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5cb00683b0 0x7f5cb006a8c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5cd014b340 tx=0x7f5cbc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:19.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.621+0000 7f5cc7400640 1 -- 172.21.15.29:0/2198319575 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f5cc0016090 con 0x7f5cd006c660 2024-04-17T20:04:19.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.821+0000 7f5cd6800640 1 -- 172.21.15.29:0/2198319575 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5cd014b8e0 con 0x7f5cd006c660 2024-04-17T20:04:19.823 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.825+0000 7f5cc7400640 1 -- 172.21.15.29:0/2198319575 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6913 (secure 0 0 0) 0x7f5cc00f0910 con 0x7f5cd006c660 2024-04-17T20:04:19.823 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:19.823 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6076,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:25.246297+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5946,5775,5631,5350,5212]},"id":1}]} 2024-04-17T20:04:19.823 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 62 2024-04-17T20:04:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.825+0000 7f5cd6800640 1 -- 172.21.15.29:0/2198319575 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5cb00683b0 msgr2=0x7f5cb006a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.825+0000 7f5cd6800640 1 --2- 172.21.15.29:0/2198319575 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5cb00683b0 0x7f5cb006a8c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5cd014b340 tx=0x7f5cbc009040 comp rx=0 tx=0).stop 2024-04-17T20:04:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.825+0000 7f5cd6800640 1 -- 172.21.15.29:0/2198319575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cd006c660 msgr2=0x7f5cd014bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.825+0000 7f5cd6800640 1 --2- 172.21.15.29:0/2198319575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cd006c660 0x7f5cd014bfa0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5cc001a0d0 tx=0x7f5cc0019240 comp rx=0 tx=0).stop 2024-04-17T20:04:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.825+0000 7f5cd6800640 1 -- 172.21.15.29:0/2198319575 shutdown_connections 2024-04-17T20:04:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.825+0000 7f5cd6800640 1 --2- 172.21.15.29:0/2198319575 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f5cb00683b0 0x7f5cb006a8c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.825+0000 7f5cd6800640 1 --2- 172.21.15.29:0/2198319575 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f5cd0159c90 0x7f5cd01f8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.825+0000 7f5cd6800640 1 --2- 172.21.15.29:0/2198319575 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f5cd00ca7f0 0x7f5cd0148ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.825+0000 7f5cd6800640 1 --2- 172.21.15.29:0/2198319575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5cd006c660 0x7f5cd014bfa0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.825+0000 7f5cd6800640 1 -- 172.21.15.29:0/2198319575 >> 172.21.15.29:0/2198319575 conn(0x7f5cd006fde0 msgr2=0x7f5cd01590c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.825+0000 7f5cd6800640 1 -- 172.21.15.29:0/2198319575 shutdown_connections 2024-04-17T20:04:19.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:19.825+0000 7f5cd6800640 1 -- 172.21.15.29:0/2198319575 wait complete. 2024-04-17T20:04:19.844 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-17T20:04:19.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-04-17T20:04:21.846 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:22.015 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.017+0000 7efe18800640 1 Processor -- start 2024-04-17T20:04:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.017+0000 7efe18800640 1 -- start start 2024-04-17T20:04:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.017+0000 7efe18800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe100cc8c0 0x7efe100cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.017+0000 7efe18800640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7efe10159390 0x7efe101597a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.017+0000 7efe18800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efe100ca330 0x7efe100ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.017+0000 7efe18800640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7efe100c88b0 con 0x7efe100ca330 2024-04-17T20:04:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.017+0000 7efe18800640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7efe100c6ce0 con 0x7efe10159390 2024-04-17T20:04:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.017+0000 7efe18800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7efe100c6fc0 con 0x7efe100cc8c0 2024-04-17T20:04:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.017+0000 7efe16800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efe100ca330 0x7efe100ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.017+0000 7efe16800640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efe100ca330 0x7efe100ca740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33972/0 (socket says 172.21.15.29:33972) 2024-04-17T20:04:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.017+0000 7efe16800640 1 -- 172.21.15.29:0/2229039373 learned_addr learned my addr 172.21.15.29:0/2229039373 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.017+0000 7efe15e00640 1 --2- 172.21.15.29:0/2229039373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe100cc8c0 0x7efe100cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.017+0000 7efe15400640 1 --2- 172.21.15.29:0/2229039373 >> v2:172.21.15.37:3300/0 conn(0x7efe10159390 0x7efe101597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.021+0000 7efe16800640 1 -- 172.21.15.29:0/2229039373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe100cc8c0 msgr2=0x7efe100cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.021+0000 7efe16800640 1 --2- 172.21.15.29:0/2229039373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe100cc8c0 0x7efe100cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.021+0000 7efe16800640 1 -- 172.21.15.29:0/2229039373 >> v2:172.21.15.37:3300/0 conn(0x7efe10159390 msgr2=0x7efe101597a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.021+0000 7efe16800640 1 --2- 172.21.15.29:0/2229039373 >> v2:172.21.15.37:3300/0 conn(0x7efe10159390 0x7efe101597a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.021+0000 7efe16800640 1 -- 172.21.15.29:0/2229039373 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe100c72a0 con 0x7efe100ca330 2024-04-17T20:04:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.021+0000 7efe15400640 1 --2- 172.21.15.29:0/2229039373 >> v2:172.21.15.37:3300/0 conn(0x7efe10159390 0x7efe101597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:04:22.021 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.021+0000 7efe16800640 1 --2- 172.21.15.29:0/2229039373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe100ca330 0x7efe100ca740 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7efe100ce400 tx=0x7efe0c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bcfc23a5a747fc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.021+0000 7efe14a00640 1 -- 172.21.15.29:0/2229039373 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c0a5670 con 0x7efe100ca330 2024-04-17T20:04:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.021+0000 7efe14a00640 1 -- 172.21.15.29:0/2229039373 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe0c0a5810 con 0x7efe100ca330 2024-04-17T20:04:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.021+0000 7efe14a00640 1 -- 172.21.15.29:0/2229039373 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c0a5be0 con 0x7efe100ca330 2024-04-17T20:04:22.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.025+0000 7efe18800640 1 -- 172.21.15.29:0/2229039373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe100ca330 msgr2=0x7efe100ca740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:22.023 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.025+0000 7efe18800640 1 --2- 172.21.15.29:0/2229039373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe100ca330 0x7efe100ca740 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7efe100ce400 tx=0x7efe0c01c9f0 comp rx=0 tx=0).stop 2024-04-17T20:04:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.025+0000 7efe18800640 1 -- 172.21.15.29:0/2229039373 shutdown_connections 2024-04-17T20:04:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.025+0000 7efe18800640 1 --2- 172.21.15.29:0/2229039373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe100ca330 0x7efe100ca740 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.025+0000 7efe18800640 1 --2- 172.21.15.29:0/2229039373 >> v2:172.21.15.37:3300/0 conn(0x7efe10159390 0x7efe101597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.025+0000 7efe18800640 1 --2- 172.21.15.29:0/2229039373 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe100cc8c0 0x7efe100cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.025+0000 7efe18800640 1 -- 172.21.15.29:0/2229039373 >> 172.21.15.29:0/2229039373 conn(0x7efe1006fde0 msgr2=0x7efe100701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.025+0000 7efe18800640 1 -- 172.21.15.29:0/2229039373 shutdown_connections 2024-04-17T20:04:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.025+0000 7efe18800640 1 -- 172.21.15.29:0/2229039373 wait complete. 2024-04-17T20:04:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.025+0000 7efe18800640 1 Processor -- start 2024-04-17T20:04:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.025+0000 7efe18800640 1 -- start start 2024-04-17T20:04:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe18800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe100ca330 0x7efe1015aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe18800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe100cc8c0 0x7efe101f8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe18800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe10159390 0x7efe10162ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe18800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7efe1015ca60 con 0x7efe100ca330 2024-04-17T20:04:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe18800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7efe100c6a90 con 0x7efe10159390 2024-04-17T20:04:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe18800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7efe1015cbe0 con 0x7efe100cc8c0 2024-04-17T20:04:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe15e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe100ca330 0x7efe1015aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe15e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe100ca330 0x7efe1015aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33980/0 (socket says 172.21.15.29:33980) 2024-04-17T20:04:22.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe15e00640 1 -- 172.21.15.29:0/2763811498 learned_addr learned my addr 172.21.15.29:0/2763811498 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:22.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe15400640 1 --2- 172.21.15.29:0/2763811498 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe100cc8c0 0x7efe101f8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:22.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe16800640 1 --2- 172.21.15.29:0/2763811498 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe10159390 0x7efe10162ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:22.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe15e00640 1 -- 172.21.15.29:0/2763811498 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe100cc8c0 msgr2=0x7efe101f8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:22.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe15e00640 1 --2- 172.21.15.29:0/2763811498 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe100cc8c0 0x7efe101f8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:22.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe15e00640 1 -- 172.21.15.29:0/2763811498 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe10159390 msgr2=0x7efe10162ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:22.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe15e00640 1 --2- 172.21.15.29:0/2763811498 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe10159390 0x7efe10162ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:22.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe15e00640 1 -- 172.21.15.29:0/2763811498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe101f8f70 con 0x7efe100ca330 2024-04-17T20:04:22.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe15e00640 1 --2- 172.21.15.29:0/2763811498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe100ca330 0x7efe1015aaa0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7efe0400e870 tx=0x7efe0400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:22.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe0a000640 1 -- 172.21.15.29:0/2763811498 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe04017070 con 0x7efe100ca330 2024-04-17T20:04:22.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe0a000640 1 -- 172.21.15.29:0/2763811498 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe0400ee60 con 0x7efe100ca330 2024-04-17T20:04:22.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe18800640 1 -- 172.21.15.29:0/2763811498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe101f5e20 con 0x7efe100ca330 2024-04-17T20:04:22.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe0a000640 1 -- 172.21.15.29:0/2763811498 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0400cb80 con 0x7efe100ca330 2024-04-17T20:04:22.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe18800640 1 -- 172.21.15.29:0/2763811498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe10156ee0 con 0x7efe100ca330 2024-04-17T20:04:22.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.029+0000 7efe18800640 1 -- 172.21.15.29:0/2763811498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe101f8670 con 0x7efe100ca330 2024-04-17T20:04:22.030 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.033+0000 7efe0a000640 1 -- 172.21.15.29:0/2763811498 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7efe0401a070 con 0x7efe100ca330 2024-04-17T20:04:22.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.033+0000 7efe0a000640 1 --2- 172.21.15.29:0/2763811498 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efdf4068380 0x7efdf406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:22.031 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.033+0000 7efe0a000640 1 -- 172.21.15.29:0/2763811498 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7efe04087250 con 0x7efe100ca330 2024-04-17T20:04:22.032 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.033+0000 7efe15400640 1 --2- 172.21.15.29:0/2763811498 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efdf4068380 0x7efdf406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:22.033 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.033+0000 7efe15400640 1 --2- 172.21.15.29:0/2763811498 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efdf4068380 0x7efdf406a890 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7efe10139d40 tx=0x7efe00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:22.036 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.037+0000 7efe0a000640 1 -- 172.21.15.29:0/2763811498 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7efe040571b0 con 0x7efe100ca330 2024-04-17T20:04:22.236 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.237+0000 7efe18800640 1 -- 172.21.15.29:0/2763811498 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe10156a30 con 0x7efe100ca330 2024-04-17T20:04:22.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.237+0000 7efe0a000640 1 -- 172.21.15.29:0/2763811498 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6913 (secure 0 0 0) 0x7efe04056ab0 con 0x7efe100ca330 2024-04-17T20:04:22.237 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:22.237 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6076,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:25.246297+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5946,5775,5631,5350,5212]},"id":1}]} 2024-04-17T20:04:22.238 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 62 2024-04-17T20:04:22.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.241+0000 7efe18800640 1 -- 172.21.15.29:0/2763811498 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efdf4068380 msgr2=0x7efdf406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.241+0000 7efe18800640 1 --2- 172.21.15.29:0/2763811498 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efdf4068380 0x7efdf406a890 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7efe10139d40 tx=0x7efe00008040 comp rx=0 tx=0).stop 2024-04-17T20:04:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.241+0000 7efe18800640 1 -- 172.21.15.29:0/2763811498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe100ca330 msgr2=0x7efe1015aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.241+0000 7efe18800640 1 --2- 172.21.15.29:0/2763811498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe100ca330 0x7efe1015aaa0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7efe0400e870 tx=0x7efe0400ed30 comp rx=0 tx=0).stop 2024-04-17T20:04:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.241+0000 7efe18800640 1 -- 172.21.15.29:0/2763811498 shutdown_connections 2024-04-17T20:04:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.241+0000 7efe18800640 1 --2- 172.21.15.29:0/2763811498 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7efdf4068380 0x7efdf406a890 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:22.240 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.241+0000 7efe18800640 1 --2- 172.21.15.29:0/2763811498 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7efe10159390 0x7efe10162ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.241+0000 7efe18800640 1 --2- 172.21.15.29:0/2763811498 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7efe100cc8c0 0x7efe101f8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.241+0000 7efe18800640 1 --2- 172.21.15.29:0/2763811498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe100ca330 0x7efe1015aaa0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.241+0000 7efe18800640 1 -- 172.21.15.29:0/2763811498 >> 172.21.15.29:0/2763811498 conn(0x7efe1006fde0 msgr2=0x7efe10140da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.241+0000 7efe18800640 1 -- 172.21.15.29:0/2763811498 shutdown_connections 2024-04-17T20:04:22.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:22.241+0000 7efe18800640 1 -- 172.21.15.29:0/2763811498 wait complete. 2024-04-17T20:04:22.259 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-17T20:04:24.260 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:24.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.421+0000 7fa322a00640 1 Processor -- start 2024-04-17T20:04:24.420 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.421+0000 7fa322a00640 1 -- start start 2024-04-17T20:04:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.421+0000 7fa322a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa31c155da0 0x7fa31c1561b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.425+0000 7fa322a00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa31c1566f0 con 0x7fa31c0cbf90 2024-04-17T20:04:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.425+0000 7fa322a00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa31c0cbba0 con 0x7fa31c0c7f50 2024-04-17T20:04:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.425+0000 7fa322a00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fa31c06bdc0 con 0x7fa31c155da0 2024-04-17T20:04:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.425+0000 7fa31be00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa31c0cbf90 0x7fa31c15ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51504/0 (socket says 172.21.15.29:51504) 2024-04-17T20:04:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.425+0000 7fa31be00640 1 -- 172.21.15.29:0/2132722456 learned_addr learned my addr 172.21.15.29:0/2132722456 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:04:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.425+0000 7fa320a00640 1 --2- 172.21.15.29:0/2132722456 >> v2:172.21.15.29:3300/0 conn(0x7fa31c155da0 0x7fa31c1561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.425+0000 7fa31b400640 1 -- 172.21.15.29:0/2132722456 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1633425494 0 0) 0x7fa31c1566f0 con 0x7fa31c0cbf90 2024-04-17T20:04:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.425+0000 7fa31b400640 1 -- 172.21.15.29:0/2132722456 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa300003530 con 0x7fa31c0cbf90 2024-04-17T20:04:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.425+0000 7fa320a00640 1 -- 172.21.15.29:0/2132722456 >> v1:172.21.15.37:6789/0 conn(0x7fa31c0c7f50 legacy=0x7fa31c0ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.425+0000 7fa320a00640 1 -- 172.21.15.29:0/2132722456 >> v1:172.21.15.29:6789/0 conn(0x7fa31c0cbf90 legacy=0x7fa31c15ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.425+0000 7fa320a00640 1 -- 172.21.15.29:0/2132722456 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa31c06c010 con 0x7fa31c155da0 2024-04-17T20:04:24.425 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.425+0000 7fa320a00640 1 --2- 172.21.15.29:0/2132722456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa31c155da0 0x7fa31c1561b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fa314009d90 tx=0x7fa31401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ddde17fac8915e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.425+0000 7fa31b400640 1 -- 172.21.15.29:0/2132722456 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3140b2070 con 0x7fa31c155da0 2024-04-17T20:04:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.425+0000 7fa31b400640 1 -- 172.21.15.29:0/2132722456 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3140a5830 con 0x7fa31c155da0 2024-04-17T20:04:24.426 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.425+0000 7fa31b400640 1 -- 172.21.15.29:0/2132722456 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3140a5c00 con 0x7fa31c155da0 2024-04-17T20:04:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.429+0000 7fa322a00640 1 -- 172.21.15.29:0/2132722456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa31c155da0 msgr2=0x7fa31c1561b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.429+0000 7fa322a00640 1 --2- 172.21.15.29:0/2132722456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa31c155da0 0x7fa31c1561b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fa314009d90 tx=0x7fa31401c970 comp rx=0 tx=0).stop 2024-04-17T20:04:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.429+0000 7fa322a00640 1 -- 172.21.15.29:0/2132722456 shutdown_connections 2024-04-17T20:04:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.429+0000 7fa322a00640 1 --2- 172.21.15.29:0/2132722456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa31c155da0 0x7fa31c1561b0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.429+0000 7fa322a00640 1 -- 172.21.15.29:0/2132722456 >> 172.21.15.29:0/2132722456 conn(0x7fa31c06fde0 msgr2=0x7fa31c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.429+0000 7fa322a00640 1 -- 172.21.15.29:0/2132722456 shutdown_connections 2024-04-17T20:04:24.427 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.429+0000 7fa322a00640 1 -- 172.21.15.29:0/2132722456 wait complete. 2024-04-17T20:04:24.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.429+0000 7fa322a00640 1 Processor -- start 2024-04-17T20:04:24.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.429+0000 7fa322a00640 1 -- start start 2024-04-17T20:04:24.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa322a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa31c0c7f50 0x7fa31c1400d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:24.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa322a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa31c0cbf90 0x7fa31c13b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:24.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa322a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa31c155da0 0x7fa31c13bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:24.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa322a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c0cb8e0 con 0x7fa31c155da0 2024-04-17T20:04:24.431 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa322a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c0cb760 con 0x7fa31c0c7f50 2024-04-17T20:04:24.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa322a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa31c16a5a0 con 0x7fa31c0cbf90 2024-04-17T20:04:24.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa320a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa31c155da0 0x7fa31c13bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:24.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa320a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa31c155da0 0x7fa31c13bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33998/0 (socket says 172.21.15.29:33998) 2024-04-17T20:04:24.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa320a00640 1 -- 172.21.15.29:0/2163057052 learned_addr learned my addr 172.21.15.29:0/2163057052 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:24.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa313e00640 1 --2- 172.21.15.29:0/2163057052 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa31c0cbf90 0x7fa31c13b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:24.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa31be00640 1 --2- 172.21.15.29:0/2163057052 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa31c0c7f50 0x7fa31c1400d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:24.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa320a00640 1 -- 172.21.15.29:0/2163057052 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa31c0cbf90 msgr2=0x7fa31c13b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:24.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa320a00640 1 --2- 172.21.15.29:0/2163057052 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa31c0cbf90 0x7fa31c13b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:24.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa320a00640 1 -- 172.21.15.29:0/2163057052 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa31c0c7f50 msgr2=0x7fa31c1400d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:24.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa320a00640 1 --2- 172.21.15.29:0/2163057052 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa31c0c7f50 0x7fa31c1400d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:24.432 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa320a00640 1 -- 172.21.15.29:0/2163057052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa31c140610 con 0x7fa31c155da0 2024-04-17T20:04:24.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa320a00640 1 --2- 172.21.15.29:0/2163057052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa31c155da0 0x7fa31c13bea0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fa3140b0040 tx=0x7fa314019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:24.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa318c00640 1 -- 172.21.15.29:0/2163057052 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3140b2050 con 0x7fa31c155da0 2024-04-17T20:04:24.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa318c00640 1 -- 172.21.15.29:0/2163057052 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3140be070 con 0x7fa31c155da0 2024-04-17T20:04:24.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa318c00640 1 -- 172.21.15.29:0/2163057052 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3140ad870 con 0x7fa31c155da0 2024-04-17T20:04:24.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa322a00640 1 -- 172.21.15.29:0/2163057052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa31c1478a0 con 0x7fa31c155da0 2024-04-17T20:04:24.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa322a00640 1 -- 172.21.15.29:0/2163057052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa31c146f50 con 0x7fa31c155da0 2024-04-17T20:04:24.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.433+0000 7fa322a00640 1 -- 172.21.15.29:0/2163057052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa31c147610 con 0x7fa31c155da0 2024-04-17T20:04:24.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.437+0000 7fa318c00640 1 -- 172.21.15.29:0/2163057052 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa3140ada10 con 0x7fa31c155da0 2024-04-17T20:04:24.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.437+0000 7fa318c00640 1 --2- 172.21.15.29:0/2163057052 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa2f8068380 0x7fa2f806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:24.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.437+0000 7fa318c00640 1 -- 172.21.15.29:0/2163057052 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7fa314121ff0 con 0x7fa31c155da0 2024-04-17T20:04:24.435 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.437+0000 7fa31be00640 1 --2- 172.21.15.29:0/2163057052 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa2f8068380 0x7fa2f806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:24.436 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.437+0000 7fa31be00640 1 --2- 172.21.15.29:0/2163057052 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa2f8068380 0x7fa2f806a890 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa30c0011f0 tx=0x7fa30c007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:24.439 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.441+0000 7fa318c00640 1 -- 172.21.15.29:0/2163057052 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa314016560 con 0x7fa31c155da0 2024-04-17T20:04:24.638 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.641+0000 7fa322a00640 1 -- 172.21.15.29:0/2163057052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa31c069b90 con 0x7fa31c155da0 2024-04-17T20:04:24.639 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.641+0000 7fa318c00640 1 -- 172.21.15.29:0/2163057052 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6913 (secure 0 0 0) 0x7fa3140f18f0 con 0x7fa31c155da0 2024-04-17T20:04:24.639 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:24.639 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6076,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:25.246297+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5946,5775,5631,5350,5212]},"id":1}]} 2024-04-17T20:04:24.639 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 62 2024-04-17T20:04:24.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.641+0000 7fa322a00640 1 -- 172.21.15.29:0/2163057052 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa2f8068380 msgr2=0x7fa2f806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:24.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.641+0000 7fa322a00640 1 --2- 172.21.15.29:0/2163057052 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa2f8068380 0x7fa2f806a890 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fa30c0011f0 tx=0x7fa30c007d40 comp rx=0 tx=0).stop 2024-04-17T20:04:24.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.641+0000 7fa322a00640 1 -- 172.21.15.29:0/2163057052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa31c155da0 msgr2=0x7fa31c13bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:24.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.641+0000 7fa322a00640 1 --2- 172.21.15.29:0/2163057052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa31c155da0 0x7fa31c13bea0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fa3140b0040 tx=0x7fa314019710 comp rx=0 tx=0).stop 2024-04-17T20:04:24.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.641+0000 7fa322a00640 1 -- 172.21.15.29:0/2163057052 shutdown_connections 2024-04-17T20:04:24.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.641+0000 7fa322a00640 1 --2- 172.21.15.29:0/2163057052 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa2f8068380 0x7fa2f806a890 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:24.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.641+0000 7fa322a00640 1 --2- 172.21.15.29:0/2163057052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa31c155da0 0x7fa31c13bea0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:24.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.641+0000 7fa322a00640 1 --2- 172.21.15.29:0/2163057052 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa31c0cbf90 0x7fa31c13b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:24.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.641+0000 7fa322a00640 1 --2- 172.21.15.29:0/2163057052 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa31c0c7f50 0x7fa31c1400d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:24.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.641+0000 7fa322a00640 1 -- 172.21.15.29:0/2163057052 >> 172.21.15.29:0/2163057052 conn(0x7fa31c06fde0 msgr2=0x7fa31c1561b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:24.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.641+0000 7fa322a00640 1 -- 172.21.15.29:0/2163057052 shutdown_connections 2024-04-17T20:04:24.642 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:24.641+0000 7fa322a00640 1 -- 172.21.15.29:0/2163057052 wait complete. 2024-04-17T20:04:24.661 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-17T20:04:26.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:26.670 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-04-17T20:04:26.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.821+0000 7feb4fa00640 1 Processor -- start 2024-04-17T20:04:26.822 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.821+0000 7feb4fa00640 1 -- start start 2024-04-17T20:04:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb4fa00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7feb481523d0 0x7feb481527e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb4fa00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb480c60e0 con 0x7feb480c75c0 2024-04-17T20:04:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb4fa00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb480c6350 con 0x7feb480c71b0 2024-04-17T20:04:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb4fa00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7feb4815ca30 con 0x7feb481523d0 2024-04-17T20:04:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb3fe00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7feb480c75c0 0x7feb4806bf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51516/0 (socket says 172.21.15.29:51516) 2024-04-17T20:04:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb3fe00640 1 -- 172.21.15.29:0/2310287175 learned_addr learned my addr 172.21.15.29:0/2310287175 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:04:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb4da00640 1 --2- 172.21.15.29:0/2310287175 >> v2:172.21.15.37:3300/0 conn(0x7feb481523d0 0x7feb481527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:26.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb47e00640 1 -- 172.21.15.29:0/2310287175 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1235876767 0 0) 0x7feb480c60e0 con 0x7feb480c75c0 2024-04-17T20:04:26.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb47e00640 1 -- 172.21.15.29:0/2310287175 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb2c003530 con 0x7feb480c75c0 2024-04-17T20:04:26.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb47e00640 1 -- 172.21.15.29:0/2310287175 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2934577567 0 0) 0x7feb2c003530 con 0x7feb480c75c0 2024-04-17T20:04:26.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb47e00640 1 -- 172.21.15.29:0/2310287175 >> v2:172.21.15.37:3300/0 conn(0x7feb481523d0 msgr2=0x7feb481527e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:26.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb47e00640 1 --2- 172.21.15.29:0/2310287175 >> v2:172.21.15.37:3300/0 conn(0x7feb481523d0 0x7feb481527e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:26.826 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb47e00640 1 -- 172.21.15.29:0/2310287175 >> v1:172.21.15.37:6789/0 conn(0x7feb480c71b0 legacy=0x7feb4815c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb47e00640 1 -- 172.21.15.29:0/2310287175 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb480c6630 con 0x7feb480c75c0 2024-04-17T20:04:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb47e00640 1 -- 172.21.15.29:0/2310287175 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7feb30001680 con 0x7feb480c75c0 2024-04-17T20:04:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb47e00640 1 -- 172.21.15.29:0/2310287175 >> v1:172.21.15.29:6789/0 conn(0x7feb480c75c0 legacy=0x7feb4806bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb47e00640 1 --2- 172.21.15.29:0/2310287175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb2c003a30 0x7feb2c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb47e00640 1 --2- 172.21.15.29:0/2310287175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb2c004810 0x7feb2c004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb47e00640 1 --2- 172.21.15.29:0/2310287175 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb2c005230 0x7feb2c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb47e00640 1 -- 172.21.15.29:0/2310287175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feb4815ca30 con 0x7feb2c004810 2024-04-17T20:04:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb47e00640 1 -- 172.21.15.29:0/2310287175 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7feb2c00c1b0 con 0x7feb2c003a30 2024-04-17T20:04:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb47e00640 1 -- 172.21.15.29:0/2310287175 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7feb2c00c3b0 con 0x7feb2c005230 2024-04-17T20:04:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb3fe00640 1 --2- 172.21.15.29:0/2310287175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb2c004810 0x7feb2c004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb4da00640 1 --2- 172.21.15.29:0/2310287175 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb2c005230 0x7feb2c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb4d000640 1 --2- 172.21.15.29:0/2310287175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb2c003a30 0x7feb2c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb3fe00640 1 -- 172.21.15.29:0/2310287175 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb2c005230 msgr2=0x7feb2c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb3fe00640 1 --2- 172.21.15.29:0/2310287175 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb2c005230 0x7feb2c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb3fe00640 1 -- 172.21.15.29:0/2310287175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb2c003a30 msgr2=0x7feb2c003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:26.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb3fe00640 1 --2- 172.21.15.29:0/2310287175 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb2c003a30 0x7feb2c003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:26.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb3fe00640 1 -- 172.21.15.29:0/2310287175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb2c003530 con 0x7feb2c004810 2024-04-17T20:04:26.828 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.825+0000 7feb4da00640 1 -- 172.21.15.29:0/2310287175 reap_dead start 2024-04-17T20:04:26.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.829+0000 7feb3fe00640 1 --2- 172.21.15.29:0/2310287175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb2c004810 0x7feb2c004c20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7feb3000c240 tx=0x7feb3001cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9183fc769a85138 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:26.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.829+0000 7feb47e00640 1 -- 172.21.15.29:0/2310287175 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb30007d20 con 0x7feb2c004810 2024-04-17T20:04:26.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.829+0000 7feb47e00640 1 -- 172.21.15.29:0/2310287175 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb300a9070 con 0x7feb2c004810 2024-04-17T20:04:26.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.829+0000 7feb47e00640 1 -- 172.21.15.29:0/2310287175 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb300a5da0 con 0x7feb2c004810 2024-04-17T20:04:26.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.829+0000 7feb4fa00640 1 -- 172.21.15.29:0/2310287175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb2c004810 msgr2=0x7feb2c004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:26.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.829+0000 7feb4fa00640 1 --2- 172.21.15.29:0/2310287175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb2c004810 0x7feb2c004c20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7feb3000c240 tx=0x7feb3001cc00 comp rx=0 tx=0).stop 2024-04-17T20:04:26.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.829+0000 7feb4fa00640 1 -- 172.21.15.29:0/2310287175 shutdown_connections 2024-04-17T20:04:26.830 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.829+0000 7feb4fa00640 1 --2- 172.21.15.29:0/2310287175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb2c004810 0x7feb2c004c20 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:26.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.829+0000 7feb4fa00640 1 -- 172.21.15.29:0/2310287175 >> 172.21.15.29:0/2310287175 conn(0x7feb4806fde0 msgr2=0x7feb480701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:26.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.829+0000 7feb4fa00640 1 -- 172.21.15.29:0/2310287175 shutdown_connections 2024-04-17T20:04:26.831 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.829+0000 7feb4fa00640 1 -- 172.21.15.29:0/2310287175 wait complete. 2024-04-17T20:04:26.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.833+0000 7feb4fa00640 1 Processor -- start 2024-04-17T20:04:26.832 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.833+0000 7feb4fa00640 1 -- start start 2024-04-17T20:04:26.834 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.833+0000 7feb4fa00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb2c004810 0x7feb48072bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.833+0000 7feb4fa00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb481f5970 0x7feb481e1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.833+0000 7feb4fa00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb48142f20 0x7feb48143330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.833+0000 7feb4fa00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7feb480cb140 con 0x7feb481f5970 2024-04-17T20:04:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.833+0000 7feb4fa00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7feb4815be90 con 0x7feb48142f20 2024-04-17T20:04:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.833+0000 7feb4fa00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7feb480c6910 con 0x7feb2c004810 2024-04-17T20:04:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.833+0000 7feb3fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb481f5970 0x7feb481e1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.833+0000 7feb3fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb481f5970 0x7feb481e1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34008/0 (socket says 172.21.15.29:34008) 2024-04-17T20:04:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.833+0000 7feb3fe00640 1 -- 172.21.15.29:0/1396838481 learned_addr learned my addr 172.21.15.29:0/1396838481 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.837+0000 7feb4da00640 1 --2- 172.21.15.29:0/1396838481 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb48142f20 0x7feb48143330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.837+0000 7feb4d000640 1 --2- 172.21.15.29:0/1396838481 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb2c004810 0x7feb48072bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.837+0000 7feb3fe00640 1 -- 172.21.15.29:0/1396838481 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb2c004810 msgr2=0x7feb48072bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.837+0000 7feb3fe00640 1 --2- 172.21.15.29:0/1396838481 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb2c004810 0x7feb48072bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.837+0000 7feb3fe00640 1 -- 172.21.15.29:0/1396838481 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb48142f20 msgr2=0x7feb48143330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.837+0000 7feb3fe00640 1 --2- 172.21.15.29:0/1396838481 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb48142f20 0x7feb48143330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:26.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.837+0000 7feb3fe00640 1 -- 172.21.15.29:0/1396838481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb48143870 con 0x7feb481f5970 2024-04-17T20:04:26.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.837+0000 7feb4da00640 1 --2- 172.21.15.29:0/1396838481 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb48142f20 0x7feb48143330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:04:26.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.837+0000 7feb3fe00640 1 --2- 172.21.15.29:0/1396838481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb481f5970 0x7feb481e1a90 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7feb3000c240 tx=0x7feb2c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:26.836 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.837+0000 7feb45600640 1 -- 172.21.15.29:0/1396838481 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb300a5b10 con 0x7feb481f5970 2024-04-17T20:04:26.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.837+0000 7feb45600640 1 -- 172.21.15.29:0/1396838481 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb300a5cb0 con 0x7feb481f5970 2024-04-17T20:04:26.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.837+0000 7feb45600640 1 -- 172.21.15.29:0/1396838481 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb300a5e60 con 0x7feb481f5970 2024-04-17T20:04:26.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.837+0000 7feb4fa00640 1 -- 172.21.15.29:0/1396838481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb481463f0 con 0x7feb481f5970 2024-04-17T20:04:26.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.837+0000 7feb4fa00640 1 -- 172.21.15.29:0/1396838481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb48146ca0 con 0x7feb481f5970 2024-04-17T20:04:26.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.837+0000 7feb4fa00640 1 -- 172.21.15.29:0/1396838481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb48069b90 con 0x7feb481f5970 2024-04-17T20:04:26.837 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.837+0000 7feb45600640 1 -- 172.21.15.29:0/1396838481 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7feb30007360 con 0x7feb481f5970 2024-04-17T20:04:26.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.841+0000 7feb45600640 1 --2- 172.21.15.29:0/1396838481 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7feb240683b0 0x7feb2406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:26.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.841+0000 7feb45600640 1 -- 172.21.15.29:0/1396838481 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7feb30120db0 con 0x7feb481f5970 2024-04-17T20:04:26.838 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.841+0000 7feb4d000640 1 --2- 172.21.15.29:0/1396838481 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7feb240683b0 0x7feb2406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:26.839 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.841+0000 7feb4d000640 1 --2- 172.21.15.29:0/1396838481 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7feb240683b0 0x7feb2406a8c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7feb38000c10 tx=0x7feb38009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:26.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:26.845+0000 7feb45600640 1 -- 172.21.15.29:0/1396838481 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7feb30125070 con 0x7feb481f5970 2024-04-17T20:04:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:27.045+0000 7feb4fa00640 1 -- 172.21.15.29:0/1396838481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb4806bf90 con 0x7feb481f5970 2024-04-17T20:04:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:27.045+0000 7feb45600640 1 -- 172.21.15.29:0/1396838481 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6913 (secure 0 0 0) 0x7feb48143870 con 0x7feb481f5970 2024-04-17T20:04:27.046 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:27.046 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6076,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:25.246297+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5946,5775,5631,5350,5212]},"id":1}]} 2024-04-17T20:04:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 62 2024-04-17T20:04:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:27.049+0000 7feb4fa00640 1 -- 172.21.15.29:0/1396838481 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7feb240683b0 msgr2=0x7feb2406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:27.049+0000 7feb4fa00640 1 --2- 172.21.15.29:0/1396838481 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7feb240683b0 0x7feb2406a8c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7feb38000c10 tx=0x7feb38009040 comp rx=0 tx=0).stop 2024-04-17T20:04:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:27.049+0000 7feb4fa00640 1 -- 172.21.15.29:0/1396838481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb481f5970 msgr2=0x7feb481e1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:27.049+0000 7feb4fa00640 1 --2- 172.21.15.29:0/1396838481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb481f5970 0x7feb481e1a90 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7feb3000c240 tx=0x7feb2c005840 comp rx=0 tx=0).stop 2024-04-17T20:04:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:27.049+0000 7feb4fa00640 1 -- 172.21.15.29:0/1396838481 shutdown_connections 2024-04-17T20:04:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:27.049+0000 7feb4fa00640 1 --2- 172.21.15.29:0/1396838481 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7feb240683b0 0x7feb2406a8c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:27.049+0000 7feb4fa00640 1 --2- 172.21.15.29:0/1396838481 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7feb48142f20 0x7feb48143330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:27.049+0000 7feb4fa00640 1 --2- 172.21.15.29:0/1396838481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb481f5970 0x7feb481e1a90 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:27.049+0000 7feb4fa00640 1 --2- 172.21.15.29:0/1396838481 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7feb2c004810 0x7feb48072bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:27.049+0000 7feb4fa00640 1 -- 172.21.15.29:0/1396838481 >> 172.21.15.29:0/1396838481 conn(0x7feb4806fde0 msgr2=0x7feb480ca3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:27.049+0000 7feb4fa00640 1 -- 172.21.15.29:0/1396838481 shutdown_connections 2024-04-17T20:04:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:27.049+0000 7feb4fa00640 1 -- 172.21.15.29:0/1396838481 wait complete. 2024-04-17T20:04:27.068 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-17T20:04:29.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:29.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.237+0000 7f2ef4200640 1 Processor -- start 2024-04-17T20:04:29.237 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.237+0000 7f2ef4200640 1 -- start start 2024-04-17T20:04:29.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2ef4200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2eec0c7f50 0x7f2eec0ca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:29.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2ef4200640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eec0cbba0 con 0x7f2eec0cbf90 2024-04-17T20:04:29.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2ef4200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eec06c010 con 0x7f2eec155da0 2024-04-17T20:04:29.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2ef4200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2eec06bdc0 con 0x7f2eec0c7f50 2024-04-17T20:04:29.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2ef1800640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2eec0cbf90 0x7f2eec15ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52926/0 (socket says 172.21.15.29:52926) 2024-04-17T20:04:29.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2ef1800640 1 -- 172.21.15.29:0/3980793195 learned_addr learned my addr 172.21.15.29:0/3980793195 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:04:29.239 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2ef0e00640 1 --2- 172.21.15.29:0/3980793195 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2eec0c7f50 0x7f2eec0ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:29.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2eebe00640 1 -- 172.21.15.29:0/3980793195 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 887399582 0 0) 0x7f2eec0cbba0 con 0x7f2eec0cbf90 2024-04-17T20:04:29.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2eebe00640 1 -- 172.21.15.29:0/3980793195 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ed4003530 con 0x7f2eec0cbf90 2024-04-17T20:04:29.241 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2eebe00640 1 -- 172.21.15.29:0/3980793195 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1164760548 0 0) 0x7f2ed4003530 con 0x7f2eec0cbf90 2024-04-17T20:04:29.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2eebe00640 1 -- 172.21.15.29:0/3980793195 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2eec0c7f50 msgr2=0x7f2eec0ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:29.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2eebe00640 1 --2- 172.21.15.29:0/3980793195 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2eec0c7f50 0x7f2eec0ca410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:29.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2eebe00640 1 -- 172.21.15.29:0/3980793195 >> v1:172.21.15.37:6789/0 conn(0x7f2eec155da0 legacy=0x7f2eec1561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:29.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2eebe00640 1 -- 172.21.15.29:0/3980793195 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eec06c2f0 con 0x7f2eec0cbf90 2024-04-17T20:04:29.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2eebe00640 1 -- 172.21.15.29:0/3980793195 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f2ee0001680 con 0x7f2eec0cbf90 2024-04-17T20:04:29.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2eebe00640 1 -- 172.21.15.29:0/3980793195 >> v1:172.21.15.29:6789/0 conn(0x7f2eec0cbf90 legacy=0x7f2eec15ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:29.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2eebe00640 1 --2- 172.21.15.29:0/3980793195 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2eec0c7f50 0x7f2ed4003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:29.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2eebe00640 1 --2- 172.21.15.29:0/3980793195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed4003fe0 0x7f2ed4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:29.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2eebe00640 1 --2- 172.21.15.29:0/3980793195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ed4004d50 0x7f2ed40051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:29.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2eebe00640 1 -- 172.21.15.29:0/3980793195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec06bdc0 con 0x7f2ed4003fe0 2024-04-17T20:04:29.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2eebe00640 1 -- 172.21.15.29:0/3980793195 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2ed400bbe0 con 0x7f2ed4004d50 2024-04-17T20:04:29.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2eebe00640 1 -- 172.21.15.29:0/3980793195 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2ed400bde0 con 0x7f2eec0c7f50 2024-04-17T20:04:29.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2ef0e00640 1 --2- 172.21.15.29:0/3980793195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed4003fe0 0x7f2ed4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:29.242 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2ef1800640 1 --2- 172.21.15.29:0/3980793195 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2eec0c7f50 0x7f2ed4003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:29.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2ef2200640 1 --2- 172.21.15.29:0/3980793195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ed4004d50 0x7f2ed40051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:29.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2ef0e00640 1 -- 172.21.15.29:0/3980793195 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2eec0c7f50 msgr2=0x7f2ed4003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:29.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2ef0e00640 1 --2- 172.21.15.29:0/3980793195 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2eec0c7f50 0x7f2ed4003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:29.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2ef0e00640 1 -- 172.21.15.29:0/3980793195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ed4004d50 msgr2=0x7f2ed40051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:29.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2ef0e00640 1 --2- 172.21.15.29:0/3980793195 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2ed4004d50 0x7f2ed40051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:29.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.241+0000 7f2ef0e00640 1 -- 172.21.15.29:0/3980793195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ed4003530 con 0x7f2ed4003fe0 2024-04-17T20:04:29.243 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.245+0000 7f2ef0e00640 1 --2- 172.21.15.29:0/3980793195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed4003fe0 0x7f2ed4004810 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009ae0 tx=0x7f2ed801ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce739d2556d90199 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:29.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.245+0000 7f2eebe00640 1 -- 172.21.15.29:0/3980793195 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed80b2070 con 0x7f2ed4003fe0 2024-04-17T20:04:29.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.245+0000 7f2eebe00640 1 -- 172.21.15.29:0/3980793195 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ed80a5570 con 0x7f2ed4003fe0 2024-04-17T20:04:29.244 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.245+0000 7f2eebe00640 1 -- 172.21.15.29:0/3980793195 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed80096b0 con 0x7f2ed4003fe0 2024-04-17T20:04:29.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.245+0000 7f2ef4200640 1 -- 172.21.15.29:0/3980793195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed4003fe0 msgr2=0x7f2ed4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:29.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.245+0000 7f2ef4200640 1 --2- 172.21.15.29:0/3980793195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed4003fe0 0x7f2ed4004810 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8009ae0 tx=0x7f2ed801ca90 comp rx=0 tx=0).stop 2024-04-17T20:04:29.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.245+0000 7f2ef2200640 1 -- 172.21.15.29:0/3980793195 reap_dead start 2024-04-17T20:04:29.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.245+0000 7f2ef4200640 1 -- 172.21.15.29:0/3980793195 shutdown_connections 2024-04-17T20:04:29.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.245+0000 7f2ef4200640 1 -- 172.21.15.29:0/3980793195 >> 172.21.15.29:0/3980793195 conn(0x7f2eec06fde0 msgr2=0x7f2eec0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:29.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.245+0000 7f2ef4200640 1 -- 172.21.15.29:0/3980793195 shutdown_connections 2024-04-17T20:04:29.245 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.245+0000 7f2ef4200640 1 -- 172.21.15.29:0/3980793195 wait complete. 2024-04-17T20:04:29.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef4200640 1 Processor -- start 2024-04-17T20:04:29.247 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef4200640 1 -- start start 2024-04-17T20:04:29.253 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef4200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed4003fe0 0x7f2eec1e69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:29.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef4200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2eec14b2c0 0x7f2eec0723d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:29.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef4200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2eec073870 0x7f2eec072910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:29.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef4200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec0c84e0 con 0x7f2ed4003fe0 2024-04-17T20:04:29.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef4200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f2eec0c8360 con 0x7f2eec073870 2024-04-17T20:04:29.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef4200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f2eec0cb760 con 0x7f2eec14b2c0 2024-04-17T20:04:29.254 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef1800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed4003fe0 0x7f2eec1e69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:29.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef1800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed4003fe0 0x7f2eec1e69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58412/0 (socket says 172.21.15.29:58412) 2024-04-17T20:04:29.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef1800640 1 -- 172.21.15.29:0/3029097588 learned_addr learned my addr 172.21.15.29:0/3029097588 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:29.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef0e00640 1 --2- 172.21.15.29:0/3029097588 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2eec14b2c0 0x7f2eec0723d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:29.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef2200640 1 --2- 172.21.15.29:0/3029097588 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2eec073870 0x7f2eec072910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:29.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef1800640 1 -- 172.21.15.29:0/3029097588 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2eec14b2c0 msgr2=0x7f2eec0723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:29.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef1800640 1 --2- 172.21.15.29:0/3029097588 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2eec14b2c0 0x7f2eec0723d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:29.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef1800640 1 -- 172.21.15.29:0/3029097588 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2eec073870 msgr2=0x7f2eec072910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:29.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef1800640 1 --2- 172.21.15.29:0/3029097588 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2eec073870 0x7f2eec072910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:29.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef1800640 1 -- 172.21.15.29:0/3029097588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eec15b7d0 con 0x7f2ed4003fe0 2024-04-17T20:04:29.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef0e00640 1 --2- 172.21.15.29:0/3029097588 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2eec14b2c0 0x7f2eec0723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:04:29.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef1800640 1 --2- 172.21.15.29:0/3029097588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed4003fe0 0x7f2eec1e69c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0011790 tx=0x7f2ee0011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:29.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ee9600640 1 -- 172.21.15.29:0/3029097588 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee0010040 con 0x7f2ed4003fe0 2024-04-17T20:04:29.255 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ee9600640 1 -- 172.21.15.29:0/3029097588 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ee001a070 con 0x7f2ed4003fe0 2024-04-17T20:04:29.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ee9600640 1 -- 172.21.15.29:0/3029097588 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ee0011ea0 con 0x7f2ed4003fe0 2024-04-17T20:04:29.256 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.249+0000 7f2ef4200640 1 -- 172.21.15.29:0/3029097588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eec1e6f00 con 0x7f2ed4003fe0 2024-04-17T20:04:29.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.253+0000 7f2ef4200640 1 -- 172.21.15.29:0/3029097588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eec14bf50 con 0x7f2ed4003fe0 2024-04-17T20:04:29.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.253+0000 7f2ef4200640 1 -- 172.21.15.29:0/3029097588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eec069b90 con 0x7f2ed4003fe0 2024-04-17T20:04:29.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.253+0000 7f2ee9600640 1 -- 172.21.15.29:0/3029097588 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f2ee001f460 con 0x7f2ed4003fe0 2024-04-17T20:04:29.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.253+0000 7f2ee9600640 1 --2- 172.21.15.29:0/3029097588 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2ecc0683b0 0x7f2ecc06a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:29.257 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.253+0000 7f2ee9600640 1 -- 172.21.15.29:0/3029097588 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7f2ee008a210 con 0x7f2ed4003fe0 2024-04-17T20:04:29.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.253+0000 7f2ef0e00640 1 --2- 172.21.15.29:0/3029097588 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2ecc0683b0 0x7f2ecc06a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:29.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.257+0000 7f2ef0e00640 1 --2- 172.21.15.29:0/3029097588 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2ecc0683b0 0x7f2ecc06a8c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2ed80af000 tx=0x7f2ed80046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:29.258 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.257+0000 7f2ee9600640 1 -- 172.21.15.29:0/3029097588 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2ee005a0d0 con 0x7f2ed4003fe0 2024-04-17T20:04:29.456 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.457+0000 7f2ef4200640 1 -- 172.21.15.29:0/3029097588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2eec0cddc0 con 0x7f2ed4003fe0 2024-04-17T20:04:29.457 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.457+0000 7f2ee9600640 1 -- 172.21.15.29:0/3029097588 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6913 (secure 0 0 0) 0x7f2ee00599d0 con 0x7f2ed4003fe0 2024-04-17T20:04:29.457 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:29.457 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6076,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:25.246297+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5946,5775,5631,5350,5212]},"id":1}]} 2024-04-17T20:04:29.457 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 62 2024-04-17T20:04:29.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.461+0000 7f2ef4200640 1 -- 172.21.15.29:0/3029097588 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2ecc0683b0 msgr2=0x7f2ecc06a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:29.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.461+0000 7f2ef4200640 1 --2- 172.21.15.29:0/3029097588 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2ecc0683b0 0x7f2ecc06a8c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2ed80af000 tx=0x7f2ed80046d0 comp rx=0 tx=0).stop 2024-04-17T20:04:29.459 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.461+0000 7f2ef4200640 1 -- 172.21.15.29:0/3029097588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed4003fe0 msgr2=0x7f2eec1e69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:29.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.461+0000 7f2ef4200640 1 --2- 172.21.15.29:0/3029097588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed4003fe0 0x7f2eec1e69c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0011790 tx=0x7f2ee0011c50 comp rx=0 tx=0).stop 2024-04-17T20:04:29.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.461+0000 7f2ef4200640 1 -- 172.21.15.29:0/3029097588 shutdown_connections 2024-04-17T20:04:29.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.461+0000 7f2ef4200640 1 --2- 172.21.15.29:0/3029097588 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f2ecc0683b0 0x7f2ecc06a8c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:29.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.461+0000 7f2ef4200640 1 --2- 172.21.15.29:0/3029097588 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f2eec073870 0x7f2eec072910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:29.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.461+0000 7f2ef4200640 1 --2- 172.21.15.29:0/3029097588 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f2eec14b2c0 0x7f2eec0723d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:29.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.461+0000 7f2ef4200640 1 --2- 172.21.15.29:0/3029097588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed4003fe0 0x7f2eec1e69c0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:29.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.461+0000 7f2ef4200640 1 -- 172.21.15.29:0/3029097588 >> 172.21.15.29:0/3029097588 conn(0x7f2eec06fde0 msgr2=0x7f2eec15ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:29.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.461+0000 7f2ef4200640 1 -- 172.21.15.29:0/3029097588 shutdown_connections 2024-04-17T20:04:29.460 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:29.461+0000 7f2ef4200640 1 -- 172.21.15.29:0/3029097588 wait complete. 2024-04-17T20:04:29.479 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-17T20:04:29.981 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:04:29.985 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:04:31.481 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.649+0000 7f3c83200640 1 Processor -- start 2024-04-17T20:04:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.649+0000 7f3c83200640 1 -- start start 2024-04-17T20:04:31.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.649+0000 7f3c83200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3c840c3530 0x7f3c8415c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:31.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.649+0000 7f3c83200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c8415c660 0x7f3c841634b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:31.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.649+0000 7f3c83200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c840c95e0 con 0x7f3c84152d90 2024-04-17T20:04:31.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.649+0000 7f3c83200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f3c84071e30 con 0x7f3c840c3530 2024-04-17T20:04:31.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.649+0000 7f3c83200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3c8415ce00 con 0x7f3c8415c660 2024-04-17T20:04:31.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.649+0000 7f3c81400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3c840c3530 0x7f3c8415c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:31.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.649+0000 7f3c81400640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f3c840c3530 0x7f3c8415c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:39426/0 (socket says 172.21.15.29:39426) 2024-04-17T20:04:31.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.649+0000 7f3c82800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c8415c660 0x7f3c841634b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:31.649 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.649+0000 7f3c81400640 1 -- 172.21.15.29:0/1847164677 learned_addr learned my addr 172.21.15.29:0/1847164677 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:31.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.649+0000 7f3c81400640 1 -- 172.21.15.29:0/1847164677 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c8415c660 msgr2=0x7f3c841634b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:31.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.649+0000 7f3c81400640 1 --2- 172.21.15.29:0/1847164677 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c8415c660 0x7f3c841634b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:31.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.649+0000 7f3c81400640 1 -- 172.21.15.29:0/1847164677 >> v1:172.21.15.37:6789/0 conn(0x7f3c84152d90 legacy=0x7f3c840c2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:31.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.649+0000 7f3c81400640 1 -- 172.21.15.29:0/1847164677 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c84073eb0 con 0x7f3c840c3530 2024-04-17T20:04:31.650 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.649+0000 7f3c82800640 1 --2- 172.21.15.29:0/1847164677 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c8415c660 0x7f3c841634b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T20:04:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.653+0000 7f3c81400640 1 --2- 172.21.15.29:0/1847164677 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c840c3530 0x7f3c8415c120 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000bb10 tx=0x7f3c7001ec00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9838b203837da732 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.653+0000 7f3c80a00640 1 -- 172.21.15.29:0/1847164677 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c700b4070 con 0x7f3c840c3530 2024-04-17T20:04:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.653+0000 7f3c80a00640 1 -- 172.21.15.29:0/1847164677 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c700a7b60 con 0x7f3c840c3530 2024-04-17T20:04:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.653+0000 7f3c80a00640 1 -- 172.21.15.29:0/1847164677 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c700af440 con 0x7f3c840c3530 2024-04-17T20:04:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.653+0000 7f3c83200640 1 -- 172.21.15.29:0/1847164677 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c840c3530 msgr2=0x7f3c8415c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.653+0000 7f3c83200640 1 --2- 172.21.15.29:0/1847164677 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c840c3530 0x7f3c8415c120 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3c7000bb10 tx=0x7f3c7001ec00 comp rx=0 tx=0).stop 2024-04-17T20:04:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.653+0000 7f3c83200640 1 -- 172.21.15.29:0/1847164677 shutdown_connections 2024-04-17T20:04:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.653+0000 7f3c83200640 1 --2- 172.21.15.29:0/1847164677 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c8415c660 0x7f3c841634b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.653+0000 7f3c83200640 1 --2- 172.21.15.29:0/1847164677 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c840c3530 0x7f3c8415c120 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.653+0000 7f3c83200640 1 -- 172.21.15.29:0/1847164677 >> 172.21.15.29:0/1847164677 conn(0x7f3c840ccf80 msgr2=0x7f3c840cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.653+0000 7f3c83200640 1 -- 172.21.15.29:0/1847164677 shutdown_connections 2024-04-17T20:04:31.654 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.653+0000 7f3c83200640 1 -- 172.21.15.29:0/1847164677 wait complete. 2024-04-17T20:04:31.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c83200640 1 Processor -- start 2024-04-17T20:04:31.655 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c83200640 1 -- start start 2024-04-17T20:04:31.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c83200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c840c3530 0x7f3c841f8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:31.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c83200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c84152d90 0x7f3c841eb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:31.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c83200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8415c660 0x7f3c8415bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:31.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c83200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8406d390 con 0x7f3c8415c660 2024-04-17T20:04:31.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c83200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8406d210 con 0x7f3c840c3530 2024-04-17T20:04:31.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c83200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f3c840c1fa0 con 0x7f3c84152d90 2024-04-17T20:04:31.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c82800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8415c660 0x7f3c8415bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:31.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c82800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8415c660 0x7f3c8415bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58422/0 (socket says 172.21.15.29:58422) 2024-04-17T20:04:31.659 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c82800640 1 -- 172.21.15.29:0/3532725529 learned_addr learned my addr 172.21.15.29:0/3532725529 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:31.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c81400640 1 --2- 172.21.15.29:0/3532725529 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c84152d90 0x7f3c841eb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:31.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c81e00640 1 --2- 172.21.15.29:0/3532725529 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c840c3530 0x7f3c841f8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:31.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c82800640 1 -- 172.21.15.29:0/3532725529 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c84152d90 msgr2=0x7f3c841eb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:31.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c82800640 1 --2- 172.21.15.29:0/3532725529 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c84152d90 0x7f3c841eb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:31.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c82800640 1 -- 172.21.15.29:0/3532725529 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c840c3530 msgr2=0x7f3c841f8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:31.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c82800640 1 --2- 172.21.15.29:0/3532725529 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c840c3530 0x7f3c841f8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:31.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c82800640 1 -- 172.21.15.29:0/3532725529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c841f9350 con 0x7f3c8415c660 2024-04-17T20:04:31.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.657+0000 7f3c81e00640 1 --2- 172.21.15.29:0/3532725529 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c840c3530 0x7f3c841f8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:04:31.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.661+0000 7f3c82800640 1 --2- 172.21.15.29:0/3532725529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8415c660 0x7f3c8415bc80 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009e10 tx=0x7f3c7c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:31.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.661+0000 7f3c7a000640 1 -- 172.21.15.29:0/3532725529 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c018020 con 0x7f3c8415c660 2024-04-17T20:04:31.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.661+0000 7f3c7a000640 1 -- 172.21.15.29:0/3532725529 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c7c014480 con 0x7f3c8415c660 2024-04-17T20:04:31.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.661+0000 7f3c7a000640 1 -- 172.21.15.29:0/3532725529 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c014850 con 0x7f3c8415c660 2024-04-17T20:04:31.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.661+0000 7f3c83200640 1 -- 172.21.15.29:0/3532725529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c8415a170 con 0x7f3c8415c660 2024-04-17T20:04:31.660 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.661+0000 7f3c83200640 1 -- 172.21.15.29:0/3532725529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c841f8610 con 0x7f3c8415c660 2024-04-17T20:04:31.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.661+0000 7f3c83200640 1 -- 172.21.15.29:0/3532725529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c84159ee0 con 0x7f3c8415c660 2024-04-17T20:04:31.661 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.661+0000 7f3c7a000640 1 -- 172.21.15.29:0/3532725529 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f3c7c01b020 con 0x7f3c8415c660 2024-04-17T20:04:31.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.661+0000 7f3c7a000640 1 --2- 172.21.15.29:0/3532725529 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3c64068380 0x7f3c6406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:31.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.665+0000 7f3c7a000640 1 -- 172.21.15.29:0/3532725529 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7f3c7c087d20 con 0x7f3c8415c660 2024-04-17T20:04:31.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.665+0000 7f3c81e00640 1 --2- 172.21.15.29:0/3532725529 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3c64068380 0x7f3c6406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:31.663 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.665+0000 7f3c81e00640 1 --2- 172.21.15.29:0/3532725529 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3c64068380 0x7f3c6406a890 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3c74000f70 tx=0x7f3c74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:31.667 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.669+0000 7f3c7a000640 1 -- 172.21.15.29:0/3532725529 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f3c7c022030 con 0x7f3c8415c660 2024-04-17T20:04:31.895 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.897+0000 7f3c83200640 1 -- 172.21.15.29:0/3532725529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c841f8420 con 0x7f3c8415c660 2024-04-17T20:04:31.896 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.897+0000 7f3c7a000640 1 -- 172.21.15.29:0/3532725529 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+6913 (secure 0 0 0) 0x7f3c7c010040 con 0x7f3c8415c660 2024-04-17T20:04:31.896 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:31.896 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6076,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:02:25.246297+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":5212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5212":{"gid":5212,"name":"b","rank":4,"incarnation":26,"state":"up:active","state_seq":31,"addr":"172.21.15.37:6838/767681664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":767681664},{"type":"v1","addr":"172.21.15.37:6838","nonce":767681664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5946,5775,5631,5350,5212]},"id":1}]} 2024-04-17T20:04:31.896 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 62 2024-04-17T20:04:31.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.897+0000 7f3c83200640 1 -- 172.21.15.29:0/3532725529 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3c64068380 msgr2=0x7f3c6406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:31.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.897+0000 7f3c83200640 1 --2- 172.21.15.29:0/3532725529 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3c64068380 0x7f3c6406a890 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f3c74000f70 tx=0x7f3c74009040 comp rx=0 tx=0).stop 2024-04-17T20:04:31.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.897+0000 7f3c83200640 1 -- 172.21.15.29:0/3532725529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8415c660 msgr2=0x7f3c8415bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:31.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.897+0000 7f3c83200640 1 --2- 172.21.15.29:0/3532725529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8415c660 0x7f3c8415bc80 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c009e10 tx=0x7f3c7c00c490 comp rx=0 tx=0).stop 2024-04-17T20:04:31.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.897+0000 7f3c83200640 1 -- 172.21.15.29:0/3532725529 shutdown_connections 2024-04-17T20:04:31.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.897+0000 7f3c83200640 1 --2- 172.21.15.29:0/3532725529 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f3c64068380 0x7f3c6406a890 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:31.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.897+0000 7f3c83200640 1 --2- 172.21.15.29:0/3532725529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8415c660 0x7f3c8415bc80 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:31.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.897+0000 7f3c83200640 1 --2- 172.21.15.29:0/3532725529 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f3c84152d90 0x7f3c841eb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:31.898 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.897+0000 7f3c83200640 1 --2- 172.21.15.29:0/3532725529 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f3c840c3530 0x7f3c841f8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:31.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.897+0000 7f3c83200640 1 -- 172.21.15.29:0/3532725529 >> 172.21.15.29:0/3532725529 conn(0x7f3c840ccf80 msgr2=0x7f3c840c8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:31.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.897+0000 7f3c83200640 1 -- 172.21.15.29:0/3532725529 shutdown_connections 2024-04-17T20:04:31.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:31.897+0000 7f3c83200640 1 -- 172.21.15.29:0/3532725529 wait complete. 2024-04-17T20:04:31.918 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-17T20:04:33.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:34.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.093+0000 7fe568e00640 1 Processor -- start 2024-04-17T20:04:34.091 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.093+0000 7fe568e00640 1 -- start start 2024-04-17T20:04:34.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.093+0000 7fe568e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe5640c6f70 0x7fe5640c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:34.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.093+0000 7fe568e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe56406c660 0x7fe56406ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:34.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.093+0000 7fe568e00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe56415a0a0 con 0x7fe564159c90 2024-04-17T20:04:34.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.093+0000 7fe568e00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fe56415abe0 con 0x7fe5640c6f70 2024-04-17T20:04:34.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.093+0000 7fe568e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe56415a340 con 0x7fe56406c660 2024-04-17T20:04:34.092 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.093+0000 7fe55be00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe5640c6f70 0x7fe5640c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:34.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.093+0000 7fe55be00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe5640c6f70 0x7fe5640c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58430/0 (socket says 172.21.15.29:58430) 2024-04-17T20:04:34.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.093+0000 7fe55be00640 1 -- 172.21.15.29:0/4033017177 learned_addr learned my addr 172.21.15.29:0/4033017177 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:34.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.093+0000 7fe561e00640 1 --1- 172.21.15.29:0/4033017177 >> v1:172.21.15.29:6789/0 conn(0x7fe564159c90 0x7fe5640ca2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52942/0 (socket says 172.21.15.29:52942) 2024-04-17T20:04:34.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.093+0000 7fe562800640 1 --2- 172.21.15.29:0/4033017177 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe56406c660 0x7fe56406ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:34.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.093+0000 7fe55be00640 1 -- 172.21.15.29:0/4033017177 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe56406c660 msgr2=0x7fe56406ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:34.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.093+0000 7fe55be00640 1 --2- 172.21.15.29:0/4033017177 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe56406c660 0x7fe56406ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.093 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.093+0000 7fe55be00640 1 -- 172.21.15.29:0/4033017177 >> v1:172.21.15.29:6789/0 conn(0x7fe564159c90 legacy=0x7fe5640ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:34.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.093+0000 7fe561400640 1 -- 172.21.15.29:0/4033017177 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3775462178 0 0) 0x7fe56415a0a0 con 0x7fe564159c90 2024-04-17T20:04:34.094 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.093+0000 7fe55be00640 1 -- 172.21.15.29:0/4033017177 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe56415a620 con 0x7fe5640c6f70 2024-04-17T20:04:34.095 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.097+0000 7fe55be00640 1 --2- 172.21.15.29:0/4033017177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5640c6f70 0x7fe5640c7380 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe54c009870 tx=0x7fe54c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07f4b39e84dd31f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:34.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.097+0000 7fe561400640 1 -- 172.21.15.29:0/4033017177 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe54c0b2070 con 0x7fe5640c6f70 2024-04-17T20:04:34.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.097+0000 7fe561400640 1 -- 172.21.15.29:0/4033017177 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe54c0a9070 con 0x7fe5640c6f70 2024-04-17T20:04:34.096 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.097+0000 7fe561400640 1 -- 172.21.15.29:0/4033017177 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe54c0ad880 con 0x7fe5640c6f70 2024-04-17T20:04:34.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.097+0000 7fe568e00640 1 -- 172.21.15.29:0/4033017177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5640c6f70 msgr2=0x7fe5640c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:34.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.097+0000 7fe568e00640 1 --2- 172.21.15.29:0/4033017177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5640c6f70 0x7fe5640c7380 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe54c009870 tx=0x7fe54c01c560 comp rx=0 tx=0).stop 2024-04-17T20:04:34.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.097+0000 7fe568e00640 1 -- 172.21.15.29:0/4033017177 shutdown_connections 2024-04-17T20:04:34.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.097+0000 7fe568e00640 1 --2- 172.21.15.29:0/4033017177 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe56406c660 0x7fe56406ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.097+0000 7fe568e00640 1 --2- 172.21.15.29:0/4033017177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5640c6f70 0x7fe5640c7380 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.097+0000 7fe568e00640 1 -- 172.21.15.29:0/4033017177 >> 172.21.15.29:0/4033017177 conn(0x7fe56406fde0 msgr2=0x7fe5640701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:34.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.097+0000 7fe568e00640 1 -- 172.21.15.29:0/4033017177 shutdown_connections 2024-04-17T20:04:34.097 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.097+0000 7fe568e00640 1 -- 172.21.15.29:0/4033017177 wait complete. 2024-04-17T20:04:34.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe568e00640 1 Processor -- start 2024-04-17T20:04:34.099 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe568e00640 1 -- start start 2024-04-17T20:04:34.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe568e00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe56406c660 0x7fe5641f53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:34.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe568e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5640c6f70 0x7fe5641470a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:34.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe568e00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe564159c90 0x7fe5641ebb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:34.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe568e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fe5640ca9c0 con 0x7fe5640c6f70 2024-04-17T20:04:34.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe568e00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fe56406cfb0 con 0x7fe56406c660 2024-04-17T20:04:34.101 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe568e00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fe5640cab40 con 0x7fe564159c90 2024-04-17T20:04:34.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe55be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5640c6f70 0x7fe5641470a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:34.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe55be00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5640c6f70 0x7fe5641470a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58440/0 (socket says 172.21.15.29:58440) 2024-04-17T20:04:34.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe55be00640 1 -- 172.21.15.29:0/907885352 learned_addr learned my addr 172.21.15.29:0/907885352 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:34.102 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe562800640 1 --2- 172.21.15.29:0/907885352 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe564159c90 0x7fe5641ebb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:34.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe561e00640 1 --2- 172.21.15.29:0/907885352 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe56406c660 0x7fe5641f53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:34.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe55be00640 1 -- 172.21.15.29:0/907885352 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe564159c90 msgr2=0x7fe5641ebb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:34.103 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe55be00640 1 --2- 172.21.15.29:0/907885352 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe564159c90 0x7fe5641ebb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe55be00640 1 -- 172.21.15.29:0/907885352 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe56406c660 msgr2=0x7fe5641f53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:34.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe55be00640 1 --2- 172.21.15.29:0/907885352 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe56406c660 0x7fe5641f53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.104 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe55be00640 1 -- 172.21.15.29:0/907885352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5641475e0 con 0x7fe5640c6f70 2024-04-17T20:04:34.105 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe55be00640 1 --2- 172.21.15.29:0/907885352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5640c6f70 0x7fe5641470a0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe54c01ac10 tx=0x7fe54c01ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:34.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe55a000640 1 -- 172.21.15.29:0/907885352 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe54c0bc020 con 0x7fe5640c6f70 2024-04-17T20:04:34.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe55a000640 1 -- 172.21.15.29:0/907885352 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe54c0ade30 con 0x7fe5640c6f70 2024-04-17T20:04:34.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe568e00640 1 -- 172.21.15.29:0/907885352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe56414f080 con 0x7fe5640c6f70 2024-04-17T20:04:34.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe55a000640 1 -- 172.21.15.29:0/907885352 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe54c0b2040 con 0x7fe5640c6f70 2024-04-17T20:04:34.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.101+0000 7fe568e00640 1 -- 172.21.15.29:0/907885352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5641f93d0 con 0x7fe5640c6f70 2024-04-17T20:04:34.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.105+0000 7fe568e00640 1 -- 172.21.15.29:0/907885352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe564069b90 con 0x7fe5640c6f70 2024-04-17T20:04:34.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.105+0000 7fe55a000640 1 -- 172.21.15.29:0/907885352 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fe54c0bf020 con 0x7fe5640c6f70 2024-04-17T20:04:34.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.105+0000 7fe55a000640 1 --2- 172.21.15.29:0/907885352 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe540068380 0x7fe54006a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:34.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.105+0000 7fe55a000640 1 -- 172.21.15.29:0/907885352 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7337+0+0 (secure 0 0 0) 0x7fe54c121300 con 0x7fe5640c6f70 2024-04-17T20:04:34.106 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.105+0000 7fe561e00640 1 --2- 172.21.15.29:0/907885352 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe540068380 0x7fe54006a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:34.107 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.109+0000 7fe561e00640 1 --2- 172.21.15.29:0/907885352 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe540068380 0x7fe54006a890 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe554000f70 tx=0x7fe554007c30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:34.109 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.109+0000 7fe55a000640 1 -- 172.21.15.29:0/907885352 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fe54c0c6030 con 0x7fe5640c6f70 2024-04-17T20:04:34.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.341+0000 7fe568e00640 1 -- 172.21.15.29:0/907885352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe56414cd00 con 0x7fe5640c6f70 2024-04-17T20:04:34.342 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.345+0000 7fe55a000640 1 -- 172.21.15.29:0/907885352 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6086 (secure 0 0 0) 0x7fe54c016150 con 0x7fe5640c6f70 2024-04-17T20:04:34.343 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:34.343 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:04:34.139050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":6076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6076":{"gid":6076,"name":"f","rank":4,"incarnation":63,"state":"up:replay","state_seq":1,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5350,5631,5775,5946]},"id":1}]} 2024-04-17T20:04:34.343 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 63 2024-04-17T20:04:34.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.345+0000 7fe568e00640 1 -- 172.21.15.29:0/907885352 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe540068380 msgr2=0x7fe54006a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:34.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.345+0000 7fe568e00640 1 --2- 172.21.15.29:0/907885352 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe540068380 0x7fe54006a890 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe554000f70 tx=0x7fe554007c30 comp rx=0 tx=0).stop 2024-04-17T20:04:34.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.345+0000 7fe568e00640 1 -- 172.21.15.29:0/907885352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5640c6f70 msgr2=0x7fe5641470a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:34.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.345+0000 7fe568e00640 1 --2- 172.21.15.29:0/907885352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5640c6f70 0x7fe5641470a0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fe54c01ac10 tx=0x7fe54c01ada0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.345+0000 7fe568e00640 1 -- 172.21.15.29:0/907885352 shutdown_connections 2024-04-17T20:04:34.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.345+0000 7fe568e00640 1 --2- 172.21.15.29:0/907885352 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fe540068380 0x7fe54006a890 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.345+0000 7fe568e00640 1 --2- 172.21.15.29:0/907885352 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fe564159c90 0x7fe5641ebb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.345+0000 7fe568e00640 1 --2- 172.21.15.29:0/907885352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5640c6f70 0x7fe5641470a0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.345+0000 7fe568e00640 1 --2- 172.21.15.29:0/907885352 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fe56406c660 0x7fe5641f53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.345+0000 7fe568e00640 1 -- 172.21.15.29:0/907885352 >> 172.21.15.29:0/907885352 conn(0x7fe56406fde0 msgr2=0x7fe5640754b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:34.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.345+0000 7fe568e00640 1 -- 172.21.15.29:0/907885352 shutdown_connections 2024-04-17T20:04:34.346 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.345+0000 7fe568e00640 1 -- 172.21.15.29:0/907885352 wait complete. 2024-04-17T20:04:34.368 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2024-04-17T20:04:34.368 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2024-04-17T20:04:34.369 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.525+0000 7fb109a00640 1 Processor -- start 2024-04-17T20:04:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.525+0000 7fb109a00640 1 -- start start 2024-04-17T20:04:34.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.525+0000 7fb109a00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb1040cc7f0 0x7fb1040c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:34.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.525+0000 7fb109a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb10415fdb0 0x7fb1041601c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:34.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.525+0000 7fb109a00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1040cbc20 con 0x7fb1040cae50 2024-04-17T20:04:34.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.525+0000 7fb109a00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fb10415d140 con 0x7fb1040cc7f0 2024-04-17T20:04:34.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.525+0000 7fb109a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb10415c180 con 0x7fb10415fdb0 2024-04-17T20:04:34.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.525+0000 7fb102a00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb1040cc7f0 0x7fb1040c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:34.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.525+0000 7fb102000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb10415fdb0 0x7fb1041601c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:34.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.525+0000 7fb102a00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fb1040cc7f0 0x7fb1040c9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:39450/0 (socket says 172.21.15.29:39450) 2024-04-17T20:04:34.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.525+0000 7fb102000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb10415fdb0 0x7fb1041601c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:46486/0 (socket says 172.21.15.29:46486) 2024-04-17T20:04:34.525 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.525+0000 7fb102a00640 1 -- 172.21.15.29:0/655429961 learned_addr learned my addr 172.21.15.29:0/655429961 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.525+0000 7fb102000640 1 -- 172.21.15.29:0/655429961 >> v2:172.21.15.37:3300/0 conn(0x7fb1040cc7f0 msgr2=0x7fb1040c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.525+0000 7fb102000640 1 --2- 172.21.15.29:0/655429961 >> v2:172.21.15.37:3300/0 conn(0x7fb1040cc7f0 0x7fb1040c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.525+0000 7fb102000640 1 -- 172.21.15.29:0/655429961 >> v1:172.21.15.37:6789/0 conn(0x7fb1040cae50 legacy=0x7fb1040cf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.525+0000 7fb102000640 1 -- 172.21.15.29:0/655429961 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10406ca70 con 0x7fb10415fdb0 2024-04-17T20:04:34.528 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.529+0000 7fb102000640 1 --2- 172.21.15.29:0/655429961 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb10415fdb0 0x7fb1041601c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb0f00097c0 tx=0x7fb0f001c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2ca81462eb3eb00 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:34.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.529+0000 7fb101600640 1 -- 172.21.15.29:0/655429961 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f00b0070 con 0x7fb10415fdb0 2024-04-17T20:04:34.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.529+0000 7fb101600640 1 -- 172.21.15.29:0/655429961 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f00ab600 con 0x7fb10415fdb0 2024-04-17T20:04:34.529 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.529+0000 7fb101600640 1 -- 172.21.15.29:0/655429961 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f00ab9d0 con 0x7fb10415fdb0 2024-04-17T20:04:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.529+0000 7fb109a00640 1 -- 172.21.15.29:0/655429961 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb10415fdb0 msgr2=0x7fb1041601c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.529+0000 7fb109a00640 1 --2- 172.21.15.29:0/655429961 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb10415fdb0 0x7fb1041601c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb0f00097c0 tx=0x7fb0f001c380 comp rx=0 tx=0).stop 2024-04-17T20:04:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.529+0000 7fb109a00640 1 -- 172.21.15.29:0/655429961 shutdown_connections 2024-04-17T20:04:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.529+0000 7fb109a00640 1 --2- 172.21.15.29:0/655429961 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb10415fdb0 0x7fb1041601c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.529+0000 7fb109a00640 1 --2- 172.21.15.29:0/655429961 >> v2:172.21.15.37:3300/0 conn(0x7fb1040cc7f0 0x7fb1040c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.529+0000 7fb109a00640 1 -- 172.21.15.29:0/655429961 >> 172.21.15.29:0/655429961 conn(0x7fb104067eb0 msgr2=0x7fb104070040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.529+0000 7fb109a00640 1 -- 172.21.15.29:0/655429961 shutdown_connections 2024-04-17T20:04:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.529+0000 7fb109a00640 1 -- 172.21.15.29:0/655429961 wait complete. 2024-04-17T20:04:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.533+0000 7fb109a00640 1 Processor -- start 2024-04-17T20:04:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.533+0000 7fb109a00640 1 -- start start 2024-04-17T20:04:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.533+0000 7fb109a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1040cae50 0x7fb104156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.533+0000 7fb109a00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1040cc7f0 0x7fb104157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.533+0000 7fb109a00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb10415fdb0 0x7fb10415ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.533+0000 7fb109a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fb10406c790 con 0x7fb1040cae50 2024-04-17T20:04:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.533+0000 7fb109a00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fb10406c4b0 con 0x7fb1040cc7f0 2024-04-17T20:04:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.533+0000 7fb109a00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fb1040cb670 con 0x7fb10415fdb0 2024-04-17T20:04:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.533+0000 7fb102a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1040cae50 0x7fb104156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.533+0000 7fb102a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1040cae50 0x7fb104156bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58452/0 (socket says 172.21.15.29:58452) 2024-04-17T20:04:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.533+0000 7fb102a00640 1 -- 172.21.15.29:0/2366693935 learned_addr learned my addr 172.21.15.29:0/2366693935 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.533+0000 7fb102000640 1 --2- 172.21.15.29:0/2366693935 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1040cc7f0 0x7fb104157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.533+0000 7fb103400640 1 --2- 172.21.15.29:0/2366693935 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb10415fdb0 0x7fb10415ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.537+0000 7fb102a00640 1 -- 172.21.15.29:0/2366693935 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb10415fdb0 msgr2=0x7fb10415ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.537+0000 7fb102a00640 1 --2- 172.21.15.29:0/2366693935 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb10415fdb0 0x7fb10415ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.537+0000 7fb102a00640 1 -- 172.21.15.29:0/2366693935 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1040cc7f0 msgr2=0x7fb104157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.537+0000 7fb102a00640 1 --2- 172.21.15.29:0/2366693935 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1040cc7f0 0x7fb104157b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.537 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.537+0000 7fb102a00640 1 -- 172.21.15.29:0/2366693935 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1041f8d30 con 0x7fb1040cae50 2024-04-17T20:04:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.537+0000 7fb102000640 1 --2- 172.21.15.29:0/2366693935 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1040cc7f0 0x7fb104157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:04:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.537+0000 7fb102a00640 1 --2- 172.21.15.29:0/2366693935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1040cae50 0x7fb104156bf0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400e820 tx=0x7fb0f400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.537+0000 7fb0faa00640 1 -- 172.21.15.29:0/2366693935 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4017070 con 0x7fb1040cae50 2024-04-17T20:04:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.537+0000 7fb0faa00640 1 -- 172.21.15.29:0/2366693935 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f400c8e0 con 0x7fb1040cae50 2024-04-17T20:04:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.537+0000 7fb109a00640 1 -- 172.21.15.29:0/2366693935 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1040b2ea0 con 0x7fb1040cae50 2024-04-17T20:04:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.537+0000 7fb0faa00640 1 -- 172.21.15.29:0/2366693935 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f400ccd0 con 0x7fb1040cae50 2024-04-17T20:04:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.537+0000 7fb109a00640 1 -- 172.21.15.29:0/2366693935 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1041f5bb0 con 0x7fb1040cae50 2024-04-17T20:04:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.537+0000 7fb109a00640 1 -- 172.21.15.29:0/2366693935 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb104069b90 con 0x7fb1040cae50 2024-04-17T20:04:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.537+0000 7fb0faa00640 1 -- 172.21.15.29:0/2366693935 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fb0f401a070 con 0x7fb1040cae50 2024-04-17T20:04:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.541+0000 7fb0faa00640 1 --2- 172.21.15.29:0/2366693935 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb0e4068380 0x7fb0e406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.541+0000 7fb0faa00640 1 -- 172.21.15.29:0/2366693935 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7423+0+0 (secure 0 0 0) 0x7fb0f4087730 con 0x7fb1040cae50 2024-04-17T20:04:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.541+0000 7fb102000640 1 --2- 172.21.15.29:0/2366693935 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb0e4068380 0x7fb0e406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.541+0000 7fb102000640 1 --2- 172.21.15.29:0/2366693935 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb0e4068380 0x7fb0e406a890 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb0f00097c0 tx=0x7fb0f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:34.541 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.541+0000 7fb0faa00640 1 -- 172.21.15.29:0/2366693935 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fb0f4057620 con 0x7fb1040cae50 2024-04-17T20:04:34.745 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.745+0000 7fb109a00640 1 -- 172.21.15.29:0/2366693935 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb1040cce20 con 0x7fb1040cae50 2024-04-17T20:04:34.746 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.749+0000 7fb0faa00640 1 -- 172.21.15.29:0/2366693935 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6086 (secure 0 0 0) 0x7fb0f4056f20 con 0x7fb1040cae50 2024-04-17T20:04:34.746 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:34.746 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:04:34.139050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":6076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6076":{"gid":6076,"name":"f","rank":4,"incarnation":63,"state":"up:replay","state_seq":1,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5350,5631,5775,5946]},"id":1}]} 2024-04-17T20:04:34.747 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 63 2024-04-17T20:04:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.749+0000 7fb109a00640 1 -- 172.21.15.29:0/2366693935 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb0e4068380 msgr2=0x7fb0e406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.749+0000 7fb109a00640 1 --2- 172.21.15.29:0/2366693935 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb0e4068380 0x7fb0e406a890 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb0f00097c0 tx=0x7fb0f0008040 comp rx=0 tx=0).stop 2024-04-17T20:04:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.749+0000 7fb109a00640 1 -- 172.21.15.29:0/2366693935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1040cae50 msgr2=0x7fb104156bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.749+0000 7fb109a00640 1 --2- 172.21.15.29:0/2366693935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1040cae50 0x7fb104156bf0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fb0f400e820 tx=0x7fb0f400ece0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.749+0000 7fb109a00640 1 -- 172.21.15.29:0/2366693935 shutdown_connections 2024-04-17T20:04:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.749+0000 7fb109a00640 1 --2- 172.21.15.29:0/2366693935 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fb0e4068380 0x7fb0e406a890 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.749+0000 7fb109a00640 1 --2- 172.21.15.29:0/2366693935 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fb10415fdb0 0x7fb10415ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.749+0000 7fb109a00640 1 --2- 172.21.15.29:0/2366693935 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fb1040cc7f0 0x7fb104157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.749+0000 7fb109a00640 1 --2- 172.21.15.29:0/2366693935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1040cae50 0x7fb104156bf0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.749+0000 7fb109a00640 1 -- 172.21.15.29:0/2366693935 >> 172.21.15.29:0/2366693935 conn(0x7fb104067eb0 msgr2=0x7fb1041562b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.749+0000 7fb109a00640 1 -- 172.21.15.29:0/2366693935 shutdown_connections 2024-04-17T20:04:34.749 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:34.749+0000 7fb109a00640 1 -- 172.21.15.29:0/2366693935 wait complete. 2024-04-17T20:04:34.769 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 63, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-17T19:47:49.315515+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 63, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5350": { "addr": "172.21.15.37:6837/2495953425", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6834", "nonce": 2495953425, "type": "v2" }, { "addr": "172.21.15.37:6837", "nonce": 2495953425, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5350, "incarnation": 32, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 18 }, "gid_5631": { "addr": "172.21.15.29:6836/2376553580", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6832", "nonce": 2376553580, "type": "v2" }, { "addr": "172.21.15.29:6836", "nonce": 2376553580, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5631, "incarnation": 44, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 42 }, "gid_5775": { "addr": "172.21.15.29:6835/1352350250", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6834", "nonce": 1352350250, "type": "v2" }, { "addr": "172.21.15.29:6835", "nonce": 1352350250, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5775, "incarnation": 50, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 39 }, "gid_5946": { "addr": "172.21.15.29:6837/3087508289", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6833", "nonce": 3087508289, "type": "v2" }, { "addr": "172.21.15.29:6837", "nonce": 3087508289, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5946, "incarnation": 57, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 25 }, "gid_6076": { "addr": "172.21.15.37:6839/2273456671", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6836", "nonce": 2273456671, "type": "v2" }, { "addr": "172.21.15.37:6839", "nonce": 2273456671, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6076, "incarnation": 63, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 76, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-17T20:04:34.139050+0000", "qdb_cluster": [ 5350, 5631, 5775, 5946 ], "qdb_leader": 5775, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5775, "mds_1": 5631, "mds_2": 5946, "mds_3": 5350, "mds_4": 6076 } } } ], "standbys": [] } 2024-04-17T20:04:36.770 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.936+0000 7f759cc00640 1 Processor -- start 2024-04-17T20:04:36.935 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.936+0000 7f759cc00640 1 -- start start 2024-04-17T20:04:36.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.936+0000 7f759cc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7598156370 0x7f75980ca110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:36.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.936+0000 7f759cc00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f759806c730 con 0x7f75980cabe0 2024-04-17T20:04:36.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.936+0000 7f759cc00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f759806c9f0 con 0x7f759815c970 2024-04-17T20:04:36.937 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.936+0000 7f759cc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f759806ccd0 con 0x7f7598156370 2024-04-17T20:04:36.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.936+0000 7f758fe00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f75980cabe0 0x7f759815c260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52946/0 (socket says 172.21.15.29:52946) 2024-04-17T20:04:36.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.936+0000 7f758fe00640 1 -- 172.21.15.29:0/3526005726 learned_addr learned my addr 172.21.15.29:0/3526005726 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:04:36.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.936+0000 7f7597400640 1 --2- 172.21.15.29:0/3526005726 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7598156370 0x7f75980ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:36.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.936+0000 7f7596a00640 1 -- 172.21.15.29:0/3526005726 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3739365178 0 0) 0x7f759806c730 con 0x7f75980cabe0 2024-04-17T20:04:36.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.936+0000 7f7596a00640 1 -- 172.21.15.29:0/3526005726 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f757c003530 con 0x7f75980cabe0 2024-04-17T20:04:36.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.936+0000 7f7596a00640 1 -- 172.21.15.29:0/3526005726 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2583446497 0 0) 0x7f757c003530 con 0x7f75980cabe0 2024-04-17T20:04:36.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.936+0000 7f7597400640 1 -- 172.21.15.29:0/3526005726 >> v1:172.21.15.37:6789/0 conn(0x7f759815c970 legacy=0x7f7598166740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:36.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.936+0000 7f7597400640 1 -- 172.21.15.29:0/3526005726 >> v1:172.21.15.29:6789/0 conn(0x7f75980cabe0 legacy=0x7f759815c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:36.938 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.936+0000 7f7597400640 1 -- 172.21.15.29:0/3526005726 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f759806cfb0 con 0x7f7598156370 2024-04-17T20:04:36.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.940+0000 7f7597400640 1 --2- 172.21.15.29:0/3526005726 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7598156370 0x7f75980ca110 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f75880097f0 tx=0x7f758801c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cb5b570e85ff6d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:36.941 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.940+0000 7f7596a00640 1 -- 172.21.15.29:0/3526005726 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75880b0070 con 0x7f7598156370 2024-04-17T20:04:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.940+0000 7f7596a00640 1 -- 172.21.15.29:0/3526005726 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75880ab400 con 0x7f7598156370 2024-04-17T20:04:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.940+0000 7f7596a00640 1 -- 172.21.15.29:0/3526005726 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7580001680 con 0x7f7598156370 2024-04-17T20:04:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.944+0000 7f759cc00640 1 -- 172.21.15.29:0/3526005726 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7598156370 msgr2=0x7f75980ca110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:36.942 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.944+0000 7f759cc00640 1 --2- 172.21.15.29:0/3526005726 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7598156370 0x7f75980ca110 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f75880097f0 tx=0x7f758801c340 comp rx=0 tx=0).stop 2024-04-17T20:04:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.944+0000 7f759cc00640 1 -- 172.21.15.29:0/3526005726 shutdown_connections 2024-04-17T20:04:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.944+0000 7f759cc00640 1 --2- 172.21.15.29:0/3526005726 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7598156370 0x7f75980ca110 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.944+0000 7f759cc00640 1 -- 172.21.15.29:0/3526005726 >> 172.21.15.29:0/3526005726 conn(0x7f75980680e0 msgr2=0x7f759806dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.944+0000 7f759cc00640 1 -- 172.21.15.29:0/3526005726 shutdown_connections 2024-04-17T20:04:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.944+0000 7f759cc00640 1 -- 172.21.15.29:0/3526005726 wait complete. 2024-04-17T20:04:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.944+0000 7f759cc00640 1 Processor -- start 2024-04-17T20:04:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.944+0000 7f759cc00640 1 -- start start 2024-04-17T20:04:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.944+0000 7f759cc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75980cabe0 0x7f75981e29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.944+0000 7f759cc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598156370 0x7f75980c8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.944+0000 7f759cc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f759815c970 0x7f75980c7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.944+0000 7f759cc00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f75980ca650 con 0x7f7598156370 2024-04-17T20:04:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.944+0000 7f759cc00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f759806d570 con 0x7f759815c970 2024-04-17T20:04:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.944+0000 7f759cc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f759806e7c0 con 0x7f75980cabe0 2024-04-17T20:04:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f758fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598156370 0x7f75980c8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f758fe00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598156370 0x7f75980c8a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58466/0 (socket says 172.21.15.29:58466) 2024-04-17T20:04:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f758fe00640 1 -- 172.21.15.29:0/3363124532 learned_addr learned my addr 172.21.15.29:0/3363124532 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f7597400640 1 --2- 172.21.15.29:0/3363124532 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75980cabe0 0x7f75981e29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f7597e00640 1 --2- 172.21.15.29:0/3363124532 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f759815c970 0x7f75980c7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f758fe00640 1 -- 172.21.15.29:0/3363124532 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75980cabe0 msgr2=0x7f75981e29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f758fe00640 1 --2- 172.21.15.29:0/3363124532 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75980cabe0 0x7f75981e29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f758fe00640 1 -- 172.21.15.29:0/3363124532 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f759815c970 msgr2=0x7f75980c7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f758fe00640 1 --2- 172.21.15.29:0/3363124532 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f759815c970 0x7f75980c7920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f758fe00640 1 -- 172.21.15.29:0/3363124532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75980c8fb0 con 0x7f7598156370 2024-04-17T20:04:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f7597e00640 1 --2- 172.21.15.29:0/3363124532 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f759815c970 0x7f75980c7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:04:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f758fe00640 1 --2- 172.21.15.29:0/3363124532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598156370 0x7f75980c8a70 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7580007fd0 tx=0x7f758000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f758f400640 1 -- 172.21.15.29:0/3363124532 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7580018070 con 0x7f7598156370 2024-04-17T20:04:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f758f400640 1 -- 172.21.15.29:0/3363124532 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75800072d0 con 0x7f7598156370 2024-04-17T20:04:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f758f400640 1 -- 172.21.15.29:0/3363124532 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75800047f0 con 0x7f7598156370 2024-04-17T20:04:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f759cc00640 1 -- 172.21.15.29:0/3363124532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75981f3760 con 0x7f7598156370 2024-04-17T20:04:36.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f759cc00640 1 -- 172.21.15.29:0/3363124532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75980c68d0 con 0x7f7598156370 2024-04-17T20:04:36.950 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f759cc00640 1 -- 172.21.15.29:0/3363124532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f759804ff10 con 0x7f7598156370 2024-04-17T20:04:36.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.948+0000 7f758f400640 1 -- 172.21.15.29:0/3363124532 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f758001b070 con 0x7f7598156370 2024-04-17T20:04:36.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.952+0000 7f758f400640 1 --2- 172.21.15.29:0/3363124532 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7574068380 0x7f757406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:36.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.952+0000 7f758f400640 1 -- 172.21.15.29:0/3363124532 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7423+0+0 (secure 0 0 0) 0x7f7580086410 con 0x7f7598156370 2024-04-17T20:04:36.951 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.952+0000 7f7597400640 1 --2- 172.21.15.29:0/3363124532 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7574068380 0x7f757406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:36.952 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.952+0000 7f7597400640 1 --2- 172.21.15.29:0/3363124532 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7574068380 0x7f757406a890 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f75880ad000 tx=0x7f75880a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:36.956+0000 7f758f400640 1 -- 172.21.15.29:0/3363124532 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7580055cd0 con 0x7f7598156370 2024-04-17T20:04:37.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:37.152+0000 7f759cc00640 1 -- 172.21.15.29:0/3363124532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f759806dd80 con 0x7f7598156370 2024-04-17T20:04:37.154 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:37.156+0000 7f758f400640 1 -- 172.21.15.29:0/3363124532 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6086 (secure 0 0 0) 0x7f7580059fd0 con 0x7f7598156370 2024-04-17T20:04:37.155 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:37.155 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:04:34.139050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":6076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6076":{"gid":6076,"name":"f","rank":4,"incarnation":63,"state":"up:replay","state_seq":1,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5350,5631,5775,5946]},"id":1}]} 2024-04-17T20:04:37.155 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 63 2024-04-17T20:04:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:37.156+0000 7f759cc00640 1 -- 172.21.15.29:0/3363124532 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7574068380 msgr2=0x7f757406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:37.156+0000 7f759cc00640 1 --2- 172.21.15.29:0/3363124532 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7574068380 0x7f757406a890 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f75880ad000 tx=0x7f75880a6000 comp rx=0 tx=0).stop 2024-04-17T20:04:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:37.156+0000 7f759cc00640 1 -- 172.21.15.29:0/3363124532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598156370 msgr2=0x7f75980c8a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:37.156+0000 7f759cc00640 1 --2- 172.21.15.29:0/3363124532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598156370 0x7f75980c8a70 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f7580007fd0 tx=0x7f758000cbe0 comp rx=0 tx=0).stop 2024-04-17T20:04:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:37.156+0000 7f759cc00640 1 -- 172.21.15.29:0/3363124532 shutdown_connections 2024-04-17T20:04:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:37.156+0000 7f759cc00640 1 --2- 172.21.15.29:0/3363124532 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7574068380 0x7f757406a890 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:37.156+0000 7f759cc00640 1 --2- 172.21.15.29:0/3363124532 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f759815c970 0x7f75980c7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:37.156+0000 7f759cc00640 1 --2- 172.21.15.29:0/3363124532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598156370 0x7f75980c8a70 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:37.156+0000 7f759cc00640 1 --2- 172.21.15.29:0/3363124532 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f75980cabe0 0x7f75981e29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:37.156+0000 7f759cc00640 1 -- 172.21.15.29:0/3363124532 >> 172.21.15.29:0/3363124532 conn(0x7f75980680e0 msgr2=0x7f75980c58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:37.156+0000 7f759cc00640 1 -- 172.21.15.29:0/3363124532 shutdown_connections 2024-04-17T20:04:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:37.156+0000 7f759cc00640 1 -- 172.21.15.29:0/3363124532 wait complete. 2024-04-17T20:04:37.176 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:04:39.178 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:39.343 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.344+0000 7f45ddc00640 1 Processor -- start 2024-04-17T20:04:39.344 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.344+0000 7f45ddc00640 1 -- start start 2024-04-17T20:04:39.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.344+0000 7f45ddc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45d80c6f70 0x7f45d80c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:39.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.344+0000 7f45ddc00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45d815a0a0 con 0x7f45d8159c90 2024-04-17T20:04:39.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.344+0000 7f45ddc00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45d815abe0 con 0x7f45d806c660 2024-04-17T20:04:39.345 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.344+0000 7f45ddc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f45d815a340 con 0x7f45d80c6f70 2024-04-17T20:04:39.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.344+0000 7f45d6a00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f45d8159c90 0x7f45d80ca2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46028/0 (socket says 172.21.15.29:46028) 2024-04-17T20:04:39.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.344+0000 7f45d6a00640 1 -- 172.21.15.29:0/1964710708 learned_addr learned my addr 172.21.15.29:0/1964710708 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:04:39.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.344+0000 7f45ce000640 1 --2- 172.21.15.29:0/1964710708 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45d80c6f70 0x7f45d80c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:39.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.344+0000 7f45d6000640 1 -- 172.21.15.29:0/1964710708 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 469200367 0 0) 0x7f45d815a0a0 con 0x7f45d8159c90 2024-04-17T20:04:39.347 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.344+0000 7f45d6000640 1 -- 172.21.15.29:0/1964710708 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45bc003530 con 0x7f45d8159c90 2024-04-17T20:04:39.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45d6000640 1 -- 172.21.15.29:0/1964710708 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2225763429 0 0) 0x7f45d815abe0 con 0x7f45d806c660 2024-04-17T20:04:39.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45d6000640 1 -- 172.21.15.29:0/1964710708 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45d815a0a0 con 0x7f45d806c660 2024-04-17T20:04:39.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45d6000640 1 -- 172.21.15.29:0/1964710708 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1314490141 0 0) 0x7f45bc003530 con 0x7f45d8159c90 2024-04-17T20:04:39.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45d6000640 1 -- 172.21.15.29:0/1964710708 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45d80c6f70 msgr2=0x7f45d80c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:39.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45d6000640 1 --2- 172.21.15.29:0/1964710708 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45d80c6f70 0x7f45d80c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:39.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45d6000640 1 -- 172.21.15.29:0/1964710708 >> v1:172.21.15.37:6789/0 conn(0x7f45d806c660 legacy=0x7f45d806ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:39.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45d6000640 1 -- 172.21.15.29:0/1964710708 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d815a620 con 0x7f45d8159c90 2024-04-17T20:04:39.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45d6000640 1 -- 172.21.15.29:0/1964710708 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f45c80016a0 con 0x7f45d8159c90 2024-04-17T20:04:39.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45d6000640 1 -- 172.21.15.29:0/1964710708 >> v1:172.21.15.29:6789/0 conn(0x7f45d8159c90 legacy=0x7f45d80ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:39.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45d6000640 1 --2- 172.21.15.29:0/1964710708 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45bc003bf0 0x7f45bc004040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:39.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45d6000640 1 --2- 172.21.15.29:0/1964710708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45bc004810 0x7f45bc004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:39.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45d6000640 1 --2- 172.21.15.29:0/1964710708 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45bc005220 0x7f45bc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:39.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45d6000640 1 -- 172.21.15.29:0/1964710708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f45d815a340 con 0x7f45bc004810 2024-04-17T20:04:39.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45d6000640 1 -- 172.21.15.29:0/1964710708 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f45bc00c100 con 0x7f45bc003bf0 2024-04-17T20:04:39.348 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45d6000640 1 -- 172.21.15.29:0/1964710708 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f45bc00c300 con 0x7f45bc005220 2024-04-17T20:04:39.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45ce000640 1 --2- 172.21.15.29:0/1964710708 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45d80c6f70 0x7f45d80c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T20:04:39.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45d6a00640 1 --2- 172.21.15.29:0/1964710708 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45bc003bf0 0x7f45bc004040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:39.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45ce000640 1 --2- 172.21.15.29:0/1964710708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45bc004810 0x7f45bc004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:39.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45d7400640 1 --2- 172.21.15.29:0/1964710708 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45bc005220 0x7f45bc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:39.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45ce000640 1 -- 172.21.15.29:0/1964710708 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45bc005220 msgr2=0x7f45bc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:39.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45ce000640 1 --2- 172.21.15.29:0/1964710708 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45bc005220 0x7f45bc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:39.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45ce000640 1 -- 172.21.15.29:0/1964710708 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45bc003bf0 msgr2=0x7f45bc004040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:39.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45ce000640 1 --2- 172.21.15.29:0/1964710708 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45bc003bf0 0x7f45bc004040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:39.349 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.348+0000 7f45ce000640 1 -- 172.21.15.29:0/1964710708 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45bc003530 con 0x7f45bc004810 2024-04-17T20:04:39.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.352+0000 7f45ce000640 1 --2- 172.21.15.29:0/1964710708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45bc004810 0x7f45bc004c20 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f45d80cb260 tx=0x7f45c001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eecf5c87244d9b1e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:39.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.352+0000 7f45d6000640 1 -- 172.21.15.29:0/1964710708 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c00097f0 con 0x7f45bc004810 2024-04-17T20:04:39.351 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.352+0000 7f45d6000640 1 -- 172.21.15.29:0/1964710708 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45c0004030 con 0x7f45bc004810 2024-04-17T20:04:39.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.352+0000 7f45d6000640 1 -- 172.21.15.29:0/1964710708 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c0004400 con 0x7f45bc004810 2024-04-17T20:04:39.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.352+0000 7f45ddc00640 1 -- 172.21.15.29:0/1964710708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45bc004810 msgr2=0x7f45bc004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:39.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.352+0000 7f45ddc00640 1 --2- 172.21.15.29:0/1964710708 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45bc004810 0x7f45bc004c20 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f45d80cb260 tx=0x7f45c001ca90 comp rx=0 tx=0).stop 2024-04-17T20:04:39.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.352+0000 7f45d7400640 1 -- 172.21.15.29:0/1964710708 reap_dead start 2024-04-17T20:04:39.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.352+0000 7f45ddc00640 1 -- 172.21.15.29:0/1964710708 shutdown_connections 2024-04-17T20:04:39.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.352+0000 7f45ddc00640 1 -- 172.21.15.29:0/1964710708 >> 172.21.15.29:0/1964710708 conn(0x7f45d806fde0 msgr2=0x7f45d80701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:39.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.352+0000 7f45ddc00640 1 -- 172.21.15.29:0/1964710708 shutdown_connections 2024-04-17T20:04:39.352 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.352+0000 7f45ddc00640 1 -- 172.21.15.29:0/1964710708 wait complete. 2024-04-17T20:04:39.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.352+0000 7f45ddc00640 1 Processor -- start 2024-04-17T20:04:39.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.352+0000 7f45ddc00640 1 -- start start 2024-04-17T20:04:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45ddc00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45bc004810 0x7f45d814c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45ddc00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45d8073690 0x7f45d814b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45ddc00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45d814d1d0 0x7f45d80c2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45ddc00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f45d806c140 con 0x7f45bc004810 2024-04-17T20:04:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45ddc00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f45d806be60 con 0x7f45d8073690 2024-04-17T20:04:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45ddc00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f45d80ca9c0 con 0x7f45d814d1d0 2024-04-17T20:04:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45d6a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45bc004810 0x7f45d814c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45d6a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45bc004810 0x7f45d814c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49290/0 (socket says 172.21.15.29:49290) 2024-04-17T20:04:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45d6a00640 1 -- 172.21.15.29:0/2730645473 learned_addr learned my addr 172.21.15.29:0/2730645473 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45d7400640 1 --2- 172.21.15.29:0/2730645473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45d814d1d0 0x7f45d80c2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45ce000640 1 --2- 172.21.15.29:0/2730645473 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45d8073690 0x7f45d814b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:39.356 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45d6a00640 1 -- 172.21.15.29:0/2730645473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45d814d1d0 msgr2=0x7f45d80c2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:39.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45d6a00640 1 --2- 172.21.15.29:0/2730645473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45d814d1d0 0x7f45d80c2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:39.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45d6a00640 1 -- 172.21.15.29:0/2730645473 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45d8073690 msgr2=0x7f45d814b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:39.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45d6a00640 1 --2- 172.21.15.29:0/2730645473 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45d8073690 0x7f45d814b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:39.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45d6a00640 1 -- 172.21.15.29:0/2730645473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d814cdc0 con 0x7f45bc004810 2024-04-17T20:04:39.357 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45d6a00640 1 --2- 172.21.15.29:0/2730645473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45bc004810 0x7f45d814c880 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f45c8004e80 tx=0x7f45c8007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:39.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45cf400640 1 -- 172.21.15.29:0/2730645473 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c800f7b0 con 0x7f45bc004810 2024-04-17T20:04:39.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45cf400640 1 -- 172.21.15.29:0/2730645473 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45c8018070 con 0x7f45bc004810 2024-04-17T20:04:39.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45ddc00640 1 -- 172.21.15.29:0/2730645473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d814bb30 con 0x7f45bc004810 2024-04-17T20:04:39.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45cf400640 1 -- 172.21.15.29:0/2730645473 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c800fb70 con 0x7f45bc004810 2024-04-17T20:04:39.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45ddc00640 1 -- 172.21.15.29:0/2730645473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d80c4d90 con 0x7f45bc004810 2024-04-17T20:04:39.358 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.356+0000 7f45ddc00640 1 -- 172.21.15.29:0/2730645473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45d8069b90 con 0x7f45bc004810 2024-04-17T20:04:39.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.360+0000 7f45cf400640 1 -- 172.21.15.29:0/2730645473 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f45c800fd10 con 0x7f45bc004810 2024-04-17T20:04:39.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.360+0000 7f45cf400640 1 --2- 172.21.15.29:0/2730645473 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f45b40683b0 0x7f45b406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:39.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.360+0000 7f45cf400640 1 -- 172.21.15.29:0/2730645473 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7423+0+0 (secure 0 0 0) 0x7f45c8086e70 con 0x7f45bc004810 2024-04-17T20:04:39.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.360+0000 7f45ce000640 1 --2- 172.21.15.29:0/2730645473 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f45b40683b0 0x7f45b406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:39.360 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.360+0000 7f45ce000640 1 --2- 172.21.15.29:0/2730645473 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f45b40683b0 0x7f45b406a8c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f45c00ad000 tx=0x7f45c00af040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:39.364 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.364+0000 7f45cf400640 1 -- 172.21.15.29:0/2730645473 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f45c8056d80 con 0x7f45bc004810 2024-04-17T20:04:39.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.564+0000 7f45ddc00640 1 -- 172.21.15.29:0/2730645473 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45d80c7660 con 0x7f45bc004810 2024-04-17T20:04:39.566 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.564+0000 7f45cf400640 1 -- 172.21.15.29:0/2730645473 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) ==== 94+0+6086 (secure 0 0 0) 0x7f45c8056680 con 0x7f45bc004810 2024-04-17T20:04:39.566 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:39.566 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:04:34.139050+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":6076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6076":{"gid":6076,"name":"f","rank":4,"incarnation":63,"state":"up:replay","state_seq":1,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5350,5631,5775,5946]},"id":1}]} 2024-04-17T20:04:39.567 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 63 2024-04-17T20:04:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.568+0000 7f45ddc00640 1 -- 172.21.15.29:0/2730645473 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f45b40683b0 msgr2=0x7f45b406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.568+0000 7f45ddc00640 1 --2- 172.21.15.29:0/2730645473 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f45b40683b0 0x7f45b406a8c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f45c00ad000 tx=0x7f45c00af040 comp rx=0 tx=0).stop 2024-04-17T20:04:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.568+0000 7f45ddc00640 1 -- 172.21.15.29:0/2730645473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45bc004810 msgr2=0x7f45d814c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.568+0000 7f45ddc00640 1 --2- 172.21.15.29:0/2730645473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45bc004810 0x7f45d814c880 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f45c8004e80 tx=0x7f45c8007430 comp rx=0 tx=0).stop 2024-04-17T20:04:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.568+0000 7f45ddc00640 1 -- 172.21.15.29:0/2730645473 shutdown_connections 2024-04-17T20:04:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.568+0000 7f45ddc00640 1 --2- 172.21.15.29:0/2730645473 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f45b40683b0 0x7f45b406a8c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.568+0000 7f45ddc00640 1 --2- 172.21.15.29:0/2730645473 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f45d814d1d0 0x7f45d80c2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.568+0000 7f45ddc00640 1 --2- 172.21.15.29:0/2730645473 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f45d8073690 0x7f45d814b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.568+0000 7f45ddc00640 1 --2- 172.21.15.29:0/2730645473 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45bc004810 0x7f45d814c880 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:39.568 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.568+0000 7f45ddc00640 1 -- 172.21.15.29:0/2730645473 >> 172.21.15.29:0/2730645473 conn(0x7f45d806fde0 msgr2=0x7f45d814ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:39.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.568+0000 7f45ddc00640 1 -- 172.21.15.29:0/2730645473 shutdown_connections 2024-04-17T20:04:39.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:39.568+0000 7f45ddc00640 1 -- 172.21.15.29:0/2730645473 wait complete. 2024-04-17T20:04:39.588 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:04:41.589 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:41.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.752+0000 7fa2aba00640 1 Processor -- start 2024-04-17T20:04:41.754 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.752+0000 7fa2aba00640 1 -- start start 2024-04-17T20:04:41.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2aba00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fa2a40c75c0 0x7fa2a406bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:41.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2aba00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2a41523d0 0x7fa2a41527e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:41.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2aba00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2a40c60e0 con 0x7fa2a40c71b0 2024-04-17T20:04:41.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2aba00640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fa2a415ca30 con 0x7fa2a40c75c0 2024-04-17T20:04:41.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2aba00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa2a415be90 con 0x7fa2a41523d0 2024-04-17T20:04:41.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a9000640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa2a40c71b0 0x7fa2a415c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46034/0 (socket says 172.21.15.29:46034) 2024-04-17T20:04:41.756 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a9000640 1 -- 172.21.15.29:0/1693444776 learned_addr learned my addr 172.21.15.29:0/1693444776 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:04:41.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a9a00640 1 --2- 172.21.15.29:0/1693444776 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2a41523d0 0x7fa2a41527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:41.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3e00640 1 --2- 172.21.15.29:0/1693444776 >> v2:172.21.15.37:3300/0 conn(0x7fa2a40c75c0 0x7fa2a406bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:41.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3400640 1 -- 172.21.15.29:0/1693444776 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4186721106 0 0) 0x7fa2a40c60e0 con 0x7fa2a40c71b0 2024-04-17T20:04:41.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3400640 1 -- 172.21.15.29:0/1693444776 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa28c003450 con 0x7fa2a40c71b0 2024-04-17T20:04:41.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3400640 1 -- 172.21.15.29:0/1693444776 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 900994216 0 0) 0x7fa28c003450 con 0x7fa2a40c71b0 2024-04-17T20:04:41.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3400640 1 -- 172.21.15.29:0/1693444776 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2a41523d0 msgr2=0x7fa2a41527e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:41.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3400640 1 --2- 172.21.15.29:0/1693444776 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2a41523d0 0x7fa2a41527e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:41.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3400640 1 -- 172.21.15.29:0/1693444776 >> v2:172.21.15.37:3300/0 conn(0x7fa2a40c75c0 msgr2=0x7fa2a406bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:41.758 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3400640 1 --2- 172.21.15.29:0/1693444776 >> v2:172.21.15.37:3300/0 conn(0x7fa2a40c75c0 0x7fa2a406bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3400640 1 -- 172.21.15.29:0/1693444776 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2a40c6350 con 0x7fa2a40c71b0 2024-04-17T20:04:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3400640 1 -- 172.21.15.29:0/1693444776 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7fa2980016a0 con 0x7fa2a40c71b0 2024-04-17T20:04:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3400640 1 -- 172.21.15.29:0/1693444776 >> v1:172.21.15.29:6789/0 conn(0x7fa2a40c71b0 legacy=0x7fa2a415c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3400640 1 --2- 172.21.15.29:0/1693444776 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa28c003b20 0x7fa28c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3400640 1 --2- 172.21.15.29:0/1693444776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c008900 0x7fa28c00ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3400640 1 --2- 172.21.15.29:0/1693444776 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2a41523d0 0x7fa28c00d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3400640 1 -- 172.21.15.29:0/1693444776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a415ca30 con 0x7fa28c008900 2024-04-17T20:04:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3400640 1 -- 172.21.15.29:0/1693444776 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a415be90 con 0x7fa28c003b20 2024-04-17T20:04:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3400640 1 -- 172.21.15.29:0/1693444776 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa28c00dd40 con 0x7fa2a41523d0 2024-04-17T20:04:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3e00640 1 --2- 172.21.15.29:0/1693444776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c008900 0x7fa28c00ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a9000640 1 --2- 172.21.15.29:0/1693444776 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa28c003b20 0x7fa28c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a9a00640 1 --2- 172.21.15.29:0/1693444776 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2a41523d0 0x7fa28c00d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3e00640 1 -- 172.21.15.29:0/1693444776 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2a41523d0 msgr2=0x7fa28c00d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3e00640 1 --2- 172.21.15.29:0/1693444776 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa2a41523d0 0x7fa28c00d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:41.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3e00640 1 -- 172.21.15.29:0/1693444776 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa28c003b20 msgr2=0x7fa28c003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:41.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3e00640 1 --2- 172.21.15.29:0/1693444776 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa28c003b20 0x7fa28c003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:41.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.756+0000 7fa2a3e00640 1 -- 172.21.15.29:0/1693444776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa28c003450 con 0x7fa28c008900 2024-04-17T20:04:41.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.760+0000 7fa2a3e00640 1 --2- 172.21.15.29:0/1693444776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c008900 0x7fa28c00ade0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fa290009ae0 tx=0x7fa29001cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92dc85cecc39fd06 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:41.760 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.760+0000 7fa2a3400640 1 -- 172.21.15.29:0/1693444776 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2900b2070 con 0x7fa28c008900 2024-04-17T20:04:41.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.760+0000 7fa2a3400640 1 -- 172.21.15.29:0/1693444776 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2900a56a0 con 0x7fa28c008900 2024-04-17T20:04:41.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.760+0000 7fa2a3400640 1 -- 172.21.15.29:0/1693444776 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2900096b0 con 0x7fa28c008900 2024-04-17T20:04:41.761 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.760+0000 7fa2aba00640 1 -- 172.21.15.29:0/1693444776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c008900 msgr2=0x7fa28c00ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:41.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.760+0000 7fa2aba00640 1 --2- 172.21.15.29:0/1693444776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa28c008900 0x7fa28c00ade0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fa290009ae0 tx=0x7fa29001cab0 comp rx=0 tx=0).stop 2024-04-17T20:04:41.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.760+0000 7fa2a9a00640 1 -- 172.21.15.29:0/1693444776 reap_dead start 2024-04-17T20:04:41.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.760+0000 7fa2aba00640 1 -- 172.21.15.29:0/1693444776 shutdown_connections 2024-04-17T20:04:41.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.760+0000 7fa2aba00640 1 -- 172.21.15.29:0/1693444776 >> 172.21.15.29:0/1693444776 conn(0x7fa2a406fde0 msgr2=0x7fa2a40701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:41.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.760+0000 7fa2aba00640 1 -- 172.21.15.29:0/1693444776 shutdown_connections 2024-04-17T20:04:41.762 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.760+0000 7fa2aba00640 1 -- 172.21.15.29:0/1693444776 wait complete. 2024-04-17T20:04:41.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.764+0000 7fa2aba00640 1 Processor -- start 2024-04-17T20:04:41.764 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.764+0000 7fa2aba00640 1 -- start start 2024-04-17T20:04:41.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.764+0000 7fa2aba00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c008900 0x7fa2a414fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:41.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.764+0000 7fa2aba00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2a4151f70 0x7fa2a414e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:41.766 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.764+0000 7fa2aba00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a414cfd0 0x7fa2a414d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:41.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.764+0000 7fa2aba00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a40cb140 con 0x7fa2a414cfd0 2024-04-17T20:04:41.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.764+0000 7fa2aba00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a415bb20 con 0x7fa2a4151f70 2024-04-17T20:04:41.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.764+0000 7fa2aba00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fa2a40c6910 con 0x7fa28c008900 2024-04-17T20:04:41.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.764+0000 7fa2a9a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a414cfd0 0x7fa2a414d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:41.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.764+0000 7fa2a9a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a414cfd0 0x7fa2a414d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49318/0 (socket says 172.21.15.29:49318) 2024-04-17T20:04:41.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.764+0000 7fa2a9a00640 1 -- 172.21.15.29:0/691833143 learned_addr learned my addr 172.21.15.29:0/691833143 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:41.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.764+0000 7fa2a9000640 1 --2- 172.21.15.29:0/691833143 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c008900 0x7fa2a414fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:41.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.764+0000 7fa2a3e00640 1 --2- 172.21.15.29:0/691833143 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2a4151f70 0x7fa2a414e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:41.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.768+0000 7fa2a9a00640 1 -- 172.21.15.29:0/691833143 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c008900 msgr2=0x7fa2a414fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:41.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.768+0000 7fa2a9a00640 1 --2- 172.21.15.29:0/691833143 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c008900 0x7fa2a414fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:41.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.768+0000 7fa2a9a00640 1 -- 172.21.15.29:0/691833143 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2a4151f70 msgr2=0x7fa2a414e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:41.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.768+0000 7fa2a9a00640 1 --2- 172.21.15.29:0/691833143 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2a4151f70 0x7fa2a414e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:41.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.768+0000 7fa2a9a00640 1 -- 172.21.15.29:0/691833143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2a41502c0 con 0x7fa2a414cfd0 2024-04-17T20:04:41.767 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.768+0000 7fa2a3e00640 1 --2- 172.21.15.29:0/691833143 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2a4151f70 0x7fa2a414e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:04:41.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.768+0000 7fa2a9a00640 1 --2- 172.21.15.29:0/691833143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a414cfd0 0x7fa2a414d3e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa2a406eef0 tx=0x7fa29c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:41.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.768+0000 7fa2a0c00640 1 -- 172.21.15.29:0/691833143 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c0039a0 con 0x7fa2a414cfd0 2024-04-17T20:04:41.768 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.768+0000 7fa2a0c00640 1 -- 172.21.15.29:0/691833143 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa29c01b070 con 0x7fa2a414cfd0 2024-04-17T20:04:41.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.768+0000 7fa2aba00640 1 -- 172.21.15.29:0/691833143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2a414eac0 con 0x7fa2a414cfd0 2024-04-17T20:04:41.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.768+0000 7fa2a0c00640 1 -- 172.21.15.29:0/691833143 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c003b80 con 0x7fa2a414cfd0 2024-04-17T20:04:41.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.768+0000 7fa2aba00640 1 -- 172.21.15.29:0/691833143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2a41e19f0 con 0x7fa2a414cfd0 2024-04-17T20:04:41.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.768+0000 7fa2aba00640 1 -- 172.21.15.29:0/691833143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2a414efe0 con 0x7fa2a414cfd0 2024-04-17T20:04:41.769 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.768+0000 7fa2a0c00640 1 -- 172.21.15.29:0/691833143 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fa29c004030 con 0x7fa2a414cfd0 2024-04-17T20:04:41.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.768+0000 7fa2a0c00640 1 --2- 172.21.15.29:0/691833143 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa2840683b0 0x7fa28406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:41.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.768+0000 7fa2a0c00640 1 -- 172.21.15.29:0/691833143 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7423+0+0 (secure 0 0 0) 0x7fa29c087900 con 0x7fa2a414cfd0 2024-04-17T20:04:41.770 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.772+0000 7fa2a9000640 1 --2- 172.21.15.29:0/691833143 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa2840683b0 0x7fa28406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:41.771 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.772+0000 7fa2a9000640 1 --2- 172.21.15.29:0/691833143 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa2840683b0 0x7fa28406a8c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa298001060 tx=0x7fa298007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:41.774 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.776+0000 7fa2a0c00640 1 -- 172.21.15.29:0/691833143 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fa2a414efe0 con 0x7fa2a414cfd0 2024-04-17T20:04:41.977 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.976+0000 7fa2aba00640 1 -- 172.21.15.29:0/691833143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2a4069b90 con 0x7fa2a414cfd0 2024-04-17T20:04:41.978 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.980+0000 7fa2a0c00640 1 -- 172.21.15.29:0/691833143 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+6088 (secure 0 0 0) 0x7fa29c025020 con 0x7fa2a414cfd0 2024-04-17T20:04:41.978 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:41.978 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:04:40.014925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":6076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6076":{"gid":6076,"name":"f","rank":4,"incarnation":63,"state":"up:resolve","state_seq":33,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5946,5775,5631,5350]},"id":1}]} 2024-04-17T20:04:41.979 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 64 2024-04-17T20:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.980+0000 7fa2aba00640 1 -- 172.21.15.29:0/691833143 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa2840683b0 msgr2=0x7fa28406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.980+0000 7fa2aba00640 1 --2- 172.21.15.29:0/691833143 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa2840683b0 0x7fa28406a8c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa298001060 tx=0x7fa298007040 comp rx=0 tx=0).stop 2024-04-17T20:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.980+0000 7fa2aba00640 1 -- 172.21.15.29:0/691833143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a414cfd0 msgr2=0x7fa2a414d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.980+0000 7fa2aba00640 1 --2- 172.21.15.29:0/691833143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a414cfd0 0x7fa2a414d3e0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fa2a406eef0 tx=0x7fa29c0115a0 comp rx=0 tx=0).stop 2024-04-17T20:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.980+0000 7fa2aba00640 1 -- 172.21.15.29:0/691833143 shutdown_connections 2024-04-17T20:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.980+0000 7fa2aba00640 1 --2- 172.21.15.29:0/691833143 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fa2840683b0 0x7fa28406a8c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.980+0000 7fa2aba00640 1 --2- 172.21.15.29:0/691833143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2a414cfd0 0x7fa2a414d3e0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.980+0000 7fa2aba00640 1 --2- 172.21.15.29:0/691833143 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fa2a4151f70 0x7fa2a414e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.980+0000 7fa2aba00640 1 --2- 172.21.15.29:0/691833143 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fa28c008900 0x7fa2a414fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.980+0000 7fa2aba00640 1 -- 172.21.15.29:0/691833143 >> 172.21.15.29:0/691833143 conn(0x7fa2a406fde0 msgr2=0x7fa2a40ca210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.980+0000 7fa2aba00640 1 -- 172.21.15.29:0/691833143 shutdown_connections 2024-04-17T20:04:41.981 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:41.980+0000 7fa2aba00640 1 -- 172.21.15.29:0/691833143 wait complete. 2024-04-17T20:04:42.003 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:04:44.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:44.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.168+0000 7f7c11200640 1 Processor -- start 2024-04-17T20:04:44.169 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.168+0000 7f7c11200640 1 -- start start 2024-04-17T20:04:44.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.172+0000 7f7c11200640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7c0c159c90 0x7f7c0c0ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:44.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.172+0000 7f7c11200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c0c06c660 0x7f7c0c06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:44.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.172+0000 7f7c11200640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c0c15a0a0 con 0x7f7c0c0ca7f0 2024-04-17T20:04:44.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.172+0000 7f7c11200640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7f7c0c15abe0 con 0x7f7c0c159c90 2024-04-17T20:04:44.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.172+0000 7f7c11200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7c0c15a340 con 0x7f7c0c06c660 2024-04-17T20:04:44.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.172+0000 7f7c0a800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c0c06c660 0x7f7c0c06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:44.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.172+0000 7f7c09e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7c0c159c90 0x7f7c0c0ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:44.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.172+0000 7f7c0a800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c0c06c660 0x7f7c0c06ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:37658/0 (socket says 172.21.15.29:37658) 2024-04-17T20:04:44.171 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.172+0000 7f7c0a800640 1 -- 172.21.15.29:0/648952220 learned_addr learned my addr 172.21.15.29:0/648952220 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:44.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.172+0000 7f7c09e00640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7f7c0c159c90 0x7f7c0c0ca2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3300/0 says I am v2:172.21.15.29:37666/0 (socket says 172.21.15.29:37666) 2024-04-17T20:04:44.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.172+0000 7f7c09e00640 1 -- 172.21.15.29:0/648952220 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c0c06c660 msgr2=0x7f7c0c06ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:44.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.172+0000 7f7c09e00640 1 --2- 172.21.15.29:0/648952220 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c0c06c660 0x7f7c0c06ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:44.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.172+0000 7f7c09e00640 1 -- 172.21.15.29:0/648952220 >> v1:172.21.15.37:6789/0 conn(0x7f7c0c0ca7f0 legacy=0x7f7c0c0c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:44.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.172+0000 7f7c09e00640 1 -- 172.21.15.29:0/648952220 --> v2:172.21.15.37:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0c15a620 con 0x7f7c0c159c90 2024-04-17T20:04:44.172 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.172+0000 7f7c0a800640 1 --2- 172.21.15.29:0/648952220 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c0c06c660 0x7f7c0c06ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T20:04:44.175 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.176+0000 7f7c09e00640 1 --2- 172.21.15.29:0/648952220 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c0c159c90 0x7f7c0c0ca2b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc0097c0 tx=0x7f7bfc01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59944d3e4866b20d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:44.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.176+0000 7f7c09400640 1 -- 172.21.15.29:0/648952220 <== mon.1 v2:172.21.15.37:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc0b1070 con 0x7f7c0c159c90 2024-04-17T20:04:44.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.176+0000 7f7c09400640 1 -- 172.21.15.29:0/648952220 <== mon.1 v2:172.21.15.37:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bfc0ac410 con 0x7f7c0c159c90 2024-04-17T20:04:44.176 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.176+0000 7f7c09400640 1 -- 172.21.15.29:0/648952220 <== mon.1 v2:172.21.15.37:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc0ac7e0 con 0x7f7c0c159c90 2024-04-17T20:04:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.176+0000 7f7c11200640 1 -- 172.21.15.29:0/648952220 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c0c159c90 msgr2=0x7f7c0c0ca2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.176+0000 7f7c11200640 1 --2- 172.21.15.29:0/648952220 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c0c159c90 0x7f7c0c0ca2b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc0097c0 tx=0x7f7bfc01c4a0 comp rx=0 tx=0).stop 2024-04-17T20:04:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.176+0000 7f7c11200640 1 -- 172.21.15.29:0/648952220 shutdown_connections 2024-04-17T20:04:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.176+0000 7f7c11200640 1 --2- 172.21.15.29:0/648952220 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c0c06c660 0x7f7c0c06ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.176+0000 7f7c11200640 1 --2- 172.21.15.29:0/648952220 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c0c159c90 0x7f7c0c0ca2b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.176+0000 7f7c11200640 1 -- 172.21.15.29:0/648952220 >> 172.21.15.29:0/648952220 conn(0x7f7c0c06fde0 msgr2=0x7f7c0c0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.176+0000 7f7c11200640 1 -- 172.21.15.29:0/648952220 shutdown_connections 2024-04-17T20:04:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.176+0000 7f7c11200640 1 -- 172.21.15.29:0/648952220 wait complete. 2024-04-17T20:04:44.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c11200640 1 Processor -- start 2024-04-17T20:04:44.179 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c11200640 1 -- start start 2024-04-17T20:04:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c11200640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c0c06c660 0x7f7c0c1f9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c11200640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c0c0ca7f0 0x7f7c0c146aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c11200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c0c159c90 0x7f7c0c0c5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c11200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c0c7680 con 0x7f7c0c159c90 2024-04-17T20:04:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c11200640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f7c0c06cfb0 con 0x7f7c0c06c660 2024-04-17T20:04:44.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c11200640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f7c0c0c7800 con 0x7f7c0c0ca7f0 2024-04-17T20:04:44.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c0a800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c0c159c90 0x7f7c0c0c5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:44.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c0a800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c0c159c90 0x7f7c0c0c5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49328/0 (socket says 172.21.15.29:49328) 2024-04-17T20:04:44.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c0a800640 1 -- 172.21.15.29:0/3707048267 learned_addr learned my addr 172.21.15.29:0/3707048267 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:44.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c09e00640 1 --2- 172.21.15.29:0/3707048267 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c0c06c660 0x7f7c0c1f9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:44.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c03e00640 1 --2- 172.21.15.29:0/3707048267 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c0c0ca7f0 0x7f7c0c146aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:44.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c0a800640 1 -- 172.21.15.29:0/3707048267 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c0c0ca7f0 msgr2=0x7f7c0c146aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:44.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c0a800640 1 --2- 172.21.15.29:0/3707048267 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c0c0ca7f0 0x7f7c0c146aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:44.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c0a800640 1 -- 172.21.15.29:0/3707048267 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c0c06c660 msgr2=0x7f7c0c1f9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:44.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c0a800640 1 --2- 172.21.15.29:0/3707048267 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c0c06c660 0x7f7c0c1f9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:44.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c0a800640 1 -- 172.21.15.29:0/3707048267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0c1f57f0 con 0x7f7c0c159c90 2024-04-17T20:04:44.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c09e00640 1 --2- 172.21.15.29:0/3707048267 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c0c06c660 0x7f7c0c1f9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:04:44.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c03e00640 1 --2- 172.21.15.29:0/3707048267 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c0c0ca7f0 0x7f7c0c146aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:04:44.183 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c0a800640 1 --2- 172.21.15.29:0/3707048267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c0c159c90 0x7f7c0c0c5b80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400efc0 tx=0x7f7c0400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c02000640 1 -- 172.21.15.29:0/3707048267 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0400cb80 con 0x7f7c0c159c90 2024-04-17T20:04:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c02000640 1 -- 172.21.15.29:0/3707048267 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c0400cd20 con 0x7f7c0c159c90 2024-04-17T20:04:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c11200640 1 -- 172.21.15.29:0/3707048267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c0c1540c0 con 0x7f7c0c159c90 2024-04-17T20:04:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.180+0000 7f7c02000640 1 -- 172.21.15.29:0/3707048267 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c04003bc0 con 0x7f7c0c159c90 2024-04-17T20:04:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.184+0000 7f7c11200640 1 -- 172.21.15.29:0/3707048267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c0c164040 con 0x7f7c0c159c90 2024-04-17T20:04:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.184+0000 7f7c11200640 1 -- 172.21.15.29:0/3707048267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c0c069b90 con 0x7f7c0c159c90 2024-04-17T20:04:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.184+0000 7f7c02000640 1 -- 172.21.15.29:0/3707048267 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f7c0401b030 con 0x7f7c0c159c90 2024-04-17T20:04:44.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.184+0000 7f7c02000640 1 --2- 172.21.15.29:0/3707048267 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7be8068380 0x7f7be806a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:44.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.184+0000 7f7c02000640 1 -- 172.21.15.29:0/3707048267 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7423+0+0 (secure 0 0 0) 0x7f7c040873a0 con 0x7f7c0c159c90 2024-04-17T20:04:44.185 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.184+0000 7f7c09e00640 1 --2- 172.21.15.29:0/3707048267 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7be8068380 0x7f7be806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:44.186 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.188+0000 7f7c09e00640 1 --2- 172.21.15.29:0/3707048267 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7be8068380 0x7f7be806a890 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc0ae000 tx=0x7f7bfc0a6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:44.189 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.188+0000 7f7c02000640 1 -- 172.21.15.29:0/3707048267 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f7c04022020 con 0x7f7c0c159c90 2024-04-17T20:04:44.392 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.392+0000 7f7c11200640 1 -- 172.21.15.29:0/3707048267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c0c158a60 con 0x7f7c0c159c90 2024-04-17T20:04:44.393 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.392+0000 7f7c02000640 1 -- 172.21.15.29:0/3707048267 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+6088 (secure 0 0 0) 0x7f7c04010040 con 0x7f7c0c159c90 2024-04-17T20:04:44.393 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:44.393 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:04:40.014925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":6076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6076":{"gid":6076,"name":"f","rank":4,"incarnation":63,"state":"up:resolve","state_seq":33,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5946,5775,5631,5350]},"id":1}]} 2024-04-17T20:04:44.393 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 64 2024-04-17T20:04:44.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.396+0000 7f7c11200640 1 -- 172.21.15.29:0/3707048267 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7be8068380 msgr2=0x7f7be806a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:44.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.396+0000 7f7c11200640 1 --2- 172.21.15.29:0/3707048267 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7be8068380 0x7f7be806a890 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc0ae000 tx=0x7f7bfc0a6040 comp rx=0 tx=0).stop 2024-04-17T20:04:44.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.396+0000 7f7c11200640 1 -- 172.21.15.29:0/3707048267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c0c159c90 msgr2=0x7f7c0c0c5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:44.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.396+0000 7f7c11200640 1 --2- 172.21.15.29:0/3707048267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c0c159c90 0x7f7c0c0c5b80 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7c0400efc0 tx=0x7f7c0400c490 comp rx=0 tx=0).stop 2024-04-17T20:04:44.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.396+0000 7f7c11200640 1 -- 172.21.15.29:0/3707048267 shutdown_connections 2024-04-17T20:04:44.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.396+0000 7f7c11200640 1 --2- 172.21.15.29:0/3707048267 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f7be8068380 0x7f7be806a890 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:44.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.396+0000 7f7c11200640 1 --2- 172.21.15.29:0/3707048267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c0c159c90 0x7f7c0c0c5b80 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:44.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.396+0000 7f7c11200640 1 --2- 172.21.15.29:0/3707048267 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f7c0c0ca7f0 0x7f7c0c146aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:44.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.396+0000 7f7c11200640 1 --2- 172.21.15.29:0/3707048267 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f7c0c06c660 0x7f7c0c1f9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:44.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.396+0000 7f7c11200640 1 -- 172.21.15.29:0/3707048267 >> 172.21.15.29:0/3707048267 conn(0x7f7c0c06fde0 msgr2=0x7f7c0c14d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:44.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.396+0000 7f7c11200640 1 -- 172.21.15.29:0/3707048267 shutdown_connections 2024-04-17T20:04:44.396 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:44.396+0000 7f7c11200640 1 -- 172.21.15.29:0/3707048267 wait complete. 2024-04-17T20:04:44.415 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:04:46.416 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.568+0000 7f6211800640 1 Processor -- start 2024-04-17T20:04:46.569 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.568+0000 7f6211800640 1 -- start start 2024-04-17T20:04:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f6211800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f620c06bdc0 0x7f620c15fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f6211800640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f620c06cbf0 con 0x7f620c06c1d0 2024-04-17T20:04:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f6211800640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f620c06ceb0 con 0x7f620c1603a0 2024-04-17T20:04:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f6211800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f620c06d190 con 0x7f620c06bdc0 2024-04-17T20:04:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620b400640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f620c06c1d0 0x7f620c166a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46038/0 (socket says 172.21.15.29:46038) 2024-04-17T20:04:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620b400640 1 -- 172.21.15.29:0/3702670286 learned_addr learned my addr 172.21.15.29:0/3702670286 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:04:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620aa00640 1 --2- 172.21.15.29:0/3702670286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f620c06bdc0 0x7f620c15fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620a000640 1 -- 172.21.15.29:0/3702670286 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2487110326 0 0) 0x7f620c06cbf0 con 0x7f620c06c1d0 2024-04-17T20:04:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620a000640 1 -- 172.21.15.29:0/3702670286 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61f0003530 con 0x7f620c06c1d0 2024-04-17T20:04:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620a000640 1 -- 172.21.15.29:0/3702670286 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2338077562 0 0) 0x7f61f0003530 con 0x7f620c06c1d0 2024-04-17T20:04:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620a000640 1 -- 172.21.15.29:0/3702670286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f620c06bdc0 msgr2=0x7f620c15fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620a000640 1 --2- 172.21.15.29:0/3702670286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f620c06bdc0 0x7f620c15fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620a000640 1 -- 172.21.15.29:0/3702670286 >> v1:172.21.15.37:6789/0 conn(0x7f620c1603a0 legacy=0x7f620c0cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620a000640 1 -- 172.21.15.29:0/3702670286 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f620c06d470 con 0x7f620c06c1d0 2024-04-17T20:04:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620a000640 1 -- 172.21.15.29:0/3702670286 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3609959452 0 0) 0x7f6204001a00 con 0x7f620c06c1d0 2024-04-17T20:04:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620a000640 1 -- 172.21.15.29:0/3702670286 >> v1:172.21.15.29:6789/0 conn(0x7f620c06c1d0 legacy=0x7f620c166a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620a000640 1 --2- 172.21.15.29:0/3702670286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61f0003a60 0x7f61f0003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620a000640 1 --2- 172.21.15.29:0/3702670286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61f0004810 0x7f61f0004c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620a000640 1 --2- 172.21.15.29:0/3702670286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f620c06bdc0 0x7f61f0005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620a000640 1 -- 172.21.15.29:0/3702670286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f620c06d190 con 0x7f61f0004810 2024-04-17T20:04:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620a000640 1 -- 172.21.15.29:0/3702670286 --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f61f000bae0 con 0x7f61f0003a60 2024-04-17T20:04:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620a000640 1 -- 172.21.15.29:0/3702670286 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f61f000bce0 con 0x7f620c06bdc0 2024-04-17T20:04:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f6203e00640 1 --2- 172.21.15.29:0/3702670286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61f0004810 0x7f61f0004c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620aa00640 1 --2- 172.21.15.29:0/3702670286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61f0003a60 0x7f61f0003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620b400640 1 --2- 172.21.15.29:0/3702670286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f620c06bdc0 0x7f61f0005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f6203e00640 1 -- 172.21.15.29:0/3702670286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f620c06bdc0 msgr2=0x7f61f0005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f6203e00640 1 --2- 172.21.15.29:0/3702670286 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f620c06bdc0 0x7f61f0005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f6203e00640 1 -- 172.21.15.29:0/3702670286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61f0003a60 msgr2=0x7f61f0003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f6203e00640 1 --2- 172.21.15.29:0/3702670286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61f0003a60 0x7f61f0003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f6203e00640 1 -- 172.21.15.29:0/3702670286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f0003530 con 0x7f61f0004810 2024-04-17T20:04:46.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.572+0000 7f620aa00640 1 --2- 172.21.15.29:0/3702670286 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f61f0003a60 0x7f61f0003e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:04:46.575 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.576+0000 7f6203e00640 1 --2- 172.21.15.29:0/3702670286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61f0004810 0x7f61f0004c20 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f620c073f10 tx=0x7f61f401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a01fa8327c527d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:46.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.576+0000 7f620a000640 1 -- 172.21.15.29:0/3702670286 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f40b1070 con 0x7f61f0004810 2024-04-17T20:04:46.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.576+0000 7f620a000640 1 -- 172.21.15.29:0/3702670286 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61f40ac640 con 0x7f61f0004810 2024-04-17T20:04:46.576 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.576+0000 7f620a000640 1 -- 172.21.15.29:0/3702670286 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f4001460 con 0x7f61f0004810 2024-04-17T20:04:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.576+0000 7f6211800640 1 -- 172.21.15.29:0/3702670286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61f0004810 msgr2=0x7f61f0004c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.576+0000 7f6211800640 1 --2- 172.21.15.29:0/3702670286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61f0004810 0x7f61f0004c20 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f620c073f10 tx=0x7f61f401c4e0 comp rx=0 tx=0).stop 2024-04-17T20:04:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.576+0000 7f620b400640 1 -- 172.21.15.29:0/3702670286 reap_dead start 2024-04-17T20:04:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.576+0000 7f6211800640 1 -- 172.21.15.29:0/3702670286 shutdown_connections 2024-04-17T20:04:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.576+0000 7f6211800640 1 -- 172.21.15.29:0/3702670286 >> 172.21.15.29:0/3702670286 conn(0x7f620c06e3d0 msgr2=0x7f620c06e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.576+0000 7f6211800640 1 -- 172.21.15.29:0/3702670286 shutdown_connections 2024-04-17T20:04:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.576+0000 7f6211800640 1 -- 172.21.15.29:0/3702670286 wait complete. 2024-04-17T20:04:46.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6211800640 1 Processor -- start 2024-04-17T20:04:46.579 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6211800640 1 -- start start 2024-04-17T20:04:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6211800640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61f0004810 0x7f620c0b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6211800640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f620c1f6000 0x7f620c14bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6211800640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f620c152ff0 0x7f620c14c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6211800640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f620c15cff0 con 0x7f620c1f6000 2024-04-17T20:04:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6211800640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f620c15c1c0 con 0x7f620c152ff0 2024-04-17T20:04:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6211800640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f620c15d170 con 0x7f61f0004810 2024-04-17T20:04:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6203e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f620c1f6000 0x7f620c14bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6203e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f620c1f6000 0x7f620c14bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49342/0 (socket says 172.21.15.29:49342) 2024-04-17T20:04:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6203e00640 1 -- 172.21.15.29:0/2853657762 learned_addr learned my addr 172.21.15.29:0/2853657762 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f620b400640 1 --2- 172.21.15.29:0/2853657762 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f620c152ff0 0x7f620c14c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f620aa00640 1 --2- 172.21.15.29:0/2853657762 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61f0004810 0x7f620c0b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6203e00640 1 -- 172.21.15.29:0/2853657762 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61f0004810 msgr2=0x7f620c0b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6203e00640 1 --2- 172.21.15.29:0/2853657762 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61f0004810 0x7f620c0b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6203e00640 1 -- 172.21.15.29:0/2853657762 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f620c152ff0 msgr2=0x7f620c14c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6203e00640 1 --2- 172.21.15.29:0/2853657762 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f620c152ff0 0x7f620c14c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6203e00640 1 -- 172.21.15.29:0/2853657762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f620c0cd3c0 con 0x7f620c1f6000 2024-04-17T20:04:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6203e00640 1 --2- 172.21.15.29:0/2853657762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f620c1f6000 0x7f620c14bc50 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f61f401c310 tx=0x7f61f0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:46.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6202a00640 1 -- 172.21.15.29:0/2853657762 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f40b1040 con 0x7f620c1f6000 2024-04-17T20:04:46.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6202a00640 1 -- 172.21.15.29:0/2853657762 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61f40bc070 con 0x7f620c1f6000 2024-04-17T20:04:46.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6202a00640 1 -- 172.21.15.29:0/2853657762 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f4005270 con 0x7f620c1f6000 2024-04-17T20:04:46.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6211800640 1 -- 172.21.15.29:0/2853657762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f620c0ce140 con 0x7f620c1f6000 2024-04-17T20:04:46.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.580+0000 7f6211800640 1 -- 172.21.15.29:0/2853657762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f620c0c5d50 con 0x7f620c1f6000 2024-04-17T20:04:46.583 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.584+0000 7f6211800640 1 -- 172.21.15.29:0/2853657762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f620c069b90 con 0x7f620c1f6000 2024-04-17T20:04:46.584 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.584+0000 7f6202a00640 1 -- 172.21.15.29:0/2853657762 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f61f40067e0 con 0x7f620c1f6000 2024-04-17T20:04:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.584+0000 7f6202a00640 1 --2- 172.21.15.29:0/2853657762 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f61e80683b0 0x7f61e806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.584+0000 7f6202a00640 1 -- 172.21.15.29:0/2853657762 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7423+0+0 (secure 0 0 0) 0x7f61f41212d0 con 0x7f620c1f6000 2024-04-17T20:04:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.584+0000 7f620aa00640 1 --2- 172.21.15.29:0/2853657762 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f61e80683b0 0x7f61e806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.588+0000 7f620aa00640 1 --2- 172.21.15.29:0/2853657762 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f61e80683b0 0x7f61e806a8c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f61fc009ab0 tx=0x7f61fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:46.594 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.592+0000 7f6202a00640 1 -- 172.21.15.29:0/2853657762 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f61f40161f0 con 0x7f620c1f6000 2024-04-17T20:04:46.794 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.796+0000 7f6211800640 1 -- 172.21.15.29:0/2853657762 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f620c0b1810 con 0x7f620c1f6000 2024-04-17T20:04:46.795 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.796+0000 7f6202a00640 1 -- 172.21.15.29:0/2853657762 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+6090 (secure 0 0 0) 0x7f61f40f10b0 con 0x7f620c1f6000 2024-04-17T20:04:46.795 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:46.795 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:04:45.671297+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":6076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6076":{"gid":6076,"name":"f","rank":4,"incarnation":63,"state":"up:reconnect","state_seq":35,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5350,5631,5775,5946]},"id":1}]} 2024-04-17T20:04:46.796 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 65 2024-04-17T20:04:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.796+0000 7f6211800640 1 -- 172.21.15.29:0/2853657762 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f61e80683b0 msgr2=0x7f61e806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.796+0000 7f6211800640 1 --2- 172.21.15.29:0/2853657762 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f61e80683b0 0x7f61e806a8c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f61fc009ab0 tx=0x7f61fc008040 comp rx=0 tx=0).stop 2024-04-17T20:04:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.796+0000 7f6211800640 1 -- 172.21.15.29:0/2853657762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f620c1f6000 msgr2=0x7f620c14bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.796+0000 7f6211800640 1 --2- 172.21.15.29:0/2853657762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f620c1f6000 0x7f620c14bc50 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f61f401c310 tx=0x7f61f0005840 comp rx=0 tx=0).stop 2024-04-17T20:04:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.796+0000 7f6211800640 1 -- 172.21.15.29:0/2853657762 shutdown_connections 2024-04-17T20:04:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.796+0000 7f6211800640 1 --2- 172.21.15.29:0/2853657762 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f61e80683b0 0x7f61e806a8c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.796+0000 7f6211800640 1 --2- 172.21.15.29:0/2853657762 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f620c152ff0 0x7f620c14c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.796+0000 7f6211800640 1 --2- 172.21.15.29:0/2853657762 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f620c1f6000 0x7f620c14bc50 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:46.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.796+0000 7f6211800640 1 --2- 172.21.15.29:0/2853657762 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f61f0004810 0x7f620c0b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:46.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.796+0000 7f6211800640 1 -- 172.21.15.29:0/2853657762 >> 172.21.15.29:0/2853657762 conn(0x7f620c06e3d0 msgr2=0x7f620c0cc320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:46.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.796+0000 7f6211800640 1 -- 172.21.15.29:0/2853657762 shutdown_connections 2024-04-17T20:04:46.799 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:46.796+0000 7f6211800640 1 -- 172.21.15.29:0/2853657762 wait complete. 2024-04-17T20:04:46.819 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 65, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-17T19:47:49.315515+0000", "damaged": [], "data_pools": [ 3, 4 ], "enabled": true, "epoch": 65, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2, 3, 4 ], "info": { "gid_5350": { "addr": "172.21.15.37:6837/2495953425", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6834", "nonce": 2495953425, "type": "v2" }, { "addr": "172.21.15.37:6837", "nonce": 2495953425, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5350, "incarnation": 32, "join_fscid": -1, "name": "d", "rank": 3, "state": "up:active", "state_seq": 18 }, "gid_5631": { "addr": "172.21.15.29:6836/2376553580", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6832", "nonce": 2376553580, "type": "v2" }, { "addr": "172.21.15.29:6836", "nonce": 2376553580, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5631, "incarnation": 44, "join_fscid": -1, "name": "a", "rank": 1, "state": "up:active", "state_seq": 42 }, "gid_5775": { "addr": "172.21.15.29:6835/1352350250", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6834", "nonce": 1352350250, "type": "v2" }, { "addr": "172.21.15.29:6835", "nonce": 1352350250, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5775, "incarnation": 50, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 39 }, "gid_5946": { "addr": "172.21.15.29:6837/3087508289", "addrs": { "addrvec": [ { "addr": "172.21.15.29:6833", "nonce": 3087508289, "type": "v2" }, { "addr": "172.21.15.29:6837", "nonce": 3087508289, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5946, "incarnation": 57, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:active", "state_seq": 25 }, "gid_6076": { "addr": "172.21.15.37:6839/2273456671", "addrs": { "addrvec": [ { "addr": "172.21.15.37:6836", "nonce": 2273456671, "type": "v2" }, { "addr": "172.21.15.37:6839", "nonce": 2273456671, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6076, "incarnation": 63, "join_fscid": -1, "name": "f", "rank": 4, "state": "up:reconnect", "state_seq": 35 } }, "last_failure": 0, "last_failure_osd_epoch": 76, "max_file_size": 1099511627776, "max_mds": 5, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-17T20:04:45.671297+0000", "qdb_cluster": [ 5350, 5631, 5775, 5946 ], "qdb_leader": 5775, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5775, "mds_1": 5631, "mds_2": 5946, "mds_3": 5350, "mds_4": 6076 } } } ], "standbys": [] } 2024-04-17T20:04:48.820 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:48.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.984+0000 7fba8a600640 1 Processor -- start 2024-04-17T20:04:48.986 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.988+0000 7fba8a600640 1 -- start start 2024-04-17T20:04:48.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.988+0000 7fba8a600640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fba840cabe0 0x7fba840caff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:48.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.988+0000 7fba8a600640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7fba8415c260 0x7fba8415c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:48.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.988+0000 7fba8a600640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba8406c730 con 0x7fba84156370 2024-04-17T20:04:48.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.988+0000 7fba8a600640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7fba8406c9f0 con 0x7fba840cabe0 2024-04-17T20:04:48.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.988+0000 7fba8a600640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7fba8406ccd0 con 0x7fba8415c260 2024-04-17T20:04:48.988 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.988+0000 7fba83e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fba840cabe0 0x7fba840caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:48.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.988+0000 7fba89200640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fba84156370 0x7fba840ca110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44418/0 (socket says 172.21.15.29:44418) 2024-04-17T20:04:48.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.988+0000 7fba89200640 1 -- 172.21.15.29:0/1301142065 learned_addr learned my addr 172.21.15.29:0/1301142065 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:04:48.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.988+0000 7fba89c00640 1 --2- 172.21.15.29:0/1301142065 >> v2:172.21.15.37:3300/0 conn(0x7fba8415c260 0x7fba8415c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:48.989 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.988+0000 7fba83400640 1 -- 172.21.15.29:0/1301142065 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3079364946 0 0) 0x7fba8406c730 con 0x7fba84156370 2024-04-17T20:04:48.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.988+0000 7fba83400640 1 -- 172.21.15.29:0/1301142065 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba6c003530 con 0x7fba84156370 2024-04-17T20:04:48.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.988+0000 7fba83e00640 1 -- 172.21.15.29:0/1301142065 >> v2:172.21.15.37:3300/0 conn(0x7fba8415c260 msgr2=0x7fba8415c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:48.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.988+0000 7fba83e00640 1 --2- 172.21.15.29:0/1301142065 >> v2:172.21.15.37:3300/0 conn(0x7fba8415c260 0x7fba8415c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:48.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.988+0000 7fba83e00640 1 -- 172.21.15.29:0/1301142065 >> v1:172.21.15.29:6789/0 conn(0x7fba84156370 legacy=0x7fba840ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:48.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.988+0000 7fba83e00640 1 -- 172.21.15.29:0/1301142065 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba8406cfb0 con 0x7fba840cabe0 2024-04-17T20:04:48.990 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.988+0000 7fba89c00640 1 --2- 172.21.15.29:0/1301142065 >> v2:172.21.15.37:3300/0 conn(0x7fba8415c260 0x7fba8415c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:04:48.991 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.992+0000 7fba83e00640 1 --2- 172.21.15.29:0/1301142065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba840cabe0 0x7fba840caff0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fba840714c0 tx=0x7fba7001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a96645734d1ca731 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:48.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.992+0000 7fba83400640 1 -- 172.21.15.29:0/1301142065 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba700b5070 con 0x7fba840cabe0 2024-04-17T20:04:48.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.992+0000 7fba83400640 1 -- 172.21.15.29:0/1301142065 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba700a8970 con 0x7fba840cabe0 2024-04-17T20:04:48.992 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.992+0000 7fba83400640 1 -- 172.21.15.29:0/1301142065 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba700a8d40 con 0x7fba840cabe0 2024-04-17T20:04:48.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.992+0000 7fba8a600640 1 -- 172.21.15.29:0/1301142065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba840cabe0 msgr2=0x7fba840caff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:48.993 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.992+0000 7fba8a600640 1 --2- 172.21.15.29:0/1301142065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba840cabe0 0x7fba840caff0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fba840714c0 tx=0x7fba7001ec70 comp rx=0 tx=0).stop 2024-04-17T20:04:48.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.992+0000 7fba8a600640 1 -- 172.21.15.29:0/1301142065 shutdown_connections 2024-04-17T20:04:48.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.992+0000 7fba8a600640 1 --2- 172.21.15.29:0/1301142065 >> v2:172.21.15.37:3300/0 conn(0x7fba8415c260 0x7fba8415c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:48.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.992+0000 7fba8a600640 1 --2- 172.21.15.29:0/1301142065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba840cabe0 0x7fba840caff0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:48.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.992+0000 7fba8a600640 1 -- 172.21.15.29:0/1301142065 >> 172.21.15.29:0/1301142065 conn(0x7fba840680e0 msgr2=0x7fba8406dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:48.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.992+0000 7fba8a600640 1 -- 172.21.15.29:0/1301142065 shutdown_connections 2024-04-17T20:04:48.994 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.992+0000 7fba8a600640 1 -- 172.21.15.29:0/1301142065 wait complete. 2024-04-17T20:04:48.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba8a600640 1 Processor -- start 2024-04-17T20:04:48.995 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba8a600640 1 -- start start 2024-04-17T20:04:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba8a600640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba840cabe0 0x7fba840af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba8a600640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba84156370 0x7fba841f3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba8a600640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba8415c260 0x7fba840b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba8a600640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fba8415cbb0 con 0x7fba84156370 2024-04-17T20:04:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba8a600640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fba8406d570 con 0x7fba8415c260 2024-04-17T20:04:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba8a600640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fba8406e7c0 con 0x7fba840cabe0 2024-04-17T20:04:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba83e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba84156370 0x7fba841f3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba83e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba84156370 0x7fba841f3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40836/0 (socket says 172.21.15.29:40836) 2024-04-17T20:04:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba83e00640 1 -- 172.21.15.29:0/1529446302 learned_addr learned my addr 172.21.15.29:0/1529446302 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba89200640 1 --2- 172.21.15.29:0/1529446302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba840cabe0 0x7fba840af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba89c00640 1 --2- 172.21.15.29:0/1529446302 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba8415c260 0x7fba840b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba83e00640 1 -- 172.21.15.29:0/1529446302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba840cabe0 msgr2=0x7fba840af9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba83e00640 1 --2- 172.21.15.29:0/1529446302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba840cabe0 0x7fba840af9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba83e00640 1 -- 172.21.15.29:0/1529446302 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba8415c260 msgr2=0x7fba840b3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba83e00640 1 --2- 172.21.15.29:0/1529446302 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba8415c260 0x7fba840b3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba83e00640 1 -- 172.21.15.29:0/1529446302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba841f3dc0 con 0x7fba84156370 2024-04-17T20:04:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba83e00640 1 --2- 172.21.15.29:0/1529446302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba84156370 0x7fba841f3880 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fba700b3040 tx=0x7fba7001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba80c00640 1 -- 172.21.15.29:0/1529446302 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba700bf070 con 0x7fba84156370 2024-04-17T20:04:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba80c00640 1 -- 172.21.15.29:0/1529446302 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba700b0720 con 0x7fba84156370 2024-04-17T20:04:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba80c00640 1 -- 172.21.15.29:0/1529446302 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba700b5040 con 0x7fba84156370 2024-04-17T20:04:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:48.996+0000 7fba8a600640 1 -- 172.21.15.29:0/1529446302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba840c9450 con 0x7fba84156370 2024-04-17T20:04:49.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.000+0000 7fba8a600640 1 -- 172.21.15.29:0/1529446302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba840c8e70 con 0x7fba84156370 2024-04-17T20:04:49.001 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.000+0000 7fba8a600640 1 -- 172.21.15.29:0/1529446302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba8404ff10 con 0x7fba84156370 2024-04-17T20:04:49.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.000+0000 7fba80c00640 1 -- 172.21.15.29:0/1529446302 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fba700b0b10 con 0x7fba84156370 2024-04-17T20:04:49.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.000+0000 7fba80c00640 1 --2- 172.21.15.29:0/1529446302 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fba640683b0 0x7fba6406a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:49.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.000+0000 7fba80c00640 1 -- 172.21.15.29:0/1529446302 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7423+0+0 (secure 0 0 0) 0x7fba70123830 con 0x7fba84156370 2024-04-17T20:04:49.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.000+0000 7fba89200640 1 --2- 172.21.15.29:0/1529446302 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fba640683b0 0x7fba6406a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:49.002 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.004+0000 7fba89200640 1 --2- 172.21.15.29:0/1529446302 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fba640683b0 0x7fba6406a8c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fba78005890 tx=0x7fba78009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:49.008 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.004+0000 7fba80c00640 1 -- 172.21.15.29:0/1529446302 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fba700187d0 con 0x7fba84156370 2024-04-17T20:04:49.208 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.208+0000 7fba8a600640 1 -- 172.21.15.29:0/1529446302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba840cb480 con 0x7fba84156370 2024-04-17T20:04:49.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.208+0000 7fba80c00640 1 -- 172.21.15.29:0/1529446302 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+6090 (secure 0 0 0) 0x7fba700f3730 con 0x7fba84156370 2024-04-17T20:04:49.209 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:49.209 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:04:45.671297+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":6076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6076":{"gid":6076,"name":"f","rank":4,"incarnation":63,"state":"up:reconnect","state_seq":35,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5350,5631,5775,5946]},"id":1}]} 2024-04-17T20:04:49.209 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 65 2024-04-17T20:04:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.212+0000 7fba8a600640 1 -- 172.21.15.29:0/1529446302 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fba640683b0 msgr2=0x7fba6406a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.212+0000 7fba8a600640 1 --2- 172.21.15.29:0/1529446302 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fba640683b0 0x7fba6406a8c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fba78005890 tx=0x7fba78009040 comp rx=0 tx=0).stop 2024-04-17T20:04:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.212+0000 7fba8a600640 1 -- 172.21.15.29:0/1529446302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba84156370 msgr2=0x7fba841f3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.212+0000 7fba8a600640 1 --2- 172.21.15.29:0/1529446302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba84156370 0x7fba841f3880 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fba700b3040 tx=0x7fba7001ba10 comp rx=0 tx=0).stop 2024-04-17T20:04:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.212+0000 7fba8a600640 1 -- 172.21.15.29:0/1529446302 shutdown_connections 2024-04-17T20:04:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.212+0000 7fba8a600640 1 --2- 172.21.15.29:0/1529446302 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fba640683b0 0x7fba6406a8c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.212+0000 7fba8a600640 1 --2- 172.21.15.29:0/1529446302 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fba8415c260 0x7fba840b3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.212+0000 7fba8a600640 1 --2- 172.21.15.29:0/1529446302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fba84156370 0x7fba841f3880 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:49.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.212+0000 7fba8a600640 1 --2- 172.21.15.29:0/1529446302 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fba840cabe0 0x7fba840af9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:49.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.212+0000 7fba8a600640 1 -- 172.21.15.29:0/1529446302 >> 172.21.15.29:0/1529446302 conn(0x7fba840680e0 msgr2=0x7fba84071630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:49.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.212+0000 7fba8a600640 1 -- 172.21.15.29:0/1529446302 shutdown_connections 2024-04-17T20:04:49.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:49.212+0000 7fba8a600640 1 -- 172.21.15.29:0/1529446302 wait complete. 2024-04-17T20:04:49.231 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:04:51.231 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:51.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.384+0000 7f26b6c00640 1 Processor -- start 2024-04-17T20:04:51.384 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.384+0000 7f26b6c00640 1 -- start start 2024-04-17T20:04:51.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.384+0000 7f26b6c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26b015fc80 0x7f26b0160090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:51.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.384+0000 7f26b6c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f26b00cc4d0 0x7f26b00cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:51.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.384+0000 7f26b6c00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26b006c900 con 0x7f26b00caf80 2024-04-17T20:04:51.386 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.384+0000 7f26b6c00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f26b015c050 con 0x7f26b00cc4d0 2024-04-17T20:04:51.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.384+0000 7f26b6c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f26b006c690 con 0x7f26b015fc80 2024-04-17T20:04:51.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.384+0000 7f26b4c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f26b00cc4d0 0x7f26b00cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:51.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.384+0000 7f26b4c00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f26b00cc4d0 0x7f26b00cc8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40846/0 (socket says 172.21.15.29:40846) 2024-04-17T20:04:51.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.384+0000 7f26b4c00640 1 -- 172.21.15.29:0/3152951774 learned_addr learned my addr 172.21.15.29:0/3152951774 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:51.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.384+0000 7f26a7e00640 1 --2- 172.21.15.29:0/3152951774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26b015fc80 0x7f26b0160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:51.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.388+0000 7f26b4c00640 1 -- 172.21.15.29:0/3152951774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26b015fc80 msgr2=0x7f26b0160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:51.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.388+0000 7f26b4c00640 1 --2- 172.21.15.29:0/3152951774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26b015fc80 0x7f26b0160090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:51.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.388+0000 7f26b4c00640 1 -- 172.21.15.29:0/3152951774 >> v1:172.21.15.37:6789/0 conn(0x7f26b00caf80 legacy=0x7f26b015c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:51.387 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.388+0000 7f26b4c00640 1 -- 172.21.15.29:0/3152951774 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26b006cb80 con 0x7f26b00cc4d0 2024-04-17T20:04:51.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.388+0000 7f26b4c00640 1 --2- 172.21.15.29:0/3152951774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b00cc4d0 0x7f26b00cc8e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f26a8009e50 tx=0x7f26a801f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=226ead5416b0bce6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:51.389 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.388+0000 7f26af400640 1 -- 172.21.15.29:0/3152951774 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a80b4070 con 0x7f26b00cc4d0 2024-04-17T20:04:51.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.388+0000 7f26af400640 1 -- 172.21.15.29:0/3152951774 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26a80af400 con 0x7f26b00cc4d0 2024-04-17T20:04:51.390 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.388+0000 7f26af400640 1 -- 172.21.15.29:0/3152951774 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a80af7d0 con 0x7f26b00cc4d0 2024-04-17T20:04:51.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.392+0000 7f26b6c00640 1 -- 172.21.15.29:0/3152951774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b00cc4d0 msgr2=0x7f26b00cc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:51.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.392+0000 7f26b6c00640 1 --2- 172.21.15.29:0/3152951774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b00cc4d0 0x7f26b00cc8e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f26a8009e50 tx=0x7f26a801f210 comp rx=0 tx=0).stop 2024-04-17T20:04:51.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.392+0000 7f26b6c00640 1 -- 172.21.15.29:0/3152951774 shutdown_connections 2024-04-17T20:04:51.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.392+0000 7f26b6c00640 1 --2- 172.21.15.29:0/3152951774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b00cc4d0 0x7f26b00cc8e0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:51.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.392+0000 7f26b6c00640 1 --2- 172.21.15.29:0/3152951774 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26b015fc80 0x7f26b0160090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:51.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.392+0000 7f26b6c00640 1 -- 172.21.15.29:0/3152951774 >> 172.21.15.29:0/3152951774 conn(0x7f26b006e3d0 msgr2=0x7f26b006e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:51.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.392+0000 7f26b6c00640 1 -- 172.21.15.29:0/3152951774 shutdown_connections 2024-04-17T20:04:51.391 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.392+0000 7f26b6c00640 1 -- 172.21.15.29:0/3152951774 wait complete. 2024-04-17T20:04:51.394 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.392+0000 7f26b6c00640 1 Processor -- start 2024-04-17T20:04:51.395 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26b6c00640 1 -- start start 2024-04-17T20:04:51.399 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26b6c00640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26b00caf80 0x7f26b00b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26b6c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b00cc4d0 0x7f26b01e5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26b6c00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26b015fc80 0x7f26b01e71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26b6c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f26b006d420 con 0x7f26b00cc4d0 2024-04-17T20:04:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26b6c00640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f26b006d140 con 0x7f26b00caf80 2024-04-17T20:04:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26b6c00640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f26b015f760 con 0x7f26b015fc80 2024-04-17T20:04:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26a7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b00cc4d0 0x7f26b01e5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26a7e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b00cc4d0 0x7f26b01e5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40854/0 (socket says 172.21.15.29:40854) 2024-04-17T20:04:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26a7e00640 1 -- 172.21.15.29:0/3847980647 learned_addr learned my addr 172.21.15.29:0/3847980647 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26afe00640 1 --2- 172.21.15.29:0/3847980647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26b00caf80 0x7f26b00b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26b4c00640 1 --2- 172.21.15.29:0/3847980647 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26b015fc80 0x7f26b01e71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26a7e00640 1 -- 172.21.15.29:0/3847980647 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26b015fc80 msgr2=0x7f26b01e71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26a7e00640 1 --2- 172.21.15.29:0/3847980647 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26b015fc80 0x7f26b01e71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26a7e00640 1 -- 172.21.15.29:0/3847980647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26b00caf80 msgr2=0x7f26b00b0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26a7e00640 1 --2- 172.21.15.29:0/3847980647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26b00caf80 0x7f26b00b0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26a7e00640 1 -- 172.21.15.29:0/3847980647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26b00b0970 con 0x7f26b00cc4d0 2024-04-17T20:04:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26b4c00640 1 --2- 172.21.15.29:0/3847980647 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26b015fc80 0x7f26b01e71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:04:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26a7e00640 1 --2- 172.21.15.29:0/3847980647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b00cc4d0 0x7f26b01e5370 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f269800d620 tx=0x7f269800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26acc00640 1 -- 172.21.15.29:0/3847980647 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f269800b620 con 0x7f26b00cc4d0 2024-04-17T20:04:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26acc00640 1 -- 172.21.15.29:0/3847980647 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2698019070 con 0x7f26b00cc4d0 2024-04-17T20:04:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26acc00640 1 -- 172.21.15.29:0/3847980647 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f269800b9e0 con 0x7f26b00cc4d0 2024-04-17T20:04:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.396+0000 7f26b6c00640 1 -- 172.21.15.29:0/3847980647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26b01f24b0 con 0x7f26b00cc4d0 2024-04-17T20:04:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.400+0000 7f26b6c00640 1 -- 172.21.15.29:0/3847980647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26b01f2a70 con 0x7f26b00cc4d0 2024-04-17T20:04:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.400+0000 7f26b6c00640 1 -- 172.21.15.29:0/3847980647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26b0069b90 con 0x7f26b00cc4d0 2024-04-17T20:04:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.400+0000 7f26acc00640 1 -- 172.21.15.29:0/3847980647 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f269800bb80 con 0x7f26b00cc4d0 2024-04-17T20:04:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.400+0000 7f26acc00640 1 --2- 172.21.15.29:0/3847980647 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f268c068380 0x7f268c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.400+0000 7f26acc00640 1 -- 172.21.15.29:0/3847980647 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7423+0+0 (secure 0 0 0) 0x7f2698087420 con 0x7f26b00cc4d0 2024-04-17T20:04:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.400+0000 7f26afe00640 1 --2- 172.21.15.29:0/3847980647 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f268c068380 0x7f268c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.404+0000 7f26afe00640 1 --2- 172.21.15.29:0/3847980647 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f268c068380 0x7f268c06a890 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f26a0001770 tx=0x7f26a0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.404+0000 7f26acc00640 1 -- 172.21.15.29:0/3847980647 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f2698057330 con 0x7f26b00cc4d0 2024-04-17T20:04:51.610 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.612+0000 7f26b6c00640 1 -- 172.21.15.29:0/3847980647 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26b00cad60 con 0x7f26b00cc4d0 2024-04-17T20:04:51.611 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.612+0000 7f26acc00640 1 -- 172.21.15.29:0/3847980647 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+6087 (secure 0 0 0) 0x7f2698056c30 con 0x7f26b00cc4d0 2024-04-17T20:04:51.611 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:51.611 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:04:50.691756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":6076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6076":{"gid":6076,"name":"f","rank":4,"incarnation":63,"state":"up:rejoin","state_seq":37,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5946,5775,5631,5350]},"id":1}]} 2024-04-17T20:04:51.611 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 66 2024-04-17T20:04:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.612+0000 7f26b6c00640 1 -- 172.21.15.29:0/3847980647 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f268c068380 msgr2=0x7f268c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.612+0000 7f26b6c00640 1 --2- 172.21.15.29:0/3847980647 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f268c068380 0x7f268c06a890 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f26a0001770 tx=0x7f26a0009040 comp rx=0 tx=0).stop 2024-04-17T20:04:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.612+0000 7f26b6c00640 1 -- 172.21.15.29:0/3847980647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b00cc4d0 msgr2=0x7f26b01e5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.612+0000 7f26b6c00640 1 --2- 172.21.15.29:0/3847980647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b00cc4d0 0x7f26b01e5370 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f269800d620 tx=0x7f269800dae0 comp rx=0 tx=0).stop 2024-04-17T20:04:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.612+0000 7f26b6c00640 1 -- 172.21.15.29:0/3847980647 shutdown_connections 2024-04-17T20:04:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.612+0000 7f26b6c00640 1 --2- 172.21.15.29:0/3847980647 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f268c068380 0x7f268c06a890 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.612+0000 7f26b6c00640 1 --2- 172.21.15.29:0/3847980647 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f26b015fc80 0x7f26b01e71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.612+0000 7f26b6c00640 1 --2- 172.21.15.29:0/3847980647 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26b00cc4d0 0x7f26b01e5370 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.612+0000 7f26b6c00640 1 --2- 172.21.15.29:0/3847980647 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f26b00caf80 0x7f26b00b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.612+0000 7f26b6c00640 1 -- 172.21.15.29:0/3847980647 >> 172.21.15.29:0/3847980647 conn(0x7f26b006e3d0 msgr2=0x7f26b00af7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.612+0000 7f26b6c00640 1 -- 172.21.15.29:0/3847980647 shutdown_connections 2024-04-17T20:04:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:51.612+0000 7f26b6c00640 1 -- 172.21.15.29:0/3847980647 wait complete. 2024-04-17T20:04:51.635 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:04:53.636 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:53.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.800+0000 7faa77000640 1 Processor -- start 2024-04-17T20:04:53.802 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.800+0000 7faa77000640 1 -- start start 2024-04-17T20:04:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.804+0000 7faa77000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa700cccd0 0x7faa700ca330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.804+0000 7faa77000640 1 --2- >> v2:172.21.15.37:3300/0 conn(0x7faa700ca870 0x7faa701635e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.804+0000 7faa77000640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa700cd0e0 con 0x7faa700cc8c0 2024-04-17T20:04:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.804+0000 7faa77000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7faa700c88b0 con 0x7faa700cccd0 2024-04-17T20:04:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.804+0000 7faa77000640 1 -- --> v2:172.21.15.37:3300/0 -- mon_getmap magic: 0 -- 0x7faa700c6ce0 con 0x7faa700ca870 2024-04-17T20:04:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.804+0000 7faa6fe00640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faa700cc8c0 0x7faa70159390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44428/0 (socket says 172.21.15.29:44428) 2024-04-17T20:04:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.804+0000 7faa6fe00640 1 -- 172.21.15.29:0/838391083 learned_addr learned my addr 172.21.15.29:0/838391083 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:04:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.804+0000 7faa67400640 1 --2- 172.21.15.29:0/838391083 >> v2:172.21.15.29:3300/0 conn(0x7faa700cccd0 0x7faa700ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:53.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.804+0000 7faa75000640 1 --2- 172.21.15.29:0/838391083 >> v2:172.21.15.37:3300/0 conn(0x7faa700ca870 0x7faa701635e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:53.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.804+0000 7faa67400640 1 -- 172.21.15.29:0/838391083 >> v2:172.21.15.37:3300/0 conn(0x7faa700ca870 msgr2=0x7faa701635e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:53.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.804+0000 7faa67400640 1 --2- 172.21.15.29:0/838391083 >> v2:172.21.15.37:3300/0 conn(0x7faa700ca870 0x7faa701635e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:53.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.804+0000 7faa67400640 1 -- 172.21.15.29:0/838391083 >> v1:172.21.15.29:6789/0 conn(0x7faa700cc8c0 legacy=0x7faa70159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:53.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.804+0000 7faa67400640 1 -- 172.21.15.29:0/838391083 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa700c6fc0 con 0x7faa700cccd0 2024-04-17T20:04:53.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.804+0000 7faa6f400640 1 -- 172.21.15.29:0/838391083 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 601077288 0 0) 0x7faa700cd0e0 con 0x7faa700cc8c0 2024-04-17T20:04:53.805 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.804+0000 7faa75000640 1 --2- 172.21.15.29:0/838391083 >> v2:172.21.15.37:3300/0 conn(0x7faa700ca870 0x7faa701635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T20:04:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.808+0000 7faa67400640 1 --2- 172.21.15.29:0/838391083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa700cccd0 0x7faa700ca330 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7faa700722d0 tx=0x7faa5801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1f059cdad7185bc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.808+0000 7faa6f400640 1 -- 172.21.15.29:0/838391083 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa580b0070 con 0x7faa700cccd0 2024-04-17T20:04:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.808+0000 7faa6f400640 1 -- 172.21.15.29:0/838391083 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa580ab600 con 0x7faa700cccd0 2024-04-17T20:04:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.808+0000 7faa6f400640 1 -- 172.21.15.29:0/838391083 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa580ab9d0 con 0x7faa700cccd0 2024-04-17T20:04:53.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.808+0000 7faa77000640 1 -- 172.21.15.29:0/838391083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa700cccd0 msgr2=0x7faa700ca330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:53.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.808+0000 7faa77000640 1 --2- 172.21.15.29:0/838391083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa700cccd0 0x7faa700ca330 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7faa700722d0 tx=0x7faa5801c480 comp rx=0 tx=0).stop 2024-04-17T20:04:53.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.808+0000 7faa77000640 1 -- 172.21.15.29:0/838391083 shutdown_connections 2024-04-17T20:04:53.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.808+0000 7faa77000640 1 --2- 172.21.15.29:0/838391083 >> v2:172.21.15.37:3300/0 conn(0x7faa700ca870 0x7faa701635e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:53.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.808+0000 7faa77000640 1 --2- 172.21.15.29:0/838391083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa700cccd0 0x7faa700ca330 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:53.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.808+0000 7faa77000640 1 -- 172.21.15.29:0/838391083 >> 172.21.15.29:0/838391083 conn(0x7faa7006fde0 msgr2=0x7faa700701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:53.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.808+0000 7faa77000640 1 -- 172.21.15.29:0/838391083 shutdown_connections 2024-04-17T20:04:53.809 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.808+0000 7faa77000640 1 -- 172.21.15.29:0/838391083 wait complete. 2024-04-17T20:04:53.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa77000640 1 Processor -- start 2024-04-17T20:04:53.811 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa77000640 1 -- start start 2024-04-17T20:04:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa77000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa700ca870 0x7faa70155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa77000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa700cc8c0 0x7faa701541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa77000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa700cccd0 0x7faa7013d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa77000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7faa70159c20 con 0x7faa700cccd0 2024-04-17T20:04:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa77000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7faa70159aa0 con 0x7faa700cc8c0 2024-04-17T20:04:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa77000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7faa700c6880 con 0x7faa700ca870 2024-04-17T20:04:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa75000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa700cccd0 0x7faa7013d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa75000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa700cccd0 0x7faa7013d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40876/0 (socket says 172.21.15.29:40876) 2024-04-17T20:04:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa75000640 1 -- 172.21.15.29:0/169131000 learned_addr learned my addr 172.21.15.29:0/169131000 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:53.814 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa6fe00640 1 --2- 172.21.15.29:0/169131000 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa700ca870 0x7faa70155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa67400640 1 --2- 172.21.15.29:0/169131000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa700cc8c0 0x7faa701541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa75000640 1 -- 172.21.15.29:0/169131000 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa700ca870 msgr2=0x7faa70155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa75000640 1 --2- 172.21.15.29:0/169131000 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa700ca870 0x7faa70155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa75000640 1 -- 172.21.15.29:0/169131000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa700cc8c0 msgr2=0x7faa701541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa75000640 1 --2- 172.21.15.29:0/169131000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa700cc8c0 0x7faa701541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa75000640 1 -- 172.21.15.29:0/169131000 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa70156440 con 0x7faa700cccd0 2024-04-17T20:04:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa75000640 1 --2- 172.21.15.29:0/169131000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa700cccd0 0x7faa7013d370 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7faa68007e90 tx=0x7faa6800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa6cc00640 1 -- 172.21.15.29:0/169131000 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6801a070 con 0x7faa700cccd0 2024-04-17T20:04:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa6cc00640 1 -- 172.21.15.29:0/169131000 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa68015400 con 0x7faa700cccd0 2024-04-17T20:04:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa77000640 1 -- 172.21.15.29:0/169131000 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa70155740 con 0x7faa700cccd0 2024-04-17T20:04:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa6cc00640 1 -- 172.21.15.29:0/169131000 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa680157f0 con 0x7faa700cccd0 2024-04-17T20:04:53.815 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.812+0000 7faa77000640 1 -- 172.21.15.29:0/169131000 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa7015a560 con 0x7faa700cccd0 2024-04-17T20:04:53.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.816+0000 7faa77000640 1 -- 172.21.15.29:0/169131000 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa70069b90 con 0x7faa700cccd0 2024-04-17T20:04:53.816 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.816+0000 7faa6cc00640 1 -- 172.21.15.29:0/169131000 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7faa6801d070 con 0x7faa700cccd0 2024-04-17T20:04:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.816+0000 7faa6cc00640 1 --2- 172.21.15.29:0/169131000 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faa4c068380 0x7faa4c06a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.816+0000 7faa6cc00640 1 -- 172.21.15.29:0/169131000 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7423+0+0 (secure 0 0 0) 0x7faa68087400 con 0x7faa700cccd0 2024-04-17T20:04:53.817 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.816+0000 7faa6fe00640 1 --2- 172.21.15.29:0/169131000 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faa4c068380 0x7faa4c06a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:53.818 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.816+0000 7faa6fe00640 1 --2- 172.21.15.29:0/169131000 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faa4c068380 0x7faa4c06a890 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faa60000f70 tx=0x7faa60007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:53.821 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:53.820+0000 7faa6cc00640 1 -- 172.21.15.29:0/169131000 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7faa68010070 con 0x7faa700cccd0 2024-04-17T20:04:54.025 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:54.024+0000 7faa77000640 1 -- 172.21.15.29:0/169131000 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa7015a7a0 con 0x7faa700cccd0 2024-04-17T20:04:54.026 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:54.024+0000 7faa6cc00640 1 -- 172.21.15.29:0/169131000 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+6087 (secure 0 0 0) 0x7faa680571e0 con 0x7faa700cccd0 2024-04-17T20:04:54.026 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:54.026 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:04:50.691756+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":6076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6076":{"gid":6076,"name":"f","rank":4,"incarnation":63,"state":"up:rejoin","state_seq":37,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5946,5775,5631,5350]},"id":1}]} 2024-04-17T20:04:54.026 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 66 2024-04-17T20:04:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:54.028+0000 7faa77000640 1 -- 172.21.15.29:0/169131000 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faa4c068380 msgr2=0x7faa4c06a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:54.028+0000 7faa77000640 1 --2- 172.21.15.29:0/169131000 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faa4c068380 0x7faa4c06a890 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7faa60000f70 tx=0x7faa60007c10 comp rx=0 tx=0).stop 2024-04-17T20:04:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:54.028+0000 7faa77000640 1 -- 172.21.15.29:0/169131000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa700cccd0 msgr2=0x7faa7013d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:54.028+0000 7faa77000640 1 --2- 172.21.15.29:0/169131000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa700cccd0 0x7faa7013d370 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7faa68007e90 tx=0x7faa6800d4a0 comp rx=0 tx=0).stop 2024-04-17T20:04:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:54.028+0000 7faa77000640 1 -- 172.21.15.29:0/169131000 shutdown_connections 2024-04-17T20:04:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:54.028+0000 7faa77000640 1 --2- 172.21.15.29:0/169131000 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7faa4c068380 0x7faa4c06a890 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:54.028+0000 7faa77000640 1 --2- 172.21.15.29:0/169131000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa700cccd0 0x7faa7013d370 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:54.028+0000 7faa77000640 1 --2- 172.21.15.29:0/169131000 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7faa700cc8c0 0x7faa701541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:54.028+0000 7faa77000640 1 --2- 172.21.15.29:0/169131000 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7faa700ca870 0x7faa70155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:54.028+0000 7faa77000640 1 -- 172.21.15.29:0/169131000 >> 172.21.15.29:0/169131000 conn(0x7faa7006fde0 msgr2=0x7faa7013df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:54.028+0000 7faa77000640 1 -- 172.21.15.29:0/169131000 shutdown_connections 2024-04-17T20:04:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:54.028+0000 7faa77000640 1 -- 172.21.15.29:0/169131000 wait complete. 2024-04-17T20:04:54.048 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2024-04-17T20:04:56.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:56.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.204+0000 7fac9c400640 1 Processor -- start 2024-04-17T20:04:56.203 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.204+0000 7fac9c400640 1 -- start start 2024-04-17T20:04:56.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.204+0000 7fac9c400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fac9406cd70 0x7fac940c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:56.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.204+0000 7fac9c400640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac9415a0a0 con 0x7fac940c6f70 2024-04-17T20:04:56.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.204+0000 7fac9c400640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac9415abe0 con 0x7fac94159c90 2024-04-17T20:04:56.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.204+0000 7fac9c400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fac9415a340 con 0x7fac9406cd70 2024-04-17T20:04:56.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.204+0000 7fac99000640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fac940c6f70 0x7fac9406c660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44430/0 (socket says 172.21.15.29:44430) 2024-04-17T20:04:56.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.204+0000 7fac99000640 1 -- 172.21.15.29:0/1138163111 learned_addr learned my addr 172.21.15.29:0/1138163111 (peer_addr_for_me v1:172.21.15.29:0/0) 2024-04-17T20:04:56.205 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.204+0000 7fac9a400640 1 --2- 172.21.15.29:0/1138163111 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fac9406cd70 0x7fac940c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:56.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.204+0000 7fac93e00640 1 -- 172.21.15.29:0/1138163111 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2170015715 0 0) 0x7fac9415a0a0 con 0x7fac940c6f70 2024-04-17T20:04:56.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.204+0000 7fac93e00640 1 -- 172.21.15.29:0/1138163111 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac7c003530 con 0x7fac940c6f70 2024-04-17T20:04:56.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.204+0000 7fac93e00640 1 -- 172.21.15.29:0/1138163111 <== mon.1 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 391103949 0 0) 0x7fac9415abe0 con 0x7fac94159c90 2024-04-17T20:04:56.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.204+0000 7fac93e00640 1 -- 172.21.15.29:0/1138163111 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac9415a0a0 con 0x7fac94159c90 2024-04-17T20:04:56.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.204+0000 7fac93e00640 1 -- 172.21.15.29:0/1138163111 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3045524754 0 0) 0x7fac7c003530 con 0x7fac940c6f70 2024-04-17T20:04:56.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.204+0000 7fac9a400640 1 -- 172.21.15.29:0/1138163111 >> v1:172.21.15.37:6789/0 conn(0x7fac94159c90 legacy=0x7fac940ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:56.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.204+0000 7fac9a400640 1 -- 172.21.15.29:0/1138163111 >> v1:172.21.15.29:6789/0 conn(0x7fac940c6f70 legacy=0x7fac9406c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:56.206 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.204+0000 7fac9a400640 1 -- 172.21.15.29:0/1138163111 --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac9415a620 con 0x7fac9406cd70 2024-04-17T20:04:56.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.208+0000 7fac9a400640 1 --2- 172.21.15.29:0/1138163111 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fac9406cd70 0x7fac940c7380 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009dc0 tx=0x7fac8c01c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a79b15134e9f899a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:56.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.208+0000 7fac93e00640 1 -- 172.21.15.29:0/1138163111 <== mon.2 v2:172.21.15.37:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac800016a0 con 0x7fac9406cd70 2024-04-17T20:04:56.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.208+0000 7fac93e00640 1 -- 172.21.15.29:0/1138163111 <== mon.2 v2:172.21.15.37:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac8c0a6500 con 0x7fac9406cd70 2024-04-17T20:04:56.209 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.208+0000 7fac93e00640 1 -- 172.21.15.29:0/1138163111 <== mon.2 v2:172.21.15.37:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8c0a68b0 con 0x7fac9406cd70 2024-04-17T20:04:56.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.208+0000 7fac9c400640 1 -- 172.21.15.29:0/1138163111 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fac9406cd70 msgr2=0x7fac940c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:56.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.208+0000 7fac9c400640 1 --2- 172.21.15.29:0/1138163111 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fac9406cd70 0x7fac940c7380 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009dc0 tx=0x7fac8c01c8b0 comp rx=0 tx=0).stop 2024-04-17T20:04:56.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.212+0000 7fac9c400640 1 -- 172.21.15.29:0/1138163111 shutdown_connections 2024-04-17T20:04:56.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.212+0000 7fac9c400640 1 --2- 172.21.15.29:0/1138163111 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fac9406cd70 0x7fac940c7380 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:56.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.212+0000 7fac9c400640 1 -- 172.21.15.29:0/1138163111 >> 172.21.15.29:0/1138163111 conn(0x7fac9406fde0 msgr2=0x7fac940701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:56.210 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.212+0000 7fac9c400640 1 -- 172.21.15.29:0/1138163111 shutdown_connections 2024-04-17T20:04:56.211 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.212+0000 7fac9c400640 1 -- 172.21.15.29:0/1138163111 wait complete. 2024-04-17T20:04:56.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.212+0000 7fac9c400640 1 Processor -- start 2024-04-17T20:04:56.212 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.212+0000 7fac9c400640 1 -- start start 2024-04-17T20:04:56.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.212+0000 7fac9c400640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fac9406cd70 0x7fac941560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:56.214 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.212+0000 7fac9c400640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac940c6f70 0x7fac94147ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.212+0000 7fac9c400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94159c90 0x7fac941569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.212+0000 7fac9c400640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7fac940cab40 con 0x7fac94159c90 2024-04-17T20:04:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.212+0000 7fac9c400640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7fac940ca9c0 con 0x7fac940c6f70 2024-04-17T20:04:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.212+0000 7fac9c400640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7fac94167e40 con 0x7fac9406cd70 2024-04-17T20:04:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.212+0000 7fac9a400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94159c90 0x7fac941569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.212+0000 7fac9a400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94159c90 0x7fac941569a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40892/0 (socket says 172.21.15.29:40892) 2024-04-17T20:04:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.212+0000 7fac9a400640 1 -- 172.21.15.29:0/4050149489 learned_addr learned my addr 172.21.15.29:0/4050149489 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.216+0000 7fac99a00640 1 --2- 172.21.15.29:0/4050149489 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fac9406cd70 0x7fac941560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.216+0000 7fac99000640 1 --2- 172.21.15.29:0/4050149489 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac940c6f70 0x7fac94147ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.216+0000 7fac9a400640 1 -- 172.21.15.29:0/4050149489 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fac9406cd70 msgr2=0x7fac941560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.216+0000 7fac9a400640 1 --2- 172.21.15.29:0/4050149489 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fac9406cd70 0x7fac941560c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.216+0000 7fac9a400640 1 -- 172.21.15.29:0/4050149489 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac940c6f70 msgr2=0x7fac94147ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.216+0000 7fac9a400640 1 --2- 172.21.15.29:0/4050149489 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac940c6f70 0x7fac94147ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.216+0000 7fac9a400640 1 -- 172.21.15.29:0/4050149489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac941594d0 con 0x7fac94159c90 2024-04-17T20:04:56.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.216+0000 7fac9a400640 1 --2- 172.21.15.29:0/4050149489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94159c90 0x7fac941569a0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fac8c01a4e0 tx=0x7fac8c019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:56.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.216+0000 7fac91600640 1 -- 172.21.15.29:0/4050149489 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8c0a6ac0 con 0x7fac94159c90 2024-04-17T20:04:56.216 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.216+0000 7fac91600640 1 -- 172.21.15.29:0/4050149489 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac8c0bf070 con 0x7fac94159c90 2024-04-17T20:04:56.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.216+0000 7fac91600640 1 -- 172.21.15.29:0/4050149489 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8c004040 con 0x7fac94159c90 2024-04-17T20:04:56.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.216+0000 7fac9c400640 1 -- 172.21.15.29:0/4050149489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac941550c0 con 0x7fac94159c90 2024-04-17T20:04:56.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.216+0000 7fac9c400640 1 -- 172.21.15.29:0/4050149489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac941f8c30 con 0x7fac94159c90 2024-04-17T20:04:56.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.216+0000 7fac9c400640 1 -- 172.21.15.29:0/4050149489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac94069b90 con 0x7fac94159c90 2024-04-17T20:04:56.217 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.216+0000 7fac91600640 1 -- 172.21.15.29:0/4050149489 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7fac8c0041e0 con 0x7fac94159c90 2024-04-17T20:04:56.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.220+0000 7fac91600640 1 --2- 172.21.15.29:0/4050149489 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fac74068380 0x7fac7406a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:56.218 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.220+0000 7fac91600640 1 -- 172.21.15.29:0/4050149489 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7423+0+0 (secure 0 0 0) 0x7fac8c122ee0 con 0x7fac94159c90 2024-04-17T20:04:56.219 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.220+0000 7fac99a00640 1 --2- 172.21.15.29:0/4050149489 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fac74068380 0x7fac7406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:56.221 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.220+0000 7fac99a00640 1 --2- 172.21.15.29:0/4050149489 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fac74068380 0x7fac7406a890 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fac880011f0 tx=0x7fac88009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:56.227 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.228+0000 7fac91600640 1 -- 172.21.15.29:0/4050149489 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7fac8c0164a0 con 0x7fac94159c90 2024-04-17T20:04:56.429 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.428+0000 7fac9c400640 1 -- 172.21.15.29:0/4050149489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac941525d0 con 0x7fac94159c90 2024-04-17T20:04:56.430 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.432+0000 7fac91600640 1 -- 172.21.15.29:0/4050149489 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6092 (secure 0 0 0) 0x7fac8c0f2e60 con 0x7fac94159c90 2024-04-17T20:04:56.431 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:56.431 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:04:55.737912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":6076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6076":{"gid":6076,"name":"f","rank":4,"incarnation":63,"state":"up:active","state_seq":39,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5350,5631,5775,5946,6076]},"id":1}]} 2024-04-17T20:04:56.431 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 67 2024-04-17T20:04:56.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.432+0000 7fac9c400640 1 -- 172.21.15.29:0/4050149489 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fac74068380 msgr2=0x7fac7406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:56.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.432+0000 7fac9c400640 1 --2- 172.21.15.29:0/4050149489 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fac74068380 0x7fac7406a890 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fac880011f0 tx=0x7fac88009040 comp rx=0 tx=0).stop 2024-04-17T20:04:56.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.432+0000 7fac9c400640 1 -- 172.21.15.29:0/4050149489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94159c90 msgr2=0x7fac941569a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:56.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.432+0000 7fac9c400640 1 --2- 172.21.15.29:0/4050149489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94159c90 0x7fac941569a0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fac8c01a4e0 tx=0x7fac8c019650 comp rx=0 tx=0).stop 2024-04-17T20:04:56.433 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.432+0000 7fac9c400640 1 -- 172.21.15.29:0/4050149489 shutdown_connections 2024-04-17T20:04:56.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.432+0000 7fac9c400640 1 --2- 172.21.15.29:0/4050149489 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7fac74068380 0x7fac7406a890 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:56.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.432+0000 7fac9c400640 1 --2- 172.21.15.29:0/4050149489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac94159c90 0x7fac941569a0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:56.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.432+0000 7fac9c400640 1 --2- 172.21.15.29:0/4050149489 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7fac940c6f70 0x7fac94147ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:56.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.432+0000 7fac9c400640 1 --2- 172.21.15.29:0/4050149489 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7fac9406cd70 0x7fac941560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:56.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.432+0000 7fac9c400640 1 -- 172.21.15.29:0/4050149489 >> 172.21.15.29:0/4050149489 conn(0x7fac9406fde0 msgr2=0x7fac940755f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:56.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.432+0000 7fac9c400640 1 -- 172.21.15.29:0/4050149489 shutdown_connections 2024-04-17T20:04:56.434 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:56.432+0000 7fac9c400640 1 -- 172.21.15.29:0/4050149489 wait complete. 2024-04-17T20:04:56.459 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=4, replacing gid=5212 2024-04-17T20:04:56.459 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.b 2024-04-17T20:04:57.491 INFO:tasks.ceph.mds.e.smithi029.stderr:2024-04-17T20:04:57.492+0000 7f956e000640 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-17T20:04:57.566 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 16% (40/245) Updating files: 16% (41/245) Updating files: 17% (42/245) Updating files: 18% (45/245) Updating files: 19% (47/245) Updating files: 20% (49/245) Updating files: 21% (52/245) Updating files: 22% (54/245) Updating files: 23% (57/245) Updating files: 24% (59/245) Updating files: 25% (62/245) Updating files: 26% (64/245) Updating files: 27% (67/245) Updating files: 28% (69/245) Updating files: 29% (72/245) Updating files: 30% (74/245) Updating files: 31% (76/245) Updating files: 32% (79/245) Updating files: 33% (81/245) Updating files: 34% (84/245) Updating files: 35% (86/245) Updating files: 36% (89/245) Updating files: 37% (91/245) Updating files: 38% (94/245) Updating files: 39% (96/245) Updating files: 40% (98/245) Updating files: 41% (101/245) Updating files: 42% (103/245) Updating files: 43% (106/245) Updating files: 44% (108/245) Updating files: 45% (111/245) Updating files: 46% (113/245) Updating files: 47% (116/245) Updating files: 48% (118/245) Updating files: 49% (121/245) Updating files: 50% (123/245) Updating files: 51% (125/245) Updating files: 52% (128/245) Updating files: 53% (130/245) Updating files: 54% (133/245) Updating files: 55% (135/245) Updating files: 56% (138/245) Updating files: 57% (140/245) Updating files: 58% (143/245) Updating files: 59% (145/245) Updating files: 60% (147/245) Updating files: 61% (150/245) Updating files: 62% (152/245) Updating files: 63% (155/245) Updating files: 64% (157/245) Updating files: 65% (160/245) Updating files: 66% (162/245) Updating files: 67% (165/245) Updating files: 68% (167/245) Updating files: 69% (170/245) Updating files: 70% (172/245) Updating files: 71% (174/245) Updating files: 72% (177/245) Updating files: 73% (179/245) Updating files: 74% (182/245) Updating files: 75% (184/245) Updating files: 75% (185/245) Updating files: 76% (187/245) Updating files: 77% (189/245) Updating files: 78% (192/245) Updating files: 79% (194/245) Updating files: 80% (196/245) Updating files: 81% (199/245) Updating files: 82% (201/245) Updating files: 83% (204/245) Updating files: 84% (206/245) Updating files: 85% (209/245) Updating files: 86% (211/245) Updating files: 87% (214/245) Updating files: 88% (216/245) Updating files: 89% (219/245) Updating files: 90% (221/245) Updating files: 91% (223/245) Updating files: 92% (226/245) Updating files: 93% (228/245) Updating files: 94% (231/245) Updating files: 95% (233/245) Updating files: 96% (236/245) Updating files: 97% (238/245) Updating files: 98% (241/245) Updating files: 99% (243/245) Updating files: 100% (245/245) Updating files: 100% (245/245), done. 2024-04-17T20:04:58.459 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2024-04-17T20:04:58.459 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T20:04:58.460 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T20:04:58.462 INFO:tasks.ceph.mds.b:Started 2024-04-17T20:04:58.463 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T20:04:58.574 INFO:tasks.ceph.mds.b.smithi037.stdout:starting mds.b at 2024-04-17T20:04:58.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.616+0000 7f20b2000640 1 Processor -- start 2024-04-17T20:04:58.616 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.616+0000 7f20b2000640 1 -- start start 2024-04-17T20:04:58.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.616+0000 7f20b2000640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f20ac0c6f70 0x7f20ac0c7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:58.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.616+0000 7f20b2000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20ac06c660 0x7f20ac06ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:58.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.616+0000 7f20b2000640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20ac15a0a0 con 0x7f20ac159c90 2024-04-17T20:04:58.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.616+0000 7f20b2000640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 -- 0x7f20ac15abe0 con 0x7f20ac0c6f70 2024-04-17T20:04:58.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.616+0000 7f20b2000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f20ac15a340 con 0x7f20ac06c660 2024-04-17T20:04:58.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.616+0000 7f20a3e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f20ac0c6f70 0x7f20ac0c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:58.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.616+0000 7f20abe00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20ac06c660 0x7f20ac06ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:58.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.616+0000 7f20abe00640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20ac06c660 0x7f20ac06ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.37:3301/0 says I am v2:172.21.15.29:35236/0 (socket says 172.21.15.29:35236) 2024-04-17T20:04:58.618 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.616+0000 7f20abe00640 1 -- 172.21.15.29:0/358505438 learned_addr learned my addr 172.21.15.29:0/358505438 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:58.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.616+0000 7f20a3e00640 1 --2- 172.21.15.29:0/358505438 >> v2:172.21.15.29:3300/0 conn(0x7f20ac0c6f70 0x7f20ac0c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40898/0 (socket says 172.21.15.29:40898) 2024-04-17T20:04:58.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.620+0000 7f20a3e00640 1 -- 172.21.15.29:0/358505438 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20ac06c660 msgr2=0x7f20ac06ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T20:04:58.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.620+0000 7f20a3e00640 1 --2- 172.21.15.29:0/358505438 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20ac06c660 0x7f20ac06ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:58.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.620+0000 7f20a3e00640 1 -- 172.21.15.29:0/358505438 >> v1:172.21.15.37:6789/0 conn(0x7f20ac159c90 legacy=0x7f20ac0ca2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:58.619 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.620+0000 7f20a3e00640 1 -- 172.21.15.29:0/358505438 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20ac15a620 con 0x7f20ac0c6f70 2024-04-17T20:04:58.621 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.620+0000 7f20a3e00640 1 --2- 172.21.15.29:0/358505438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20ac0c6f70 0x7f20ac0c7380 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f20940097c0 tx=0x7f209401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9ad00acb35e8b40 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:58.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.620+0000 7f20aaa00640 1 -- 172.21.15.29:0/358505438 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20940b0070 con 0x7f20ac0c6f70 2024-04-17T20:04:58.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.620+0000 7f20aaa00640 1 -- 172.21.15.29:0/358505438 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20940ab600 con 0x7f20ac0c6f70 2024-04-17T20:04:58.622 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.620+0000 7f20aaa00640 1 -- 172.21.15.29:0/358505438 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20940ab9d0 con 0x7f20ac0c6f70 2024-04-17T20:04:58.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.624+0000 7f20b2000640 1 -- 172.21.15.29:0/358505438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20ac0c6f70 msgr2=0x7f20ac0c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:58.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.624+0000 7f20b2000640 1 --2- 172.21.15.29:0/358505438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20ac0c6f70 0x7f20ac0c7380 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f20940097c0 tx=0x7f209401c4a0 comp rx=0 tx=0).stop 2024-04-17T20:04:58.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.624+0000 7f20b2000640 1 -- 172.21.15.29:0/358505438 shutdown_connections 2024-04-17T20:04:58.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.624+0000 7f20b2000640 1 --2- 172.21.15.29:0/358505438 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20ac06c660 0x7f20ac06ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:58.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.624+0000 7f20b2000640 1 --2- 172.21.15.29:0/358505438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20ac0c6f70 0x7f20ac0c7380 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:58.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.624+0000 7f20b2000640 1 -- 172.21.15.29:0/358505438 >> 172.21.15.29:0/358505438 conn(0x7f20ac06fde0 msgr2=0x7f20ac0701f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:58.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.624+0000 7f20b2000640 1 -- 172.21.15.29:0/358505438 shutdown_connections 2024-04-17T20:04:58.623 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.624+0000 7f20b2000640 1 -- 172.21.15.29:0/358505438 wait complete. 2024-04-17T20:04:58.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.624+0000 7f20b2000640 1 Processor -- start 2024-04-17T20:04:58.625 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.624+0000 7f20b2000640 1 -- start start 2024-04-17T20:04:58.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20b2000640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20ac06c660 0x7f20ac1f6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:58.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20b2000640 1 --2- >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20ac0c6f70 0x7f20ac152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:58.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20b2000640 1 --2- >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20ac159c90 0x7f20ac149f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:58.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20b2000640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 -- 0x7f20ac0ca9c0 con 0x7f20ac06c660 2024-04-17T20:04:58.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20b2000640 1 -- --> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] -- mon_getmap magic: 0 -- 0x7f20ac06cfb0 con 0x7f20ac159c90 2024-04-17T20:04:58.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20b2000640 1 -- --> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] -- mon_getmap magic: 0 -- 0x7f20ac0cab40 con 0x7f20ac0c6f70 2024-04-17T20:04:58.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20ab400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20ac06c660 0x7f20ac1f6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:58.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20ab400640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20ac06c660 0x7f20ac1f6260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40904/0 (socket says 172.21.15.29:40904) 2024-04-17T20:04:58.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20ab400640 1 -- 172.21.15.29:0/404185653 learned_addr learned my addr 172.21.15.29:0/404185653 (peer_addr_for_me v2:172.21.15.29:0/0) 2024-04-17T20:04:58.627 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20abe00640 1 --2- 172.21.15.29:0/404185653 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20ac159c90 0x7f20ac149f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:58.628 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20a3e00640 1 --2- 172.21.15.29:0/404185653 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20ac0c6f70 0x7f20ac152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:58.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20ab400640 1 -- 172.21.15.29:0/404185653 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20ac0c6f70 msgr2=0x7f20ac152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:58.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20ab400640 1 --2- 172.21.15.29:0/404185653 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20ac0c6f70 0x7f20ac152a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:58.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20ab400640 1 -- 172.21.15.29:0/404185653 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20ac159c90 msgr2=0x7f20ac149f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:58.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20ab400640 1 --2- 172.21.15.29:0/404185653 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20ac159c90 0x7f20ac149f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:58.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20ab400640 1 -- 172.21.15.29:0/404185653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20ac153340 con 0x7f20ac06c660 2024-04-17T20:04:58.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20abe00640 1 --2- 172.21.15.29:0/404185653 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20ac159c90 0x7f20ac149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T20:04:58.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20ab400640 1 --2- 172.21.15.29:0/404185653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20ac06c660 0x7f20ac1f6260 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f209c00c3f0 tx=0x7f209c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:58.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20a3400640 1 -- 172.21.15.29:0/404185653 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209c017070 con 0x7f20ac06c660 2024-04-17T20:04:58.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20a3400640 1 -- 172.21.15.29:0/404185653 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f209c0052f0 con 0x7f20ac06c660 2024-04-17T20:04:58.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20b2000640 1 -- 172.21.15.29:0/404185653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20ac153600 con 0x7f20ac06c660 2024-04-17T20:04:58.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20a3400640 1 -- 172.21.15.29:0/404185653 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209c003860 con 0x7f20ac06c660 2024-04-17T20:04:58.629 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20b2000640 1 -- 172.21.15.29:0/404185653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20ac1f5a60 con 0x7f20ac06c660 2024-04-17T20:04:58.630 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.628+0000 7f20b2000640 1 -- 172.21.15.29:0/404185653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20ac069b90 con 0x7f20ac06c660 2024-04-17T20:04:58.631 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.632+0000 7f20a3400640 1 -- 172.21.15.29:0/404185653 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) ==== 87987+0+0 (secure 0 0 0) 0x7f209c01a070 con 0x7f20ac06c660 2024-04-17T20:04:58.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.632+0000 7f20a3400640 1 --2- 172.21.15.29:0/404185653 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f20880683b0 0x7f208806a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T20:04:58.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.632+0000 7f20a3400640 1 -- 172.21.15.29:0/404185653 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 7423+0+0 (secure 0 0 0) 0x7f209c0862a0 con 0x7f20ac06c660 2024-04-17T20:04:58.632 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.632+0000 7f20a3e00640 1 --2- 172.21.15.29:0/404185653 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f20880683b0 0x7f208806a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T20:04:58.633 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.632+0000 7f20a3e00640 1 --2- 172.21.15.29:0/404185653 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f20880683b0 0x7f208806a8c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f20940ad000 tx=0x7f2094008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T20:04:58.641 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.640+0000 7f20a3400640 1 -- 172.21.15.29:0/404185653 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+179799 (secure 0 0 0) 0x7f209c055b40 con 0x7f20ac06c660 2024-04-17T20:04:58.842 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.840+0000 7f20b2000640 1 -- 172.21.15.29:0/404185653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20ac0c7510 con 0x7f20ac06c660 2024-04-17T20:04:58.843 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.844+0000 7f20a3400640 1 -- 172.21.15.29:0/404185653 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) ==== 94+0+6916 (secure 0 0 0) 0x7f209c059e40 con 0x7f20ac06c660 2024-04-17T20:04:58.843 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-04-17T20:04:58.843 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5678,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.37:6838/2313672825","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6835","nonce":2313672825},{"type":"v1","addr":"172.21.15.37:6838","nonce":2313672825}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T19:47:49.315515+0000","modified":"2024-04-17T20:04:55.737912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":5775,"mds_1":5631,"mds_2":5946,"mds_3":5350,"mds_4":6076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5350":{"gid":5350,"name":"d","rank":3,"incarnation":32,"state":"up:active","state_seq":18,"addr":"172.21.15.37:6837/2495953425","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6834","nonce":2495953425},{"type":"v1","addr":"172.21.15.37:6837","nonce":2495953425}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5631":{"gid":5631,"name":"a","rank":1,"incarnation":44,"state":"up:active","state_seq":42,"addr":"172.21.15.29:6836/2376553580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":2376553580},{"type":"v1","addr":"172.21.15.29:6836","nonce":2376553580}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5775":{"gid":5775,"name":"e","rank":0,"incarnation":50,"state":"up:active","state_seq":39,"addr":"172.21.15.29:6835/1352350250","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1352350250},{"type":"v1","addr":"172.21.15.29:6835","nonce":1352350250}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_5946":{"gid":5946,"name":"c","rank":2,"incarnation":57,"state":"up:active","state_seq":25,"addr":"172.21.15.29:6837/3087508289","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6833","nonce":3087508289},{"type":"v1","addr":"172.21.15.29:6837","nonce":3087508289}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_6076":{"gid":6076,"name":"f","rank":4,"incarnation":63,"state":"up:active","state_seq":39,"addr":"172.21.15.37:6839/2273456671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.37:6836","nonce":2273456671},{"type":"v1","addr":"172.21.15.37:6839","nonce":2273456671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[6076,5946,5775,5631,5350]},"id":1}]} 2024-04-17T20:04:58.843 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 68 2024-04-17T20:04:58.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.844+0000 7f20b2000640 1 -- 172.21.15.29:0/404185653 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f20880683b0 msgr2=0x7f208806a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:58.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.844+0000 7f20b2000640 1 --2- 172.21.15.29:0/404185653 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f20880683b0 0x7f208806a8c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f20940ad000 tx=0x7f2094008040 comp rx=0 tx=0).stop 2024-04-17T20:04:58.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.844+0000 7f20b2000640 1 -- 172.21.15.29:0/404185653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20ac06c660 msgr2=0x7f20ac1f6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T20:04:58.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.844+0000 7f20b2000640 1 --2- 172.21.15.29:0/404185653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20ac06c660 0x7f20ac1f6260 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f209c00c3f0 tx=0x7f209c00c8b0 comp rx=0 tx=0).stop 2024-04-17T20:04:58.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.844+0000 7f20b2000640 1 -- 172.21.15.29:0/404185653 shutdown_connections 2024-04-17T20:04:58.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.844+0000 7f20b2000640 1 --2- 172.21.15.29:0/404185653 >> [v2:172.21.15.37:6832/1588862286,v1:172.21.15.37:6833/1588862286] conn(0x7f20880683b0 0x7f208806a8c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:58.845 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.844+0000 7f20b2000640 1 --2- 172.21.15.29:0/404185653 >> [v2:172.21.15.37:3300/0,v1:172.21.15.37:6789/0] conn(0x7f20ac159c90 0x7f20ac149f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:58.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.844+0000 7f20b2000640 1 --2- 172.21.15.29:0/404185653 >> [v2:172.21.15.37:3301/0,v1:172.21.15.37:6790/0] conn(0x7f20ac0c6f70 0x7f20ac152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:58.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.844+0000 7f20b2000640 1 --2- 172.21.15.29:0/404185653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f20ac06c660 0x7f20ac1f6260 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T20:04:58.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.844+0000 7f20b2000640 1 -- 172.21.15.29:0/404185653 >> 172.21.15.29:0/404185653 conn(0x7f20ac06fde0 msgr2=0x7f20ac0cb530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T20:04:58.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.844+0000 7f20b2000640 1 -- 172.21.15.29:0/404185653 shutdown_connections 2024-04-17T20:04:58.846 INFO:teuthology.orchestra.run.smithi029.stderr:2024-04-17T20:04:58.844+0000 7f20b2000640 1 -- 172.21.15.29:0/404185653 wait complete. 2024-04-17T20:04:58.864 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2024-04-17T20:04:58.865 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 99 secs before thrashing 2024-04-17T20:05:00.050 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:05:00.053 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:05:00.732 INFO:tasks.workunit.client.0.smithi029.stdout:HEAD is now at 1dd420957f9 journal: detect size of raw block devices properly 2024-04-17T20:05:00.738 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/v0.5 2024-04-17T20:05:00.752 INFO:tasks.workunit.client.0.smithi029.stderr:+ [ 6 -eq 48 ] 2024-04-17T20:05:00.752 INFO:tasks.workunit.client.0.smithi029.stderr:+ ver=v0.6 2024-04-17T20:05:00.752 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo v0.6 2024-04-17T20:05:00.752 INFO:tasks.workunit.client.0.smithi029.stderr:+ git reset --hard v0.6 2024-04-17T20:05:00.752 INFO:tasks.workunit.client.0.smithi029.stdout:v0.6 2024-04-17T20:05:11.245 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 53% (121/227) Updating files: 54% (123/227) Updating files: 55% (125/227) Updating files: 56% (128/227) Updating files: 57% (130/227) Updating files: 58% (132/227) Updating files: 59% (134/227) Updating files: 60% (137/227) Updating files: 61% (139/227) Updating files: 62% (141/227) Updating files: 63% (144/227) Updating files: 64% (146/227) Updating files: 65% (148/227) Updating files: 66% (150/227) Updating files: 67% (153/227) Updating files: 68% (155/227) Updating files: 69% (157/227) Updating files: 70% (159/227) Updating files: 71% (162/227) Updating files: 72% (164/227) Updating files: 73% (166/227) Updating files: 74% (168/227) Updating files: 75% (171/227) Updating files: 76% (173/227) Updating files: 77% (175/227) Updating files: 78% (178/227) Updating files: 79% (180/227) Updating files: 80% (182/227) Updating files: 81% (184/227) Updating files: 82% (187/227) Updating files: 83% (189/227) Updating files: 84% (191/227) Updating files: 85% (193/227) Updating files: 86% (196/227) Updating files: 87% (198/227) Updating files: 88% (200/227) Updating files: 89% (203/227) Updating files: 90% (205/227) Updating files: 91% (207/227) Updating files: 92% (209/227) Updating files: 93% (212/227) Updating files: 94% (214/227) Updating files: 95% (216/227) Updating files: 96% (218/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 99% (226/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2024-04-17T20:05:11.268 INFO:tasks.workunit.client.0.smithi029.stdout:HEAD is now at a4c752df713 v0.6 2024-04-17T20:05:11.272 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/v0.6 2024-04-17T20:05:15.805 INFO:tasks.workunit.client.0.smithi029.stderr:+ [ 7 -eq 48 ] 2024-04-17T20:05:15.805 INFO:tasks.workunit.client.0.smithi029.stderr:+ ver=v0.7 2024-04-17T20:05:15.806 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo v0.7 2024-04-17T20:05:15.806 INFO:tasks.workunit.client.0.smithi029.stdout:v0.7 2024-04-17T20:05:15.806 INFO:tasks.workunit.client.0.smithi029.stderr:+ git reset --hard v0.7 2024-04-17T20:05:17.213 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 66% (134/201) Updating files: 67% (135/201) Updating files: 68% (137/201) Updating files: 69% (139/201) Updating files: 70% (141/201) Updating files: 71% (143/201) Updating files: 72% (145/201) Updating files: 73% (147/201) Updating files: 74% (149/201) Updating files: 75% (151/201) Updating files: 76% (153/201) Updating files: 77% (155/201) Updating files: 78% (157/201) Updating files: 79% (159/201) Updating files: 80% (161/201) Updating files: 81% (163/201) Updating files: 82% (165/201) Updating files: 83% (167/201) Updating files: 84% (169/201) Updating files: 85% (171/201) Updating files: 86% (173/201) Updating files: 87% (175/201) Updating files: 88% (177/201) Updating files: 89% (179/201) Updating files: 90% (181/201) Updating files: 91% (183/201) Updating files: 92% (185/201) Updating files: 93% (187/201) Updating files: 94% (189/201) Updating files: 95% (191/201) Updating files: 96% (193/201) Updating files: 97% (195/201) Updating files: 98% (197/201) Updating files: 99% (199/201) Updating files: 100% (201/201) Updating files: 100% (201/201), done. 2024-04-17T20:05:17.250 INFO:tasks.workunit.client.0.smithi029.stdout:HEAD is now at c46e1b320c6 mkcephfs: btrfsctl -a between mkfs and mount 2024-04-17T20:05:17.255 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/v0.7 2024-04-17T20:05:18.439 INFO:tasks.workunit.client.0.smithi029.stdout:v0.8 2024-04-17T20:05:18.439 INFO:tasks.workunit.client.0.smithi029.stderr:+ [ 8 -eq 48 ] 2024-04-17T20:05:18.440 INFO:tasks.workunit.client.0.smithi029.stderr:+ ver=v0.8 2024-04-17T20:05:18.440 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo v0.8 2024-04-17T20:05:18.440 INFO:tasks.workunit.client.0.smithi029.stderr:+ git reset --hard v0.8 2024-04-17T20:05:26.978 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 37% (85/225) Updating files: 38% (86/225) Updating files: 39% (88/225) Updating files: 40% (90/225) Updating files: 41% (93/225) Updating files: 42% (95/225) Updating files: 43% (97/225) Updating files: 44% (99/225) Updating files: 45% (102/225) Updating files: 46% (104/225) Updating files: 47% (106/225) Updating files: 48% (108/225) Updating files: 49% (111/225) Updating files: 50% (113/225) Updating files: 51% (115/225) Updating files: 52% (117/225) Updating files: 53% (120/225) Updating files: 54% (122/225) Updating files: 55% (124/225) Updating files: 56% (126/225) Updating files: 57% (129/225) Updating files: 58% (131/225) Updating files: 59% (133/225) Updating files: 60% (135/225) Updating files: 61% (138/225) Updating files: 62% (140/225) Updating files: 63% (142/225) Updating files: 63% (143/225) Updating files: 64% (144/225) Updating files: 65% (147/225) Updating files: 66% (149/225) Updating files: 67% (151/225) Updating files: 68% (153/225) Updating files: 69% (156/225) Updating files: 70% (158/225) Updating files: 71% (160/225) Updating files: 72% (162/225) Updating files: 73% (165/225) Updating files: 74% (167/225) Updating files: 75% (169/225) Updating files: 76% (171/225) Updating files: 77% (174/225) Updating files: 78% (176/225) Updating files: 79% (178/225) Updating files: 80% (180/225) Updating files: 81% (183/225) Updating files: 82% (185/225) Updating files: 83% (187/225) Updating files: 84% (189/225) Updating files: 85% (192/225) Updating files: 86% (194/225) Updating files: 87% (196/225) Updating files: 88% (198/225) Updating files: 89% (201/225) Updating files: 90% (203/225) Updating files: 91% (205/225) Updating files: 92% (207/225) Updating files: 93% (210/225) Updating files: 94% (212/225) Updating files: 95% (214/225) Updating files: 96% (216/225) Updating files: 97% (219/225) Updating files: 98% (221/225) Updating files: 99% (223/225) Updating files: 100% (225/225) Updating files: 100% (225/225), done. 2024-04-17T20:05:26.997 INFO:tasks.workunit.client.0.smithi029.stdout:HEAD is now at 284161c00ab kclient: kill dput/d_drop debug cruft 2024-04-17T20:05:27.004 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/v0.8 2024-04-17T20:05:30.106 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:05:30.110 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:05:31.937 INFO:tasks.workunit.client.0.smithi029.stdout:v0.9 2024-04-17T20:05:31.937 INFO:tasks.workunit.client.0.smithi029.stderr:+ [ 9 -eq 48 ] 2024-04-17T20:05:31.938 INFO:tasks.workunit.client.0.smithi029.stderr:+ ver=v0.9 2024-04-17T20:05:31.938 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo v0.9 2024-04-17T20:05:31.938 INFO:tasks.workunit.client.0.smithi029.stderr:+ git reset --hard v0.9 2024-04-17T20:05:35.907 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 5% (11/193) Updating files: 6% (12/193) Updating files: 7% (14/193) Updating files: 8% (16/193) Updating files: 9% (18/193) Updating files: 10% (20/193) Updating files: 11% (22/193) Updating files: 12% (24/193) Updating files: 13% (26/193) Updating files: 14% (28/193) Updating files: 15% (29/193) Updating files: 16% (31/193) Updating files: 17% (33/193) Updating files: 18% (35/193) Updating files: 19% (37/193) Updating files: 20% (39/193) Updating files: 21% (41/193) Updating files: 22% (43/193) Updating files: 23% (45/193) Updating files: 24% (47/193) Updating files: 25% (49/193) Updating files: 26% (51/193) Updating files: 27% (53/193) Updating files: 28% (55/193) Updating files: 29% (56/193) Updating files: 30% (58/193) Updating files: 31% (60/193) Updating files: 32% (62/193) Updating files: 33% (64/193) Updating files: 34% (66/193) Updating files: 35% (68/193) Updating files: 36% (70/193) Updating files: 37% (72/193) Updating files: 38% (74/193) Updating files: 39% (76/193) Updating files: 40% (78/193) Updating files: 41% (80/193) Updating files: 42% (82/193) Updating files: 43% (83/193) Updating files: 44% (85/193) Updating files: 45% (87/193) Updating files: 46% (89/193) Updating files: 47% (91/193) Updating files: 48% (93/193) Updating files: 49% (95/193) Updating files: 50% (97/193) Updating files: 51% (99/193) Updating files: 52% (101/193) Updating files: 53% (103/193) Updating files: 54% (105/193) Updating files: 55% (107/193) Updating files: 56% (109/193) Updating files: 57% (111/193) Updating files: 58% (112/193) Updating files: 59% (114/193) Updating files: 60% (116/193) Updating files: 61% (118/193) Updating files: 62% (120/193) Updating files: 63% (122/193) Updating files: 64% (124/193) Updating files: 65% (126/193) Updating files: 66% (128/193) Updating files: 67% (130/193) Updating files: 68% (132/193) Updating files: 69% (134/193) Updating files: 70% (136/193) Updating files: 71% (138/193) Updating files: 72% (139/193) Updating files: 73% (141/193) Updating files: 74% (143/193) Updating files: 75% (145/193) Updating files: 76% (147/193) Updating files: 77% (149/193) Updating files: 78% (151/193) Updating files: 79% (153/193) Updating files: 80% (155/193) Updating files: 81% (157/193) Updating files: 82% (159/193) Updating files: 83% (161/193) Updating files: 84% (163/193) Updating files: 85% (165/193) Updating files: 86% (166/193) Updating files: 87% (168/193) Updating files: 88% (170/193) Updating files: 89% (172/193) Updating files: 90% (174/193) Updating files: 91% (176/193) Updating files: 92% (178/193) Updating files: 93% (180/193) Updating files: 94% (182/193) Updating files: 95% (184/193) Updating files: 96% (186/193) Updating files: 97% (188/193) Updating files: 98% (190/193) Updating files: 99% (192/193) Updating files: 100% (193/193) Updating files: 100% (193/193), done. 2024-04-17T20:05:36.125 INFO:tasks.workunit.client.0.smithi029.stdout:HEAD is now at 41481853924 kclient: strip out kernel version compatibility cruft 2024-04-17T20:05:36.129 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/v0.9 2024-04-17T20:05:36.147 INFO:tasks.workunit.client.0.smithi029.stderr:+ [ 10 -eq 48 ] 2024-04-17T20:05:36.147 INFO:tasks.workunit.client.0.smithi029.stderr:+ ver=v0.10 2024-04-17T20:05:36.148 INFO:tasks.workunit.client.0.smithi029.stdout:v0.10 2024-04-17T20:05:36.148 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo v0.10 2024-04-17T20:05:36.148 INFO:tasks.workunit.client.0.smithi029.stderr:+ git reset --hard v0.10 2024-04-17T20:05:37.500 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 70% (116/164) Updating files: 71% (117/164) Updating files: 72% (119/164) Updating files: 73% (120/164) Updating files: 74% (122/164) Updating files: 75% (123/164) Updating files: 76% (125/164) Updating files: 77% (127/164) Updating files: 78% (128/164) Updating files: 79% (130/164) Updating files: 80% (132/164) Updating files: 81% (133/164) Updating files: 82% (135/164) Updating files: 83% (137/164) Updating files: 84% (138/164) Updating files: 85% (140/164) Updating files: 86% (142/164) Updating files: 87% (143/164) Updating files: 88% (145/164) Updating files: 89% (146/164) Updating files: 90% (148/164) Updating files: 91% (150/164) Updating files: 92% (151/164) Updating files: 93% (153/164) Updating files: 94% (155/164) Updating files: 95% (156/164) Updating files: 96% (158/164) Updating files: 97% (160/164) Updating files: 98% (161/164) Updating files: 99% (163/164) Updating files: 100% (164/164) Updating files: 100% (164/164), done. 2024-04-17T20:05:37.541 INFO:tasks.workunit.client.0.smithi029.stdout:HEAD is now at 007313fe844 v0.10 2024-04-17T20:05:37.546 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/v0.10 2024-04-17T20:05:37.830 INFO:tasks.workunit.client.0.smithi029.stdout:v0.11 2024-04-17T20:05:37.831 INFO:tasks.workunit.client.0.smithi029.stderr:+ [ 11 -eq 48 ] 2024-04-17T20:05:37.831 INFO:tasks.workunit.client.0.smithi029.stderr:+ ver=v0.11 2024-04-17T20:05:37.831 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo v0.11 2024-04-17T20:05:37.831 INFO:tasks.workunit.client.0.smithi029.stderr:+ git reset --hard v0.11 2024-04-17T20:05:38.549 INFO:tasks.workunit.client.0.smithi029.stdout:HEAD is now at 9fa6f5e7d09 v0.11 2024-04-17T20:05:38.553 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/v0.11 2024-04-17T20:05:38.618 INFO:tasks.workunit.client.0.smithi029.stdout:v0.12 2024-04-17T20:05:38.619 INFO:tasks.workunit.client.0.smithi029.stderr:+ [ 12 -eq 48 ] 2024-04-17T20:05:38.619 INFO:tasks.workunit.client.0.smithi029.stderr:+ ver=v0.12 2024-04-17T20:05:38.619 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo v0.12 2024-04-17T20:05:38.619 INFO:tasks.workunit.client.0.smithi029.stderr:+ git reset --hard v0.12 2024-04-17T20:05:39.833 INFO:tasks.workunit.client.0.smithi029.stdout:HEAD is now at f524150822a v0.12 2024-04-17T20:05:39.837 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/v0.12 2024-04-17T20:05:39.977 INFO:tasks.workunit.client.0.smithi029.stdout:v0.13 2024-04-17T20:05:39.977 INFO:tasks.workunit.client.0.smithi029.stderr:+ [ 13 -eq 48 ] 2024-04-17T20:05:39.977 INFO:tasks.workunit.client.0.smithi029.stderr:+ ver=v0.13 2024-04-17T20:05:39.977 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo v0.13 2024-04-17T20:05:39.977 INFO:tasks.workunit.client.0.smithi029.stderr:+ git reset --hard v0.13 2024-04-17T20:05:41.720 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 80% (107/133) Updating files: 81% (108/133) Updating files: 82% (110/133) Updating files: 83% (111/133) Updating files: 84% (112/133) Updating files: 85% (114/133) Updating files: 86% (115/133) Updating files: 87% (116/133) Updating files: 88% (118/133) Updating files: 89% (119/133) Updating files: 90% (120/133) Updating files: 91% (122/133) Updating files: 92% (123/133) Updating files: 93% (124/133) Updating files: 94% (126/133) Updating files: 95% (127/133) Updating files: 96% (128/133) Updating files: 97% (130/133) Updating files: 98% (131/133) Updating files: 99% (132/133) Updating files: 100% (133/133) Updating files: 100% (133/133), done. 2024-04-17T20:05:41.759 INFO:tasks.workunit.client.0.smithi029.stdout:HEAD is now at 2b32297267d v0.13 2024-04-17T20:05:41.762 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/v0.13 2024-04-17T20:05:42.709 INFO:tasks.workunit.client.0.smithi029.stdout:v0.14 2024-04-17T20:05:42.709 INFO:tasks.workunit.client.0.smithi029.stderr:+ [ 14 -eq 48 ] 2024-04-17T20:05:42.709 INFO:tasks.workunit.client.0.smithi029.stderr:+ ver=v0.14 2024-04-17T20:05:42.709 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo v0.14 2024-04-17T20:05:42.709 INFO:tasks.workunit.client.0.smithi029.stderr:+ git reset --hard v0.14 2024-04-17T20:06:00.160 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:06:00.164 DEBUG:teuthology.orchestra.run.smithi037:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T20:06:00.363 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files: 35% (53/151) Updating files: 36% (55/151) Updating files: 37% (56/151) Updating files: 38% (58/151) Updating files: 39% (59/151) Updating files: 40% (61/151) Updating files: 41% (62/151) Updating files: 42% (64/151) Updating files: 43% (65/151) Updating files: 44% (67/151) Updating files: 45% (68/151) Updating files: 46% (70/151) Updating files: 47% (71/151) Updating files: 48% (73/151) Updating files: 49% (74/151) Updating files: 50% (76/151) Updating files: 51% (78/151) Updating files: 52% (79/151) Updating files: 53% (81/151) Updating files: 54% (82/151) Updating files: 55% (84/151) Updating files: 56% (85/151) Updating files: 57% (87/151) Updating files: 58% (88/151) Updating files: 59% (90/151) Updating files: 60% (91/151) Updating files: 61% (93/151) Updating files: 62% (94/151) Updating files: 63% (96/151) Updating files: 64% (97/151) Updating files: 65% (99/151) Updating files: 66% (100/151) Updating files: 67% (102/151) Updating files: 68% (103/151) Updating files: 68% (104/151) Updating files: 69% (105/151) Updating files: 70% (106/151) Updating files: 71% (108/151) Updating files: 72% (109/151) Updating files: 73% (111/151) Updating files: 74% (112/151) Updating files: 75% (114/151) Updating files: 76% (115/151) Updating files: 77% (117/151) Updating files: 78% (118/151) Updating files: 79% (120/151) Updating files: 80% (121/151) Updating files: 81% (123/151) Updating files: 82% (124/151) Updating files: 83% (126/151) Updating files: 84% (127/151) Updating files: 85% (129/151) Updating files: 86% (130/151) Updating files: 87% (132/151) Updating files: 88% (133/151) Updating files: 89% (135/151) Updating files: 90% (136/151) Updating files: 91% (138/151) Updating files: 92% (139/151) Updating files: 93% (141/151) Updating files: 94% (142/151) Updating files: 95% (144/151) Updating files: 96% (145/151) Updating files: 97% (147/151) Updating files: 98% (148/151) Updating files: 99% (150/151) Updating files: 100% (151/151) Updating files: 100% (151/151), done. 2024-04-17T20:06:00.656 INFO:tasks.workunit.client.0.smithi029.stdout:HEAD is now at 1ec1e8d6fb6 v0.14 2024-04-17T20:06:00.660 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/v0.14 2024-04-17T20:06:00.756 INFO:tasks.workunit.client.0.smithi029.stdout:v0.15 2024-04-17T20:06:00.756 INFO:tasks.workunit.client.0.smithi029.stderr:+ [ 15 -eq 48 ] 2024-04-17T20:06:00.756 INFO:tasks.workunit.client.0.smithi029.stderr:+ ver=v0.15 2024-04-17T20:06:00.756 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo v0.15 2024-04-17T20:06:00.756 INFO:tasks.workunit.client.0.smithi029.stderr:+ git reset --hard v0.15 2024-04-17T20:06:01.944 INFO:tasks.workunit.client.0.smithi029.stdout:HEAD is now at ab3c421eb95 v0.15 2024-04-17T20:06:01.947 INFO:tasks.workunit.client.0.smithi029.stderr:+ mkdir .snap/v0.15 2024-04-17T20:06:03.689 INFO:tasks.workunit.client.0.smithi029.stdout:v0.16 2024-04-17T20:06:03.689 INFO:tasks.workunit.client.0.smithi029.stderr:+ [ 16 -eq 48 ] 2024-04-17T20:06:03.689 INFO:tasks.workunit.client.0.smithi029.stderr:+ ver=v0.16 2024-04-17T20:06:03.689 INFO:tasks.workunit.client.0.smithi029.stderr:+ echo v0.16 2024-04-17T20:06:03.690 INFO:tasks.workunit.client.0.smithi029.stderr:+ git reset --hard v0.16 2024-04-17T20:06:07.060 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-04-17T20:06:07.060 DEBUG:teuthology.task.console_log:Killing console logger for smithi029 2024-04-17T20:06:07.061 DEBUG:teuthology.task.console_log:Killing console logger for smithi037 2024-04-17T20:06:07.061 DEBUG:teuthology.exit:Finished running handlers